Refocusing Edtech Research on Implementation, Not Efficacy

At the "Better Connecting Research Evidence and Product Innovation" session at the 2013 SIIA Ed Tech Industry Summit, three directors and a VP--all from SRI International--gathered to discuss findings around how to gather evidence and data to support research on the efficacy of online programs. The discussion touched upon some of the points raised in the recent U.S. Dept. of Ed.'s policy report, "Expanding Evidence Approaches for Learning in A Digital World." (Be sure to check out EdSurge contributor Andrew Plemmons Pratt's three-part summary.)

The panel's general message could be summed up along the lines of:

"Hold your horses! There's plenty of data--but we're not sure what this all means without context and framework."

While that may be a bit disappointing (don't we always expect scientists to have the answers?), we respect the urge for patience and caution.

Robert Murphy, SRI's Evaluation Program Director, believed it may be more prudent to re-frame the research agenda away from 'efficacy' studies. "Implementation research should be the focus at this stage," he asserted. "The question 'Does it work?' when the 'it' is still being defined and changing year from year is a little bit off the mark."

Indeed, the outcomes from online programs depend on the context and extent to which they are deployed. How is the class set up? Are teachers properly trained? What are their expectations for these tools? How are they designing their teaching around them?

Murphy's call to shift the focus towards implementation was based on his observations at schools like Alliance, Rocketship, FirstLine, KIPP EA, and Summit, all early adopters of blended learning programs:

• Blended learning models remain in a state of flux--look at Rocketship, for instance, where the learning labs will no longer be separate from the classroom. In lab models, classroom teachers can feel disconnected from what's going on in the computer labs;
  
• Very few teachers are attempting to flip the classroom, despite all buzz around resources like Khan Academy. Most online programs are used for additional practice or enrichment tools rather than the primary source of instruction;
• Integrating multiple outputs from different vendors into a dashboard that teachers can use remains a challenge. And performance data is often not interpretable in terms of a school's curriculum objective--at least not in a format that a teacher can quickly go into the system and see if a student mastered a specific skill;
  
• Along those lines, there are still questions over the validity of online program assessments. Administrators and teachers trust their own judgment and instincts over those from online systems;
  
• Teachers are asking for online programs that are both adaptive and assignable (which to us sound a little bit at odds).

The overall takeaway? The blended learning experiment continues.