At a time when more than 11,000 edtech tools are on the market and schools are embracing learning technology like never before, there is a stunning lack of research and evidence to support the efficacy of those products.
A report released this spring from LearnPlatform, a company that helps districts better understand and manage the technology they’re using, found that of the 100 most accessed edtech products in K-12 classrooms in the first half of the 2022-23 school year, just 26 have published research aligned to one of the four tiers of evidence in the federal Every Student Succeeds Act (ESSA).
Of those 26 edtech solutions with ESSA-aligned research available, 17 had earned the entry-level ESSA tier, Level IV, which asks only that a product “demonstrates a rationale.” Four of the companies had earned ESSA Level III (“promising evidence”), none had earned ESSA Level II (“moderate evidence”), and five had earned the highest tier, ESSA Level I (“strong evidence”).
That doesn’t mean the other products don’t work the way they’re designed to work. It means there is no way of knowing — yet — whether, when, for whom and under what conditions those products work as intended. And that leaves districts to do a great deal of guesswork.
At a time when students are trying to make up for instructional time lost during the pandemic and many school districts have seen two decades of progress wiped out in reading and math, the last thing that should be happening in education is trial and error.
During a panel I moderated at the 2023 ASU+GSV Summit in San Diego in April, edtech providers and proponents, along with a leader from the U.S. Department of Education, came together to consider how the sector can close the gap between the edtech that exists and the emerging evidence that supports it. You can watch a recording of the discussion or read a summary of it below.
What’s clear is that many — perhaps most, and hopefully all — edtech providers in this space believe their products work and are in the education field because they want to help students and educators succeed, panelists said.
“There's nobody here that wants to invest in anything that's not working,” said Karl Rectanus, co-founder and CEO of LearnPlatform. “I don't talk to anybody who says, like, ‘Our revenues are through the roof — and get this: It doesn't work.’”
The barriers, in many cases, are the time, money and resources it takes to prove that a tool works.
“It’s difficult,” conceded Sunil Gunderia, chief innovation officer at Age of Learning, which has earned numerous ESSA certifications for its game-based math and literacy products, from ESSA Level IV through Level I, and has completed two randomized controlled trials. “It's a process. You have to go to a third-party evaluator. That's expensive. It's a costly, high-stake endeavor.”
But, Gunderia added, “It’s definitely worth it.”
Most edtech companies are a long way off from randomized controlled trials — considered the gold standard in research. The natural starting point for edtech providers, instead, is pursuing ESSA Level IV certification.
To “demonstrate rationale” and earn ESSA Level IV certification, companies need only show a logic model or theory of action and indicate that they are planning to study the effects of their invention. It is not a high bar to clear.
“Everybody should have a Level IV,” Gunderia said plainly. “You shouldn't be in educational technology if you don't have a logic model — if you don't know what you're solving.”
One of the key reasons the uptake to ESSA evidence has been so slow, according to Kristina Ishmael, deputy director of the Education Department’s Office of Educational Technology, is that many in the field have found the ESSA guidance and certifications to be unapproachable.
For years, Ishmael added, department officials have been asked to provide more technical assistance and professional learning opportunities for folks seeking to understand ESSA.
“We heard from the field, back in September, that there could be literally a training every single day for educational leaders and decision makers, whether at the school level or the state level, and we still would not have full saturation across the country for folks that know what [ESSA] means,” she said, emphasizing the charge for the Education Department.
Since last fall, department officials have been working on delivering the type of accessible, digestible professional learning around ESSA and edtech evidence that educators and solution providers say they need. The result, released the week of the ASU+GSV Summit, is an “EdTech Evidence Toolkit,” complete with four one-pagers, each aligned to a different tier of ESSA, and a series of corresponding blog posts.
Each one-pager — which is dense with text and only counts as “one” page if you print it double-sided — gives background information on a different tier and provides a district case study to offer additional context on the ESSA Level.
The blog posts — two so far, with more expected — offer education leaders additional background and implementation support and suggestions.
“This,” Ishmael said, holding up physical copies of the one-pagers to the audience, “is very approachable.”
It is, of course, much too early to say how the toolkit is being received. But the hope is that, as districts become more sophisticated with technology, as many have had to do in recent years, they will be ready to ask more difficult questions about the products they’re already using or considering adopting. And when those questions arise, the Education Department hopes to brandish the answers (or at least some of them — edtech providers must hold up their end of the deal too).
Districts are already proving to be more interested in seeing evidence of edtech efficacy and more savvy about which products they purchase, Rectanus noted.
Two of the largest school districts in the country, he said, have made evidence central to their purchasing decisions. Los Angeles Unified has requested evidence from all of its solution providers, and Chicago Public Schools recently added evidence requirements to its procurement process.
But these are still early days. The sector has yet to experience a mindset shift around evidence. It is still treated as a nice-to-have when it should be viewed as a necessity, the panelists said.
So how does the field get from here to there? What’s next?
“Put together a logic model and get ESSA IV to start,” Gunderia said, offering guidance to edtech providers. “Then start planning how you’re going to do your first tier III, a correlation analysis, and move up.”
Dana Bryson, senior vice president of social impact at Study.com, said that it’s critical for edtech staff to get buy-in inside their organizations about the importance of evidence.
“If you are selling to a public school district … and you're not looking for your evidence base, you are behind. It is imperative,” Bryson said. “Whatever's in your roadmap or your priorities, go ahead and move this to the top. This is where we're going. This is a requirement. These are public funds. These are student outcomes. We're not messing around. I think that it's that imperative that you've gotta bring to it and a sense of urgency to get it done.”
Ishmael, from the Education Department, added that it’s important to start somewhere, even if it means starting small.
“Whether you're gonna start with a logic model, or whatever entry point you want to do, those small steps and those small wins add up very quickly,” Ishmael noted, “and we need that collectively across our entire system.”
