With thousands of education apps available today, it can seem like students' success is in the palm of their hands. But easy to forget is that technology is not in itself a solution. For it to work as intended, it must be paired with other critical elements: professional development for teachers, thoughtful implementation and consistent engagement.
A new report that analyzes 1.48 million hours of technology usage by 390,000 students across 48 U.S. school districts underscores this point.
School leaders expect students and teachers to be using their licensed apps all the time, says Ryan Baker, director of the University of Pennsylvania Center for Learning Analytics. But in reality, most of the apps schools adopt are neglected.
“Schools will buy these licenses, and then they never really get touched,” says Baker, who analyzed the data and co-authored the report.
That’s an especially big problem for the education apps that are designed for “relatively intensive usage,” he adds, because only at that level can the apps be expected to lead to their promised learning outcomes.
The study, from K-12 data management company BrightBytes, used data collected from the company’s analytics platform to measure learning outcomes from digital apps through three primary lenses. First is investment—how much a subscription costs, how many licenses a school has and how active its users are. Second is engagement—how many students use an app, how long they use it, how often they use it and how they feel about it. The third lens is impact—whether students’ use of an app has any bearing, positive or negative, on standardized test scores.
If you’re a school leader who has invested big in education apps, or a proponent of digital apps in the classroom, the findings from this report are bleak.
Many apps, independent of how long or often students use them, have no relationship to learning outcomes. And the majority of purchased licenses don’t get used. A median of 97.6 percent of licenses analyzed in Baker’s study were never used “intensively” (for 10 or more hours between assessments), despite the recommendations of the edtech app providers.
Baker found only minimal overlap between apps with the most licenses purchased and the largest number of intensive users. The apps with the most licenses purchased are ConnectEd, WeVideo, Blender Learn Discovery and Education Streaming Plus. The apps with the highest intensive users are Google Drive, Canvas, Dreambox Learning, Lexia Reading Core5 and IXL. (Some apps, such as Google Drive, have more users than licenses purchased because they offer their services for free.)
One of the most interesting takeaways from this analysis, Baker notes, is that app effectiveness varies widely, including within natural categorical breakdowns. For example, McGraw-Hill Education's math and science app ALEKS led to higher performance in a school located in a mid-sized city in the Midwest, but not in smaller towns. Another math program, Dreambox Learning, was associated with higher learning outcomes in a mid-sized city in the Midwest, but not in a similar neighboring city.
“I would’ve put down my hypothesis at the beginning that urban districts would look like other urban districts, rural districts would look like rural districts and small districts would look like small districts,” Baker says. “But I really think it has to do with local implementation conditions. Implementation matters.”
Another reason schools are seeing such grim usage rates is school leaders have not been very intentional about tracking and assessing the data, says Tracy Kleine, the vice president of marketing at BrightBytes.
“The answer is never to just buy something and let it run,” says Kleine, a former teacher. “They have to look at the results and take action.”
