The Harvard researchers evaluating DreamBox Learning start their study bluntly: "There is little evidence yet that educational software is actually helping students progress more rapidly." Maybe, if we’re looking on the bright side, there’s nowhere to go but up? Or maybe DreamBox is another in a long succession of tools with nice promises that don't pan out.
Luckily for Bellevue, Washington-based company, theirs may be a tool that is the exception rather than the norm. Harvard’s study on the effects of the adaptive math tool, “DreamBox Learning Growth in the Howard County Public School System and Rocketship Education,” strikes a positive tone.
The study began in 2014, when Harvard's Center for Education Policy Research started working with the Howard County Public School System (HCPSS) and Rocketship Education's charter school network. The three partnered to measure the impact of DreamBox Learning on student achievement and to develop a "streamlined, low-cost evaluation model that could be replicated easily," Harvard researchers write.
Some of Harvard's key findings:
1. Students who spent more time on the DreamBox software saw larger gains in achievement, and those who followed the company's lesson recommendations saw faster gains. DreamBox operates by engaging students in math lessons, gauging their progress then recommending other DreamBox lessons in order to, theoretically, optimize mastery. Following these suggestions, Harvard found, helps students more so than repeating previous lessons.
2. Researchers wrote, "The DreamBox progress measure was positively associated with achievement gains on state tests and other interim assessments." Measuring students beginning the year at the 50th percentile in math scores, Harvard found that Rocketship students reached the 54th percentile by the end of the year and that HCPSS students advanced to the between the 54th and 55th percentile. The measured students were using DreamBox the average amount for their schools.
3. The results of recommended usage were positive, but Harvard found that "most students did not reach the recommended levels of usage of the DreamBox software." DreamBox urges students to engage with the software for between 60 and 90 minutes per week, but students at Rocketship only did so for an average of 44 minutes per week. HCPSS students likewise used DreamBox an average of just 35 minutes per week. The university called the results "encouraging but mixed."
Jessie Woolley-Wilson, CEO of DreamBox, said that the study was in the works long before researchers started taking notes.
“Startup companies have tough choices: do I hire more people, take the product to the next level or invest in a multi-year study?” she said. “So we waited until we made the product the best it could be and we could get on the radar of a reputable research team and we had a critical mass of users.”
Not all companies volunteer their products for third-party review; there's always a risk that the tools don't live up to the marketing. In DreamBox's case, however, Woolley-Wilson said the positive stories from her users gave her confidence in its efficacy.
“It could have been negative,” she said, “But we’d gotten so much good anecdotal feedback from our users that we wanted to understand what our impact was and under what conditions we could make our biggest impact.”
University researchers appended a few caveats to the end of their study. They reasoned that above-average use of DreamBox might correlate with a student’s propensity to study more rather than solely to the beneficial effects of DreamBox. They wrote that the study is “promising but not conclusive.”
