When edtech entrepreneurs design and develop their products, high-quality research is not always at the top of their minds. After all, it takes extra time, money and effort for edtech developers to use research as they design and develop their products. And while many entrepreneurs claim that their products are 'research-based," it's difficult for consumers to determine whether the studies they cite or conduct are of a certain caliber.
As Alpha Public Schools administrator Jin-Soo Huh told EdSurge this past summer, “Edtech companies have a responsibility when putting out studies to understand data clearly and present it accurately.”
Nonprofit Digital Promise agrees, and on August 9, the nonprofit launched an effort to collection and evaluate quality examples of how companies conduct and research in designing and improving their products.
Today, the nonprofit reports that 53 companies shared an array of approaches in three submission categories: Learning Sciences (applying learning science theory in the design of new products), User Research (understanding user behaviors and needs), and Evaluation Research (launching evaluations of product effectiveness). Representatives from Teachers College, Columbia University reviewed applications, selecting one exemplary company and two honorable mentions in each of the three categories.
Here are the top companies in each category.
The Learning Sciences Category: BrainQuake, Inc.
Top marks in this category went to BrainQuake, Inc., which applied specifically with its Wuzzit Trouble product—a math game for middle grades—in mind. According to the BrainQuake application, cofounder Keith Devlin brings a wealth of research with him to the table, having been a researcher at Stanford University and focusing on alternative representations of mathematics in games for the past eight years.
“Developers who are new to research should partner with someone who knows that material. It takes years to get immersed. Team up with someone with that background, rather than try to master the material and assimilate it,” Devlin says.
When designing Wuzzit Trouble, BrainQuake worked to combine research on math pedagogy with research on learning games. "We also didn’t try to force the research to support what we thought we could do,” adds Randy Weiner, BrainQuake CEO and Co-Founder. “Instead of going for breadth, or looking to cover a lot of the math curriculum, we asked: are there high-leverage parts of the curriculum that could change kids’ lives if they were done well?"
Additionally, the team endeavored to focus on equity, finding research that will support solutions to “move the needle” for girls and children of color in terms of their math outcomes. Devlin reports that BrainQuake’s underlying mathematics pedagogy is grounded in the National Research Council 2001 study, “Adding it Up: Helping Children Learn Mathematics, National Academies Press,” and heavily influenced by the Nunes et al 1993 text “Street Mathematics and School Mathematics.”
Honorable mentions in the Learning Sciences category went to Teachley, an educational game studio created by three Teacher’s College Ph.Ds and cognitive science researchers, and Woot Math, a math supplement that teaches number sense and conceptual understanding of fractions and decimals.
The User Research Category: Cogent Education
Cogent Education, producer of a set of interactive science case studies called “Interactive Cases,”puts students in the shoes of a professional scientist.The team believes in the utility of “testing early, testing often” with users.
Cogent Education employs a variety of methods—from informal 1:1 interviews to focus groups to classroom observations—to gather and look at data to see how students are progressing on the case studies. Each time the company updates a portion of the product, it reviews student and teacher feedback from product trials conducted by the University of Georgia, as well as data from current users, in the design and development of the product.
“[Students] don’t have a filter. If they don’t like something, they just tell you. It’s the most honest feedback you can get,” reports Tom Robertson, cofounder, scientist and CEO of Cogent Education. “We made sure we listened to the students and the teachers, and made sure we were incorporating that into the next version of [Interactive Studies].”
An example of a concrete piece of improvement came recently when teachers reported that they wanted to be able to see and help a student struggling “in the moment.” In response to this, Robertson says that the team created a “heat map” feature “to get teachers the data they needed and they valued in real-time.”
Honorable mentions in the User Research category went to Accelerate Learning, creator of curriculum platform STEMscopes, and Smithsonian Learning Lab, an open educational resource collection of images drawn from the Smithsonian’s 19 museums, nine major research centers and the National Zoo.
The Evaluation Research Category: Woot Math
While Woot Math received an honorable mention in the Learning Sciences category, the company received the highest score in the category of Evaluation Research for its evaluation of Woot Math: Fractions Unlocked activities, geared toward grade 3-7.
After receiving SBIR (Small Business Innovation Research) funding to do two phases of research, Woot Math chose to focus first on the adaptivity of the product, and then on whether the product impacts learning. The team works closely with David Webb, a University of Colorado associate professor, to help them do the job. Although most Phase I SBIR studies are small, informal tests, Webb helped Woot Math enroll 600 students in an RCT (randomized control trial).
Brent Milne, Woot Math’s vice president of research, recommends that all companies consider using a scientific approach when developing their products. “Be willing to ask questions… How does this work? How well does this work? It’s applying principles of science, making hypotheses and testing,” he says. And, while “RCT’s are the gold standard,” he acknowledges that there are other ways to collect important data on a product.
Another important thing, he adds, is that edtech organizations focus on a topic that is a major pain point or piece of the curriculum that can be done well, versus trying to take on an entire curricular area when going through the evaluation process.
Honorable mentions in the Evaluation Research category went to Cell-Ed, a “cellphone language learning program” for adult learners, and Dreambox Learning, a collection of K-8 math games that adapts to the learner's level of knowledge.
