“Schools deserve more bang for their buck.”
It’s a common refrain, and it was the theme of my favorite panel at SXSWedu. Four panelists, including EdSurge’s own Leonard Medlock, argued that companies have a responsibility to prove that products and services improve targeted student outcomes.
At the discussion’s core was a call for the edtech sector to increase its investment in effectiveness research. For companies of all sizes and at all points of their life cycle, effectiveness research is affordable, necessary, and extremely beneficial. A good researcher can design a study that respects the resource constraints of small- and medium-sized firms while still providing clients proof of product and service effectiveness, delivering valuable insights to drive improvement, and aiding marketing and sales efforts.
The ESSA and the new standard for effectiveness research
The perception that effectiveness research is too expensive for all but the largest market actors likely comes from No Child Left Behind (NCLB), which set an ambitious but ambiguous standard for research to qualify as “scientifically-based.” The only way companies could be sure they would comply was to pursue the “gold standard” of randomized controlled trials. In education, randomized controlled trials are run by university schools of education or the very largest research firms, take years to complete, and have thousands of participating students.
However, as all researchers know, rigorous research that produces critical findings is possible using methods other than randomized controlled trials. In keeping with this, the Every Student Succeeds Acts (ESSA) legislation recognizes three additional tiers of research that state governments and schools may accept as proof of program or service effectiveness. These tiers allow for the use of a variety of rigorous statistical techniques that can simulate experimental conditions.
The ESSA allows organizations that cannot afford randomized control trials to still fund and benefit from a robust research agenda and allows state and local actors to use such evidence to justify purchasing decisions. In short, small- and medium-sized companies can now afford research that makes them eligible for a host of new revenue streams.
What effectiveness research adds to a company
My firm’s experience serves as an example of the benefits of this new approach. An organization asked us to assess whether its program and services helped children learn targeted math and science skills. The organization lacked funds for a randomized controlled study, but had a commitment to using research to generate an honest look at its strengths and weaknesses. Using existing test items from the organization’s worksheets we constructed a pretest/posttest that organization personnel administered to thousands of children nationwide.
Any ethical researcher must acknowledge the limitations of a study’s design. A pre-test/post-test design is not sufficient to establish a full causal link between program participation and student growth. Item writers know that worksheet questions are very different from standardized test items and may question how well the former can be modified into the latter.
However, the goal of effectiveness research is not publication in a peer-reviewed journal but the production of direct, helpful information that allows consumers to make more informed choices. Where rigorous research shows that a program or service is associated with the exact gains it was designed to teach, it’s a reasonable assumption to accept that the program or service likely had some effect.
Despite these limitations, we conducted a study that provided valid, valuable answers to questions critical to the organization. Using the pretest/posttest design, the study found that the organization’s programs and services were associated with gains in targeted skills. While this evidence is not up to the gold standard of randomized controlled trials, it does strongly suggest that the organization’s programs and services are a valuable investment for educators and parents to make.
For a limited cost, the organization now has a rigorous study that demonstrates a link between their programs and services and gains in their targeted outcome and provides a clear marketing advantage over competitors who have not undertaken similar research.
The power of improvement
It’s important to remember that the approach outlined here creates ethical quandaries for both companies and researchers. Researchers must take care to emphasize that all research designs can only suggest, rather than prove, program effectiveness, and companies must provide a truthful account of those findings to the public. Both parties must acknowledge and accept that effectiveness studies must be open to the possibility that programs and services under examination do not produce all of their intended benefits.
Fortunately, companies have strong incentive to hope research uncovers their products’ limitations as well as its strengths. When a study does not uncover evidence of a hypothesized benefit, it creates opportunities for companies to intelligently reassess their products. Does the lack of evidence indicate a flaw in product design or a lack of support services? Are changes necessary to ensure that products provide intended benefits to schools and their children?
Questions like these can disrupt a company’s existing plans and provide a clear direction and mandate for improvement. Companies should view research as part of a program evaluation process that will ultimately leave the company with stronger programs better able to compete and win in the marketplace. All of which will help the EdTech community move towards the larger goal of providing schools with the maximum return on investment. The bang for their buck.
