It should be obvious to everyone working in education that in order to measure how well we are doing, we can’t just look at our revenues, profit margins, or funds raised. Nor can we just look at our user growth and activity metrics. How can you truly claim success if your product doesn’t make an impact on student learning and therefore student lives?
While this seems obvious, it’s far from widespread in our ecosystem.
At Pearson, we are on a journey to better measure our success in these areas, across all of our lines of business. We define efficacy as “a measurable impact on learner’s lives,” and we evaluate our products based on the learner outcomes they deliver in addition to their financial results. By focusing on efficacy, we’re transforming ourselves from a company that used to create mainly inputs (e.g., textbooks, tests) to one that is focused on outcomes.
Pearson’s efficacy approach consists of the following:
- Define intended learner outcomes for our products: The types of outcomes Pearson products and services deliver for learners should guide our business decisions. We’ve defined what makes a “good” outcome. It takes the form of a statement: “the learner/teacher will be able to X [verb]…” as a result of using a Pearson product or service. We have also defined the different types of learner outcomes. Some are simple: improved access across platforms, for example, or a positive user experience. Others are more ambitious, like progression to the next course of study, training, or employment.
- Review products to ensure they are positioned to deliver on those outcomes, and put in place efficacy improvement programs: The Pearson efficacy framework is used internally to evaluate how well-positioned a particular product or service is to enable learner outcomes. Its purpose is to set out the ways a product or service could be improved, and to create a path forward for product and business teams. (The framework is available for public use as well!)
- Conduct research to measure the efficacy of products in use, and feed insights back into product development. To check that things are working the way we say they will, we collect and analyze all kinds of evidence related to outcomes – from surveys, to on-platform analytics, to academic-caliber studies or randomized controlled trials. (See here for some examples in our K12 portfolio.) We believe different outcomes will need to be upheld by different kinds of evidence at different levels of robustness.
- Support customers to effectively implement the products to get the best outcomes possible. Efficacy doesn’t happen in a controlled laboratory; it depends on educators, institutions, and systems – our partners. We have teams on the ground developing best practices for incorporating our products and services into instructional environments, or building educational institutions. (See here and here for some examples.)
Let me be clear that I believe all educational companies - regardless of size or tax status - should have a rigorous approach to efficacy because it is the right thing to do. Efficacy brings accountability to the mission that drives educational companies. It answers the question, 'Are we really helping the students we serve?
But for large and established education companies, there is a true business case for efficacy as well. First, it’s about building good products that match the needs of customers. In our most recent earnings call, Pearson CEO John Fallon said that “access, impact, and outcomes… are the hallmarks of the most commercially successful products we have in Pearson today.”
Second, the move from outputs to outcomes is a smart strategy given the realities of doing business in the education sector. Tests and textbooks are just a small slice of the money spent on education globally; money spent on teaching, learning, and related services are a much larger and growing share. In order to participate in this sector companies need to get smarter on how to support customers in their interactions with learners, and to share accountability for learner outcomes. Last, efficacy taps into new opportunities enabled by technology: personalized learning, better analytics, flexibility of formats, and diversity of content.
For start-ups, many of these arguments are the same. Just as with large companies, the market will increasingly put a price on efficacy as products become more sophisticated and outcomes are easier to track and manage. Education companies are increasingly asked to extend their reach, making sure things work the way they should for maximal impact, and ed-tech startups are no exception. You will be asked to provide services and teacher PD in addition to great products, which requires an understanding of pedagogy and learning theory in addition to how the education system works. Last, strong evidence of efficacy increases your likelihood of increasing your attractiveness to other companies as partners, investors, or acquirers.
A large company like Pearson has longstanding relationships with customers, a necessary factor in evaluating whether our products and services work the way we say they will. We also have a diverse portfolio, from products that are just ideas to products on the verge of retirement. Different stages of this lifecycle offer different views into efficacy. Compared to the average start-up, we have more resources: teams of researchers, more data, and access to experts in the field. But start-ups are focused in one sector and on a smaller set of customers and learners, so you can sharply tailor your offerings. You can move faster, in many cases, to draw insights from your user data and apply them to new features and functionality. Everyone in your company can see a direct path from their work to helping learners – in ours, that can be a challenge. Perhaps most importantly, you are helping to shape the conversation by thinking differently about education and challenging long-held views about how people should learn, and what companies in education should provide.
Companies of all sizes need to make efficacy central to everything they do. We all want to make an impact on student learning: let’s put a base of rigor and accountability underneath this shared purpose, so that we actually see results.
