What Reviews Reveal About Online Learning

What do course reviews suggest about what it takes to create a compelling online learning experience for students? CourseTalk, a “Yelp for MOOCs” company founded in 2012 (and acquired by AcademixDirect in 2013), has tallied more than 74,000 user ratings for over 7,500 courses from 46 providers to get a glimpse of what students are saying.

Dollars seem to make a difference. CourseTalk’s report, “What Reviews Divulge About Online Education” (PDF), found that on a five-star rating system, paid courses on average were rated 1.4 stars higher than free courses. That finding shouldn’t be entirely surprising, however; blindfolded wine tasters have shown that costs can also influence the perceived value and quality of any product

“If students pay for something, they’re more likely to appreciate it,” Don Loonam, CEO of Academix Direct, told EdSurge. Paying can also be a signal for commitment and effort. He added that there is a “virtuous cycle where if you’re paying for a course, you’ll spend more time researching. And if you research, you’re more likely to find courses that are a better fit—and write better reviews.”

Leaving feedback can be a valuable part of the online learning experience. “Students who have the opportunity to provide feedback and are engaged in the forums…often build this awesome community within the course, and it’s the kind of community that sticks around,” said edX product designer Marco Morales at a NewCo session organized by CourseTalk on June 9.

Additionally, “there is significant pedagogical value in the process of evaluating a learning experience, said Connor Diemand-Yauman, Manager of Business Development and Strategic Partnerships at Coursera. “Many times the problem with a lot of online learning experiences is that users don’t take the time to evaluate, ‘What did I learn?’” The feedback process “helps learners retain the information and iterate on the learning process.”

But Diemand-Yauman also warned that “review can also be dangerous pedagogically as well,” citing instances where teachers sometimes earn high reviews by creating an entertaining—but not educational—experience.

Instructors of top-rated courses were often described as “humorous and personable,” according to the report, which also found “no direct link” between the hours of study expected of students and user ratings. Other attributes of highly rated courses include on-demand scheduling, active student forums and ample support materials.

As the number of online learning courses and providers grows, CourseTalk is not alone in attempting to provide information to help students sift through their options. Another service, Class Central, encourages students to leave reviews for free MOOCs.

Any user review should be taken with a grain of salt, and CourseTalk’s findings beg further analysis of students’ backgrounds, motivation and expectations—data which can be harder to find. Is a student signing up for a course to acquire a hard skill, or to satiate his or her curiosity? Answers to this question could provide helpful context around the quality of reviews and other data points such as completion rates, which many industry observers still carefully watch.

Karl Mehta, founder and CEO of EdCast, isn’t worried about that metric, however. “The power of inspiration is more powerful than the power of knowledge,” said Karl Mehta, founder and CEO of EdCast. “What MOOCs have helped to do is inspire millions and millions of people to at least get started.”

Mehta’s perspective is informed by his experience in the gaming industry, where his previous company, Playspan, built a monetization platform for free online games. At first, only two percent of players were paying, leading critics to doubt the sustainability of his business model. But “those 98 percent of people who are not paying,” he said, “are adding a lot more value to the conversation and interaction for the other 2 percent. Over time they are converting.” (His company was later acquired by Visa for $190 million in cash in 2011.)

Similarly, the online learning industry demands patience. “We are at an early stage of the MOOC revolution,” he added.