Wesley Engers has an unusual hobby: beta testing online courses from well-known colleges and universities.
He doesn’t get paid, but he helps improve the quality of courses by catching mistakes in quizzes and pointing out befuddling bits of video lectures, which can then be clarified before professors release the course to students.
“I find it much more engaging than reading a book,” says Engers, a 27-year-old data scientist, when asked why he does it. “And I do enjoy giving back and trying to contribute to a community and help future students.”
He’s one of about 2,500 volunteer beta testers for Coursera, and part of an expanded quality-control effort the company started in the past year. For a typical course that goes through the process, 25 to 35 beta testers spend a week sampling lectures and quizzes and giving feedback and suggestions. Each tester isn’t expected to complete the entire course, but to go through at least a week’s worth of the four- to six-week syllabus (with different testers trying different parts of the course).
Engers says he only participates every so often, but when he does sign up to test a course, he spends about four to six hours trying out fragments of it. In some cases he decides to eventually take the whole thing, and Coursera waives the usual fee for him to get a certificate if he finishes.
Coursera was a pioneer in offering MOOCs, or massive open online courses, in partnership with hundreds of top colleges. While attention around MOOCs has died down, the company seems to have found a business model for free courses with something it calls Specializations. They’re essentially partial graduate degrees, on the cheap, requiring students to take a series of month-long courses on a focused topic such as data science. Though the material is free to anyone to watch, students must pay a fee per course—usually about $70—for a verified certificate proving they successfully worked through it. That means these newfangled microdegrees cost only a few hundred dollars in total.
Coursera’s leaders refer to their Specializations as “products,” and their beta-testing program takes a page from the playbook of major software companies.
Andreina Parisi-Amon, the company’s manager of teaching and learning, says that Coursera takes the suggestions of the testers seriously, and that it has delayed the release of courses in at least a few cases to give professors time to address the feedback. In one case, Parisi-Amon says, testers found the tone of material in a course to be sexist (or, as she put it, “gender biased”), and the college partner worked to address the issue before the course launched.
Coursera officials say that every course that is part of a Specialization now goes through the beta testing process, and that more than 300 have been tested so far. Testers are asked to go through a brief training MOOC, but their knowledge on the topics they review is varied—the idea is they are giving the student perspective.
At a recent workshop that Coursera held for professors offering their first online course, participants had some questions about how the beta testing worked. Some wondered how thorough the volunteers could be, considering they are given only one week to test a course. Others asked whether they could ask testers to focus on particular parts of the course (they can, in a special online forum set up for the beta testers).
The overall mood was that the professors wanted as much testing as possible so they didn’t embarrass themselves when tens of thousands of students entered the virtual classroom. “I’d rather find out sooner rather than later” if there’s an issue, said Nancy Benovich Gilby, director of entrepreneurship at the University of Michigan, who is developing an online course about how to develop iPhone apps. “I have to say, a week of beta testing is not nearly enough.”
What Colleges Do
Testing online courses is not standard practice at traditional colleges, though in the past few years colleges have started doing more to ensure the quality of online offerings.
This week at the annual conference of the EDUCAUSE Learning Initiative, Dan Feinberg, senior instructional designer for the State University of New York system, outlined a course-review framework developed by SUNY called OSCQR (pronounced Oscar). It is licensed under creative commons so that any college can use it.
Feinberg says the challenge, though, has been figuring out incentives for professors or instructional designers to conduct the quality checks.
“On most campuses, instructional designers have their hands full and don’t have time to review the courses before they go live,” he says.
One college in the SUNY system, Niagra County Community College, offers to pay external reviewers to go through online courses, offering about $150 per course. Other campuses have experimented with giving release time to their own instructors in exchange for reviewing courses.
“We're still trying to find the magic bullet that motivates people to review other people's courses when they're not being paid,” he says. “Even when people have the best intentions, they're busy.”
And he says that the review framework focuses on design clarity and accessibility issues rather than content. In an interview, Feinberg says a review would probably not catch whether a quiz answer was wrong or confusing. He says professors are wary of anything that felt like “big brother” telling them how to teach. “Faculty do have academic freedom, and that leads to innovation and diversity,” he says.
Another popular review framework, called Quality Matters, is used at 1,300 colleges around the world, though it is also focused on course design.
“It would be lovely if universities would consider ways of adopting the practice of beta testing,” says Phillip Long, chief innovation officer and associate vice provost for learning sciences at the University of Texas at Austin. One factor, though, is cost. “How do you scale that at a university that has thousands of courses being taught,” he asks.
To Long, the idea of beta testers for online courses raises larger issues, such as whether even face-to-face courses could benefit from some kind of review before they are offered. “The idea of the lone faculty member who can teach a course entirely on their own is an anachronism that can’t stand,” he argues.
A Range of Approaches
Coursera is hardly alone among upstart education providers in trying beta testing programs.
Udacity, another MOOC pioneer that has shifted its focus to selling what it calls nanodegrees, has its own process for beta testing courses before they go live. “The feedback received in our test groups is used to adjust the courses (i.e. some courses are re-recorded) or to recalibrate the way certain concepts are taught,” says Shernaz Daver, Udacity’s chief marketing officer, in an e-mail interview. In 2012, the company canceled a course on “Logic and Discrete Mathematics,” even though about 20,000 students had signed up for it, after its leaders decided the video lectures didn’t live up to their standards.
Leaders of edX, the nonprofit MOOC platform started by the Massachusetts Institute of Technology and Harvard University, encourages all course developers to do beta testing, though it doesn’t require it. But every team creating a course for edX must go through a MOOC Development Checklist “to ensure they have created a course that will deliver a high-quality learner experience,” said Rachel Lapal, director of communications for edX, in an e-mail interview.
In some courses, volunteer “community TAs” have been asked to rigorously check problem sets before they go live. In one popular course, “we had three testers for each problem,” she said. “They caught several bugs which we were able to fix before the course was released to learners.”
Such checking may be more like the way textbook publishers operate than traditional professors. So perhaps the question of how much testing to do an online course, is how much it resembles an interactive textbook, or how much it is essentially a live teaching experience that is more difficult to treat like a product.
