Can college students tell the difference between a human teaching assistant and a chatbot? A Georgia Tech computer scientist has been pursuing that question for several years. And the answer has profound implications for the future of college teaching.
Ashok K. Goel, a professor of computer science and cognitive science, and his team have refined perhaps the first robot teaching assistant, dubbed Jill Watson. Here’s the scenario: Last year in an online course on artificial intelligence with 400 students, two chatbots joined 13 human TAs to answer student questions about the course and its content. Students were told that software robots were in the mix, and they were challenged to identify which of the voices they were interacting with were human and which were machine.
The robot TAs were given the names Stacy Sisko and Ian Braun. One bot was designed to be a bit more personable than the other: if a student mentioned she was from Chicago, Stacy Sisko would make a comment about the city. Ian Braun was all business, and weighed in on fewer student questions (he was an older version of the software that the researchers keep refining).
At the end of the semester about half of the students correctly guessed that Stacy was merely computer code. Only ten percent correctly identified Ian as a bot. And ten percent mistakenly thought that two of the human TAs were chatbots.
This semester, students did better at unmasking the bots, but not because they sounded more robotic, says Goel. “The performance has reached the level where she is performing so well that students say ‘This one must be a bot,’” he says. For instance, students can look in the online forum and see that the chatbot quickly responded to dozens of students, while the human TAs have only had time to type out replies to a few in the same amount of time.
Goel stresses that he has not cut back on human TAs since bringing in the Jill Watson software—he hires the same amount for the same number of hours per week. But he says what those humans spend their time doing has changed.
“The human TAs were just answering the more mundane questions again and again and again,” Goel says. “By automating some of the very mundane things, we’re freeing up time,” he says, to do things like give more-detailed answers to other student questions. “There are so many things to do as a teacher that students can take as much of your time as you have,” he argues.
Georgia Tech is not alone in developing teaching bots. IBM Watson Education is building its own chatbot that it has added to some commercial textbooks as part of a pilot project. (The Watson AI system, you may recall, famously beat a Jeopardy champion on the trivia game show a few years back). And Deakin University, in Australia, is developing a comprehensive student assistant it calls Deakin Genie that it hopes will help in the classroom, as well as with more administrative aspects of campus life. There are also a handful of startups creating AI chatbots to help with everyday questions students might have on about, say, financial aid deadlines.
Outside of higher education, the era of assistive bots has arrived (just ask Siri on your iPhone or Alexa in your kitchen). But there has been resistance by many in higher education to bringing them onto campus because they are associated with being too automated and too impersonal, says Brian Fleming, executive director of an innovation center at Southern New Hampshire University called the Sandbox ColLABorative.
“There’s a right degree of skepticism,” he notes. “The question for adoptions in higher ed is, ‘What is the problem bots solve?’” That will differ for different institutions, he adds, praising Deakin Genie and experiments with bots for things like admissions, enrollment and advising.
But he doubts that experiments like Jill Watson will work outside of a few computer-science classes (where, after all, the students are learning to build bots themselves). “We have still never seen conclusive evidence that anyone wants to be taught by a bot,” says Fleming. “Teaching and learning will remain a human endeavor.”
Goel, however, sees a future where chatbots are part of the teaching mix in a range of disciplines (though he agrees that humans will always take the lead). He plans to test his software in a biology course soon, and believes it could work for poetry as well. His plan is to publish scripts for his chatbots that others could use or adapt to build their own versions of Jill Watson. “My sense is that we are at least two or three years away from even the first script,” he says.
So what part of the teaching role can a chatbot never do? Goel recognizes that his work raises the question. “If in 10 years every student has such an assistant and every teacher has such an assistant, than the roles of the teacher fundamentally changes, I would think.”
His latest idea is to create what he calls a “family of bots,” each with a specialization. One might be designed to explain the syllabus, another to answer course content, and another to welcome students (and perhaps other micro-roles).
And this semester the researchers are trying out new names from different ethnicities to see if that impacts how students interact and view the virtual TAs.
Even if such chatbots never make it past the research phase, they seem certain to teach humans something about what it means to teach.
