“We dropped over $10,000 last year on a math product, but I’m not sure if it’s really helped our students learn.”
Sound familiar?
Over the past few years, “efficacy” has entered the edtech lexicon as a word commonly used to reference whether or not edtech is driving student learning. But as educators and edtech companies look to track the most effective tools and practices, two big questions are at the fore: How do we measure technology’s effectiveness? And if there’s an answer to that question, then is the technology actually improving student learning—or has it been a waste of time and money?
With nearly 20,000 people in attendance at this week’s ISTE conference, I chatted with a few teachers, administrators, and the president of ISTE during the conference to gauge their take on these questions.
How Should We Measure Technology’s Effectiveness?
Educators frequently agree that it’s challenging to draw a direct correlation between technology and student success. On the other hand, many feel that the most useful indications they have aren’t data but instead their own observations about how students are reacting to the technology.
Shana White, for example, is a 15-year veteran P.E. teacher and tech coach from Georgia who “loves conducting classroom visits” to get a firsthand look at students on tech platforms. Similarly, over in Minnesota, a K-6 STEM teacher named Rachel Pierson relies on feedback from the students themselves. “If students are engaged, that’s success. But if they’re turned off, it’s time to try another method,” Pierson told me, while Shelly Stout, a Texas teacher, feverishly nodded in agreement. Stout adds that she’s also used SurveyMonkey to gather students’ point of view on which platforms they do and do not like.
Administrators were a bit more tempered in their responses, expressing hesitation about whether live observation is the be-all, end-all in assessing edtech impact. Blake Gould of DSST Public Schools in Denver, for example, says he pays attention to whether students are excited about learning--but that’s not enough. His team at DSST looks at data like shortform student mastery checks and longform MAPP testing scores, feeding them through data analysis platform Ed-Fi.
Mike Dorsey, Director of Secondary Curriculum, Instruction, and Assessment for the Houston Independent School District, felt the same. Even as his team has tried to integrate observations of “technology integration” into teacher evaluations, he says that it “seems awfully subjective from time to time.”
How then, can schools integrate quantitative data into those classroom observations?
Multiple choice tests and high-stakes assessment had few fans at ISTE. Minnesota’s Pierson conceded she understands the utility of standardized tests from a state perspective: “If you ask every student [the same] question, you can compare it across the state, which can be useful.” Stout, on the other hand, doesn’t like multiple-choice testing at all: “Standardized tests measure skills of a factory model.”
However, adaptive technologies and tech-enhanced projects that allow teachers to find gaps in student learning received a thumbs-up from several of EdSurge’s interviewees, with Pierson referencing one platform in particular as “the best way to get immediate feedback, both for myself and my students.”
For instance, last school year, Shana White’s P.E. students took to the Internet to do research on a nutrition unit. Students posed as dieticians, received fictitious descriptions of patients with dietary restrictions, and researched restaurants to answer questions such as: “Would someone who’s gluten-intolerant be able to dine here?” Her data points included student work and formative assessment—and after the project, two of her most overweight students ended up losing 30 pounds.
Balancing quantitative and qualitative data, say the educators, can create a holistic view of edtech’s impact—and can be practical, given how quickly edtech products change. Instructional Technology Specialist Wanda Terral (of Lakeland School System in Tennessee) warns that relying on numerical data alone can be misleading, especially if a school decides to switch its focus to, say, student portfolios during the following year.
“It’s important to find a sort of assessment that’s generic enough to be flexible,” Terral says. “Trends change. Products come and go.”
So… Is All This Tech Implementation Work Going Somewhere?
Talk to former K-12 edtech director and education thought leader Alan November, and you’ll find that he’s not too pleased with how technology has impacted classroom learning.
“Regardless of the test, the impact of tech has been negative. Google is distracting, ethics are a mess, and the industry has to come to grips with the fact that this is all a mess,” November says. He adds: “Students don’t know how to approach messy questions! There’s no inquiry!”
There’s data to back his concerns. A 2015 OECD report assessing the impact of tech across 30+ countries reported that those countries investing the most money in school technology have declined the most in student performance. “I’m quoting Stanford’s Larry Cuban when I say that technology is way oversold, and it has under-delivered,” November says.
Part of the problem, observes ISTE President Brian Lewis, is that too much emphasis has been placed on the products themselves rather than on schools designing effective rollouts or implementation strategies. “We’ve all seen that distributing hardware becomes the impetus [for all of this change],” Lewis observes, “but that’s not effective.” Instead, he recommends that schools take three steps--and do that in the following order:
- Create a plan to tie technology to curricular objectives;
- Organize professional development to get teachers prepared, and then
- Choose the best technology for the job.
Many of the educators I encountered at ISTE echoed the sentiment that the technology itself isn’t as important as how one chooses to use it or the training that educators receive. And no one made the point more strongly than Houston ISD administrator Mike Dorsey. “Unfortunately, the measures of success that public schools look at are too narrow. It’s time that we look at the whole package, evaluating teachers based on their use of the tools and resources.”
Dorsey also had a message for all of his fellow administrators out there, one that he takes to heart.
“You have to remind yourself that it’s not about the new toys. It’s about teaching kids to think.”
