I’ve gotten two calls from reporters recently asking me about the “dangers of analytics” in higher education.
I’m always quite careful to say I think there’s a lot of promise for analytics in higher education. I can’t imagine a future where we’re not using analytics extensively to try and improve what we do. And I have no doubt those analytics will guide us on where to apply resources in some high stakes areas--how we spend advising time, where we apply remediation strategies, which students we select for special attention, what gateway courses we should fund.
But it’s precisely the power and potential of analytics that makes it so important we get this right. And what getting this right means, first and foremost, is we do it in the open, and avoid the “secret sauce” mindset we’ve tended to have about such things. As we move out of the institutional experimentation phase and into the commercialization phase of analytics, institutions are increasingly being sold a black box of formulas they can neither share nor explain. And more than anything else, it’s this “magic number” mentality which makes analytics dangerous.
The anti-analytics crowd may object that openness is not enough. We are, in fact, turning governance over partially to formula, and that’s frightening. Sugar-coat it all you like, but no matter how much individual agency you preserve in your process of identifying at-risk students, you wouldn’t be using analytics if they didn’t somehow reduce or otherwise impact your decisions. You can’t simultaneously claim that analytics are powerful and harmless.
But protestations that such a shift is unprecedented are unfounded. The truth is that we are ruled by formula more than most people would admit right now. The highway fund in the United States is a percentage of the gas tax. Your Social Security benefits are indexed to inflation. Cost of living is used to compute a number of benefits and your eligibility for various forms of student aid is a function of a poverty-level formula developed in the 1960s.
The difference with education analytics, as it is being implemented, is not that we are turning some of our adjudication over to formulas, but that those formulas are often shielded from our view.
If a set of students who should be receiving aid are not, I can look at the poverty level formula and understand the ways in which that formula may be distorting public policy or exacerbating inequality because that formula is part of public record. We can have a debate about that.
But what if I am a student who is not selected for special intervention at a college, even though I clearly need help? If it is on the basis of written policy (say, advisors will contact all students with a first semester GPA of less than 2.3), we can debate that policy. If it is a matter of personal judgment, we can hold the individual responsible for that judgment accountable, and ask them to explain their reasoning.
If it’s a “secret sauce” algorithm on the other hand, what’s our option for public discourse on it? Where is our opportunity to examine it for racial bias, for bias against part-time students, or even for general error?
We’re left with “trust the programmers, they’re objective” in a world where the programmers have repeatedly proved that not only are they not objective, but quite often just plain wrong about the math. See, for example, Purdue’s Course Signals, or the strange and sad story of Google Flu Trends.
There’s absolutely much to be gained by using the data at our disposal to try and serve our students better. But deciding which students to expend extra resources on is not the same as guessing your likely personal rating for Pulp Fiction. Public policy, public money, and the public good require an accountability above and beyond individuals deciding whether they will give $8 a month to Netflix.
Analytics, yes. But no more secret sauce analytics. It’s time to open up these formulas to the light of day and get public comment on them. I understand that involves additional work on the part of institutions and corporations engaging in analytics. But it’s difficult to see how we hold institutions accountable to the public if we don’t make them show us the formulas producing their decisions.
