Kelly Dumont is an education technologist for Canyons School District in Sandy, Utah. On December 5, his district was one of many around the country that experienced a massive Chromebook error that caused hundreds of thousands of devices to temporarily disconnect from the internet.
Dumont, who is the elementary edtech team lead in his Utah district, believes that all 20,000 of the devices in CSD were affected by the issue, which was resolved by Google later that day. Now that the devices are back online, however, he and other educators question whether the error was a one-off glitch, or a foreshadowing for other unexpected issues in the cloud-based technology.
“This was temporarily catastrophic,” Dumont tells EdSurge.
On a support page setup to assist educators in re-connecting their decies, Google attributes the error to an “invalid network policy” that was pushed out to devices. The error caused devices to lose “connectivity to passphrase-protected WiFi networks configured through admin policies.”
Mass policy updates like the one that affected devices last week are not uncommon—though the effect last week is believed to be a first. “We issue network policy updates on a regular basis, which varies based on admin updates and device boots,” a representative from Google wrote in an email.
Dumont is aware that these types of updates are routinely pushed through the cloud-based system. But the event leaves him wary nonetheless.
“It’s an eye-opening situation because we are trusting the cloud so much for everything, and we rely heavily on Google services,” he says. “Google Apps for Education is our baseline for 1:1. The potential to have 20,000 devices go down at once is concerning.” Adam Henderson, the director of technology systems at Nassau County Schools in Florida, shared similar concerns with EdSurge last week while the issue was still unraveling.
Google would not disclose how many devices were affected in total, or if a specific segment of users was impacted.
In CSD, Chromebooks were down for less than an hour. But Dumont worries about what might happen if something similar occurs again. If Google were to push another buggy policy update “that would have a long term effect or can’t be reversed,” he says, “it would be a big concern.”
Following a rush of questions about the issue from educators on Twitter and Reddit, Google last Tuesday released a series of steps districts could take to resolve the problem. Those steps consisted of manually logging devices onto a different or open network, allowing the policy update to install, then logging back to the preferred network.
Educators in CSD took these steps, and devices went back on smoothly according to Dumont. And while he initially anticipated the next morning would consist of of manually logging in 20,000 devices, he says that didn’t end up being the case. Come Thursday, many of the devices were able to re-connected themselves, and the districts IT staff had to manually connect fewer than 500 Chromebooks.
Erin Strickland, a media specialist at Rome High School in Georgia, shares a similar experience. “There was not a major problem with reconnecting. We had to manually enter our WiFi password for around 20 percent of our students, which took some time, but their Chromebooks worked fine after that,” she wrote in an email.
But another common thread between the two educators is less optimistic: neither received a heads-up from Google prior to the update. “We were not given notice,” Strickland wrote.
Dumont did not either, however he believes that even if Google had released a notification, it likely would not have changed the outcome last week.
“I don’t think anything [Google] would have said when pushing that update would have indicated this may wipe out your WiFi or liscenings,” he says. “I’m hoping our IT team will pay more attention to any updates.”
Along with disconnecting devices from the internet, Dumont says his district’s Chromebooks also temporarily lost their licensing during the blackout. (To operate a Chromebook using Google management, a user—such as a district—pays to have their devices licensed by the tech giant.)
“It showed we had 20,000 Chromebooks but no management license,” says Dumont. “My understanding is the first [policy] push took that licensing out of the consoles, and that would include the profiles for the WiFi. The second patch re-enabled [internet connection] and put the licensing back in place.”
