According to reports by ETS and Rennie Center, most data systems and reports do not include adequate support for interpreting data even though they have the ability. Often, when an edtech product communicates feedback via some data reporting component, the common assumption is educators will automatically know how to best analyze that data.
Unfortunately, education data is more complex than it often appears. For example, in U.S. Dept. of Ed. studies in 2009 and 2011 conducted in districts known for strong data use, teachers achieved only 48% accuracy when making data inferences involving basic statistical concepts. This should be no fault of educators, many of whom are highly educated and intelligent. Rather, we should consider how the tools that educators are using--and the developers--can better support them.
Recent Study
During April and May of 2013, I asked 211 educators at nine schools in six California school districts to participate in a study, which has just been published: “Over-the-Counter Data’s Impact on Educators’ Data Analysis Accuracy.” The premise was to determine the impact on analysis accuracy when data system reports provided clear instructions to educators, much like how over-the-counter medication offers labels and supplemental documentation.
The study employed a random, cross-sectional sampling procedure when incorporating responses from 211 educators spanning grades PK-12 grade and a variety of demographics that matched the general population of U.S. educators.
The study took place in computer labs at nine school sites, where I administered 10 multiple-choice questions. Six of these questions related to educator demographics, as the researcher examined both educator and site demographics as secondary independent variables and verified they had no significant impact on findings. The remaining four questions examined how educators make sense of student data in the reports they received.
Findings were significant. The following are findings that should be significant for anyone--educators, vendors, or other stakeholders--communicating data to teachers.
Recommendation 1: Footer
What does it look like?
A footer is a brief set of text at the bottom of each report. It communicates information an educator would need to know to correctly understand and analyze that particular report’s data.
Why should it be added to reports?
Educators’ data analyses were 307% more accurate when a footer was present and 336% more accurate when respondents specifically indicated having used the footer (which happened 73% of the time).
How should it be added to reports?
The four footers found to be effective ranged from 34-58 words, 156-269 characters without spaces, and 224-324 characters with spaces. The footers can be monochromatic or use minimal color purposefully (e.g., “Warning” in red and “What to Do” in green), and they should succinctly communicate only the most important information an educator would need to make successful analyses.
Recommendation 2: Reference Sheet
What does it look like?
A reference sheet--often called an abstract--is a single page that accompanies a report in order to help the educator more easily understand the report and analyze its data.
Why should it accompany reports?
Educators’ data analyses were 205% more accurate when a reference sheet was present and 300% more accurate when respondents specifically indicated having used the reference sheet (which happened 50% of the time).
How should it accompany reports?
The guide should be easily accessible, such as via a link displayed near the report title (for online users) and also printable and downloadable as a PDF (since 56% of users only read printed versions of reports others used the data system to generate). Free researcher-created sheet templates are available to help you create your own sheets for your own data reports.
The four sheets found to be effective contained the report’s title, description, image, focus (content reported), and cautionary information an educator would need to avoid the most common analysis errors made when analyzing the particular data being displayed. Two of these sheets also communicated the report’s purpose (key questions the report will help answer) and additional focus information (intended audience, and format in which data is reported).
Recommendation 3: Reference Guide
What does it look like?
A reference guide--also called an interpretation guide--is a 2- or 3-page reference guide that accompanies a report in order to help the educator more easily use the report and analyze its data.
Why should it accompany reports?
Educators’ data analyses were 273% more accurate when a guide was present and 436% more accurate when respondents specifically indicated having used the guide (which happened 52% of the time).
How should it accompany reports?
The guide should be easily accessible, as stipulated above for the reference sheet. (Free researcher-created guide templates are also available.)
The guides found to be effective featured the report’s reference sheet (described above) as the 1st page, with additional pages offering instructions (how to read the report), essential questions (showing the user where to look on this report--and what to look for), and a “more info” section (offering where to get additional information on related topics).
Recommendations 4-6: Help System, Package/Display, & Contents
Reports used in the study were modeled after formats most commonly seen when surveying education data system reports. This allowed for generalization of results but reports did not embody the best data presentation practices.Data displays commonly seen in student data systems are often not the formats most conducive to correct analyses. Recommendations include:
Help System
Each year, approximately 50 million different people use WebMD each year.Since most educators analyze data while alone, it is especially vital they have access to an online help system.
A product like ScreenSteps makes building a help system a cinch (that is what I used to build lessons for an edtech company and for a school district). An edtech product or school system can easily offer a link to such a support. The help system should follow standards to offer tech lessons (i.e., how to use the product) and also data analysis lessons (rarer to find in a data system, yet supportive of accurate analyses).
Package/Display
Over-the-counter medicine more safely communicates its nature when it uses effective packaging and display. For example, if the label features a child, the user can guess the contents are intended for a child. If such a product was instead meant for adults, the packaging would be misleading and possibly lead to harm.
In the same way, data reports should utilize formats that encourage accurate analyses. This involves following standards relating to key features, good design, and easy navigation.
Content
If a “flu remedy” actually contains nothing but sugar, it will not cure your flu. Effective product content is vital to user success. In the same way, a data system’s content must be as effectual as possible. This involves following standards relating to efficient input controls, report expiration, audience appropriateness, and precautions as to how the suite of reports works as a whole.
Educators Want This
Eighty-seven percent of study participants who received no supports indicated they would have used footers, reference sheets, or reference guides if the supports had been available. Educators are working hard to give students the best possible care, and data systems or reports that offer data in an “over-the-counter” format can significantly improve data analyses conducted on behalf of these students.
Over-the-counter data is not meant to replace professional development or other interventions that improve data use, but it is refreshing to find an added solution that doesn’t cost educators more time, money, or stress. That time is better spent on what educators do best: helping kids.
