School leaders often face a particularly vexing question when using surveys: How do I know if the feedback accurately represents the opinion of my teachers, parents, students and other stakeholders? How do I know if I’m measuring what I think I’m measuring?
The answer? It depends on how you ask. Surveys represent an immensely valuable tool for educational leaders. They offer insights into the unique--and often differing--perspectives and experiences of your constituents. But only if you ask questions thoughtfully.
Consider the following example, in which two polling organizations asked nationally representative samples of respondents to share their opinions about the Common Core State Standards last year. The Phi Delta Kappa/Gallup poll asked:
Do you favor or oppose having the teachers in your community use the Common Core State Standards to guide what they teach?
Among respondents, only 33 percent reported favoring the Common Core.
Education Next, however, asked about these standards another way:
As you may know, in the last few years states have been deciding whether or not to use {the Common Core, which are} standards for reading and math that are the same across the states. In the states that have these standards, they will be used to hold public schools accountable for their performance. Do you support or oppose the use of these {the Common Core} standards in your state?
In this case, 53 percent of respondents supported the Common Core – a 20 percent boost from the PDK/Gallup poll. When the Education Next pollsters used another version of this question that removed any mention of the phrase “the Common Core”--the phrases in brackets above--support jumped to 68%!
So how many people actually support the Common Core? What’s the right answer? If you were a principal who read results from multiple surveys with these sorts of contradictory results, which data would you use to make decisions for your school?
The truth is that people’s attitudes – and particularly their attitudes about nuanced, multi-dimensional issues such as the Common Core – are complex. Different questions about the Common Core will inevitably highlight different aspects of education policy and draw out varied responses from individuals.
In the first example, the PDK/Gallup poll’s question highlighted the role of the teacher and how standards are expected to guide instruction. The Education Next poll highlighted accountability; its second version of the same question removed the hot-button phrasing, “the Common Core standards.” No single survey question could possibly capture all facets of people’s attitudes toward such a complicated issue with any precision.
In order to collect more accurate data about the multi-dimensional topics that impact our schools, surveys must ask a set of questions--what researchers call a “scale”--so that the idiosyncrasies of each individual question cancel each other out. This is exactly our approach with the Panorama Student Survey, produced by data analytics company Panorama Education. Our survey gauges students’ perceptions on classroom environment, expectations, rigor and grit, among other important aspects of learning.
For instance, to measure student engagement, the Panorama Student Survey asks the following questions:
- In this class, how much do you participate?
- When you are not in class, how often do you talk about ideas from class?
- How excited are you about going to this class?
- How often do you get so focused on class activities that you lose track of time?
- Overall, how interested are you in this class?
By asking about students’ behaviors, the persistence of their interests, their emotions towards class, etc., we get a representative cross-section of the different features of classroom engagement. After all, different students engage in class in many different ways. What’s more, employing survey “scales” rather than single questions allows educators to quickly bolster the accuracy of survey results and obtain a much more accurate and nuanced view of any given individual’s attitude about a broad topic. The result is that educators have better data with which to make better decisions.
This same approach--selecting a representative set of questions to improve data quality--could also help us better understand people’s views of the Common Core State Standards, for example. By using multiple questions to tap into different characteristics of the standards, we could better understand public opinion about the Common Core as well as other pressing educational policy issues.
