First administered in the 2000 to assess the quality of education systems across the world, the PISA (short for Program for International Student Assessment) is currently undergoing significant changes.
No longer does the test, given every three years to 15-year-olds, ask about math, reading and science. In the latest iteration in 2015, questions covered collaborative problem solving, social skills, and even psychological well-being.
The changes started before the 2012 exam, after the Organisation for Economic Co-operation and Development (OECD) recognized that the traditional multiple choice evaluations on traditional academic subjects were no longer sufficient to prepare students to succeed in the 21st century economy. But the test-makers first had to resolve this question: how can one create an exam with standard, but meaningful measures to nearly 80 countries? How can we one decide what should be measured?
Andreas Schleicher, the director for the Directorate of Education and Skills at the OECD, believes he has the answers. “We look very carefully at how the world and the skills that people need are changing and then we try to reflect that in our measure,” says the German statistician who has been involved with PISA since its beginning.
From his perspective, collaborative problem solving (CPS) is a skill that is important regardless of where one resides. After a pilot in 2012, the PISA test included a mandatory CPS section in 2015, which all students took alongside with math, reading and science. “We are not doing this because we think it’s just interesting, but we see that kind of skill playing an ever more important role for success in our society,” says Schleicher. In order to evaluate how students collaborate, the OECD developed an exam in which students use a chat tool to talk to team members and solve a problem together. For instance, the scenario may involve hosting a group of international students in their school and organizing a trip to an interesting place in the city. Students will have to consider opening hours of different venues, distance and costs to optimize the event. (Check out pages 49 to 89 for a detailed example.)
In interview with EdSurge, Schleicher talks about how PISA has been evolving, defends the importance of assessments and describes the use of technology in education as something with great potential, but still full of false hopes. “Putting technology on top of traditional teaching will not improve traditional teaching. In many countries technology is making results worse, not better.”
Q: When OECD uses PISA to measure certain skills, it is sending a clear message of what's important and of what kids should be learning. How do you decide which skills are important and how they should be measured? How do you decide to review those skills?
Schleicher: We look very carefully at the evolution of skills demanded in our society. Many of the skills that schools have traditionally emphasized—memorizing things and then recalling them—are becoming less and less important for the success of people. In contrast, creative thinking, collaborative problem solving and social skills are becoming more important. We look very carefully at how the world and the skills that people need are changing and then we try to reflect that in our measure.
For example, we know that it is important for people to collaborate, compete, connect, work with each other. In 2015, in the latest PISA, we have added one domain called collaborative problem solving. We are not doing this because we think it’s just interesting, but we see that kind of skill playing an ever more important role for success in our society. The world is becoming so complex that you can no longer solve a problem by yourself.
Q: Then we get to a gray area where we have to measure things that are not easily measurable. It puts PISA in a position of creating new ways of evaluating...
Schleicher: That’s exactly what we have done. We have created a new metric, a new measurement that we have included in our last PISA exam.
[Check out the Draft Collaborative Problem Solving Framework for details in the CPS exam and the Draft Questionnaire Framework, pages 15 and 16, for details on the social-emotional skills assessment]
Q: In 2013, OECD piloted a social-emotional assessment in Rio de Janeiro with more than 24,000 students in grades 5, 10 and 12. Are there plans to incorporate this test into PISA? Also, how do you decide which skill are you going to measure?
Schleicher: We have incorporated a first set of measures in our latest PISA exam looking at things like resilience [and] motivation. This is very important and difficult to measure, but we increasingly have added those types of dimensions to the PISA test because they make it very important to lots of people. We wrote a book, Skills for Social Progress, on this.
We look at the skills that are important for the success of people and we make sure this is appropriate for the 80 countries that take part of PISA. It should be the most important aspects of the success of people. That's a discussion. Countries debate what's important for them.
Q: Every country and state also administers exams. What should we do to make sure that we are not over-testing students?
Schleicher: These tests [PISA and local tests] are complementary. You can look to outcomes with multiple lenses in classrooms, but we have to make sure that there is coherence between these different instruments. Evaluation should be part of the pedagogy in a classroom, part of the learning process. Teaching, learning and assessing they all belong together.
Q: A recent OECD report found “no appreciable improvements in student achievement in reading, mathematics or science in the countries that had invested heavily in ICT for education.” How do you feel about the use of technology in classrooms?
Schleicher: I have two thoughts about this. On one hand, the potential of technology is very significant. It enables us to use innovative pedagogy, to learn collaboratively, to connect with other people, to connect teachers. On the other hand, the reality is that technology is very poorly used. Students sit in a class, copy and paste material from Google. This is not going to help them to learn better.
Technology can always make good teaching better, but it cannot replace poor teaching. Putting technology on top of traditional teaching will not improve traditional teaching. In many countries technology is making results worse, not better. The answer is to not adopt technology, but reinvent schooling and pedagogy to making learning more effective.
Q: Another study suggested: “In the United States, the likelihood of low performance in mathematics is higher for students who are socio-economically disadvantaged and had repeated a grade.” Are we punishing or benefiting low income students?
Schleicher: That’s [narrowing the socioeconomic gap] the current promise of technology. Technology can move good instruction to the classroom and can make an impact, a better future. Technology could be used for equity in education, but currently it is going the other way around. Teachers and students in the wealthiest areas get the better teachers, better technology. Students in remote areas have neither access to good teachers nor technology. The potential of technology is precisely to narrow the social gap, but I don't think we've had a lot of progress to realize that potential.
