Teaching and assessing problem solving skills – PISA recommendations

In 2012 PISA (Programme for International Student Assessment), which operates under the OECD, released their evaluation of problem solving skills in 15 year old students. The study defined problem solving and established an assessment regime. They then compared the results from 65 participating economies. The full report is at http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-v.htm

I don’t propose to critique, or even attempt to summarize the report, but as an educator there are some interesting points in it which I would like to refer to. I will be skipping through the report, highlighting what I see as key ideas for classroom teachers and mixing in a few personal observations and ideas. For the full context and supportive data you should refer back to the original report.

Importance of problem solving skills in the 21st Century

Across the countries it was found that a large majority of workers are expected solve a simple non-routine problem (taking less than 30 mins) once a week. One in ten workers are confronted on a daily basis with harder problems. Complex problem-solving skills are more in demand in the faster growing managerial and professional occupations.

A suggested explanation is that automated systems are increasingly dealing with routine problems, leaving the workers to deal with the unexpected or unfamiliar situations.

As a result we see the following trend in employment towards non-routine analytical and relational skills.

As educators preparing students for the world, we need to consider these trends. Amongst other things we need to be seeking to equip our students with the skills to thrive in non-routine problem-solving situations.

What are problem-solving skills?

In order to teach and assess problem-solving it is necessary to define the skills and understand them.

PISA defines problem-solving as:

…an individual’s capacity to engage in cognitive processing to understand and resolve problem situations where a method of solution is not immediately obvious. It includes the willingness to engage with such situations in order to achieve one’s potential as a constructive and reflective citizen.

(http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-v.htm pg. 30)

The key domains in the definition are identified as:

  1. Cognitive domain: The problem solver needs to engage, understand and resolve the problem.
  2. Problem domain: The problem is non-routine, meaning that the goal cannot be achieved by merely applying a proviously developed solution.
  3. Affective domain: The problem solver needs willingness to tackle the problem. I would add that willingness also pre-supposes confidence and the ability to handle failure positively.

The problem solving framework defined by PISA involves three main elements, further divided into parameters or processes:

Using this framework a problem can be categorized and the problem solving process (not necessarily linear as shown) described and evaluated.

Levels of problem-solving competence

To evaluate  the level of competence of a problem-solver there needs to be a progressive developmental framework. In previous posts I have discussed the progressions to evaluate collaborative problem-solving skills proposed by the ATC21S (Melbourne University) group and 21CLD . I have summarized these down to 4 levels relevant to secondary education https://doncollegegrant.wordpress.com/2014/08/06/digital-badges-for-collaborative-problem-solving/.

PISA present 6 progressive levels of development for problem-solving:

Level 1

  • At Level 1, students can explore a problem scenario only in a limited way, but tend to do so only when they have encountered very similar situations before. Based on their observations of familiar scenarios, these students are able only to partially describe the behaviour of a simple, everyday device. In general, students at Level 1 can solve straightforward problems provided there is only a simple condition to be satisfied and there are only one or two steps to be performed to reach the goal. Level 1 students tend not to be able to plan ahead or set sub-goals.

Level 2

  • At Level 2, students can explore an unfamiliar problem scenario and understand a small part of it. They try, but only partially succeed, to understand and control digital devices with unfamiliar controls, such as home appliances and vending machines. Level 2 problem-solvers can test a simple hypothesis that is given to them and can solve a problem that has a single, specific constraint. They can plan and carry out one step at a time to achieve a sub-goal, and have some capacity to monitor overall progress towards a solution.

Level 3

  • At Level 3, students can handle information presented in several different formats. They can explore a problem scenario and infer simple relationships among its components. They can control simple digital devices, but have trouble with more complex devices. Problem-solvers at Level 3 can fully deal with one condition, for example, by generating several solutions and checking to see whether these satisfy the condition. When there are multiple conditions or inter-related features, they can hold one variable constant to see the effect of change on the other variables. They can devise and execute tests to confirm or refute a given hypothesis. They understand the need to plan ahead and monitor progress, and are able to try a different option if necessary.

Level 4

  • At Level 4, students can explore a moderately complex problem scenario in a focused way. They grasp the links among the components of the scenario that are required to solve the problem. They can control moderately complex digital devices, such as unfamiliar vending machines or home appliances, but they don’t always do so efficiently. These students can plan a few steps ahead and monitor the progress of their plans. They are usually able to adjust these plans or reformulate a goal in light of feedback. They can systematically try out different possibilities and check whether multiple conditions have been satisfied. They can form an hypothesis about why a system is malfunctioning, and describe how to test it.

Level 5

  • At Level 5, students can systematically explore a complex problem scenario to gain an understanding of how relevant information is structured. When faced with unfamiliar, moderately complex devices, such as vending machines or home appliances, they respond quickly to feedback in order to control the device. In order to reach a solution, Level 5 problem-solvers think ahead to find the best strategy that addresses all the given constraints. They can immediately adjust their plans or backtrack when they detect unexpected difficulties or when they make mistakes that take them off course.

Level 6

  • At Level 6, students can develop complete, coherent mental models of diverse problem scenarios, enabling them to solve complex problems efficiently. They can explore a scenario in a highly strategic manner to understand all information pertaining to the problem. The information may be presented in different formats, requiring interpretation and integration of related parts. When confronted with very complex devices, such as home appliances that work in an unusual or unexpected manner, they quickly learn how to control the devices to achieve a goal in an optimal way. Level 6 problem-solvers can set up general hypotheses about a system and thoroughly test them. They can follow a premise through to a logical conclusion or recognise when there is not enough information available to reach one. In order to reach a solution, these highly proficient problem-solvers can create complex, flexible, multi-step plans that they continually monitor during execution. Where necessary, they modify their strategies, taking all constraints into account, both explicit and implicit.

I believe that this progressive development is sufficient basis for an assessment rubric and to establish zones of proximal development for students, which is the first step towards developing an educational process for teaching and assessing problem-solving. Test questions have also been developed and they can be viewed at http://www.oecd.org/pisa/test/ .


Analyzing their assessment results PISA found that some countries were doing better than others at teaching problem-solving skills. On this basis they were able to make recommendations on improving education in this area. There were suggestions for improving educational policy, but I have adapted the following 5 points from their recommendations, which can be implemented at the classroom level. I suggest you refer to the original document for a more detailed discussion.

Don’t teach solutions

In general problem solving is taught by focusing on rule-based solutions. This is most obvious in Mathematics education. This is really a two step process, the first step being formulation of the problem from a messy real-world scenario, the second step is the application of the solution. Once the solution path is established the rest of the process can be automated, so it is the first step that is the more valuable skill.

In order to assist students to develop skills in problem analysis and solution formulation they need to be exposed to numerous real-world problems.

In the language of ATC21S this means exposing students to real-world problem spaces so that they learn to develop, evaluate and select solutions. Presenting students with restricted problem spaces leading to defined solution paths is not developing effective problem-solving skills.

Teach for skill transfer by looking for connections

problem-solving skills developed in one domain do not readily transfer into another domain. Teachers can assist this transfer by using diagrams and illustrations to highlight the similarity between strategies across domains rather than the superficial differences of jargon or context.

In practice this might involve finding the similarities in the design process when designing a house or a prom dress, or calculating loads on roof trusses and optimum tacking angles for a yacht. These pairs of problems seem superficially different, but the problem spaces have things in common.

Skills are best developed in meaningful contexts

People are less likely to transfer isolated pieces of knowledge than they are to transfer parts of well-integrated hierarchical knowledge structures. The more connections a learner sees between the learning environment and the outside world, the easier the transfer will be.

(http://www.oecd.org/pisa/keyfindings/pisa-2012-results-volume-v.htm pg 121 )

Teachers need to be prepared to look at the real world, particularly the world that students live in. I have always been aware that this helps with the affective domain in problem solving (willingness and persistence), but the evidence shows that it also assists with the cognitive domain by aiding skill transfer across contexts.

Encourage metacognition

Students need to be encouraged to think about how they are thinking about a problem. Self awareness through the process is extremely powerful in developing problem-solving skills.

This can be encouraged by “thinking aloud” sequences. Solving problems in a collaborative setting is also a way of encouraging this, particularly if the communication is managed. For example if students are placed in the situation where they are collaborating through a network chat session it encourages them to communicate their thinking explicitly to one another. It also records that communication for later analysis and discussion. ATC21S made use  this strategy in their work. I have also explored this using etherpad.

Teachers also need to be courageous enough to model this behavior for the students. Because of their familiarity with the subject teachers tend to model problem-solving as a routine activity with their classes, simply because they generally go into class knowing how to solve all the problems. It is not a bad idea to sometimes attack a problem that the teacher does not know how to solve.

Utilize the visual arts

The visual arts are often devalued by teachers, particularly teachers of the core disciplines like Maths and Literacy, as a place where real problem-solving does not happen. The visual arts can be a powerful vehicle for developing problem-solving skills. On a superficial level students are learning skills and techniques, but on a deeper level participating in the visual arts involves:

  • Envisioning: Students are asked to envision what they cannot observe directly.
  • Observing: The skill of careful observation is taught.
  • Reflecting: Teachers often encourage reflection by asking open-ended questions about the work. Students are therefore encouraged to develop metacognitive awareness of their work.
  • Engaging and persisting: Students tackle projects which engage them, and they need to persist through frustration as they refine and develop their skill with the medium.
  • Stretching and exploring: Students stretch themselves and take risks in producing their work.

So the visual arts are a powerful context in which to teach the basic problem-solving tools.


This post constitutes a summary of what I learned form the PISA report. It has given me a lot to think about as I evaluate my teaching practice. 21st Century skills, like problem-solving, are the key to future success for our students. Effectively teaching them is as challenging as it is worthwhile and work by PISA, ATC21S and 21CLD are showing the way for classroom teachers like yours truly.




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: