Font Size: 

Interactive complex problem solving is an important 21st century skill (Neubert et al., 2015). Items of the PISA 2012 included interactive problems requiring exploration of a novel device (e.g., a virtual MP3-Player) and ideas for its modifications. The aim of our analysis is to show that the use of “standard lists of creative answers†used to assess creative problem solving can lead to missing some correct solutions. The object of the analysis is the fourth question of the MP3-Player Unit: “Describe how you could change the way the MP3 player works so that there is no need to have the bottom button (http://www.oecd.org/pisa/keyfindings/PISA-2012-results-volume-V.pdf). Authors of the item write: “There is no single correct answer, and students may think creatively in devising a solutionâ€. Nonetheless, they limit the number of correct answers to the six that are described in the guidelines. Other responses are not accepted and result in a score of 0 (PISA 2012; https://nces.ed.gov/surveys/pisa/doc/cb-mp3.doc). We argue that the authors’ list of correct answers is not complete: it includes neither double successive nor simultaneous clicks, even though “double-clicking†is a common ergonomic solution in modern devices. A possible paradoxical reason is that the authors agree entirely with their example of universal bans formulated about 15 years ago, before the era of multi-touch devices: “at no point two buttons have to be pressed simultaneously†(Greiff, 2012, p. 52). However, one can easily show that solutions based on different successive and simultaneous double-clicks should be included in the list of correct answers for the item (Poddiakov, 2012). The paradigm of assessing creative answers in accordance with lists of criteria prepared in advance (“standard lists of creative answersâ€) seems to be an oxymoron in the context of assessing 21st century skills and needs to be changed.