The impact of large-scale international assessments on educational systems has consistently increased  in the past decade. The results of the Trends in Maths and Science Study (TIMSS), the Progress in International Reading Literacy Study (PIRLS) and to greater extent the Programme for International Student Assessment (PISA) have shaped educational policies and reforms in various countries. While these surveys could potentially provide invaluable insight about specific educational systems, there has been considerable controversy about the quality and the validity of the instruments employed. Central to those debates was the issue of developing equivalent language versions of the same assessment that ensure a fair comparison of student achievement across countries.

Translation and adaptation procedures applied in international assessments are highly rigorous. Nevertheless, empirical evidence points to bias in items across different countries. With language being a complex variable, highly associated with culture and thinking, developing equivalent language forms of the same test is a complicated enterprise. The impact of translation is inevitable and bias will be unavoidably manifested at some level. This compromises the validity and reliability of international tests and raises concerns about the degree to which policies based on these tests are founded on sound grounds.

In this presentation, I argue that language is an integral and inextricable part of the construct and the current practices in international assessments rest on major conceptual and methodological challenges. I use examples of items in three languages (English, French and Arabic) including released PISA items to illustrate my argument. The presentation concludes with a proposed way forward.

Keywords: trans-adaptation, language bias, international comparison