Chair: Author 1

Introduction

The validity of testing rests on the presumption that scores are not contaminated by construct irrelevant factors. This symposium draws together four contributors from the Handbook of Social and Human Conditions in Assessment (Routledge, July 2016) who examine some sources of invalidity.

Contributions

Paper 1 focuses on validity challenges when students do not give good effort. Tests are vulnerable to this problem, particularly when they are administered under low-stakes conditions. Insights into test-taking motivation are evaluated and strategies that can mitigate this validity threat are highlighted.

Paper 2 argues that in multicultural societies our understanding of test validity and test procedures need to be reconsidered. Insights from culture-informed, multidisciplinary research studies identify acculturation as a neglected factor in testing in multicultural contexts. Best practices, based on extant knowledge, are described, with special attention given to mixed methods procedures.

Paper 3 considers the general public’s perceptions of educational tests. These are often based on by intuitive theories of tests and test scores, and end up encouraging the public to give support to inappropriate test policies. Examination of American surveys of public perceptions show that the uncertainties in scores introduced by psychometric theory are far removed from the minds of the public. The paper suggests that improving the assessment literacy of teachers is a key in overcoming limitations in public perceptions and thus policies.

Paper 4 analyses the bidirectional relationship between assessment and socio-cultural contexts. The authors consider the interaction of these facets from their own research and provide recommendations for working towards valid testing practices.

Conclusions

These four papers remind us of the complexity facing test designers and administrators caused by effects arising from individual students, societal and public perceptions, and enacted practices. They also point to research-based solutions that could lead to more effective testing practice and policy.

*****

Paper 1: The Validity of Assessment When Students Don’t Give Good Effort Author 2 + 3

We assess a student’s achievement to find out something about what the student knows and can do.  Obtaining valid information requires that we have a good assessment that has been competently developed.  But that is not sufficient.  To obtain valid information, the student must be motivated to devote enough effort to the assessment that the resulting information accurately reflects his or her knowledge, skills, and abilities.  Although low motivation can affect scores from any assessment, it has historically been of greatest concern with scores from assessment programs that are low-stakes from the student’s perspective.

The idea that test score validity depends on the effort expended by the student is logical and straightforward.  Yet, this idea has thus far had little impact on our psychometric models, which tacitly assume that if we administer a test item to a student, the student will give good effort.  Because instrument design makes no allowance for unmotivated students, the psychometric models currently used make a universal assumption of motivated test taking (Wise, 2015).  In reality, however, test takers frequently do not behave effortfully.  In these instances, measurement is distorted and validity diminished, because the absence of test-taking motivation represents a construct-irrelevant factor that can negatively distort test scores (Haladyna & Downing, 2004; Messick, 1984).

In this presentation, we examine low test-taking motivation and its effects on test performance.  Relevant theoretical models will be discussed, along with an array of empirical evidence that has informed our understanding of the dynamics of test-taking motivation. In addition, several strategies will be presented that have been shown to mitigate the impact of low motivation on validity, including test administration considerations, data filtering, score adjustment procedures, and the use of “smart†computer-based tests that can monitor test-taking effort (and intervene, if warranted) as a test event occurs.

Paper 2: Assessment in education in multicultural populations Author 4

It is relatively common in educational assessment for test creators to write instruments for a target audience that is linguistically and culturally homogenous, or to let their own underlying assumptions about the world and what is ‘common knowledge’ filter through. An adequate assessment of students in multicultural populations often requires a culture-informed, multidisciplinary approach. The presentation deals with issues of educational assessment in multicultural populations by presenting an overview of the main issues, challenges, and proposed solutions. The paper starts with a description of core issues like test design, validity, and use of test scores. It then reviews the concepts of bias and equivalence as an integrative framework to issues in multicultural assessment. Special attention is paid to acculturation as a neglected factor in educational assessment in multicultural classrooms. The relevance of acculturation, the way in which immigrant students deal with the ethnic and mainstream culture, is then discussed. An overview of theoretical models of acculturation, studies of school adjustment, and ways of accounting for acculturation in assessment is then provided. A taxonomy of bias and equivalence is presented to discuss problems of assessment. The final section describes ways toward best practices, highlighting procedures in large-scale educational assessments such as the Program for International Student Assessment (PISA). Both design features (i.e., the incorporation of mixed-methods studies) and statistical analyses will be described. Finally, conclusions will be drawn.

Paper 3: Public perceptions about assessment Author 5

Assessment has the potential to provide evidence about opportunity to learn, learning progress, student achievement, and effectiveness of instruction. Stakeholders (e.g., students, educators, public) will each have varying degrees of interest in what is assessed, how it is assessed, and the outcomes. A person’s experiences with assessment in the classroom or workforce can shape their perceptions. However, having taken an assessment does not give the test-taker an accurate understanding of appropriate test development and validation. Braun and Mislevy (2005) argue that most test takers and users draw on intuitive test theory when interpreting results, meaning that many assessment takers and users have a non-scientific understanding about how tests were developed and how they can/should be interpreted.

Longitudinal public opinion polls in the United States conducted by the Gallup Organization have included education and assessment topics (Elam, 1995). National education policies in the United States have increased the role of formal assessment in reform efforts. Earlier studies were conducted when there was more limited national involvement in education policy and practice. More recently, Bushaw and Calderon (2014) suggest continued support for formal assessments, but less support for national involvement in defining achievement. The changing perspectives may relate to increases in accountability expectations.

Educators play a critical role in transferring knowledge and developing learning habits, including assessment literacy. Long-term, informing public perceptions of assessment depends on ensuring that educators have appropriate perceptions and practices themselves. Klinger et al. (2015) respond in part through translation of concepts of validity, reliability, and fairness into classroom settings with a focus on foundational measurement principles and appropriate assessment use. More broadly, the measurement community will need a multilevel approach, including current members of the public, in outreach efforts. However, it is future generations of the public that would benefit the most from assessment literacy development.

Paper 4: Assessment and Sociocultural Context: A Bidirectional Relationship Author 6 + 7

While it is intuitive that the underlying beliefs, values, and assumptions within a society affect how assessment is designed, enacted, and interpreted, the effects of these constructs are seldom explored through research or seriously considered within assessment design and implementation. The conceptual framework guiding this presentation postulates that assessment and sociocultural context have a bidirectional relationship.  On the one hand, society influences assessment.  Culture and the characteristics of a society (mainly its history, social structure, priorities, values, technological development, legislation, communication styles, and ways of doing things and viewing the world) shape assessment content, who is to be assessed, how, and to which end.  On the other hand, assessment influences society: the information produced by assessment instruments and systems shape decisions made about individuals, institutions, and policy.  These mutual influences operate at different levels (e.g., test item, assessment program, country, international test comparisons).  Using multiple examples of the bi-directional relationship between assessment and sociocultural context, three conclusions are reached. First, while bi-directional, the relationship between assessment and sociocultural context is not necessarily synergistic. Second, both within a country and in international test comparisons, standardisation does not necessarily ensure equity. Third, ignoring the relationship between assessment and sociocultural context leads to wrong expectations about the ways in which assessment can contribute to fair education reform.