Overall Abstract

Issues central to addressing fairness and validity in assessments are considerations of how to accommodate the needs of linguistic minorities. This issue is relevant to multiple countries. In particular, it is of import to countries with higher immigration and student mobility rates and countries having more than one national or official language. Several challenges arise for assessment development and the validity of score-based interpretations. These issues are heightened if the home language differs from the language used in the school, community, or on the test.

In this symposium, we explicate these issues in the first presentation, which serves as a backdrop to illustrate the complexities associated with the fair and valid assessment of linguistic minorities. The other presentations illustrate these complexities along various test development stages and provide recommendations to develop fair assessments for linguistic minorities to support the validity of score-based inferences for minority populations. We also present the ITC Guidelines developed by a team of international experts in support of the fair and valid assessment of linguistic minorities.The guidelines (and our presentations) were designed to inform foundational and/or operational processes when assessing minorities spanning across test development, administration, scoring, score reporting and interpretation of test scores.

The presentations aim to inform test developers, test administrators, psychometricians, practitioners, and users on central issues relevant to the fair and valid assessment of linguistic (or cultural) minorities and raise awareness of the needs and issues that are associated with their assessment. No other such guidelines currently exist.

*****

Presentation 1: Context of Assessing Linguistic Minorities
Fons J. R. van de Vijver, María Elena Oliveri, and Paula Elosua

Linguistic diversity in countries is very common. However, the relationships between languages, dominance, and prestige of one or more languages, use of ethnic and dominant languages in education, and the use of local dialects differ across countries.

In this presentation, we provide an overview of the main issues arising when assessing linguistic minorities and the contextual factors influencing the resources, decisions, and development efforts afforded for various language groups. We exemplify linguistic differences arising in the status of the minority versus the majority language (high or low or official status); the degree of codification of the minority language; whether the minority language has an official or vernacular form and the degree of resources available for adapting tests to the minority language the language of instruction. We also discuss whether the members of the linguistic minority belong to established (long-standing) or transient (recent) immigrant groups.

We provide an overview of the relevant literature and exemplify various linguistic contexts, which serve as a backdrop for the considerations (tradeoffs and constraints) associated with assessing linguistic minorities. We exemplify issues such as differences in the status of minority languages, in the codification of languages, in the role of dialects, and in the role of the language of instruction. We illustrate that the linguistic situation in the Western part of the world is rather different from other parts of the world and discuss the implications of these differences on assessment development efforts. We conclude that a “one size fits all†strategy to deal with linguistic diversity will have limited applicability in the fair and valid assessment of linguistic minorities. We close by proposing approaches to consider population heterogeneity in the analyses and conceptualization of assessments administered to diverse populations.

Presentation 2: Guidelines on Developing Fair Assessments for Linguistic Minorities
Stephen Sireci and Molly Faulkner-Bond

Throughout the world, tests are administered to some examinees who are not fully proficient in the language in which they are tested. It has long been acknowledged that proficiency in the language in which a test is administered often affects examinees’ performance on a test. Depending on the context and intended uses for a particular assessment, linguistic proficiency may be relevant to the tested construct and subsequent interpretations, or may be a source of construct-irrelevant variance that undermines accurate interpretation of the test performance of linguistic minorities who are not proficient in the language of the assessment.

In this presentation, we describe guidelines on the fair assessment of linguistic minorities. Our section highlights key fairness considerations to be acknowledged when testing linguistic minorities, regardless of whether language is central or construct-irrelevant. We also illustrate the different types of studies test users and developers could conduct to evaluate fairness when assessing linguistic minorities and discuss diverse approaches to address challenges psychometricians, analysts, and users face when analyzing fairness with linguistic minorities. We also exemplify challenges such as providing strategies to use when conducting differential item functioning (DIF) analyses with small sample sizes or to examine possible sources of DIF such as conducting linguistic analyses of items and cognitive labs. The presented fairness issues span along various test development stages. We conclude with a list of critical factors to consider and possible strategies to mitigate various threats in test development and evaluation whenever linguistic minorities are assessed.

Presentation 3: Validity Considerations on the Assessment of Linguistic Minorities
María Elena Oliveri

While the validity of score-based inferences is important during the test development and test conceptualization stages, herein emphasis is given to early test development stages (i.e., construct conceptualization and domain analysis). A preemptive focus is of import to mitigate possible sources of construct-irrelevant variance that otherwise may be detected as differential item functioning (DIF) posttest development. A second reason for the preemptive focus is the notion that reducing irrelevant sources of construct accessibility (due to differences in curricular exposure, opportunity to learn, or item type unfamiliarity) are best addressed prior to test administration.

This presentation has three goals. First, to discuss features of a test that might introduce sources of construct-irrelevant variance at the construct conceptualization and test adaptation stages. The second objective is to exemplify approaches to help reduce irrelevant sources of construct accessibility when assessing linguistic minorities such as the use of universal design and evidence-centered methodology to develop items and constructs that are more accessible to linguistic minorities. Real-life examples derived from assessments administered to linguistic minorities are provided to illustrate possible sources of construct-irrelevant variance arising in the context of diverse populations.Third, I discuss guidelines designed to enhance the validity of score-based inferences and reduce possible sources of construct-irrelevant variance when assessing linguistic minorities.

The presentation goals are aligned with AERA, APA, and NCME (2014) guidelines which state that “test design, development, administration, and scoring procedures that minimize barriers to valid score interpretations for the widest possible range of individuals and relevant subgroups†should be used (Standard 3, p. 63).

Presentation 4: Understanding Item and Test Development and Adaptation Guidelines
René Lawless and María Elena Oliveri

As demographic shifts in a country’s population occur due to immigration and/or student mobility, the proportion of majority language speakers may also shift for reasons such as having countries with more than one official language or moving across regions. Test takers may be disadvantaged, if they do not have sufficient familiarity with the majority language to demonstrate knowledge of the assessed constructs. The goal of item development and adaptation is to minimize construct-irrelevant variance for the targeted test population through an increased awareness of the characteristics of test items that may introduce undue difficulty for linguistic minorities. These characteristics may be easily overlooked. Unless a test or item is specifically designed to assess proficiency in the language of the assessment, certain aspects of the assessment must be attended to, in an effort to minimize construct-irrelevant difficulty.

The aim of this session is to discuss operational aspects of assessment design that test developers should attend to, in order to make stronger connections to the construct and potentially eliminate those elements that may introduce undue difficulty for the linguistic minority examinees. A second goal is to discuss item development guidelines to help minimize construct-irrelevance for linguistic minorities and make tests and items more accessible in terms of the features of items and tests in terms of wording, figures, format, directions, rubrics, and topics. The third objective is to provide concrete examples that make those characteristic salient.

The presentation objectives support the AERA, APA, and NCME (2014) guidelines in relation to improving test design and development to help minimize barriers to accessibility in support of assessing linguistic minorities.

Presentation 5: Communicating Test Results: Recognizing Linguistic Diversity to Foster Test Use
April Zenisky and Ron Hambleton

Reporting results is the public face of testing. Too often across a wide range of testing contexts and examinee populations, reporting has not been prioritized on par with other aspects of test development, administration, and scoring. However, given the increased focus on test use and the higher stakes associated with test results that has emerged in recent years, reporting has taken on greater importance. Accordingly, it is a critical consideration for testing agencies the world over. With respect to linguistically diverse populations, including those where linguistic minority subgroups are present, it is especially important that report materials (report documents as well as ancillary materials) be prepared and vetted in accordance with systematic guidance from the psychometric literature. Such guidance is relevant to various elements of reporting, including score report design and content, delivery of and access to reports and other materials, and interpretation and test use.

Through our presentation, we hope to set the context for the score reporting guidelines within the full set of guidelines for linguistic minorities. There are several key considerations in report development generally, and with respect to linguistic minorities, the quality of the processes for communicating test results has significant implications for validity. Our discussion of these guidelines will focus on articulating a conceptual framework for reporting in a given testing context, and will aim to integrate the concepts embodied by these guidelines into operational report development work to ensure that test results are communicated effectively to all subgroups within the testing population.

We will provide an overview of the authored guidelines relative to reporting and illustrate practical applications and uses of these guidelines by drawing from real-life examples of assessment involving populations with diverse linguistic backgrounds.

Our objectives support the AERA, APA, and NCME (2014) guidelines in relation to improving test design and development to help minimize barriers to accessibility in support of assessing diverse individuals and subgroups.

 

Discussant: Kurt Geisinger