Introduction

Tests with novel item formats have long played a role in the assessment of cognitive abilities. With innovations in computer-based testing and performance assessment, novel and complex items are increasingly used in achievement testing as well. However, novel item formats are known to show large practice effects and give an advantage to students who are more familiar with the test format. The assumption that an item format is equally familiar to all students is especially problematic among culturally and linguistically diverse students whose experiences with testing may vary considerably.

Objectives

Our purpose was to explore the impact of test practice and test directions on the test performance for novel item formats, particularly in multicultural and cross-cultural assessment contexts.

Methodology

Quasi-experimental data was available on the test performance of English learner (EL) students in the U.S. compared to non-EL students. We tested 882 first- and second-grade students who were presented with one of three different types of test directions that were based on Universal Design principles, but emphasized different methods of providing test directions and practice. This data provides evidence of best practices to complement the literature review.

Results

Research has shown that test directions need to provide tailored feedback, consider common misunderstandings, and actively encourage students to attend to the directions. In our data, teacher proctoring of the testing process improved outcomes for students compared to self-proctoring (likely because it encouraged students to attend to directions and provided more opportunities for feedback). Importantly, we found that even young students are resistant to listening to directions and prefer a trial-and-error process for novel formats.

Conclusions

This paper reviews the literature on practice effects, provides empirical evidence on the impact of test directions and practice, and provides recommendations for best practices in test directions and test practice, particularly for computer-based testing.