Font Size: 

Obtaining valid achievement test scores requires test takers who are motivated to give good effort throughout their test events.  Without sufficient effort, validity is diminished, as the resulting scores will underestimate test taker achievement levels.  As computer-based tests (CBTs) have become more popular, objective approaches for measuring effort have emerged that are based on test taker behavior, such as item response time.  These approaches, however, are designed for multiple-choice items in which the test taker must select and answer an item (without review) in order to proceed through the test.

This paper focuses on the measurement of test-taker effort on a test that provides much more test-taker control, such as skipping, omitting, and reviewing (and possibly changing) answers.  The recently-developed OECD Test for Schools (OTS) is a student assessment that provides descriptive information and analyses on the skills and creative application of knowledge of a school’s 15-year-old students (in reading, mathematics, and science) comparable to international PISA scales. The OTS is currently transitioning from a paper-and-pencil to a CBT format that captures various information about test taker behavior, such as item response time, omitted answers, item review, and whether (and how often) answers were changed. In addition, the OTS contains both selected- and brief constructed-response items.

The diverse item types and higher test-taker control in the OTS present challenges to the measurement of effort based on test-taker behavior.  This study will describe the development of several new effort indices that can provide information for the OTS regarding both the amount of effort exhibited by test takers and how much effort individual items received.  These indices will be validated through comparisons with student self-reported effort and through correlations with external measures.  The indices will be particularly useful in mitigating the impact of low student effort on test score validity.