Introduction

Cognitive Labs are often used to gather evidence that students use the cognitive processes the items purport to measure (AERA, APA, NCME, 2014). This study is intended to assess whether Cognitive Labs can also be used to gather validity evidence based on students’ incorrect answers. In 2015, Curriculum Associates (CA) developed a set of assessments with the express purposes of measuring mastery of each Common Core State Standard (CCSS) and providing feedback regarding how students likely arrived at incorrect answers in response analyses that can be accessed through teacher reports.

Objectives

The Cognitive Labs were limited in scope, and the research questions posed for this activity were general, but targeted towards specific needs. The questions focused on three aspects: alignment of student cognition to CCSS; alignment of student cognition to response analyses for incorrect answers; and usability.

Design/Methodology

Cognitive labs were designed as a combination of the “Think Aloud†method (Someren, Barnard, Sandberg, 1994) and cognitive interviews, which helped ascertain information on usability and cognitive processes not observed during the “Think Aloud†portion (Dolan, Goodman, Strain-Seymour, Adams, Sethuraman, 2011). These questions were critical to ensure the validity of the response analyses.

Results

The results demonstrate strong evidence that students answered items incorrectly likely due to misconceptions that either matched or were related to those noted in the response analyses.

For incorrect answers, did the student demonstrate evidence of a misconception that matches the response analysis?

1. Yes 2. Related, but not exactly 3. No

Conclusion

Cognitive Labs are often not completed as part of the development process because of the logistical complexity of the tasks. The methodology that CA used for these cognitive labs demonstrates that a limited set of targeted research questions can provide compelling validity evidence for response analyses.

Bibliography

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014).Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

Dolan, R. P., Goodman, J., Strain-Seymour, E., Adams, J., and Sethuraman, S. (2011). Cognitive lab evaluation of innovative items in mathematics and English/language arts assessment of elementary, middle, and high school students. Research Report. Iowa City: Pearson. Retrieved fromhttp://www.pearsonassessments.com/hai/images/tmrs/Cognitive_Lab_Evaluation_of_Innovative_Items.pdf

Van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think-aloud method: A practical guide to modeling cognitive processes. San Diego, CA: Academic Press Ltd.