In principled assessment design, test items are generated according to a model of students’ cognitive processing. Cognitive models allow researchers to identify the features of an item that most strongly predict the item’s difficulty, thus providing a means to estimate item difficulty without pretesting items. Unfortunately, most research on cognitive models fails to consider how individual student differences impact model fit. In this study, we sought to determine how multi-digit addition and subtraction cognitive model fit varies when applying the model to high- and low-ability students. The cognitive model included variables such as the type of operation (i.e., addition or subtraction), the number of digits in operands, and several other item features obtained from prior research and theory.

Participants included 299 third-grade students completing a computer-based test with 32 automatically generated addition and subtraction items. Data were analyzed with the linear logistic test model, a technique that extends the item response theory Rasch model in order to estimate the contribution of each pre-specified item feature to item difficulty.

Although the cognitive model fit both ability groups well (R² = .88 and .94 for high- and low-ability groups, respectively), results showed that cognitive model variables were stronger predictors of item difficulty in the low-ability group. For students in the low-ability group, the weight of item features such as operation, regrouping of digits, and the interaction between operation and regrouping was roughly double the weight of those same features in a model with high-ability students. Results imply that some key features of items are less predictive of item difficulty as students become more mathematically proficient. Additionally, results suggest that test developers should use caution when applying cognitive models to precalibrate items because difficulty estimates from a cognitive model may not generalize to students of different proficiency levels.