Introduction

For measurement professionals, drawing valid inferences from scores on assessments is a paramount concern. Substantiating valid inferences can be challenging, especially when assessing higher-order thinking skills that require scenario-based items. Clinical scenarios, for example, nest interrelated items that share a dependency on a common stimulus. For this reason, testlet models are often used to handle item dependency. Unfortunately, they are not immune to differential item functioning (DIF), which occurs when equal-ability examinees from different groups differ in their probability of answering an item correctly.

Objectives

The current work provides a new perspective into investigating DIF for testlet models. Traditional DIF analyses, such as the likelihood ratio test, rely on frequentist frameworks that a priori assume items/testlets are DIF-free (i.e., null hypothesis), and they require evidence in opposition of the null hypothesis.

By contrast, we present a Bayesian analysis that makes no such a priori assumption. Instead, the analysis expresses the degree of confidence that items/testlets are DIF-free. This intuitive approach is not possible under frequentist frameworks— assuming truth in the null prevents stating evidence in its support.

Design/Methodology

We compare the likelihood ratio test to a Bayesian analysis of simulated data generated from a Rasch testlet model. Across two groups, testlets are either DIF-free or contain DIF in either item difficulties or magnitude of the testlet effect. For the Bayesian analysis, we further compare several priors and show their impact on DIF detection.

Results

The results indicate the viability of the Bayesian approach to investigating DIF, making it a reasonable alternative to the commonly used likelihood ratio test.

Conclusions

The Bayesian DIF analysis is intuitively appealing because it quantifies evidence that items/testlets are DIF-free, which is not otherwise possible. This approach affords testing programs to state the degree of confidence that their assessments are DIF-free.