Introduction

Many institutions and organizations around the world use some form of performance-based assessment to test medical candidates. These tests may include the objective structured clinical examination (OSCE) or workplace based assessments (WBA). Some challenges associated with performance testing in medicine consist of the nested structure of these assessments as well as the number of candidates eligible to participate, as candidates can range from tens to tens of thousands depending on the institution.

Objectives

One of the areas that has been underexplored within medicine is the comparability of different measurement models for evaluating performance testing results. Additionally, the nested structure that exists within many performance tests limits the use of certain models. Therefore, this study specifically investigates comparability aspects from HLM and the MFRM in order to describe what information can be gleaned and under what testing conditions each model would be suitable for use.

Design/Methodology

To better understand and compare results from the two models, data from a high-stakes OSCE examination was analyzed using both HLM and the MFRM. These data consisted of 204 medical candidates over a four-year period (i.e., 2010 to 2014). Candidates were assessed on various competencies (e.g., communication and professionalism) using a six-point scale. Examiners also rated candidates holistically. Using the same dataset, HLM and the MFRM was used to assess candidates, station competencies, and examiners.

Results

Previous research has noted similarities between HLM and the MFRM. Our findings suggest that although HLM can be viewed as equivalent to the MFRM, they do produce different parameter estimates. The extent to which these differences are significant is being further explored in ongoing analyses.

Conclusions

This study has important implications for performance testing as it highlights aspects that should be considered when selecting a measurement model. Further, this study offers alternatives for institutions with measurement constraints.