Introduction

Testing professionals encounter numerous challenges in developing reliable and valid measurements to capture higher-order cognitive constructs, such as information integration, critical thinking and clinical judgment.

Objectives

A framework consisting of four models provides a modularized yet integrated way for measuring high-order cognitive constructs. To provide a concrete context for this paper session, nursing clinical judgment (NCJ) is chosen to be the construct of interest.

Design/Methodology/Results

The paper explores three competing theories for NCJ and describes the process of building a conceptual model as the initial step of the framework. The conceptual model provides theoretical foundations for assessing the construct. The conceptual model specifies what should be measured but not how to measure. This is where a multilayer assessment model comes into play: the assessment model renders the conceptual model into a psychometric model that becomes amenable to measurement. The assessment model plays a major role in this framework in that it provides a psychometric foundation upon which the remaining models in the paper are developed. After that, the paper describes building a task model. The task model fleshes out lower-layer components in the assessment model (e.g., connecting the lower-layer components to knowledge, skills, and abilities identified from a competency study). A well-developed task model functions as a blueprint for developing structured items that elicit responses from test takers consistently. Finally, the paper presents a scoring model and discusses validation processes of the framework. The scoring model assigns numeric values to test takers’ responses to items developed from the task model. Both qualitative and quantitative validation processes are introduced that can be employed to evaluate the assessment, task, and scoring models.

Conclusions

The framework is flexible enough to be adapted by testing professionals who are considering developing measurements for high-order cognitive constructs.