Standard setting is a critical step in the test development cycle, but also a challenging process that relies heavily on expert opinion. Due to this reliance on subjective data, which is often collected as item-level difficulty ratings, the standard setting process can lose context for participants and result in an unrealistic or unsatisfactory cut score. Because subject matter experts (SMEs) are accountable for their cut score recommendations, it is crucial that the standard setting process not only be systematic and valid, but also understandable and intuitive for participants.

This study focuses on whether collecting auxiliary information about items, besides only difficulty judgments, is useful for establishing the passing standard for an examination. Several different certifying examinations were used covering a wide range of subject matter. A methodology suggested by Wainer (Sireci and Clauser, 2001) was the basis for collecting the item difficulty judgments from the SMEs.  Using the Wainer method, the SMEs answered each item on the examination as if they were a minimally-competent candidate who is just over the hypothetical borderline between acceptable and unacceptable performance (Cizek and Bunch, 2007). While taking the examination, the SMEs also answered several questions about each item including whether the item tests knowledge that is general/core or specific/specialized, whether answering the item requires simple recall or more complex thought processes, and whether or not answering the item correctly is critical to demonstrating minimal competency. The data were analyzed to assess if there is a relationship between the item difficulty judgments and the auxiliary information, and if the auxiliary information alone can be used to help set a reasonable passing standard for an examination. After the standard setting exercises, SME surveys were conducted to determine comfort with and understanding of the process. Results indicate that the proposed methodology provides valuable information for standard setting.