Item Response Theory (IRT) and related models commonly utilise the information in tests to estimate person and item parameters for improving test and item quality, yet, with limited application to standard setting. The majority of the currently used standard setting methods rely upon judges to make decisions on the expected performance of a minimally competent examinee in a given test. Some methods (e.g. Bookmarks) use information obtained from IRT analyses to support judges’ decisions.

This workshop introduces techniques utilising the information obtained from IRT/Rasch analyses to establish defensible pass/fail decisions without additional post examination judgement. These techniques are also applicable when tests are multidimensional and/or apply non-compensatory rules.

In the workshop, the participants will learn how to apply Rasch models to standard setting when tests meet these criteria. The workshop will also introduce alternative defensible solutions when IRT/Rasch models are not applicable.

The facilitator Boaz Shulruf is the head of Medical Education Research at UNSW, Sydney, and is an expert in the topic. Shulruf has developed new methods for standards setting and has published internationally in this topic. His standard setting workshops delivered in lead international Medical Education conferences received excellent feedback and initiated a number of international research collaborations.

The workshop agenda:

1. Introduction to standard setting and a brief review of current methods and knowledge.
2. Introduction to the Rasch Model and its applicability to standard setting
3. Applying the Rasch Borderline Method  (RBM) to different tests
4. Introduction to the Objective Borderline Method (OBM) as an alternative model when RMB cannot apply.
5. Critical discussion: strengths  and limitations and summary

Logistics: Participants will need to have access to jMetrik4 and Excel.

References

Shulruf, B. (2015a, 25-29 August). Using IRT for standard setting in performance based assessments (conference workshop). Paper presented at the EARLI 16 Biennial Conference, Limassol.

Shulruf, B. (2015b, 4-8 February). Using modern test theory for standard setting in medical education (pre-conference workshop). Paper presented at the The 12th APMEC, Singapore.

Shulruf, B., Booth, R., Baker, H., Bagg, R., & Barrow, M. (in press). Using the objective borderline method (OBM) to support board of examiners’ decisions in a medical programme. Journal of Further and Higher Education

Shulruf, B., & Jones, P. (2014). Using modern test theory for standard setting in medical education (pre-conference workshop). Paper presented at the Ottawa Conference: Transforming healthcare through excellence in assessment and evaluation, Ottawa.

Shulruf, B., Jones, P., & Turner, R. (2015). Using student ability and item difficulty for standard setting. Higher Education Studies, 5(4), 106-118. doi: 10.5539/hes.v5n4p106

Shulruf, B., Poole, P., Jones, P., & Wilkinson, T. (2014a). The objective borderline method (OBM): A probabilistic method for standard setting  Assessment and Evaluation in Higher Education doi: 10.1080/02602938.2014.918088

Shulruf, B., Poole, P., Jones, P., & Wilkinson, T. (2014b, 2-5 July). Validating the objective borderline method (OBM) for standard setting using a simulation study. Paper presented at the 9th Conference of the International Test Commission (ITC), San Sebastian, Spain.

Shulruf, B., Turner, R., Poole, P., & Wilkinson, T. (2013). The objective borderline method (OBM): A probability-based model for setting up an objective pass/fail cut-off score for borderline grades in medical education programmes. Advances in Health Sciences Education, 18(2), 231-244. doi: 10.1007/s10459-012-9367-y