Eugene Wong
James Dawber¹ and James Fraser¹
1 Australian College of Rural and Remote Medicine 

The Australian College of Rural and Remote Medicine (ACRRM) is one of the two GP Colleges in Australia accredited by the Australian Medical Council. Our Fellowship program (FACRRM) has been developed by rural doctors to equip Rural Generalists and specialist General Practitioners with knowledge and skills to work in rural or remote contexts. The fellowship assessment program includes multiple modalities including StAMPS - Structured Assessment using Multiple Patient Scenarios. StAMPS is an online assessment combining attributes of an OSCE and Viva Voca across eight scenarios. StAMPS aims to assess higher- order functions in a highly contextualised framework, where candidates have the opportunity to explain what they do and demonstrate their clinical reasoning. 

Recently the standard setting method for StAMPS has been adapted, utilising numeric behaviourally anchored rating scales (BARS) across six items per scenario, a global grade per scenario, a comprehensive borderline review process utilising the recorded videos of the scenarios, and live dashboarding. We show that through the combination of these features, we can collectively provide reliable and defensible pass marks. Comparisons to the borderline regression method are made, as well as experimentation with unsupervised clustering algorithms for verification. 

The successful implementation of this StAMPS standard-setting method across five previous exam occasions indicates a viable alternative to borderline regression for OSCE-style assessments. We believe that this approach to standard setting provides added rigour and validity that borderline regression and other methods are lacking. 

References (maximum three) 

McGown, P.J., Brown, C.A., Sebastian, A. et al. Is the assumption of equal distances between global assessment categories used in borderline regression valid?. BMC Med Educ 22, 708 (2022). https://doi.org/10.1186/s12909-022-03753-5 

McKinley, D. W., & Norcini, J. J. (2014). How to set standards on performance-based examinations: AMEE Guide No. 85. Medical teacher, 36(2), 97-110. 

Pell, G., Fuller, R., Homer, M., & Roberts, T. (2010). How to measure the quality of the OSCE: a review of metrics–AMEE guide no. 49. Medical teacher, 32(10), 802-811.