Richard Hankins¹
Matt Homer² and Javier Caballero¹
1 General Medical Council, UK
2 University of Leeds

Abstract 

In high stakes clinical assessments, it is common to utilise a requirement to pass a defined proportion of stations as well as to achieve the pass mark. This secondary hurdle exists to limit compensation between stations and ensure that candidates have the required breadth of knowledge. Whereas the primary standard setting methods used in modern clinical assessments such as Borderline Regression are designed to compensate for variance in exam form difficulty, the secondary hurdle is usually fixed across test administrations. This study sets out to assess if it is practical and beneficial to calculate the conjunctive standard for each diet dynamically, taking into account variance in test form difficulty to produce a more consistent standard. Homer (2023) has proposed methods for dynamically calculating the conjunctive standard in a way that takes into account differences in station difficulty and examiner stringency between administrations. Method: Using data from the OSCE component of the PLAB exam (a medical licensing exam for the UK) we have calculated the conjunctive standard dynamically over a period of three months including over 90 exam administrations. We calculated the diet specific conjunctive standard by regression of total number of stations passed on total score, thereby calculating the borderline standard for each administration. We compare the outcomes of utilising a dynamic approach of calculating the conjunctive standard versus using a fixed hurdle and consider the effect this has on fairness, defensibility and validity. Whilst many in assessment consider the use of a conjunctive standard as necessary in a high stakes setting to limit compensation, and ensure candidates have the range of skills to work safely, it is little studied and understood. Most conjunctive standards are arbitrary and fixed in nature and this study for the first time tests the practical implementation of an objective method of calculating this hurdle. 

References (maximum three) 

Homer, M. 2023. Setting defensible minimum-stations-passed standards in OSCE-type assessments. Medical Teacher., pp.1–7.