Peter Yeates¹
Becky Edwards¹ and Richard Hays²
1 Keele university
2 JCU Murtupuni Centre for Rural & Remote Health

Equivalence can be defined as the tendency for a given student to reach the same outcome in an assessment regardless of where or when they are examined. Ensuring equivalence in high stakes performance assessments (for example OSCEs or standardized patient exams) is vital for patient safety and candidate fairness, as well as a growing focus of regulators. Equivalence is important both within institutions who run large or distributed exams, as well as between institutions nationally. 

Several countries or regions internationally have announced or are moving towards national licensing exams, or share OSCE stations to aid alignment. To test clinical skills or performance, this often requires that candidates are examined across several locations, or at multiple times with different groups of examiners. Ensuring alignment of the judgements made by different groups of examiners (at separate times or in different locations) is critical to the fairness of large-scale exams but has traditionally been challenging both to investigate and to support. Recent innovations in this space (Video-based examiner score comparison and adjustment (VESCA)(1), and Video-based benchmarking (VBB)(2)) offer novel potential approaches to calibrate, compare or even equate for examiner differences which can supplement traditional approaches to faculty development. 

Workshop facilitators will invite participants to share their experiences, opinions and recommendations for enhancing OSCE equivalence. Building from current best-practice recommendations for enhancing graduation level performance assessment(3), facilitators will use case studies to illustrate the practical use of two approaches to enhancing equivalence: 

Using data from recently completed and ongoing research, facilitators will illustrate the extent and impact of examiner variability across locations within an OSCE, and its implications for equivalence. They will present data which informs the opportunities and challenges in using these relatively novel approaches to either calibrate examiners or equate for their differences, including data on the accuracy and effectiveness of these approaches, and data which describe how participants will interact, use and trust each approach. 

Working in groups, participants will critically reflect on the relative merits of trying to calibrate examiners versus adjusting for their differences. Participants will work together to produce actionable plans for enhancing their own OSCEs. 

People involved in or interested in OSCEs, particularly when run across multiple parallel tracks or different locations. 

Level of workshop 
All


Workshop outcomes:
By the end of the workshop, participants will be able to: 

References (maximum three)