Rebecca Jane Edwards¹
1 Keele University

Examiner variability challenges fairness in OSCEs; and examiner training has been largely ineffective. However, assessors are readily influenced by their recent experiences. Therefore, we theorised that a specific, near in time intervention might increase examiner alignment. 

Video-Based Benchmarking (VBB) is a novel intervention that intends to reduce OSCE examiner variability. VBB involves examiners judging and receiving feedback on a station- specific OSCE performance shortly before scoring the target OSCE station, enabling mental comparison and adjustment to performance expectations. 

If, how, why, for whom, and under what circumstances does VBB work (or not work) to influence examiner scoring in OSCEs? 

Methodology
Realist methodology: realist evaluation and experimental methods.


Findings 

Study 1 (S1) suggests that viewing a station-specific example OSCE performance close in time to assessing the target station may reduce variability by adjusting examiners’ internal marking framework, reducing the need for examiners to ‘get their eye in’, and providing an example for direct comparison with following performances. 

Conversely, study 2’s (S2) primary analysis showed no significant relationship between VBB and examiner variability. However, exploratory analyses indicated that VBB may have worked for a subset of examiners who showed greater initial variation from the group mean. 

Cross-referencing results, participants may require support from further additions to the VBB procedure to facilitate resolution of discrepancies. Alternatively, VBB may have created a sense of increased confidence or perception of adjustment in S1 without actual adjustment to frame of reference or scoring. VBB has the potential to be useful, and further research with procedural developments and methodological refinements would be beneficial to determine efficacy and utility of VBB. 

Questions
What are your thoughts about the different findings between the studies? 

What about the difference between the primary and exploratory findings?


Attending supervisor:
Dr Yeates 

References (maximum three) 

Harasym, P. H., Woloschuk, W., & Cunning, L. (2008). Undesired variance due to examiner stringency/leniency effect in communication skill scores assessed in OSCEs. Advances in Health Sciences Education, 13(5), 617-632. doi:10.1007/s10459-007-9068-0 

Yeates, P., Cardell, J., Byrne, G., & Eva, K. W. (2015). Relatively speaking: contrast effects influence assessors' scores and narrative feedback. Medical Education, 49(9), 909-919. doi:10.1111/medu.12777