Chris McManus¹
Gerrard Phillips², Kenneth Dagg³, Stuart Hood⁴, Liliana Chis⁵ and Ben Gillon⁶
1 University College London
2 Executive Medical Director, The Federation of Royal Colleges of Physicians of the United Kingdom
3 Medical Director for Assessment, MRCP(UK), The Federation of Royal Colleges of Physicians
4 Associate Medical Director, The Federation of the Royal Colleges of Physicians of the United Kingdom
5 MRCP(UK)
6 Head of Assessment Quality and Policy, The Federation of Royal Colleges of Physicians

1. Background. Clinical exams are complex and sometimes go wrong, for as Robert Burns said in 1785, “The best-laid schemes o' mice an' men / Gang aft agley”. Despite best efforts, procedural errors occur, resulting in missing, untrustworthy or invalid marks. Marking schemes then cope badly, while regulations often offer provide only an apology, expunging the attempt, fees refund, and early resits [1], though that can feel unfair to candidates with high valid marks if career progression is affected. This workshop addresses such problems using multiple imputation [2]. 

2. Why is the topic important for research and / or practice?
Procedural issues require urgent handling in exams, with a principled approach acceptable to stakeholders, including candidates, examiners and regulators. 

3. Workshop format, including participant engagement methods.
The 60 minute workshop will be four brief (5 -10 mins) presentations on various aspects of the problem, with participant discussion and sharing of experience at each stage. 

a. Examples of Procedural Problems.
MRCP(UK) examples include: 1: During Covid a local organiser unilaterally decided patients at some encounters need not be examined; 2: An examiner’s illness becoming apparent only after a clinical exam had finished; 3: A cardiac encounter with a wrongly described heart murmur; 4: Communication stations inadvertently providing candidates with examiner briefings. 

b. The nature of missing data problems.
Marking schemes usually require all candidates to have marks on all parts of an assessment, but that often is not the case with procedural problems. This section will consider general statistical approaches to handling missing data, and their strengths and weaknesses. 

c. A specific worked example.
Participants will be provided with detailed information provided to the Board of Examiners for an actual procedural problem, including plausible estimates of passing probabilities. On a technical note, psychometric issues were addressed using the R package mice() in specially written software[3], but no software will be used during this workshop. 

d. Regulatory issues.
Regulators are becoming aware about how exams handle procedural errors and the consequences. This section will be a general sharing of experience, and possible issues. 

4. Who should participate?
The concepts involved in imputation are appropriate for all levels of those organising examinations, including senior examiners, exam organisers, psychometricians and regulators. 

5. Level of workshop (beginner / intermediate / advanced).
Beginner 

6. Take-home messages / workshop outcomes / implications for further research or practice.
All participants should understand how procedural problems can be approached in a principled way when examination schemes do go awry, and the methods involved. Psychometricians in particular will become aware of computational approaches and using multiple imputation for such cases. 

7. Maximum number of participants.
Dependent on room size, perhaps 30 or so to encourage discussion. 

References (maximum three) 

1. The Federation of the Royal Colleges of Physicians of the United Kingdom: Examination Appeals Regulations. London: https://www.mrcpuk.org/sites/default/files/documents/Appeals-regulations-2018.pdf; 2018. 

2. van Buuren S: Flexible imputation of missing data (Second edition). New York: CRC Press; 2018. 

3. van Buuren S, Groothuis-Oudshoorn K: mice: Multivariate Imputation by Chained Equations in R. Journal of Statistical Software 2011, 45(3).