Richard Arnett¹
John Hines² and Peter Brennan²
1 RCSI University of Medicine & Health Sciences
2 Surgical Royal Colleges of the United Kingdom and in Ireland

1. Background
Membership of the Royal Colleges of Surgeons (MRCS) is an intercollegiate (hosted by the 4 Surgical Royal Colleges in the UK and Ireland) postgraduate surgical qualification required for entry into higher surgical training. Part A consists of 300 MCQ-type questions and Part B is an OSCE assessment. Part A is offered 3 times a year and attracts over 10,000 candidates each year. It has recently migrated from paper-based to an online assessment hosted in 400+ commercial test centres in 150+ cities around the world. 

2. Summary of work
Analysis identified a small number of candidate pairs with unusually similar response patterns. This similarity was crudely characterized by evaluating the option choices of candidate pairs namely the ratio of ‘Exact Errors In Common (EEIC) to ‘Differences’ (D). High EEIC/D values were flagged for further investigation. 

3. Results
A very small number of candidate pairs have been identified in several diets. High EEIC/D values are not detected in historic paper-based results and do not occur in candidate pairs from different test centres. Flagged pairs are followed up with the relevant test centres to ascertain seating plans and the availability of any additional evidence. 

4. Discussion
Due to the circumstantial nature of the evidence, it is important to balance the validity requirements of the awarding body with candidate reputation. Where evidence suggests a particular result might be in question, results are withheld, and candidates are offered an opportunity to resit at the next available opportunity. 

5. Conclusions
It is likely that there are isolated incidences of collusion taking place which need to be dealt with. 

6. Take-home messages/implications for further research or practice 
Even negligible incidences of potential cheating in an assessment programme threaten its validity and require investigation. The underpinning process needs to be robust, transparent & visible. 

References (maximum three) 

Jennings JS, Harpp DN, Hogan JJ. Crime in the Classroom: Part II. An Update. J Chem Educ. 1996;73(4):349. 

Harpp DN. Crime in the Classroom: Conclusions after 27 Years. J Chem Educ. 2018;95(10):1900–1 

Deborah Collier¹
1 University of Liverpool

In Liverpool we have taken a journey with our clinical assessment since 2017 of moving from a traditional sit and resit large number of stations OSCE sat by all students to a sequential model using a smaller first sequence for the full cohort and a small sequence 2 for students not exempt. 

To accommodate a large cohort in the traditional model this was originally over a number of days with a perceived unfairness of sitting the exam on day 1 compared to day 5. 

With the implementation of a sequential model all students sat 12 stations over 2 days exempting those students who perform very well. Followed by a second sequence for a smaller cohort over 2 days with different clinical scenarios. The combination of sequence 1 and 2 providing a pass/fail boundary (1). The resulting outcome was to reduce the number of fails in the cohort but also to see that the students who did fail either just managed a pass at resit or failed again. Those who did pass at resit where seen in the failing group during the next assessment cycle. These students were unable to effect a long term deeper learning following a short term remediation (3). 

Our next step was to have the resit opportunity the next academic year (the next full assessment cycle) during which time the students where enrolled of the “Support in Clinical Learning Programme”. This gives students a shared bespoke learning opportunity with a tutor helping to guide and support their learning during the repeated year. We are now seeing the students who engage with the program are not only passing but passing at sequence 1. Those students who choose not to engage continue to struggle and either choose to leave the course of subsequently fail and will have studies terminated. 

References (maximum three) 

1.Homer, M., Fuller, R., and Pell, G. “The benefits of sequential testing: Improved diagnostic accuracy and better outcomes for failing students.” Medical Teacher,40.3 (2017): 275-284 

2.Cleland, Jennifer, et al. "The remediation challenge: theoretical and methodological insights from a systematic review." Medical education 47.3 (2013): 242-251 

Kim Ashwin¹
Maxim Morin²
1 Australian Medical Council
2 Medical Council of Canada

Background:
High-stakes medical examinations are confronting mounting challenges from sharing of unauthorized test materials, organized entities seeking to profit from facilitating exam malpractice, and illicit technologies of increasing sophistication. At the same time, data analytic techniques are available to combat these trends and should be deployed wherever possible to minimise their effects. 

Summary of work: 
The Australian Medical Council has been developing analytics based on Item Response Theory (IRT) that can help to reduce the effects of malpractice on IRT exams. The key innovation is to employ the assumptions underpinning IRT to separate the effect on exam results of malpractice and the effect of the latent trait, theta. This, together with a binomial approach to quantifying probability, allow the identification of both 1) candidates with a high probability of item pre‐knowledge and 2) items with a high probability of having leaked into the public domain. 

Results: 
These techniques have been implemented for several years and have had a demonstrable effect in reducing malpractice, measured in terms of flagged candidates, leaked items and grosser measures such as pass rates. 

Discussion
The implications of these findings extend beyond mere statistical significance. They illustrate the potential of data analytics in restoring the credibility of high-stakes examinations. 

Conclusions
The integration of advanced analytics, grounded in theoretical frameworks like IRT, offers pathways to safeguard the integrity of medical examinations. 

Take-home Messages, Symposium Outcomes, and Implications for Further Research and/or Practice: 

a) An examination of how IRT-based analytical tools can be useful in detecting malpractice. 

b) Strategies to protect exam validity with reasonable trade-offs in the areas of cost of item generation, curation and retirement 

c) Areas for continued inquiry into the effectiveness and equity of emergent anti-malpractice methodologies. 

References (maximum three) 

Belov, D. (2013). Detection of test collusion via Kullback-Leibler divergence. Journal of Educational Measurement, 50, 141–163. 

Eckerly, C. A., Babcock, B., & Wollack, J. A. (2015) Preknowledge detection using a scale- purified deterministic gated IRT model. Paper presented at the annual meeting of the National Conference on Measurement in Education, Chicago, IL. 

Alison Sturrock¹
Charlotte Hammerton¹, Carys Phillips¹, Taylor Bennie¹ and Gil Myers¹
1 University College London

Background 
In 2021 UCL medical school launched a new e-portfolio, designed in partnership with students. In-line with student feedback and published literature, the new portfolio aimed to facilitate in- person real-time feedback and encouraged student empowerment by removing reported barriers of engagement and using a simplified and clear format. 

Summary of Work
The new portfolio allowed for real-time feedback. The option to send a ‘ticket’ for delayed feedback remained. All completed forms were stored within the Academic Student Record (ASR) which displayed a summary of their progress. 

In academic years 21/22 and 22/23, we evaluated the changes using a mixed methods approach incorporating focus groups (of students and educators) and descriptive statistical analysis of questionnaire data. 

Summary of results 
In 2021/22, 54% of 43,896 assessment forms were completed in-person, in 22/23 this increased to 63% of 45,114 submissions. 

Themes across both years included: ability to review progress was valued, real-time feedback was educationally better, quality of feedback varied with assessor availability and timing of feedback having most influence. Student preference for real-time feedback continued; however the removal of a “ticket” option was unacceptable to students. 

In the anonymous feedback, students described widespread cheating. This was also seen in supervisor feedback– some had no recollection of assessing a particular student. 

Discussion
We will discuss our approach to balancing promoting engagement with deterring cheating through confirmatory emails, spot checks and communications about our future plans. We will also present future improvements including introducing a time limit on submitting forms, and formalising processes around student concerns. 

Conclusion & Take Home Messages

Students value real-time in-person feedback however further work is required to improve the quality of this feedback. The value of student empowerment and engagement needs to be balanced against discouraging student cheating. 

References (maximum three) 

[1] Subha Ramani & Sharon K. Krackov(2012)Twelve tips for giving feedback effectively in the clinical environment,Medical Teacher,34:10,787- 791,DOI: 10.3109/0142159X.2012.684916 

[2] Burgess, A., van Diggele, C., Roberts, C. et al. Feedback in the clinical setting. BMC Med Educ20 (Suppl 2), 460 (2020). https://doi.org/10.1186/s12909-020-02280-5