Kim Ashwin¹
Maxim Morin²
1 Australian Medical Council
2 Medical Council of Canada

Background:
High-stakes medical examinations are confronting mounting challenges from sharing of unauthorized test materials, organized entities seeking to profit from facilitating exam malpractice, and illicit technologies of increasing sophistication. At the same time, data analytic techniques are available to combat these trends and should be deployed wherever possible to minimise their effects. 

Summary of work: 
The Australian Medical Council has been developing analytics based on Item Response Theory (IRT) that can help to reduce the effects of malpractice on IRT exams. The key innovation is to employ the assumptions underpinning IRT to separate the effect on exam results of malpractice and the effect of the latent trait, theta. This, together with a binomial approach to quantifying probability, allow the identification of both 1) candidates with a high probability of item pre‐knowledge and 2) items with a high probability of having leaked into the public domain. 

Results: 
These techniques have been implemented for several years and have had a demonstrable effect in reducing malpractice, measured in terms of flagged candidates, leaked items and grosser measures such as pass rates. 

Discussion
The implications of these findings extend beyond mere statistical significance. They illustrate the potential of data analytics in restoring the credibility of high-stakes examinations. 

Conclusions
The integration of advanced analytics, grounded in theoretical frameworks like IRT, offers pathways to safeguard the integrity of medical examinations. 

Take-home Messages, Symposium Outcomes, and Implications for Further Research and/or Practice: 

a) An examination of how IRT-based analytical tools can be useful in detecting malpractice. 

b) Strategies to protect exam validity with reasonable trade-offs in the areas of cost of item generation, curation and retirement 

c) Areas for continued inquiry into the effectiveness and equity of emergent anti-malpractice methodologies. 

References (maximum three) 

Belov, D. (2013). Detection of test collusion via Kullback-Leibler divergence. Journal of Educational Measurement, 50, 141–163. 

Eckerly, C. A., Babcock, B., & Wollack, J. A. (2015) Preknowledge detection using a scale- purified deterministic gated IRT model. Paper presented at the annual meeting of the National Conference on Measurement in Education, Chicago, IL.