Bryan Ashman¹
Adrian Cosenza¹, Jason Frank², Ian Incoll³, Chris Kondogiannis¹, Markku Nousiainen⁴, Linda Snell⁵, James Tomlinson⁶ and Sindy Vrancic¹
1 Australian Orthopaedic Association
2 University of Ottawa
3 University of Newcastle
4 University of Toronto
5 McGill University
6 Sheffield Teaching Hospitals

There are few published studies describing validity evidence for programmatic assessment on a national-level surgical training program. We describe an evaluation of assessment of competence in the Australian Orthopaedic Association national training scheme, which transitioned from time-in-training to a competency-based model in 2017 (1). 

The evaluation team used a mixed-methods research approach. The workplace-based assessment tools (WBAs) designed to give a longitudinal view of the acquisition of competence in medical, surgical, and professional skills were analysed for validity and reliability. 

The review found that the use of a mobile device-based application for WBAs and capture of the outcomes in a web-based learning management system provided the necessary data for assessment of competence and feedback on performance. Using WBA results and logbook entries, trainee competence was evaluated by quarterly progress meetings between trainees and their supervisors and end-of-term reviews with Directors of Training at each hospital. 

Cumulative data demonstrated gradual progression of autonomy of trainees during the training program and facilitated early intervention for under-performing trainees, resulting in fewer formal remediation events than in the previous time-based program. Challenges identified included resistance to regular documentation of WBAs and variability in the application of the assessment rubrics between individual trainers. 

A core principle of assessment of competence is that it should involve multiple methods and multiple assessors to build a picture of the progression of ability and independence (2). This review confirmed that the assessment system in the new program that documents progression of competence, provides validity evidence for programmatic assessment. 

Embracing the use of WBAs to document competence requires buy-in from trainees and trainers. Commitment to engage with the process of collecting and collating outcomes of assessment of performance determines the usefulness of this method of competence assessment in surgical training. 

References (maximum three) 

1. IW Incoll , J Atkin, J Owen, O Khorshid, A Cosenza, JR Frank. Australian orthopaedic surgery training: Australian Orthopaedic Association’s strategic education review. ANZ J. Surg. 2020; 90: 997-1003 

2. J Lockyer, C Carraccio, M-K Chan, D Hart, S Smee, C Touchie, ES Holmboe, JR Frank. Core principles of assessment in competency-based medical education. Medical Teacher 2017, 39(6), 609-616