Presentation Description
Kylie Fitzgerald1
Brett Vaughan1 and Jane Fitzpatrick1
1 University of Melbourne
Brett Vaughan1 and Jane Fitzpatrick1
1 University of Melbourne
Background
Evaluating selection methods informs best practice in specialist medical selection. Applicants undertake a Multiple-Mini-Interview (MMI) for trainee selection at the Australasian College of Sport and Exercise Physicians (ACSEP). The ACSEP MMI ran face-to-face in 2019, then online from 2020 due to COVID restrictions and retained to increase equity of access and mitigate costs for Australasian candidates. We report the MMI reliability for 2019-2021.
Evaluating selection methods informs best practice in specialist medical selection. Applicants undertake a Multiple-Mini-Interview (MMI) for trainee selection at the Australasian College of Sport and Exercise Physicians (ACSEP). The ACSEP MMI ran face-to-face in 2019, then online from 2020 due to COVID restrictions and retained to increase equity of access and mitigate costs for Australasian candidates. We report the MMI reliability for 2019-2021.
Summary
A prospective observational design was used across three candidate cohorts. Station themes were aligned with the ACSEP curriculum domains. All stations were developed by education and content experts together and reviewed annually, based on evaluation data of the previous year. Interviewers participated in general MMI training in 2019, then online training for their specific MMI station in 2020-21. Generalisability analysis was used to evaluate the reliability of the MMI from 2019-2021.
Results
The seven-station 2019 MMI overall reliability was 0.43 resulting in a major review. Changes for 2020 included adding an extra station for the domain of cultural safety, a shift from two to one interviewer per station, and station specific training for interviewers. The 2020 overall reliability was 0.8, however several stations were reviewed to increase their internal consistency. The 2021 overall reliability was 0.84, with 7 of 8 stations being greater than 0.7.
Discussion
Cyclical review and evaluation over three years resulted in substantial improvement in the reliability of the ACSEP MMI. The “marks” achieved by candidates likely reflect abilities across the curriculum domains, and may be utilised for high-stakes selection decisions.
Cyclical review and evaluation over three years resulted in substantial improvement in the reliability of the ACSEP MMI. The “marks” achieved by candidates likely reflect abilities across the curriculum domains, and may be utilised for high-stakes selection decisions.
Conclusions
MMI selection processes can be reliable at small-scale. This research may inform the selection processes for programs where there are small applicant numbers.
Implications for further research
Additional research is required to develop evidence to support other elements of the validity argument for the ACSEP MMI.
References (maximum three)
NA