Imogene Rothnie¹
Kelly Dore², Jill Derby², Gill Starenios², Krysia Kostrz² and Libby Newton¹ 
1 Royal Australasian College of Physicians
2 Acuity Insights, Toronto Canada 

Entry criteria for health professional programs must include the core attributes of professional practice. The Royal Australasian College of Physicians (RACP) identifies communication and teamwork as such attributes, however, assessing these during selection remains challenging. In 2022, the RACP piloted a Situational Judgement Test (SJT), based on the theory of planned behaviour, to measure targeted attributes during selection. Given the high stakes of selection and the value such measures could yield, this study evaluated the evidence for interpretating the SJT scores as accurate and fair measures of important attributes of professional practice. 

Kane’s validity framework was used to develop an ‘interpretation use argument’ (IUA) [1] for the SJT scores as measures of professionalism, including stakeholders’ descriptions of ideal and concerning behavioural representations of the targeted attributes by physicians in practice. Criteria from a measurement framework, the Rasch model (modern test theory) [2] were used to evaluate whether scores on the SJT could be explained by applicant’s proficiency level on a construct that represents a measure of professionalism, and the precision of such measures. 

189 stakeholders participated in a combination of survey and workshops to align behavioural representations of key competencies in professional practice to situational depictions in the SJT. 278 applicants completed the SJT over three test forms. Rasch analysis showed that scores on the assessment showed good fit to a hierarchical unidimensional construct that could be explained as a measure of professional and social reasoning. Applicant measures could be reliably separated (person separation index of 0.83) into three statistically significant bands of performance. 

These results provide important validity evidence for the interpretation and use of SJT scores for assessing targeted constructs of professionalism. The results also highlight the value of engaging stakeholders to identify representative behaviours of key attributes and incorporate these in situation-based assessments. 

References (maximum three) 

1 Kane, M. T. 2013. “Validating the Interpretations and Uses of Test Scores.” Journal of Educational Measurement 50 (1): 1-73. 

2. Rasch G. (1960/980) Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: University of Chicago Press. 

Chantacha Sitticharoon¹
Issarawan Keadkraichaiwat¹, Punyapat Maprapho², Nisa Jangboon³, Nadda Wannarat³ and Varanya Srisomsak¹
1 Department of Physiology, Faculty of Medicine Siriraj Hospital, Mahidol University
2 Siriraj Health science Education Excellence center, Faculty of Medicine Siriraj Hospital, Mahidol University
3 Education department, Faculty of Medicine Siriraj Hospital, Mahidol University

Background:
The admission to the Doctor of Medicine program includes four groups: the Academic-group (academic portfolio), the Quota-group (bachelor graduates/athletics and musical talents), the Test-group (central admission test), and the Rural-group (a mandatory track for students required to serve as rural doctors). This study aimed to determine academic performance (summative scores), study behaviours (performing Course Learning Outcomes (CLOs) assessment with unlimited attempt allowance), and student perspectives to enhance future medical student selection. 

Summary of work:
Data were obtained from 316 students (academic year 2021): 7.2% Academic-group, 1.8% Quota-group, 79.4% Test-group, and 9.17% Rural-group. 

Results:
Summative scores were significantly higher in the Academic-group compared to the Test-group and the Rural-group in all subjects; in the Test-group compared to the Rural-group in all subjects; and in the Quota-group compared to the Test-group or the Rural-group in some subjects (p<0.05 all). The Rural-group had lower CLO scores, fewer additional attempts after passing each CLO, but more attempts for the first passing of each CLO than other groups. They predominantly preferred a two-category pass/fail evaluation, unlike other groups favouring a three-category outstanding/pass/fail system. 

Discussion:
The Academic-group, generally from competitive backgrounds, demonstrated superior academic performance and favoured the "Outstanding" grading. Conversely, the Rural-group, often from less competitive backgrounds, showed lower academic performance, required more attempts for first-passing CLOs, and preferred a simpler pass/fail evaluation. 

Conclusions:
Admission criteria significantly influence academic performance and study behaviours in the Doctor of Medicine program. The Academic-group stands out with higher summative scores, while the Rural-group displays a unique pattern, excelling in initial attempts but facing challenges in CLO scores and additional attempts. 

Take home messages:
Students from different admission systems exhibit varying academic performances and attitudes towards the non-grading system. These insights can aid in refining admissions for better selecting future medical students who align well with the program. 

Ye Ye¹
1 National Board lf Medical Examiners

Abstract 

In recent years, many programs are evaluating how they can better integrate considerations for diversity, equity, and inclusion (DEI) within their program practices. This begins with a commitment to respecting and representing the perspectives of all candidates and those individuals and organizations they serve in professional practice. Candidates are most likely to feel this inclusivity based on whether the content of the assessment reflects their world view from either their personal background or that of those who they serve. A diverse, equitable, and inclusive physician workforce can lead to better health outcomes and patient experience for all patients, with particular benefit for traditionally underserved populations (Marrast, Zallman, Woolhandler, Bor, & McCormick, 2014). 

When we develop assessments in medical education, what can we do to avoid having bias and stereotypes in our test content? In this presentation, we would like to offer some practical strategies on a number of dimensions: subject matter experts (SME) composition and training, item writing guidelines, patient characteristics and its use in test content, statistical techniques to detect bias, and important considerations when using technology for content creation, especially in the generative AI era. 

It is important to note that developing equitable assessments is not limited to a set of checklists. Rather, we propose a holistic approach where DEI practices are embedded throughout the development process, from item development, to test assembly, to scoring and report. The authors would like to offer our practical guidelines in creating assessment through the equity lens. 

References (maximum three) 

Marrast LM, Zallman L, Woolhandler S, Bor DH, McCormick D. Minority physicians’ role in the care of underserved patients: Diversifying the physician workforce may be key in addressing health disparities. JAMA Intern Med. 2014;174(2):289-291 

Hill KA, Samuels EA, Gross CP, Desai MM, Sitkin Zelin N, Latimore D, Huot SJ, Cramer LD, Wong AH, Boatright D. Assessment of the Prevalence of Medical Student Mistreatment by Sex, Race/Ethnicity, and Sexual Orientation. JAMA Intern Med. 2020 May 1;180(5):653-665. doi: 10.1001/jamainternmed.2020.0030. PMID: 32091540; PMCID: PMC7042809. 

Ross DA, Boatright D, Nunez-Smith M, Jordan A, Chekroud A, Moore EZ. Differences in words used to describe racial and gender groups in Medical Student Performance Evaluations. 

PLoS One. 2017 Aug 9;12(8):e0181659. doi: 10.1371/journal.pone.0181659. PMID: 28792940; PMCID: PMC5549898. 

Annette Mercer¹
Edward Li² and Julie Willems³
1 Monash University
2 Pearson VUE
3 Monash University

Background:
The UCAT_ANZ is an aptitude test used in the selection of medical/dental students. It has been used by a consortium of universities in Australia and New Zealand since 2019, following delivery in the UK since 2006. It is a carefully timed computer-based test with four cognitive sections and one Situational Judgment section. The test is developed and delivered by Pearson VUE. 

Summary of work:
An annual technical report compiled by Pearson VUE psychometricians details test and item statistics, as well as performance on the test overall and by subgroups1. This project was a synthesis of the results of five years of testing in Australia and New Zealand, showing trends in the data, and demonstrating patterns in the performance of certain subgroups. In addition, a more detailed analysis was undertaken on the 2022 data and on the top 35% of those candidates. 

Results:
UCAT_ANZ showed high levels of reliability across the years. Outcomes over time for variables such as gender, age group, language spoken at home, parental education, rural status, and school type were consistent. Trends in the top 35% of candidates varied slightly from those for all candidates. 

Discussion:
Five-year trends show consistency, and the test provides a reliable component for use in selection. The UCAT_ANZ provides a level playing field for candidates applying for medicine/dentistry, compared with variations in academic score types and interview styles2. 

Conclusions:
The purpose of this project was for transparency and to provide information to a broader audience than just the consortium universities. Analysis of the top 35% provides a profile of candidates most likely to succeed at the next stage of selection. 

References (maximum three) 

Phattrawan Pisuchpen^1,2
Sira Vachatimanont^1,3, Panot Sainamthip^1,4, Nijasri Charnnarong Suwanwela⁵ and Thanakorn Jirasevijinda⁶
1 Division of Academic Affairs, Faculty of Medicine, Chulalongkorn University
2 Department of Ophthalmology, Faculty of Medicine, Chulalongkorn University
3 Department of Radiology, Faculty of Medicine, Chulalongkorn University
4 Department of Pharmacology, Faculty of Medicine, Chulalongkorn University
5 Department of Medicine, Faculty of Medicine, Chulalongkorn University
6 Department of Pediatrics, Weill Cornell Medical College

Background:
The Chulalongkorn University International Medical Program (CU-MEDi) is a new graduate level program in Thailand. For our inaugural class in 2021, we started with a norm-based approach to the MCAT, a standardized exam widely used by US medical schools, for admissions. The shortcoming of that approach was that we could not predict how the students would perform in the rigorous academic setting. Busche K, et al. reported correlations of MCAT scores with summative performance in the preclinical phase but there was no analysis of subsection scores. 

Summary of Work:
To adopt a more objective, criterion-based approach, we analyze our students' academic performance in the preclinical phase and correlate it with their overall MCAT as well as the 4 subsection scores (Biological and Biochemical Foundations, Chemical and Physical Foundations, Psychological, Social, and Biological Foundations, and Critical Analysis and Reasoning Skills)). CU-MEDi summative knowledge performance is assessed by MCQ in integrated medical sciences (IMS) I, II and III at the end of each 6-month-long semester in the preclinical phase. 

Results:
Our study revealed positive correlations between IMS-I summative performance and both the overall MCAT score and the Biological and Biochemical Foundations section score. However, no significant correlations were observed between summative performance and the remaining three MCAT sections in IMS-II and IMS-III. 

Discussion:
Based on our findings, we have transitioned to a criterion-based approach for MCAT score evaluation, incorporating the Biological and Biochemical Foundations section score as an independent admission criterion. Future investigations will assess the performance of students admitted under this new approach compared to the inaugural class. 

Conclusions:
Overall MCAT and Biological and Biochemical Foundations section scores could be used for medical school entry selection. 

Implications:
Implementing a criterion-based approach to MCAT scores has the potential to mitigate academic risks and enhance the admissions process, ultimately fostering the success of medical students. 

References (maximum three) 

Busche K, Elks ML, Hanson JT, Jackson-Williams L, Manuel RS, Parsons WL, Wofsy D, Yuan K. The Validity of Scores From the New MCAT Exam in Predicting Student Performance: Results From a Multisite Study. Acad Med. 2020 Mar;95(3):387-395. doi: 10.1097/ACM.0000000000002942. PMID: 31425189. 

Julie Willems¹
Annette Mercer¹ and Edward Li²
1 Monash University
2 Pearson VUE

The University Clinical Aptitude Test (UCAT) is a computer-based aptitude test used by some universities in Australia and New Zealand towards student selection into medicine or dentistry (UCAT_ANZ). It is also used in the United Kingdom (UKCAT). The UCAT is comprised of four cognitive skills subtests (Verbal Reasoning, Decision Making, Quantitative Reasoning, and Abstract Reasoning) and a fifth non-cognitive Situational Judgement subtest [1]. 

Traditionally, candidates sit the UCAT in dedicated test centres with configured computers, but the COVID-19 pandemic necessitated a rapid shift to online-proctored testing for many candidates. Online testing requires strict technical requirements and proctoring protocols, demanding a strong and reliable internet connection. The study examines the results of candidates taking exams both in test centres and online. 

A perception lingers that sitting the UCAT online yields higher results than at test centres. The UKCAT compared results in 2020 and found a marked difference of scores between those sitting online versus those sitting at a test centre. For UCAT_ANZ, the results were less clear. 

4. Discussion
The effectiveness of online testing appears to depend on candidate cohorts. It may work well for mature audiences, but for the UCAT_ANZ exam where most candidates are younger school leavers, test centre-based testing seems more suitable. 

To minimise variations in the test experience and avoid in-house disruptions negatively affecting candidate performance, it is recommended for all ANZ candidates, including those with special accommodations, to sit the UCAT at test centres wherever possible. 

References (maximum three) 

Holly Caretta-Weyer¹
1 Stanford University School of Medicine

Background:
CBME is designed to create systems and structures that support a continuum of training and practice rooted in facilitating continuous growth and development and the transparent transmission of assessment data across phases of training. However, residency selection fosters an achievement orientation given the purposes of comparison between applicants, perpetuating discontinuity between phases of training. To address this potential disconnect, we sought to gain a deeper understanding of GME stakeholders’ perspectives on the current UME-GME transition, their desired outcomes for the selection process, and the role of selection within the transition. 

Summary of Work: 
Semi-structured interviews were conducted with eighteen emergency medicine program directors (PDs) within the US. Participants were asked to describe their goals for the transition to GME training and the current impact of selection within the transition. Data were analyzed using template analysis primed by a prior scoping review and further expanded using thematic analysis. 

Results:
Stated priorities in the selection and transition processes centered on alignment with the program, prediction of future performance in residency, ensuring preparedness for residency, and optimizing diversity and equity. However, PDs identified the lack of transparency around assessment data, mistrust between UME and GME, dual-purposing of assessment, lack of competency standards, and the overall lack of control and seeming arbitrariness of the selection process as barriers to achieving their desired outcomes. 

Discussion/Conclusions:
The current postgraduate selection process presents significant challenges to the employment and transmission of meaningful assessment data for both growth and selection, optimal preparedness for postgraduate training, and a seamless continuum of training and practice. 

Next Steps:
Gaining a deep understanding of the current residency selection process, PDs’ aims in selection, and ways in which to align selection with preparedness for postgraduate training are key. This requires a continued exploration of the connections between assessment, transitions, and selection. 

References (maximum three) 

Caretta-Weyer HA, Eva KW, Schumacher DJ, Yarris LM, Teunissen PW. Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values. Acad Med. 2023 Aug 1. doi: 10.1097/ACM.0000000000005365. Online ahead of print.PMID: 37556804