Colleen Robb¹
Pierre Banks², Liesel Copeland³, Alex MacIntosh¹ and Kelly Dore^1,4
1 Acuity Insights
2 UTMB John Sealy School of Medicine
3 Robert Wood Johnson Medical School - Rutgers University
4 McMaster University

Holistic admissions, a process where admission committees assess each applicant’s experiences and personal characteristics in tandem with their academic preparedness, is a key strategy to admitting a diverse study body who will succeed in-program and later in the workforce(1). Accordingly, healthcare education programs are increasingly employing holistic admissions. Despite the benefits, holistic admissions are both resource and time intensive and traditional methods for assessing personal characteristics prior to interview (e.g., reference letters) have evidenced poor reliability and predictive validity(2), but other options have the potential to bring the right candidates to the interview. Open-response situational judgement tests (SJT), for example, have shown promise in providing reliable and valid measures of personal characteristics and tend to produce smaller demographic differences than close-ended assessments(3). 

Casper, an open-response SJT used widely in Health Professions programs across North America and Australia, has adopted a new format which includes an audio-video response component. While previous work has shown that this new format further reduces demographic differences, this study aims to examine the predictive validity of the new format with regard to multiple-mini interview (MMI) performance. 

Admissions data collected from 1,011 interviewees to UTMB John Sealy School of Medicine and Rutgers Robert Wood Johnson Medical School (RWJMS) was used to evaluate predictive validity. A single-predictor bivariate logistic regression model for each school, found that for every one-unit increase in raw Casper score (scale from 1-9), odds of receiving a high MMI score (as determined by each program) increased by 159.50% for UTMB (OR: 2.56, 95%CI[2.07, 3.30]) and by 66.78% for Rutgers RWJMS (OR: 1.67, 95%CI[1.08, 2.65]). 

These findings support Casper as a valuable tool within the holistic admissions framework by identifying candidates early in the admissions process with strong personal characteristics who are likely to perform well later in the interview stage. 

References (maximum three) 

(1) Glazer, G., Danek, J., Michaels, J., Bankston, K., Fair, M., Johnson, S., & Nivet, M. (2014). Holistic admissions in the health professions: Findings from a national survey. Urban Universities for HEALTH. https://cdn.ymaws.com/www.aptrweb.org/resource/collection/9BC8166F-3A40-4028-B091- C78275D40573/Holistic_Admissions_in_the_Health_Professions.pdf 

(2) Salvatori, P. (2001). Reliability and validity of admissions tools used to select students for the health professions. Advances in Health Sciences Education, 6, 159-175. https://link.springer.com/article/10.1023/A:1011489618208 

(3) Lievens, F., Sackett, P. R., Dahlke, J. A., Oostrom, J. K., DeSoete, B. (2019). Constructed response formats and their effects on minority-majority differences and validity. Journal of Applied Psychology, 104(5), 715-726. https://psycnet.apa.org/doi/10.1037/apl0000367