Deborah O'Mara¹
1 University of Sydney Medical School/ AMEE/ ANZAHPE 

Background:
The medical education literature includes many predictive studies using linear or logistic regression, particularly for selection research. The emphasis on “predictive validity” remains, though it is contrary to modern views of assessing evidence for validity, as outlined in studies such as (Kane, 1992) and (Cook et al., 2015). 

An exploratory study of the literature suggests that many studies of predictive validity evaluate selection instruments as the predictor and performance in medical school as the outcome variable. Too often studies are limited to easily measured outcomes, such as academic scores or compromised measures of a “good doctor” or explain a small proportion of variance (eg <20%). 

Summary of work
This presentation will summarise predictive studies in medical education selection and assessment in the past 20 years, identifying the amount of variance explained, types of dependent and independent variables, design and usefulness. Alternative research designs will be offered. 

Results:
It is expected that the fallacy of the predictive research design will be supported by several factors; being more suitable for heterogenous populations and data where parametric statistical principles apply; not having a sufficient sample size; low proportion for predictor variable in logistic regression such as student professionalism issues; outcome measures with limited generalisability; and most of the variation being unexplained. 

Discussion:
Alternative research designs more suitable to medication education selection, such as segmentation and longitudinal studies focusing on the differences between target groups, will be discussed. 

Conclusions:
The variables chosen as output variables frequently disappoint as do the proportion of variance explained in predictive studies of medical education. They are not suitable for new assessment systems, such as programmatic assessment or widening access selection policies. 

Take home messages:
Think carefully before assuming you need to do a predictive study for your medical education or selection research. Consider alternative research designs. 

References (maximum three) 

Cook, D. A., Brydges, R., Ginsburg, S., & Hatala, R. (2015). A contemporary approach to validity arguments: a practical guide to Kane's framework. Medical Education, 49(6), 560-575. https://doi.org/10.1111/medu.12678 

Kane, M. T. (1992). An argument-based approach to validity. Psychological bulletin, 112(3), 527.