Imogene Rothnie¹
Kelly Dore², Jill Derby², Gill Starenios², Krysia Kostrz² and Libby Newton¹ 
1 Royal Australasian College of Physicians
2 Acuity Insights, Toronto Canada 

Entry criteria for health professional programs must include the core attributes of professional practice. The Royal Australasian College of Physicians (RACP) identifies communication and teamwork as such attributes, however, assessing these during selection remains challenging. In 2022, the RACP piloted a Situational Judgement Test (SJT), based on the theory of planned behaviour, to measure targeted attributes during selection. Given the high stakes of selection and the value such measures could yield, this study evaluated the evidence for interpretating the SJT scores as accurate and fair measures of important attributes of professional practice. 

Kane’s validity framework was used to develop an ‘interpretation use argument’ (IUA) [1] for the SJT scores as measures of professionalism, including stakeholders’ descriptions of ideal and concerning behavioural representations of the targeted attributes by physicians in practice. Criteria from a measurement framework, the Rasch model (modern test theory) [2] were used to evaluate whether scores on the SJT could be explained by applicant’s proficiency level on a construct that represents a measure of professionalism, and the precision of such measures. 

189 stakeholders participated in a combination of survey and workshops to align behavioural representations of key competencies in professional practice to situational depictions in the SJT. 278 applicants completed the SJT over three test forms. Rasch analysis showed that scores on the assessment showed good fit to a hierarchical unidimensional construct that could be explained as a measure of professional and social reasoning. Applicant measures could be reliably separated (person separation index of 0.83) into three statistically significant bands of performance. 

These results provide important validity evidence for the interpretation and use of SJT scores for assessing targeted constructs of professionalism. The results also highlight the value of engaging stakeholders to identify representative behaviours of key attributes and incorporate these in situation-based assessments. 

References (maximum three) 

1 Kane, M. T. 2013. “Validating the Interpretations and Uses of Test Scores.” Journal of Educational Measurement 50 (1): 1-73. 

2. Rasch G. (1960/980) Probabilistic Models for Some Intelligence and Attainment Tests. Chicago: University of Chicago Press.