Thai Ong¹
Su Somay²
1 National Board of Medical Examiners
2 NBME

Performance assessments and human raters are often employed in medical education to evaluate critical medical competencies (e.g., clinical skills via objective structured clinical examination [OSCE]). Although human raters provide insights not easily captured by automated algorithms, “... raters are human, and they are therefore subject to all the errors to which humankind must plead guilty” (Guilford, 1936, p.272). For this reason, human- generated scores, even with well-developed rater training and scoring guidelines, are prone to various rater effects and may be confounded with construct-irrelevant variance, posing a threat to validity of score interpretations (AERA, APA, & NCME, 2014). Rater effects, or systematic differences in how raters evaluate performance, can stem from different sources, such as implicit biases and/or lack of familiarity with the rubric (Myford & Wolfe, 2003). Given the threat to validity, medical schools relying on human raters should investigate rater effects and their impact on inferences made from the scores. 

Several statistical approaches to identify and evaluate rater effects exist, ranging from simple to complex methods. The statistical approach chosen for an assessment program depends on various factors, such as number of raters, number of cases, and rater effects of interest. Therefore, having knowledge of both the assessment program (from the participants) and framework for selecting from the various statistical approaches (from the workshop) is ideal for determining the most appropriate statistical approach for an assessment program. 

In this workshop, we aim to provide participants with a) a primer on common rater biases and b) a primer on several statistical methods to identify and evaluate rater effects through applied examples from a formative OSCE designed to assess clinical reasoning skills. The workshop will be interactive in nature: 

1. How does your school/program currently evaluate rater bias? 

Participants 

We encourage all medical and assessment professionals with any level of experience with performance assessment using human raters to evaluate students to attend the workshop. 

We intend to create workshop material at the introductory to intermediate level. 

By the end of the workshop, participants will be able to:
 1. Understand sources of common rater biases
 2. Understand several statistical methods to evaluate rater effects Maximum Number of Participants
30 

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association. 

Guilford, J. P. (1936). Psychometric methods. New York, NY: McGraw-Hill 

Myford, C. M. & Wolfe, E. W. (2003). Detecting and measuring rater effects using many-facet rasch measurement: Part I. Journal of Applied Measurement, 4, 386422. 

Wendy Crebbin
Stephen Tobin¹
1 Western Sydney University

The material presented in this workshop draws on the findings of research carried out in one department, at one major University Teaching Hospital in Victoria, Australia. 

The participants were: 26:42 third-year medical students in the surgical unit, and 17:21 unaccredited registrars and accredited trainees in the surgical unit. These were medical students and doctors at early to relatively senior levels of practice. 

Key findings of this research are that for every diagnosis there is an identifiable progression in each individual’s clinical decision making (CDM) from novice to competent/proficient level of expertise, and this can be supported, monitored, mapped, assessed and mentored to ensure more effective development. 

This topic is important because, whilst clinical practice is considered essential for the development of CDM(1), in general it remains under-researched(2), and progressive assessment of CDM needs further refinement(3). The principles, whilst refined in a surgical context, are applicable to all fields of medicine. 

The workshop will occur in three parts: 

Attendees should be involved in assessing clinical decision making or wanting to benefit from better understanding the sequence of development of CDM. How to support and better inform assessment of CDM are key outcomes. 

Participants with any level of experience can attend from novices to competent/proficient doctors. Mastery for most doctors will take many years of experience – the model supports same. 

After participating in this workshop participants will be equipped with tools that they can apply to develop CDM and assess how CDM affects diagnosis and management in their own practices. 

References (maximum three)