Sandra Kemp¹
John Norcini² and Katharine Boursicot³
1 University of Wollongong
2 SUNY Upstate Medical University
3 Health Professional Assessment Consultancy 

For leaders of assessment in medical and health professions education programmes, it is important to understand how the system of assessment, as a whole, meets criteria for good systems of assessment. The Ottawa Consensus paper published in Medical Teacher – 2018 Consensus Framework for Good Assessment – outlines a useful framework and provides guidance to educators. 

Why is the topic important for research and / or practice? 

Ensuring a system of assessment is functioning well according to recognised criteria is important for defensibility of assessment in medical and health professional programmes. Although there is guidance, how to apply this in-practice is complex and challenging. 

Workshop format, including participant engagement methods The format will be: 

Who should participate?

Assessment leads of medical and health professional education programs, Program Directors, academic staff working with assessment

Level of workshop: intermediate
 Take-home messages / workshop outcomes / implications for further research or practice

- Designing a system of assessment that is more than a collection of single assessments is important

- A range of criteria are used for evaluating a system of assessment

- Different forms of evidence should be generated, to account for different stakeholder perspectives

Maximum number of participants: 50 participants



References (maximum three)

Norcini J, Anderson MB, Bollela V, Burch V, Costa MJ, Duvivier R, Hays R, Palacios Mackay MF, Roberts T, Swanson D. 2018 Consensus framework for good assessment. Med Teach. 2018 Nov;40(11):1102-1109. doi: 10.1080/0142159X.2018.1500016. 

Su Somay¹
Christopher Feddock¹ and Thai Ong²
1 NBME
2 National Board of Medical Examiners 

1. Background
Evidence-Centered Design (ECD; Mislevy, Steinberg, & Almond, 1999) is a structured approach to developing assessment content, which includes four developmental steps: specifying the purpose of assessment and the intended inferences about the learners; specifying the evidence that is necessary for evaluating the learner in the identified areas; designing the most appropriate tasks to yield that evidence; and specifying the data that should be collected for each of those tasks. 

ECD is an important framework for assessment development because it directly aligns the evidence collected in an assessment and the intended inferences about the learners. Assessments developed through ECD are conceptualized as evidentiary arguments, building the key components of test score validity. Further, using the structured steps listed above provides a unified approach to assessment development and implementation. Thus, ECD provides a comprehensive approach for even complex domains that lack clear assessment criteria and objective assessment measures. 

In this workshop, participants will learn about the ECD framework through an application to assessment of clinical reasoning, a complex, multidimensional domain. The workshop will involve hands-on activities and group discussions to provide participants with the opportunity to apply the concepts learned to their own educational contexts. 

2. Why is the topic important for research and / or practice?
Using ECD to develop assessments for clinical reasoning ensures that the assessment scores are valid, reliable, and aligned with the desired learning outcomes. ECD starts by clearly defining the learning objectives and competencies that students are expected to achieve, resulting in assessments that are directly related to the key skills and knowledge. ECD also involves a detailed task analysis, which helps identify the cognitive processes and steps involved in clinical reasoning. This analysis allows educators to break down the complex process of clinical reasoning into manageable components and design assessment approaches that measure these components effectively. 

3. Workshop format
Part I: Presentation
-Overview of ECD and its use in designing and developing assessments -Overview of a framework for Clinical Reasoning
 
Part II: Breakout Groups 
-Participants will choose a subcompetency for Clinical Reasoning and structure an assessment through the first two steps in ECD. Presenters will rotate between groups. Presenters will rotate between groups. 

Part III: Report Out and Q&A 

4. Who should participate?
Participants should include medical educators and educational researchers interested in practicing a structured process to develop assessment content. This workshop uses the domain of clinical reasoning for illustration purposes, but the process can be adapted to any content domain. 

5. Level of workshop (beginner / intermediate / advanced)
Introductory to intermediate level 

6. Take-home messages / workshop outcomes / implications for further research or practice Participants will:
 i. Understand fundamentals of ECD and its role in developing assessments.
 ii. Practice ECD through a clinical reasoning example. 

iii. Exchange ideas on how to incorporate ECD into their own medical education programs. 

7. Maximum number of participants 30 

References (maximum three) 

Daniel, M., Rencic, J., Durning, S. J., Holmboe, E., Santen, S. A., Lang, V., Ratcliffe, T., Gordon, D., Heist, B., Lubarsky, S., Estrada, C. A., Ballard, T., Artino, A. R., Jr, Sergio Da Silva, A., Cleary, T., Stojan, J., & Gruppen, L. D. (2019). Clinical Reasoning Assessment Methods: A Scoping Review and Practical Guidance. Academic Medicine: Journal of the Association of American Medical Colleges, 94(6), 902–912. https://doi.org/10.1097/ACM.0000000000002618 

Mislevy, R.J. and Riconscente, M.M. (2006). Evidence-Centered Assessment Design. In T.M. Haladyna & S. M. Downing (Eds.). Handbook of Test Development (1st ed.). Routledge. https://doi.org/10.4324/9780203874776 

Mislevy, R.J., Steinberg, L.S., Breyer, F.J., Almond, R.G., & Johnson, L. (1999). A cognitive task analysis with implications for designing simulation-based performance assessment☆. Computers in Human Behavior, 15, 335-374. https://doi.org/10.1016/S0747-5632(99)00027-8