Skip to main content
Ottawa 2024
Times are shown in your local time zone GMT

Authentic assessment

Workshop

Workshop

10:30 am

26 February 2024

M209

Session Program

PETER BARTON1
Asela Olupeliyawa2 and Sandra Kemp3
1 Monash University
2 University of Queensland
3 University of Wollongong 



1. Background: 
medical students report anxiety1,2 on their transition from university life to  the clinical workforce. Some report being unprepared for clinical responsibility. This transition is categorised by: increased clinical autonomy (including prescribing, recognition and initial involvement in deteriorating patients); work place interactions with team members, patients and their relatives; enhanced requirements for self-monitoring and self-resilience; the replacement of university personal support structures with workforce ones; irregular shift and working hour patterns; the formation of new peer and social groups and potential translocations within Australasia, and increased societal and personal expectations of the new doctor self. 

Academic work (guided by national agreement) traditionally focusses on the skills, knowledge, competencies and capabilities of graduating doctors. Preparedness for transition into practice is undertaken across Australian and New Zealand medical schools; bespoke solutions are offered at each individual school. Limited consensus exists on how best to assist the transition of students into practice. 


2. Importance of workshop:
What are the learning outcomes and assessment activities that best prepare for the “next step”? The Medical Deans of Australia and New Zealand (MDANZ) Assessment Working Group is leading this project to identify gaps in existing preparation for transition. It will report through the Medical Education Collaboration Committee (MECC) to MDANZ. The aim is to engage with key stakeholders to generate a working consensus of best assessment practice in the field of transition. 


3. Workshop format:
The activities of the workshop are focussed on how medical students can be best prepared to safely take their next step. The workshop will begin by defining a set of common intended learning outcomes; offer guidance on how to constructively align these with their associated assessments; and if possible best practice guidance on assessment methodologies and standard setting. One key overarching intention of the project is to support the quality assurance processes of institutions. The consensus gathering aims to address gaps in curricula learning outcomes and assessment. The activities will comprise: 

1. Pre-conference reading. 
2. Introduction, 15 minutes 
2. Small group work to discuss R. for S.I.P intended learning outcomes, assessment methods, and quality assurance processes. 60 minutes. 
3. Group feedback, 15 minutes 


4. Participants: 

National and International attendees who are: leaders in medical education; directors of medical workforce programs; internship/PGY supervisors (primarily the focus is on Australia and New Zealand workforce); current or recent interns; healthcare workforce employers 


5. Level of workshop:
Intermediate


6. Take-home messages / workshop outcomes / implications for further research or practice.
By the end of the workshop there will be progress towards: 

  1. A consensus on the intended learning outcomes of a Readiness for Safe Intern Practice programme. 
  2. A discussion on the appropriate assessment methods in a Readiness for Safe Intern Practice programme (and their quality assurance) 
  3. A discussion on the appropriate standard setting methods for the assessments within a Readiness for Safe Intern Practice programme (and quality assurance) 
  4. Maximum number of participants: 60 participants 



References (maximum three) 

Hunter K, Willis K, Smallwood N: The workplace and psychosocial experiences of Australian junior doctors during the COVID-19 pandemic. 2022, Internal Medicine Journal (54), 745-754 

Medical Deans medical student survey 2022. https://medicaldeans.org.au/md/2023/05/MSOD- National-Data-Report-2022.pdf 

James Don Murphy1
Brian Kwan1, Maxwell Spadafore2, Holly Caretta-Weyer3 and Michelle Daniel1
1 University of California, San Diego
2 University of Michigan
3 Stanford University School of Medicine




Background 
Patient electronic health records (EHRs) represent a ubiquitous tool within healthcare delivery, yet the EHR is an underutilized tool with respect to learner assessment in both undergraduate and graduate medical education. Trainees commonly interact with EHRs in multiple domains including clinical documentation, order entry, electronic communication between provider teams, and increasingly, direct communication with patients. This creates opportunities for authentic, workplace-based assessments (WBAs) of multiple competencies (e.g., diagnostic and management reasoning, communication, procedural skills, etc.) With the increasing capabilities afforded by data analytics tools (e.g., tableau) and artificial intelligence (e.g., natural language processing), there are clear opportunities to derive novel and potentially transformative assessment strategies across the medical education continuum. 

This workshop will provide participants with an overview of different opportunities to incorporate the EHR into assessment through a presentation of 2 specific use cases, followed by an interactive discussion: 

  1. Assessment of procedural competency through EHR documentation leveraging tableau. 

  2. Assessment of clinical reasoning competency using machine learning on clinical notes. 

Important questions surrounding validity, generalizability, logistics, data security and how to operationalize strategies at the organizational level will be discussed. 


Why is the topic important for research and / or practice? 
  • By leveraging data already in the EHR, extracted through novel tools, personalized assessment of and for learning may be achieved, advancing the agenda of precision education. 

  • The large volume of data available in the EHR, combined with tools that help make sense of this data, can provide what has been described by Cees Van der Vleuten as “1000 points of light” within programs of assessment. 


Workshop format, including participant engagement methods 
The workshop will commence with an interactive presentation on how to incorporate EHR data into different domains of assessment across medical education (20 minutes), highlighting 2 use cases. Participants will then break up into small groups for a case-based exercise to discuss ideas, strengths and limitations of different assessment approaches leveraging data in the EHR for assessment (30 minutes). The small groups will then report back to the larger group (10 minutes). 



Who should participate? 
Faculty or administrators involved in health education assessment and evaluation within undergraduate or graduate medical education. 


Level of workshop (beginner / intermediate / advanced) 
Beginner / Intermediate / Advanced 


Take-home messages / workshop outcomes / implications for further research or practice 
Participants will leave this workshop with ideas for how to incorporate EHR data into medical assessment to advance assessment of and for learning, and precision education. Furthermore, participants will gain a better understanding of the potential advantages and limitations surrounding EHR data use in a program of assessment. 


Maximum number of participants 

100 


References (maximum three) 

NA