Prashant Jhala¹
Arvin Damodaran^2,3, Adrienne Torda², Ben Taylor², Annette Katelaris², Toby Wilcox⁴, Tayla Douglas⁴ and Boaz Shulruf⁴
1 School of Health Sciences, UNSW Sydney
2 School of Clinical Medicine, UNSW Sydney
3 Prince of Wales Hospital, Randwick Campus
4 Office of Medical Education, UNSW Sydney

Background
Recognised challenges to assessing student’s competency across health programs include assessing clinical competency from preclinical to clinical phases, utilising programmatic assessment and creating shared mental models between assessors and students. EPAs have attracted global interest for their practicality, utility, and potential as ‘assessment for learning’. However, an ongoing question is how to best utilise EPAs as part of a programmatic assessment suite. During the process of (re)developing several health professional programs at UNSW Sydney, we aimed to develop a practical method of utilising EPAs within a curriculum. 

Methods
A small working group consisting of experienced academics, senior clinicians, recent graduates and educationalists from medicine and physiotherapy developed a new assessment approach through several rounds of discussion, refinement and consensus. 

Results
A novel method was developed, utilising the expanded descriptions of EPAs to map clinical curriculums from the commencement of a program. Within expanded descriptions, micro-skills and knowledge domains that were required to perform an EPA could be identified, assessed and mapped. This created a series of data points that could be used to programmatically assess student’s competency, enabling much clearer assessment ‘for’, ‘as’ and ‘of’ learning. By making this approach transparent to students and assessors throughout the program, a shared mental model was developed, creating a “roadmap” for students to develop into practitioners. 

Conclusion 

Substantial value and opportunity lie within the expanded descriptions of EPAs. Clearly identifying task specific knowledge and microskills in an EPA assessment framework may help address issues in competency development and assessment in health professions curricula. 

Take home messages: 

References (maximum three) 

Prashant Jhala1
Arvin Damodaran^2,3, Adrienne Torda², Toby Wilcox⁴, Ben Taylor², Tayla Douglas⁴, Boaz Shulruf⁴ and Annette Katelaris²
1 School of Health Sciences, UNSW Sydney
2 School of Clinical Medicine, UNSW Sydney
3 Prince of Wales Hospital, Randwick Campus
4 Office of Medical Education, UNSW Sydney 

Background:
Since the inception of entrustable professional activities (EPAs), several entrustment scales have been introduced to medicine, pharmacy, dietetics and physiotherapy. However, the language of entrustment can cause some confusion, particularly interpreting the subcategories of supervision required. Levels of entrustment may not clearly differentiate between the graded responsibility afforded to the student. No uniform scale has been proposed that is useful for multiple entry-level disciplines. This paper presents an inter-professional entrustment scale, to be used for the entire degree, that articulates the level of responsibility appropriate to the student’s skill level. 

Methods:
Literature reviews were conducted by a small working group to inform the development and implementation of EPAs into existing programs in medicine, optometry, and exercise physiology, and new programs in pharmacy, dietetics, and physiotherapy. This informed a working group of experienced academics, senior clinicians, educationalists and recent graduates from medicine and physiotherapy developed a new scale of entrustment, through several rounds of discussion, refinement and consensus. 

Results:
The new entrustment scale consists of seven levels across three key phases of curriculum namely, foundational knowledge and skill development, work integrated learning and clinical practice. Each stage of the scale was mapped to an extended Miller’s Pyramid (ten Cate et al., 2021) to enable an understanding of the student’s progression throughout a curriculum. New descriptors were proposed, attempting to provide clarity and a shared language that could be understood by academics, students and supervising clinicians. 

Conclusions:
We propose a scale that can be applied inter-professionally to entry-level health professional education. Future research will focus on the development of common EPAs that can be evaluated using this common scale. 

References (maximum three) 

Ten Cate, O., Carraccio, C., Damodaran, A., Gofton, W., Hamstra, S.J., Hart, D.E., Richardson, D., Ross, S., Schultz, K., Warm, E.J. and Whelan, A.J., 2021. Entrustment decision making: extending Miller’s pyramid. Academic Medicine, 96(2), pp.199-204. 

STEPHEN TOBIN¹
Caroline Joyce² and Kent Robinson^3,4
1 ANZAHPE, ASME, WSU Medicine
2 Western Sydney Univiersity
3 WSU Medicine
4 NSW Health

Since 2020, our School of Medicine has been using entrustable professional activities (EPAs) for final-year students in the 5-year undergraduate medicine program. Development has been enabled by the MKM Myprogress (York, England) system, including web-based app for use on smartphones in the clinical workplace. The clinical content of the EPAs, based around the common clinical tasks of PGY 1-2 doctors has remained the same. The EPAs have demonstrated face validity for students and clinicians. 

Learning analytics, custom designed for our purpose, now enables dynamic monitoring of student engagement and entrustment by supervision level. With visual representation of their own dashboard, students aim to complete the range of 16 EPAs. Key clinical administrative staff have access to the analytics. Students who require coaching are easily identified, where EPA detail is scant, and/or the students are falling short of the required 2 EPAs/per week across the year. The visual dashboards help staff and students alike. 

This progressive assessment with EPAs, whilst actually contributing to patient care provides authentic evidence of student progression towards work readiness. The data accrued and analysed across 2021-22 has shown some trends in student performance, able to be looked at in the dynamic 2023 system. In the most recent year, this has influenced direction and guidance about use of the EPAs. There is an association between this EPA performance and our checkpoint case-based discussion examination. 

This educational research and evaluation supports our innovative approach to assessment. Discussions with other medical schools suggest that this approach is not yet commonplace, albeit there is great interest. It relies on programmatic assessment principles including being part of the assessment system, the involvement of clinicians at all levels especially in hospitals and smart co-design with IT people from the vendor and the university. 

References (maximum three) 

Torre et al Ottawa 2020 consensus statements for programmatic assessment - 2. Implementation and Practice. Med.Teach. 2021; 43:1149-1160 

Hobday PM et al The Minnesota Method: A Learner-driven Entrustable Professional Activity- Based Comprehensive Program of Assessment for Medical Students. Acad.Med 2021; 96:s50-55 

Sreedhar Radhika¹
Ananya Gangopadhyaya¹, Asra Khan¹, Ana Mauro¹, Angie Fanuke¹ and Yoon Soo Park¹
1 University of Illinois College of Medicine 

Background:
One in five patients report adverse events after discharge from the hospital, resulting in greater healthcare utilization. Teaching physicians to safely discharge patients is critical to improve patient safety. 

Summary of Work:
We evaluate an Entrustable Professional Activity (EPA) aligned patient discharge curriculum for fourth-year medical students in internal medicine, administered across three academic years (2020-23). The curriculum integrates self-study didactics, direct observation of discharge, student-led post-discharge phone calls to patients and preceptor-led small group debrief activity. 

Results:
294 students completed the pre- and post-assessments demonstrating significant improvements in knowledge (Cohen’s d = 1.41), skills (Cohen’s d = 1.45), and attitudes (Cohen’s d = 1.16) related to discharge, all P < .001. 

An EPA-grounded workplace-based assessment tool was used to provide real time feedback on discharge tasks for 291/294 students. Over 75% of students were able to do the following tasks independently: explain major diagnosis, explain alarm symptoms requiring return to hospital (EPA8),elicitconcernsaboutdischargeandhomesupport,andassesspatients’ability to safely take medications (EPA 4). 

145/294 students completed post-discharge phone calls and a questionnaire that showed that 95% of patients knew their admitting diagnosis (EPA 8), 93% of patients had a follow up appointment (EPA 9) and 99% patients/caregivers knew what to do if a problem arose (EPA 8). 

Discussion 

The curriculum demonstrates improved student performance in key areas related to discharge and helps identify student readiness to perform patient care tasks. 

A patient-discharge curriculum improves entrustability for fourth-year students in key areas related to patient discharge, which can enhance improvements toward patient safety outcomes. 

Assessing medical student readiness and entrustability for discharge is an important skill which spans several EPAs and is a critical area of assessment to consider for student competence. 

References (maximum three) 

Tzu-Hung Liu¹
1 Taipei Tzu Chi Hospital 

Background

For postgraduate trainees, the individualized learning plan (ILP) program is one of the potential measures that facilitate self-directed learning.[1] The learning goals in ILP can be translated into entrustable professional activities (EPAs). Entrustment‐based discussion (EBD) with a set of four questions has been proposed as an assessment tool for EPAs.[2] Therefore, we aim to use EBD in ILP for PGY-1 trainees in our study. 

Summary of work

We developed ILP programs for PGY-1 trainees in their 3-month internal medicine rotations, including monthly mentoring meetings. In the first meeting, the trainees generate two learning goals in the form of EPAs with their clinical mentor. In the following meetings, the mentor would use the four questions in EBD to ask the trainees and determine their entrustment level using the Chen entrustment scale.[3] 

From August 2022 to July 2023, 18 PGY-1 trainees participated in this study. Their 36 EPAs aligned mainly with two ACGME competencies, patient care and medical knowledge. According to the EBD results, most trainees’ entrustment levels were level 3b (with supervisor immediately available; key findings double-checked) (47.2%) or 3c (with supervisor distantly available; findings reviewed) (30.6%) at the end of the program. 

Trainees with level 3c entrustment in their EPAs could answer all four questions well in EBD. However, trainees with level 3b entrustment in their EPAs answered varying numbers of questions in EBD correctly. For common EPAs, correctly answering the first three questions is adequate for level 3b; for difficult EPAs, answering all four questions well does not guarantee a level 3c. 

Trainees who answered more questions correctly in EBD tend to have higher entrustment levels. The entrustment level may be lower for more difficult EPAs. 

EBD can be a useful assessment tool in ILP to generate entrustment levels for the PGY-1 trainees. 

References (maximum three) 

Tess Maguire¹
Georgina Willetts² and Loretta Garvey²
1 Centre for Forensic Behavioural Science, Swinburne University of Technology and Forensicare
2 Federation University

The Dynamic Appraisal of Situational Aggression (DASA) is an instrument designed to assess risk of imminent aggression. The Aggression Prevention Protocol (APP) was designed to structure nursing intervention according to the level of risk assessed by DASA, (together DASA+APP). Two studies testing the DASA+APP produced reductions in aggression and restrictive interventions, however a barrier to implementation is training (Griffith et al., 2021; Maguire et al., 2019). A recent study investigated approaches used in aggression prevention training to determine DASA+APP education for nurses. One recommendation was to include built-in assessment to ensure learners acquire adequate levels of competence to deliver APP interventions. 

Entrustable Professional Activities (EPAs) are a way of defining and assessing complex clinical interventions (Croft et al., 2020) such as the APP interventions. This presentation will discuss the development of seven APP/EPA’s, via focus groups with 11 expert prevention of aggression trainers. The APP/EPAs were then tested with graduate nurses via training including active-learning with expert trainers and simulation actors, to determine the suitability of the APP/EPAs as an assessment framework. 

Results suggest APP/EPAs can be used to assess learners’ readiness for professional DASA+APP practice and set the standard of practice for complex skills required to prevent aggression. 

EPAs filled a practice-gap, where focus of assessment has been on physical techniques, as opposed to de-escalation and limit-setting skills. EPAs provided consistency in practice, can enhance documentation and define standards of practice. The training provided a safe environment to practice these important skills. 

Defining, applying and assessing APP activities that prevent aggression is essential, although methods to assist with this task have been lacking. Results suggest APP/EPAs and associated training provided a suitable method to define and assess practice the complex APP clinical activities. 

The APP/EPAs offer a framework to assist aggression prevention and encourage reflection and practice improvement.



References (maximum three) 

Croft, H., Gilligan, C., Rasiah, R., Levett-Jones, T., & Schneider, J. (2020). Development and inclusion of an entrustable professional activity (EPA) scale in a simulation-based medicine dispensing assessment. Currents in Pharmacy Teaching and Learning, 12(2), 203-212. 

Griffith, J. J., Meyer, D., Maguire, T., Ogloff, J. R. P., & Daffern, M. (2021). A clinical decision support system to prevent aggression and reduce restrictive practices in a forensic mental health service. Psychiatric Services, 72(8), 885-890. 

Maguire, T., Daffern, M., Bowe, S. J., & McKenna, B. (2019). Evaluating the impact of an electronic application of the Dynamic Appraisal of Situational Aggression with an embedded Aggression Prevention Protocol on aggression and restrictive interventions on a forensic mental health unit. International Journal of Mental Health Nursing, 28(5), 1186-1197. http://doi.org/10.1111/inm.12630