Presentation Description
Brian Kwan1,2
Leslie Oyama1,2, Christopher Longhurst1,2, James Murphy1,2 and Michelle Daniel3
1 University of California San Diego School of Medicine
2 University of California San Diego Health
3 AMEE
Leslie Oyama1,2, Christopher Longhurst1,2, James Murphy1,2 and Michelle Daniel3
1 University of California San Diego School of Medicine
2 University of California San Diego Health
3 AMEE
Background
Proficiency in procedures is a core competency for graduate medical education, but reporting of procedures often relies on manual logging workflows that are duplicative, inefficient, and generate data with questionable validity and accuracy. Failure to accurately gather trainee procedural completion data slows learner progression, delays procedures, and negatively impacts patient care.
Summary of Work
We built a novel automated system that extracts procedural completion data for emergency medicine resident (trainee) physicians directly from an electronic health record (EHR) system in common use and uploads this data via an application programming interface (API)-based approach to a learning management system (LMS) used by many training programs for accreditation purposes. We also designed and built dashboards to facilitate data dissemination to stakeholders.
Results
Over approximately one year, the automated system extracted 7,617 procedures from EHR data compared to 4,291 manually logged procedures in the same timeframe, representing an increase of 78%. The system uploaded procedures to the LMS with a recall/sensitivity of 97.3%, specificity of 100%, and overall accuracy of 99.5%.
Discussion
Use of our automated system to document procedure completion resulted not only in a substantial increase in the number of procedures logged, but also provided data with greater validity and accuracy. Since data extraction and upload is automatic, this system facilitates frequent evaluation of procedural completion, enabling more timely intervention if learners require additional instruction or experience to advance in training. Limitations of our work include performance at a single site with a single EHR system and LMS.
Use of our automated system to document procedure completion resulted not only in a substantial increase in the number of procedures logged, but also provided data with greater validity and accuracy. Since data extraction and upload is automatic, this system facilitates frequent evaluation of procedural completion, enabling more timely intervention if learners require additional instruction or experience to advance in training. Limitations of our work include performance at a single site with a single EHR system and LMS.
Conclusions
Reliance on manual procedure completion workflows results in significant procedure under- reporting compared to using EHR data recorded at the point-of-performance and can negatively impact learner progression and patient care.
Implications for Further Research
We plan to expand our system into other training programs, use cases, and possibly other institutions.
References (maximum three)
1. NEJM Group. Exploring the AGME Core Competencies: Patient Care and Procedural Skills (Part 3 of 7). Published September 8, 2016. Accessed March 17, 2023. https://knowledgeplus.nejm.org/blog/patient-care-procedural-skills
2. Triola MM, Burk-Rafel J. Precision Medical Education. Acad Med. 2023;98(7):775-781. doi:10.1097/ACM.0000000000005227