Presentation Description
Yoon Soo Park1
Sean Hogan2 and Eric Holmboe2
1 University of Illinois College of Medicine
2 Accreditation Council for Graduate Medical Education
Sean Hogan2 and Eric Holmboe2
1 University of Illinois College of Medicine
2 Accreditation Council for Graduate Medical Education
Background:
Postgraduate training in the United States requires formative assessments of learners using the Accreditation Council for Graduate Medical Education (ACGME) Milestones assessment system. Recently, the ACGME implemented Milestones 2.0, offering specialties to use the same “Harmonized” Milestones for Professionalism, Communication and Interpersonal Skills, Systems-Based Practice, and Practice Based Learning and Improvement. This study examines factors contributing to variability in postgraduate year (PGY)-1 Harmonized Milestones and differences between specialties.
Postgraduate training in the United States requires formative assessments of learners using the Accreditation Council for Graduate Medical Education (ACGME) Milestones assessment system. Recently, the ACGME implemented Milestones 2.0, offering specialties to use the same “Harmonized” Milestones for Professionalism, Communication and Interpersonal Skills, Systems-Based Practice, and Practice Based Learning and Improvement. This study examines factors contributing to variability in postgraduate year (PGY)-1 Harmonized Milestones and differences between specialties.
Summary of Work:
We use data from entering cohorts in 2021 and 2022 from eight largest specialties: Emergency Medicine, Family Medicine, Internal Medicine, General Surgery, OBGYN, Psychiatry, Radiology, and Pediatrics. Variance components analyses were conducted using cross-classified random-effects models, accounting for clustering at the residency program, medical school levels, and specialty.
We use data from entering cohorts in 2021 and 2022 from eight largest specialties: Emergency Medicine, Family Medicine, Internal Medicine, General Surgery, OBGYN, Psychiatry, Radiology, and Pediatrics. Variance components analyses were conducted using cross-classified random-effects models, accounting for clustering at the residency program, medical school levels, and specialty.
Results:
We analyzed data from 62,005 residents (2,919 programs). When comparing across specialties, specialty accounted for the largest variance (24%) across competencies. By specialty, variance components for trainees, residency programs, and medical schools accounted for 29%, 35%, and 2% of total variance, respectively. Learner variance within specialties demonstrated substantial variability; learner variance in Internal Medicine and General Surgery was 48% and 32% in Harmonized Milestones, meaning programs are identifying different performance levels of trainees.
We analyzed data from 62,005 residents (2,919 programs). When comparing across specialties, specialty accounted for the largest variance (24%) across competencies. By specialty, variance components for trainees, residency programs, and medical schools accounted for 29%, 35%, and 2% of total variance, respectively. Learner variance within specialties demonstrated substantial variability; learner variance in Internal Medicine and General Surgery was 48% and 32% in Harmonized Milestones, meaning programs are identifying different performance levels of trainees.
Discussion:
Understanding factors that contribute to learner variance during PGY-1 is essential in determining the focus and allocation of training resources, and to prepare learners to successfully transition into postgraduate training. Our findings show substantial differences in the use of Harmonized Milestones across specialties. Despite specialty differences, the Harmonized Milestones offer opportunities to examine variability in learner performance and developmental patterns.
Understanding factors that contribute to learner variance during PGY-1 is essential in determining the focus and allocation of training resources, and to prepare learners to successfully transition into postgraduate training. Our findings show substantial differences in the use of Harmonized Milestones across specialties. Despite specialty differences, the Harmonized Milestones offer opportunities to examine variability in learner performance and developmental patterns.
Conclusions:
The Harmonized Milestones can be used to examine developmental trajectories of learners within specialty to examine factors contributing to longitudinal variability in resident Milestones ratings.
The Harmonized Milestones can be used to examine developmental trajectories of learners within specialty to examine factors contributing to longitudinal variability in resident Milestones ratings.
Take-Home Messages:
Harmonized Milestones can identify learners of different performance levels using consistent performance categories and subcompetencies within specialty.
Harmonized Milestones can identify learners of different performance levels using consistent performance categories and subcompetencies within specialty.
References (maximum three)
- Park YS, Hamstra SJ, Yamazaki K, Holmboe E. Longitudinal reliability of Milestones- based learning trajectories in Family Medicine residents. JAMA Netw Open. 2021;4(12): e2137179. doi: 10.1001/jamanetworkopen.2021.37179.
- Park YS, Ryan MS, Hogan SO, Berg K, Eickmeyer A, Fancher T, Farnan J, Lawson L, Turner L, Westervelt M, Holmboe E, Santen SA. Transition to residency: National study of factors contributing to variability in learner performance in Emergency Medicine and Family Medicine Milestone Ratings. Acad Med. doi: 10.1097/ACM.0000000000005366.