Michelle Daniel¹
Stuart Lane², Chris Roberts, James Murphy³, Holly Caretta-Weyer⁴, Eric Holmboe⁵, Brian Kwan¹, Dario Torre⁶ and Priya Khanna⁷
1 University of California, San Diego
2 Sydney Medical School
3 University of California San Diego School of Medicine
4 Stanford University School of Medicine
5 Accreditation Council for Graduate Medical Education
6 University of Central Florida
7 School of Medicine, The University of Sydney

Programmatic assessment (PA) is changing the way we think about assessment in a post psychometric, competency-based education era. Unlike traditional “module-test” assessment models, where the focus is largely on assessment of learning, programmatic assessment offers a holistic, longitudinal approach to assessment that combines assessment of learning with assessment for learning. PA can be applied to any competency within larger systems of assessment. 

The Principles 

PA requires information about learners’ competence to be collected and analyzed continuously. Multiple assessment data points are accumulated over time from a mix of assessment methods, to create an increasingly detailed picture of learner competence. There is a constant reflective dialogue with the learner about both individual assessment data points, goals and competency trajectories, for the purpose of providing rich feedback. Decisions about competence and progress are made remote from any individual assessment, course or clerkship through the integration of data. Credible, equitable decisions are based on the principles of proportionality and triangulation. 

The Challenge 

While the theoretical principles of programmatic assessment are clear, implementation remains a significant challenge. In this workshop, we will discuss three keys to successful implementation of PA: 

1) Robust learner dashboards and data analytics (to collect and integrate multiple assessment data points, potentially leveraging newer technology (e.g., machine learning) 

2) Competency Committees (to review data separate from individual course and clerkships, and make decisions on progress) 

3) Coaching and advising programs (to support reflective dialogue and ensure feedback) 

Why is the topic important for research and / or practice?
While many educators embrace the principles of programmatic assessment, implementation is often difficult. By focusing on keys to success, we hope to address some of the frequently encountered barriers to full implementation of the model. 

Workshop format, including participant engagement methods
Initial think-pair-share exercise (What problem(s) is PA trying to solve? Why do we need to embrace PA?) (10 min); Didactic on principles of PA and an overview of implementation challenges (20 min). Panel discussion on keys to successful PA implementation (30 min). 

Who should participate?
Faculty who design / implement curricula and programs of assessment 

Level of workshop
Novice / intermediate 

Take-home messages / workshop outcomes / implications for further research or practice
Understanding and addressing implementation challenges helps ensure that programmatic assessment is executed successfully and yields its intended benefits. This, in turn, can have a positive impact on teaching and learning outcomes, enabling educators to better support their learners' growth and development. 

Maximum number of participants
50 

References (maximum three) 

Heeneman S, de Jong LH, Dawson LJ, Wilkinson TJ, Ryan A, Tait GR, Rice N, Torre D, Freeman A, van der Vleuten CP. Ottawa 2020 consensus statement for programmatic assessment–1. Agreement on the principles. Medical teacher. 2021 Oct 3;43(10):1139-48. 

Torre D, Rice NE, Ryan A, Bok H, Dawson LJ, Bierer B, Wilkinson TJ, Tait GR, Laughlin T, Veerapen K, Heeneman S. Ottawa 2020 consensus statements for programmatic assessment– 2. Implementation and practice. Medical teacher. 2021 Oct 3;43(10):1149-60.