Presentation Description
Shelley Ross1
Zoe Brody1, Mawada Tarhuni1, Tarleen Dhanoa1, Todd Milford2, Darren Nichols1 and Shirley Schipper1
1 University of Alberta
2 University of Victoria
Zoe Brody1, Mawada Tarhuni1, Tarleen Dhanoa1, Todd Milford2, Darren Nichols1 and Shirley Schipper1
1 University of Alberta
2 University of Victoria
Background:
Effective and trustworthy assessment in competency-based medical education requires the development and implementation of well-designed programmatic assessment frameworks. In 2010, we implemented the Competency-Based Achievement System (CBAS), which includes collection of multiple workplace-based formative assessments. We use learning analytics to continuously improve our learning environment and approaches to teaching and assessment, including evaluation of CBAS. In this study, we looked specifically at two evaluation concerns: 1) To what extent does CBAS capture assessment information across all of the competencies of family medicine; and 2) To what extent does CBAS show stability over time (i.e., consistency of use and engagement from teachers and learners)?
Effective and trustworthy assessment in competency-based medical education requires the development and implementation of well-designed programmatic assessment frameworks. In 2010, we implemented the Competency-Based Achievement System (CBAS), which includes collection of multiple workplace-based formative assessments. We use learning analytics to continuously improve our learning environment and approaches to teaching and assessment, including evaluation of CBAS. In this study, we looked specifically at two evaluation concerns: 1) To what extent does CBAS capture assessment information across all of the competencies of family medicine; and 2) To what extent does CBAS show stability over time (i.e., consistency of use and engagement from teachers and learners)?
Summary of work:
Using ten years of learner analytics data (workplace-based assessment data called FieldNotes), we conducted a retrospective cohort secondary data analysis. Data were analyzed using descriptives and data visualization to identify trends within and across academic years.
Using ten years of learner analytics data (workplace-based assessment data called FieldNotes), we conducted a retrospective cohort secondary data analysis. Data were analyzed using descriptives and data visualization to identify trends within and across academic years.
Results:
Engagement with CBAS increased from 2010 (5208 FieldNotes) to 2016 (6764), then plateaued until 2020. FieldNotes dropped in 2021 (5525) and 2022 (5207). Distribution of FieldNotes across competencies has been highly consistent across all years: approximately 35% (2000-2500) FieldNotes are about diagnosis and management. The remainder are about patient-centered care (13%), procedural skills (17%), communication (10%), professionalism (6%), and teaching skills (6% each). The final two competencies are related to adaptive expertise (6%) and self-regulated learning (3%).
Engagement with CBAS increased from 2010 (5208 FieldNotes) to 2016 (6764), then plateaued until 2020. FieldNotes dropped in 2021 (5525) and 2022 (5207). Distribution of FieldNotes across competencies has been highly consistent across all years: approximately 35% (2000-2500) FieldNotes are about diagnosis and management. The remainder are about patient-centered care (13%), procedural skills (17%), communication (10%), professionalism (6%), and teaching skills (6% each). The final two competencies are related to adaptive expertise (6%) and self-regulated learning (3%).
Discussion:
Our findings indicate that CBAS has been a highly stable assessment framework, with good engagement until very recently. Even though we saw decreases in FieldNotes entries in 2021 and 2022, workplace-based assessment of learners continues to address a variety of competencies, coming from multiple observers.
Our findings indicate that CBAS has been a highly stable assessment framework, with good engagement until very recently. Even though we saw decreases in FieldNotes entries in 2021 and 2022, workplace-based assessment of learners continues to address a variety of competencies, coming from multiple observers.
Conclusions:
While CBAS has good stability, attention must be paid to the recent declines in FieldNote entries.
While CBAS has good stability, attention must be paid to the recent declines in FieldNote entries.
Future research:
We are planning a more in-depth examination of CBAS, using more learning analytics data.
We are planning a more in-depth examination of CBAS, using more learning analytics data.
References (maximum three)
1. van der Vleuten CP, Schuwirth LW. Assessing professional competence: from methods to programmes. Med Educ. 2005; 39: 309–17.
2. Ross S, Lawrence K, Bethune C, van der Goes T, Pélissier-Simard L, Donoff M, Crichton T, Laughlin T, Dhillon K, Potter M, Schultz K. Development, implementation, and meta- evaluation of a national approach to programmatic assessment in family medicine residency training. Academic Medicine 2023; 98(2):188-198
3. Ross S, Poth C, Donoff M, Humphries P, Steiner I, Schipper S, Janke F, Nichols D. The Competency-Based Achievement System (CBAS): Using formative feedback to teach and assess competencies with Family Medicine residents. Canadian Family Physician. 2011;57:e323-e330.