Presentation Description
Bernard Cerutti1
Frédéric Geoffroy1, Milena Abbiati1,2, Monica Escher1,3, Aviva Chmiel1, Vanessa Lavallard1 and Mathieu Nendaz1,3
1 University of Geneva
2 Lausanne University Hospital
3 Geneva University Hospitals
Frédéric Geoffroy1, Milena Abbiati1,2, Monica Escher1,3, Aviva Chmiel1, Vanessa Lavallard1 and Mathieu Nendaz1,3
1 University of Geneva
2 Lausanne University Hospital
3 Geneva University Hospitals
Background:
Our institution recently introduced an e-Portfolio for the undergraduate medical students to promote the longitudinal recording of evidence of experience and achievement, reflection, and both self and external assessment. One of the objectives is to move from mostly iterative summative assessment to a more continuous evaluation process.
Our institution recently introduced an e-Portfolio for the undergraduate medical students to promote the longitudinal recording of evidence of experience and achievement, reflection, and both self and external assessment. One of the objectives is to move from mostly iterative summative assessment to a more continuous evaluation process.
Summary of work:
We explored the reliability of the e-Portfolio content by analysing the consistency of a subset of variables with summative assessment scores. Study population: cohort of 157 fourth-year students who ended their first year of clinical rotations (internal medicine, primary care medicine, surgery, paediatrics, psychiatry, gynaecology) in November 2022. Data: set of 13401 documented activities (technical and clinical skills, clinical situations and case reports, clerkship wrap-up reports), CanMEDS roles and autonomy in EPAs validated by supervisors, summative written exam and OSCE scores, and motivation (collected through the Academic Motivation Scale).
We explored the reliability of the e-Portfolio content by analysing the consistency of a subset of variables with summative assessment scores. Study population: cohort of 157 fourth-year students who ended their first year of clinical rotations (internal medicine, primary care medicine, surgery, paediatrics, psychiatry, gynaecology) in November 2022. Data: set of 13401 documented activities (technical and clinical skills, clinical situations and case reports, clerkship wrap-up reports), CanMEDS roles and autonomy in EPAs validated by supervisors, summative written exam and OSCE scores, and motivation (collected through the Academic Motivation Scale).
Results:
The volume of activities was associated with both written exam (rho=.169, p=.034) and OSCEs (rho=.152, p=.056) scores. EPAs evaluation, especially the prioritization of a differential diagnosis following a clinical encounter, was associated with written exams (rho=.229, p=.004) and OSCEs scores (rho=.299, p<.001). Weaker correlation was observed for the CanMEDS roles, with a strong positive association of the exams scores with the Scholar role. There was a small association between the intrinsic motivation and both the number of documented activities in the ePortfolio (rho=.192, p=.031), and to a lesser extent the written exam scores (rho=.125, p=.161)
The volume of activities was associated with both written exam (rho=.169, p=.034) and OSCEs (rho=.152, p=.056) scores. EPAs evaluation, especially the prioritization of a differential diagnosis following a clinical encounter, was associated with written exams (rho=.229, p=.004) and OSCEs scores (rho=.299, p<.001). Weaker correlation was observed for the CanMEDS roles, with a strong positive association of the exams scores with the Scholar role. There was a small association between the intrinsic motivation and both the number of documented activities in the ePortfolio (rho=.192, p=.031), and to a lesser extent the written exam scores (rho=.125, p=.161)
Conclusion:
These preliminary results support the consistency of the ePortfolio contents with summative assessment and motivational scores. Further evidence regarding these preliminary results should be brought by the data of a second cohort of students, which will be soon available.
Take-home message:
Consistency checks are of paramount importance to investigate the reliability of the e-Portfolio contents as an evaluation tool.
Consistency checks are of paramount importance to investigate the reliability of the e-Portfolio contents as an evaluation tool.
References (maximum three)
Ting JJQ, Phua GLG, Hong DZ, Lam BKY, Lim AJS, Chong EJX, Pisupati A, Tan R, Yeo JYH, Koh YZ, Quek CWN, Lim JY, Tay KT, Ong YT, Chiam M, Zhou JX, Mason S, Wijaya L, Krishna LKR. Evidence-guided approach to portfolio-guided teaching and assessing communications, ethics and professionalism for medical students and physicians: a systematic scoping review. BMJ Open. 2023 Mar 28;13:e067048. doi: 10.1136/bmjopen-2022-067048. PMID: 36977542; PMCID: PMC10069516
Belcher R, Jones A, Smith LJ, Vincent T, Naidu SB, Montgomery J, Haq I, Gill, D. Qualitative study of the impact of an authentic electronic portfolio in undergraduate medical education. BMC Med Educ. 2014 Dec 17;14:265. doi: 10.1186/s12909-014-0265-2. PMID: 25515320; PMCID: PMC4272766
Vallerand RJ, Pelletier LG, Blais MR, Briere NM, Senecal C, Vallieres EF. The academic motivation scale: a measure of intrinsic, extrinsic, and a motivation in education. Educational and Psychological Measurement 1992(52): 1003–1017