Toni Ungaretti¹
1 Johns Hopkins University

Background
Increasingly, medical institutions seek HPEGDPs graduates to lead initiatives to advance the transformation of heath care through the preparation of future providers (Humphrey, 2021). The stakes are high for HPEGDP to ensure that graduates achieve program outcomes aligned with these needs. E-portfolio assessment address knowledge, performance, and attitudes (KPA); assists formative/summative assessment, supports self-directed learning, and informs future program development. 

Summary
Using an e-portfolio assessment, learners identify evidence related to each outcome supported by rationale/reflection with formative review/feedback after each 12-week session and summative review/feedback after completion of the evidence-based teaching core, the specialization, and the final reflection. Course faculty guide formative and independent team conducts the summative process (Heeneman et al., 2020). 

Results
Three researchers reviewed/analyzed 32 learners’ portfolios using a 3-point scale and Dedoose software. Reviewers crosschecked 5 reviews for reliability. Response patterns/outliers were identified in relation to KPAs. Findings indicate twelve outcomes associated with the core and six outcomes associated with the specializations addressed at either the proficient or exemplary levels. 

Discussion
This e-portfolio provided formative/summative assessment data that informed program evaluation and guided learners to incorporate reflective practice. Results raised future questions about process, validity, and reliability. Learners’ reflections over time indicated increased critical reflexivity in addition to critical reflection focused on social responsibility (Ng et al., 2019) which have informed this program’s development. 

Conclusion
E-portfolio assessment of learners’ KPAs is promising as a tool for HPEGDP evaluation. Given the increased use of the HPEGDP degree as an indicator of competence by medical institutions, the need for additional study of content, process, and psychometrics of e-portfolios in program evaluation is urgent. 

Take home messages/implications 
Additional efforts to examine e-portfolio assessment and its contributions to HPEGDPs evaluation are needed. The inclusion of critical reflexivity in the curriculum needs to be explored. 

References (maximum three) 

Heeneman, S., de Jong, L. H., Dawson, L. J., Wilkinson, T. J., Ryan, A., Tait, G. Rice, N., Torre, D., Freeman, A, & van der Vleuten, C. P. (2021). Ottawa 2020 consensus statement for programmatic assessment–1. Agreement on the principles. Medical teacher, 43(10), 1139- 1148. 

Humphrey, H. (2021). Conversation regarding the change in credentials of Macy Scholars over the ten-year history of the program. 

Ng, S. L., Wright, S. R., & Kuper, A. (2019). The divergence and convergence of critical reflection and critical reflexivity: implications for health professions education. Academic Medicine, 94(8), 1122-1128. 

Edward Waters 
Delyse Leadbeatter 

Background:
In 2023, Sydney University commenced a new dental curriculum, based on programmatic assessment and using an e-portfolio system to collect and collate assessment results and reflections. Removable prosthodontics (RP - the provision of dentures) involves distinct laboratory and clinical stages, and is an area of dentistry that students often find challenging [1]. Threshold concepts have been described as milestones that transform the way in which a student learns, understands and practices, and have been previously proposed as a way to define student learning outcomes in RP [2]. This study aims to build upon threshold concept theory in RP by analysing student e-portfolio submissions in this field. 

Summary of work:
a set of e-portfolio forms incorporating knowledge generating, reflective and self-assessment tasks were developed to understand how students learnt and assessed themselves in laboratory tasks in RP, and how they linked laboratory and clinical aspects of RP. These submitted forms were analysed using grounded theory. 

Results:
At the time of submission, partial data analysis has identified one new potential threshold concept in RP - "align clinical goals and laboratory requirements during all steps of denture fabrication". Student submissions provided evidence of independent identification and navigation of this threshold concept. 

Discussion:
Preliminary data suggest e-portfolio forms can be a useful tool to focus student self-learning of new concepts, and to record this learning. Learning recorded in e-portfolios can help to identify threshold concepts in dental curricula and to document student navigation of these concepts. Future work with the whole data set will aim to identify further threshold concepts in RP and guide dental curriculum development. 

Conclusions and take-home messages:
Beyond e-portfolios being a record of attainment, they can be a source of educational theory and deeply rich data that should not be neglected in future research. 

References (maximum three) 

[1] Johnson, G. M., Halket, C. A., Ferguson, G. P., & Perry, J. (2017). Using standardized patients to teach complete denture procedures in second year of dental school. Journal of dental education, 81(3), 340-346. 

[2] Hyde, S., Flatau, A., & Wilson, D. (2018). Integrating threshold concepts with reflective practice: Discussing a theory‐based approach for curriculum refinement in dental education. European Journal of Dental Education, 22(4), e687-e697 

Bernard Cerutti¹
Frédéric Geoffroy¹, Milena Abbiati^1,2, Monica Escher^1,3, Aviva Chmiel¹, Vanessa Lavallard¹ and Mathieu Nendaz^1,3
1 University of Geneva
2 Lausanne University Hospital
3 Geneva University Hospitals

Background:
Our institution recently introduced an e-Portfolio for the undergraduate medical students to promote the longitudinal recording of evidence of experience and achievement, reflection, and both self and external assessment. One of the objectives is to move from mostly iterative summative assessment to a more continuous evaluation process. 

Summary of work:
We explored the reliability of the e-Portfolio content by analysing the consistency of a subset of variables with summative assessment scores. Study population: cohort of 157 fourth-year students who ended their first year of clinical rotations (internal medicine, primary care medicine, surgery, paediatrics, psychiatry, gynaecology) in November 2022. Data: set of 13401 documented activities (technical and clinical skills, clinical situations and case reports, clerkship wrap-up reports), CanMEDS roles and autonomy in EPAs validated by supervisors, summative written exam and OSCE scores, and motivation (collected through the Academic Motivation Scale). 

Results:
The volume of activities was associated with both written exam (rho=.169, p=.034) and OSCEs (rho=.152, p=.056) scores. EPAs evaluation, especially the prioritization of a differential diagnosis following a clinical encounter, was associated with written exams (rho=.229, p=.004) and OSCEs scores (rho=.299, p<.001). Weaker correlation was observed for the CanMEDS roles, with a strong positive association of the exams scores with the Scholar role. There was a small association between the intrinsic motivation and both the number of documented activities in the ePortfolio (rho=.192, p=.031), and to a lesser extent the written exam scores (rho=.125, p=.161) 

Conclusion:
These preliminary results support the consistency of the ePortfolio contents with summative assessment and motivational scores. Further evidence regarding these preliminary results should be brought by the data of a second cohort of students, which will be soon available. 

Take-home message:
Consistency checks are of paramount importance to investigate the reliability of the e-Portfolio contents as an evaluation tool. 

References (maximum three) 

Ting JJQ, Phua GLG, Hong DZ, Lam BKY, Lim AJS, Chong EJX, Pisupati A, Tan R, Yeo JYH, Koh YZ, Quek CWN, Lim JY, Tay KT, Ong YT, Chiam M, Zhou JX, Mason S, Wijaya L, Krishna LKR. Evidence-guided approach to portfolio-guided teaching and assessing communications, ethics and professionalism for medical students and physicians: a systematic scoping review. BMJ Open. 2023 Mar 28;13:e067048. doi: 10.1136/bmjopen-2022-067048. PMID: 36977542; PMCID: PMC10069516 

Belcher R, Jones A, Smith LJ, Vincent T, Naidu SB, Montgomery J, Haq I, Gill, D. Qualitative study of the impact of an authentic electronic portfolio in undergraduate medical education. BMC Med Educ. 2014 Dec 17;14:265. doi: 10.1186/s12909-014-0265-2. PMID: 25515320; PMCID: PMC4272766 

Vallerand RJ, Pelletier LG, Blais MR, Briere NM, Senecal C, Vallieres EF. The academic motivation scale: a measure of intrinsic, extrinsic, and a motivation in education. Educational and Psychological Measurement 1992(52): 1003–1017 

Carmen Abeyaratne¹
Steven Walker¹, Alex Edwards¹ and Ben Emery¹
1 Monash University 

In Australia, pharmacy practice is underpinned by the National Competency Standards Framework (NCSF).(1) Pre-registrant pharmacists (i.e. interns) complete a year-long supervised program to meet core competencies of the NCSF. 

In 2016, an extension work-integrated learning (WIL) program was established to fast-track intern development of more advancing NCSF competencies.(2) To evidence this, interns are expected to complete a series of workplace-based assessments (WBA) as part of a year-long workplace learning plan (WLP). 

In 2023, a more overt portfolio-based WLP was established. For each of the five NCSF domains (Domain 1: Professional practice, Domain 2: Communication/Collaboration, Domain 3: Medication management, Domain 4: Leadership and Domain 5: Education), interns were required to select a range of WBAs for their portfolio submission. Interns complete supervised training in either a hospital or community setting and were expected to liaise with their supervisor to select WBAs catered to their development needs and advancement goals. 

In 2023, 231 (n=110 hospital, n=121 community) interns were enrolled in the program. For Domain 2, the most commonly performed WBA was “written communication to a Doctor” (81%)forcommunityinternsandan“inpatient/progressnote”(58.1%)forhospitalinterns. For Domain 3, the most commonly performed WBA was a “primary care case” (82.6%) for community interns and a “Best Possible Medication History” (92.7%) for hospital interns. For Domain 5, the most commonly performed WBA was “supervision of a learner” (58.7%) for community interns and a “continuing education presentation” (78.2%) for hospital interns. 

This is the first pre-registrant program in Australia which uses a portfolio-based WLP to assess advancing competencies. 

Utilising portfolio-based WLP evidence allows for structured documentation of development in NCSF domains of competency.(3) 

Take-home messages/implications for further research or practice 

Similar portfolio-based WLPs may be adopted by other pre-registrant or early career training programs. 

References (maximum three) 

Rebecca Roubin¹
 1 The University of Sydney Pharmacy School 

Background
Portfolios are used for assessment of pharmacy students, assessing their ability to provide patient-centred care through documenting patient cases and justifying clinical decisions during workplace-based placements. However, traditional portfolios were found to be onerous to mark and often provided a limited view of a student's capabilities. Moreover, student performance and feedback indicated poor engagement with the portfolio assessment. 

Summary of Work
With a goal to enhance the assessment process, this project aimed to redesign portfolio assessments in a new ePortfolio system for the new pharmacy program. The development of new ePortfolios was undertaken to capture an integrated, longitudinal view of a student's growth and competence, aiming to foster greater engagement and provide a more complete understanding of student skills and attributes. 

Results
The new ePortfolio entails compiling diverse evidence, including reflections, peer feedback, tutor, and preceptor evaluations, mastery skills and clinical documentation, organised around professional competencies. Supporting learning activities were also developed to assist students, educators, and preceptors in identifying and addressing specific areas of improvement, thereby enhancing the educational experience. 

Discussion and Conclusions
It is anticipated that the student-driven focus of the new ePortfolios will enhance student engagement. Future work includes evaluating the success of the portfolio redesign. The redesigned portfolios are expected to provide a comprehensive understanding of student competence, encompassing both clinical skills and attributes such as empathy and ethical reasoning, offering a richer assessment tool. By emphasising the holistic growth of the student, these ePortfolios are designed to more authentically mirror the demands and intricacies of pharmacy practice. 

Take-Home Messages/Implications for Further Research or Practice

The redesigned ePortfolios foster a more complete and longitudinal understanding of student competence, capturing attributes that traditional portfolios might overlook. 

References (maximum three) 

No references 

Fiona Muir¹
1 Dundee University 

Most doctors engage in teaching to a greater or lesser extent and as such need to keep their skills and knowledge up to date. While there appears to be no formal requirement or pre-defined route by which trainee doctors can become competent teachers during their undergraduate medical degree, Dundee Medical School Scotland offers students a choice to undertake a BMSc (Hons) Medical Education intercalated degree. 

The BMSc is designed with modules of study, teaching portfolio with associate fellowship, educational research, and dissertation which are assessed. This BMSc is the first to be accredited by Advance HE for this specific group of students to achieve Associate Fellowship of Advance HE (AFHEA) (Advance HE, 2023a); a badge of international recognition increasingly sought by employers across the education sector and clinical academic posts. 

Students undertake an assessed teaching portfolio of authentic teaching experiences as part of the BMSc Medical Education Programme. This provides medical students an opportunity to develop core knowledge, skills, and attributes to educate peers and colleagues, and their patients (General Medical Council, 2019). 

Students achieving BMSc with AFHEA status (n=70), via an assessed teaching portfolio, places value on professional values; knowledge of teaching and research; effective and inclusive practice and the notion that the ‘student as teacher’ intercalated programme is an asset to medical education (Muir, Bruce and McConville, 2020). 

The BMSc embeds authentic teaching practice within an assessed teaching portfolio aligned with Advance HE requirements, and provides a valuable measure of success, demonstrates commitment to teaching, learning and the student experience, through engagement in a practical process; a route by which trainee doctors can develop their competence as teachers. 

There is value of embedding an assessed teaching portfolio within a Medical Education Intercalated Degree Programme with Associate Fellowship of Advance HE. 

References (maximum three) 

Advance HE. 2023a. [accessed 7 July 2023]. https://www.advance-he.ac.uk/news- andviews/Dundee-Medical-students-recognised-as-Associate-Fellows-for-first-time 

Good Medical Practice. 2019. [accessed 7 July 2023]. https://www.gmc- uk.org//media/documents/good-medical-practice---english-1215_pdf-51527435.pdf 

Muir, F. Bruce, J and McConville, K. (2020) Teaching, reflecting, and learning: The value of an intercalated medical education programme, Medical Teacher, 42:5, 523-528, DOI: 10.1080/0142159X.2019.1708290