Times are shown in your local time zone GMT
Ad-blocker Detected - Your browser has an ad-blocker enabled, please disable it to ensure your attendance is not impacted, such as CPD tracking (if relevant). For technical help, contact Support.
Item generation and COVID strategies
Oral Presentation
Oral Presentation
11:30 am
28 February 2024
M215
Session Program
11:30 am
Korakrit Imwattana1
1 Faculty of Medicine Siriraj Hospital, Mahidol University
1 Faculty of Medicine Siriraj Hospital, Mahidol University
Background:
The multiple-choice examination is an essential tool for the assessment of basic medical knowledge. Some test items may include a short clinical scenario (a stem). If used appropriately, these stems can provide clinical context and may help students prepare for their clinical-based assessment in the future, without much impact on the question quality. However, generating stems can be challenging, especially for educators without clinical background or experience.
The multiple-choice examination is an essential tool for the assessment of basic medical knowledge. Some test items may include a short clinical scenario (a stem). If used appropriately, these stems can provide clinical context and may help students prepare for their clinical-based assessment in the future, without much impact on the question quality. However, generating stems can be challenging, especially for educators without clinical background or experience.
Summary of work:
An Excel-based exam stem generator was developed in-house. This prototype focused on basic bacterial infections, following the Thai Medical Council’s criteria, for use in the Principles of Microbiology class. The program was designed to be user-friendly, requiring only 5 inputs (bacterium of interest; patient’s age, gender, extra conditions [risk factors, comorbidities etc.]; and whether to include laboratory information). It also provided suggestions for non-clinical users. The program then generated a clinical scenario with sufficient keywords for the most likely causative organism. This study evaluates the efficacy of this program.
An Excel-based exam stem generator was developed in-house. This prototype focused on basic bacterial infections, following the Thai Medical Council’s criteria, for use in the Principles of Microbiology class. The program was designed to be user-friendly, requiring only 5 inputs (bacterium of interest; patient’s age, gender, extra conditions [risk factors, comorbidities etc.]; and whether to include laboratory information). It also provided suggestions for non-clinical users. The program then generated a clinical scenario with sufficient keywords for the most likely causative organism. This study evaluates the efficacy of this program.
Results and discussion:
Compared to an experienced educator in an ideal condition (38.97 seconds/stem) this in-house stem generator significantly reduced the time to create stems (20.49 seconds/stem, p < 0.0001). Among the 50 randomly generated stems, only 12% (95% CI: 4 – 24%) required minor editing, excluding scientific name formatting, and none required major editing.
Compared to an experienced educator in an ideal condition (38.97 seconds/stem) this in-house stem generator significantly reduced the time to create stems (20.49 seconds/stem, p < 0.0001). Among the 50 randomly generated stems, only 12% (95% CI: 4 – 24%) required minor editing, excluding scientific name formatting, and none required major editing.
Conclusion:
This exam stem generator is a simple and useful tool to reduce the educator's workload in preparing exam items. Advanced users can also easily expand on this prototype to include more diseases and specific conditions to fit the need of different courses.
This exam stem generator is a simple and useful tool to reduce the educator's workload in preparing exam items. Advanced users can also easily expand on this prototype to include more diseases and specific conditions to fit the need of different courses.
Take-home message:
An Excel-based program can be designed to help reduce the educator's workload as well as standardise the workflow.
An Excel-based program can be designed to help reduce the educator's workload as well as standardise the workflow.
References (maximum three)
1 Medical Competency Assessment Criteria for National License 2012: The Medical Council of Thailand; 2012 [Retrieved from: http://www.tmc.or.th/download/medical2555.pdf].
2. Hays RB, Coventry P, Wilcock D, Hartley K. Short and long multiple-choice question stems in a primary care oriented undergraduate medical curriculum. Educ Prim Care. 2009 May;20(3):173-7.
11:45 am
Veronica Vele1
Mark Gierl2
1 Australian Dental Council
2 University of Alberta
Mark Gierl2
1 Australian Dental Council
2 University of Alberta
Background:
Building high-quality item banks is becoming increasingly difficult, given the constant need for new content. In the health sciences, this challenge is compounded by subject- matter experts (SMEs) having to juggle item development with teaching and clinical practice. Automated item generation (AIG) was piloted as a solution to increase the efficiency of item pool development for a dentistry licensing exam.
Building high-quality item banks is becoming increasingly difficult, given the constant need for new content. In the health sciences, this challenge is compounded by subject- matter experts (SMEs) having to juggle item development with teaching and clinical practice. Automated item generation (AIG) was piloted as a solution to increase the efficiency of item pool development for a dentistry licensing exam.
Summary of work:
A small group of SMEs received training in item modelling and learnt to create cognitive models. SMEs worked individually and in pairs to produce content. As they became familiar with the process, they developed increasingly complex cognitive models using images (e.g., radiographs) and writing testlets for clinical scenarios.
A small group of SMEs received training in item modelling and learnt to create cognitive models. SMEs worked individually and in pairs to produce content. As they became familiar with the process, they developed increasingly complex cognitive models using images (e.g., radiographs) and writing testlets for clinical scenarios.
Results:
By the end of the pilot, SMEs created 3 clinical scenarios with 12 distinct models, resulting in thousands of new items and testlet combinations. A sample of the generated items was reviewed by independent SMEs. These aligned with the examination blueprint and were found to be comparable to items created using traditional methods.
By the end of the pilot, SMEs created 3 clinical scenarios with 12 distinct models, resulting in thousands of new items and testlet combinations. A sample of the generated items was reviewed by independent SMEs. These aligned with the examination blueprint and were found to be comparable to items created using traditional methods.
Discussion:
The power of AIG lies in the ability to produce large numbers of items with minimal effort. However, changes to the item development process take time. SMEs’ experiences of what does and doesn't work shift and evolve as they revisit traditional methods and embrace new ways of content creation.
The power of AIG lies in the ability to produce large numbers of items with minimal effort. However, changes to the item development process take time. SMEs’ experiences of what does and doesn't work shift and evolve as they revisit traditional methods and embrace new ways of content creation.
Conclusions:
The pilot demonstrated that to gain the acceptance of the dental profession, items generated via AIG need to meet the same standards as those produced through traditional item development processes, and SMEs need to be involved and onboarded early, especially given the high-stakes nature of licensing exams.
The pilot demonstrated that to gain the acceptance of the dental profession, items generated via AIG need to meet the same standards as those produced through traditional item development processes, and SMEs need to be involved and onboarded early, especially given the high-stakes nature of licensing exams.
Take-home messages:
AIG can be used to create high-quality items using the suggestions and expertise of SMEs. The training and on-going support of SMEs is key to the success of AIG as an alternative item development method.
AIG can be used to create high-quality items using the suggestions and expertise of SMEs. The training and on-going support of SMEs is key to the success of AIG as an alternative item development method.
References (maximum three)
Gierl, M. J., Lai, H., & Tanygin, V. (2021). Advanced methods in automatic item generation. New York: Routledge.
Gierl, M. J., Swygert, K., Matovinovic, D., Kulesher, A., & Lai, H. (2022). Three sources of validation evidence needed to evaluate the quality of generated test items for medical licensure. Teaching and Learning in Medicine. https://doi.org/10.1080/ 10401334.2022.2119569.
Lai, H., Gierl, M. J., Byrne, B. E., Spielman, A., & Waldschmidt, D. (2016). Three modeling applications to promote automatic item generation for examinations in dentistry. Journal of Dental Education, 80, 339-347.
12:00 pm
Megan Kirwan1
Bryan Burford1, Gabrielle Finn2 and Gillian Vance1
1 Newcastle University
2 University of Manchester
Bryan Burford1, Gabrielle Finn2 and Gillian Vance1
1 Newcastle University
2 University of Manchester
Background:
The Covid-19 pandemic caused significant disruptions to postgraduate medical education across the world [1]. In the UK, this prompted introduction of a set of national assessment derogations, which were agreed by the General Medical Council (GMC), Statutory Education Bodies (SEBs) and the Academy of Medical Royal Colleges (AoMRC) [2]. These changes related to specialty examinations, Annual Review of Competency Panel (ARCP) processes, and ARCP outcomes. However, it is unclear what impact these changes had across specialties, and across stakeholder groups, and what learning we can gain from the changes [3]. This symposium explores the (ongoing) impact of these changes, drawing on large-scale, national, mixed-methods research carried out in the UK on behalf of the GMC. Further, it will stimulate discussion amongst attendees about their experiences of changes to the assessment framework, to identify wider learning that may support future responses to a force majeure.
The Covid-19 pandemic caused significant disruptions to postgraduate medical education across the world [1]. In the UK, this prompted introduction of a set of national assessment derogations, which were agreed by the General Medical Council (GMC), Statutory Education Bodies (SEBs) and the Academy of Medical Royal Colleges (AoMRC) [2]. These changes related to specialty examinations, Annual Review of Competency Panel (ARCP) processes, and ARCP outcomes. However, it is unclear what impact these changes had across specialties, and across stakeholder groups, and what learning we can gain from the changes [3]. This symposium explores the (ongoing) impact of these changes, drawing on large-scale, national, mixed-methods research carried out in the UK on behalf of the GMC. Further, it will stimulate discussion amongst attendees about their experiences of changes to the assessment framework, to identify wider learning that may support future responses to a force majeure.
Why is the topic important?:
Understanding the impact of assessment derogations is crucial for medical educators and policymakers as medical education systems continue to recover from the pandemic. This learning may inform responses to similar exogenous events. The symposium aims to foster collaborative discussions and inspire evidence-based strategies to build resilience and adaptability in medical education systems globally.
Understanding the impact of assessment derogations is crucial for medical educators and policymakers as medical education systems continue to recover from the pandemic. This learning may inform responses to similar exogenous events. The symposium aims to foster collaborative discussions and inspire evidence-based strategies to build resilience and adaptability in medical education systems globally.
Symposium format, including participant engagement methods:
The symposium panellists will present an overview of the research findings, including a rapid review of published evidence in this area, secondary analysis of relevant assessment data (e.g., ARCP outcomes, professional exams), and large-scale qualitative data. There will then be a panel discussion, featuring representation from various stakeholder groups and their respective experiences of the assessment derogations. They will share their experiences and learning in relation to different types of assessment derogations, and thoughts regarding ongoing impact and research. Attendees will have the opportunity to participate in the symposium through live polls, an interactive question and answer session with the panel, and through engagement on social media.
The symposium panellists will present an overview of the research findings, including a rapid review of published evidence in this area, secondary analysis of relevant assessment data (e.g., ARCP outcomes, professional exams), and large-scale qualitative data. There will then be a panel discussion, featuring representation from various stakeholder groups and their respective experiences of the assessment derogations. They will share their experiences and learning in relation to different types of assessment derogations, and thoughts regarding ongoing impact and research. Attendees will have the opportunity to participate in the symposium through live polls, an interactive question and answer session with the panel, and through engagement on social media.
Take home messages/outcomes/implications:
The symposium will share key take home messages relating to UK assessment derogations and stimulate discussion regarding wider applicability of this learning. This will include discussion regarding the importance of 3C’s – compassion, consistency, and communication – in relation to assessment derogations. A discussion regarding gaps in data will be facilitated, including the importance of further work exploring impact on minoritized groups; and the importance of acknowledging the personal impact on trainers supporting assessments during Covid-19. The relevance of the key messages extend beyond the UK and the symposium will offer valuable insights for an international audience who are dealing with similar ongoing challenges in postgraduate medical education. The collaborative discussion will add to our evolving understanding, and ongoing work in this area.
The symposium will share key take home messages relating to UK assessment derogations and stimulate discussion regarding wider applicability of this learning. This will include discussion regarding the importance of 3C’s – compassion, consistency, and communication – in relation to assessment derogations. A discussion regarding gaps in data will be facilitated, including the importance of further work exploring impact on minoritized groups; and the importance of acknowledging the personal impact on trainers supporting assessments during Covid-19. The relevance of the key messages extend beyond the UK and the symposium will offer valuable insights for an international audience who are dealing with similar ongoing challenges in postgraduate medical education. The collaborative discussion will add to our evolving understanding, and ongoing work in this area.
References (maximum three)
1: Papapanou M, Routsi E, Tsamakis K, Fotis L, Marinos G, Lidoriki I, Karamanou M, Papaioannou TG, Tsiptsios D, Smyrnis N, Rizos E. Medical education challenges and innovations during COVID-19 pandemic. Postgraduate medical journal. 2022 May;98(1159):321-7
2: General Medical Council et al. Joint statement on temporary derogations in medical education and training. Updated September 2021. Available online: https://www.jcst.org/- /media/files/jcst/key-documents/joint-statement-on-temporary-derogations--september- 2021.pdf Last accessed 8/8/23.
3: Ahmed AE. The Impact of COVID-19 on Postgraduate Training of Doctors in the United Kingdom: A Narrative Review. Cureus. 2022 Dec 31;14(12).
12:15 pm
Denisse Zuniga1
Carlos Gonzalez1
1 Pontificia Universidad Catolica de Chile
Carlos Gonzalez1
1 Pontificia Universidad Catolica de Chile
Background.
The efforts of medical schools to maintain the continuity of undergraduate education during the health crisis are well known (1). However, it isn't clear how online education affected the teaching and evaluation in courses with a practical component (2,3). Studies are needed to examine and evaluate the projection of change in the teaching. The study aims to analyze the strategies and conceptions of medical teachers to face the teaching and evaluation of the practical component in the virtual modality.
The efforts of medical schools to maintain the continuity of undergraduate education during the health crisis are well known (1). However, it isn't clear how online education affected the teaching and evaluation in courses with a practical component (2,3). Studies are needed to examine and evaluate the projection of change in the teaching. The study aims to analyze the strategies and conceptions of medical teachers to face the teaching and evaluation of the practical component in the virtual modality.
Summary of work.
A mixed research design is proposed. Medical teachers of clinical courses from 6 Chilean universities participated. Individual interviews and observations (COPUS) of class recordings of 12 teachers were conducted, and 129 participants completed a technology use and acceptance questionnaire (UTAUT-2). Qualitative data was analyzed through thematic analysis; psychometric properties and inferential statistics to analyze quantitative data.
A mixed research design is proposed. Medical teachers of clinical courses from 6 Chilean universities participated. Individual interviews and observations (COPUS) of class recordings of 12 teachers were conducted, and 129 participants completed a technology use and acceptance questionnaire (UTAUT-2). Qualitative data was analyzed through thematic analysis; psychometric properties and inferential statistics to analyze quantitative data.
Results.
Between 2020 and 2022, courses adjusted the contents and incorporated synchronous and asynchronous teaching-learning and evaluation strategies. In virtual classrooms (VC), teachers transmit information and mediate between knowledge and its application. In the evaluation, synchronous MCQ with VC support predominated. The VC was reliable for the teachers, who became accustomed to its use; however, its contribution to learning the practical component is debatable. Only half of the teachers received training in digital competencies.
Between 2020 and 2022, courses adjusted the contents and incorporated synchronous and asynchronous teaching-learning and evaluation strategies. In virtual classrooms (VC), teachers transmit information and mediate between knowledge and its application. In the evaluation, synchronous MCQ with VC support predominated. The VC was reliable for the teachers, who became accustomed to its use; however, its contribution to learning the practical component is debatable. Only half of the teachers received training in digital competencies.
Discussion.
Teachers could face and adapt to online modality. Virtuality did not seem to impact teachers' digital competencies or the use of technology in the evaluation in the same way.
Teachers could face and adapt to online modality. Virtuality did not seem to impact teachers' digital competencies or the use of technology in the evaluation in the same way.
Conclusion.
Educational methods were adapted to the virtual modality. However, the continuity of the VC in clinical courses is unclear.
Educational methods were adapted to the virtual modality. However, the continuity of the VC in clinical courses is unclear.
Home-take messages.
Teachers adapted their educational and evaluation strategies to online modality. The evaluation was challenging and less satisfactory. It will be essential to strengthen teachers' digital competencies and increase institutional support in times of artificial intelligence.
Teachers adapted their educational and evaluation strategies to online modality. The evaluation was challenging and less satisfactory. It will be essential to strengthen teachers' digital competencies and increase institutional support in times of artificial intelligence.
References (maximum three)
Stewart W. A global crash course in teaching and learning online: A thematic review of empirical Emergency Remote Teaching (ERT) studies in higher education during Year 1 of COVID-19. Open Praxis 2021; 13 (1): 89-102.
Ten Cate O, Schultz K, Frank J, Hennus M, Ross S, Schumacher D, Snell L, Whelan A, Youngon J. Questioning medical competence: Should the Covid-19 crisis affect the goals of medical education? Medical Teacher 2021, 43 (7): 817-23.
Abreu-Hernández LF, León-Bórquez R, García-Gutiérrez JF. Pandemia de Covid-19 y educación médica en Latinoamérica. Fundación Educación Médica 2020, 23(5): 237-42.