Presentation Description
Nora AL-Shawee1
Daire Lee1, Ciaran Reinhardt1, Shohdan Osman1, Emma Farrell1, Vanessa Farnan1, Mohamed Elhassan1, Suzanne Roche1 and Muirne Spooner1,
1 RCSI
Daire Lee1, Ciaran Reinhardt1, Shohdan Osman1, Emma Farrell1, Vanessa Farnan1, Mohamed Elhassan1, Suzanne Roche1 and Muirne Spooner1,
1 RCSI
Background:
Writing high-quality Multiple Choice Questions (MCQs) is time and resource- intensive, abergaing 1.5 hours per item (1). We studied the feasibility of ChatGPT in item- writing (IW).
Writing high-quality Multiple Choice Questions (MCQs) is time and resource- intensive, abergaing 1.5 hours per item (1). We studied the feasibility of ChatGPT in item- writing (IW).
Summary of work:
This is a three-part study. Part 1: Item-writers (IW) instructed ChatGPT in the generation of 120 NBME-style MCQs for a final-year medicine (FYM) exam, after which they participated in an evaluation of the process. FYM will participate in the examination of traditional IW items and AI items in October. Part 2 will compare psychometrics on IW and AI items. Part 3 is student evaluation of the items.
This is a three-part study. Part 1: Item-writers (IW) instructed ChatGPT in the generation of 120 NBME-style MCQs for a final-year medicine (FYM) exam, after which they participated in an evaluation of the process. FYM will participate in the examination of traditional IW items and AI items in October. Part 2 will compare psychometrics on IW and AI items. Part 3 is student evaluation of the items.
Results:
AI-generated items took an average of 13 minutes to create, from initial instruction to ChatGPT to completing edits. Items aligned with learning outcomes in 70% of cases. Editing was required in the vignette (70%), distractors (0.11%) and correct answers (0.3%). Distractors and correct answer required fewer edits ( 0.11 % and 0.3% respectively). Familiarity with the softwarewas required to use it. Ai-generated items were unable to add clinical images. absence of visual aids such as images and the potential lack of complexity in item tailored to FYM.
AI-generated items took an average of 13 minutes to create, from initial instruction to ChatGPT to completing edits. Items aligned with learning outcomes in 70% of cases. Editing was required in the vignette (70%), distractors (0.11%) and correct answers (0.3%). Distractors and correct answer required fewer edits ( 0.11 % and 0.3% respectively). Familiarity with the softwarewas required to use it. Ai-generated items were unable to add clinical images. absence of visual aids such as images and the potential lack of complexity in item tailored to FYM.
Conclusion:
AI expedites IW time and creates good quality items.
AI expedites IW time and creates good quality items.
Discussion:
AI can alleviate IW barriers such as time constraints and challenges in developing a creative start (2)..
AI can alleviate IW barriers such as time constraints and challenges in developing a creative start (2)..
Take-home messages / implications for further research or practice:
AI may optimise efficient item-writing, but requires some faculty development to potentiate its use.
AI may optimise efficient item-writing, but requires some faculty development to potentiate its use.
References (maximum three)
1. CASE, SUSAN M.; HOLTZMAN, KATHY; RIPKEY, DOUGLAS R.. Developing an Item Pool for CBT: A Practical Comparison of Three Models of Item Writing. Academic Medicine 76(10):p S111-S113, October 2001.
2. Karthikeyan S, O'Connor E, Hu W. Barriers and facilitators to writing quality items for medical school assessments - a scoping review. BMC Med Educ. 2019 May 2;19(1):123. doi: 10.1186/s12909-019-1544-8. PMID: 31046744; PMCID: PMC6498649