Skip to main content
Ottawa 2024
Times are shown in your local time zone GMT

Enhancing Toward Optimum MCQ-Based Assessment: Integrating Medical Educator and Large Language Model (LLM) - Quantitative Preliminary Data Investigation and Analysis

E Poster Presentation
Edit Your Submission
Edit

ePoster Presentation

11:00 am

26 February 2024

Exhibition Hall (Poster 1)

Technology and AI in assessment

ePoster

100% Page:   /  

Presentation Description

Rungroj Angwatcharaprakan1
Chaowaphon Ittiphanitphong1
1 Sawanpracharak Hospital



Background:
In medical education, assessing students' knowledge application traditionally relies on methods like Multiple-Choice Questions (MCQs). The emergence of Large Language Models (LLMs), specifically ChatGPT, can enhance medical educators' ability to formulate higher-volume of quality MCQs. However, concerns exist about LLM capabilities, especially data accuracy and credibility. Validation by medical experts is crucial for data integrity. This research proposes an approach that combines medical educators' expertise with LLM to enhance MCQ-based assessments. 


Summary of Work:
Refining prompts to communicate with LLM for MCQs creation; MCQs characterized by the application of knowledge style, with coherent key options and distractors in terms of sentence length and category; adaptability of question topics, and accommodating queries for diagnosis, differential diagnosis, further investigations, or management; cross- sectional study, gathering and analyzing clinical-level medical students' responses to forming the basis of a model for assessment enhancement. 


Results:
Our pilot study yielded dual outcomes. Firstly, collaborative MCQs creation exhibited medical educator-LMM synergy, generating complex, contextually relevant questions and options. Secondly, student responses highlighted the optimum acceptability index and efficiency of distractors, indicating assessment efficacy enhancement. 


Discussion:
LLM-generated MCQs displayed notable clinical vignette accuracy, but unclear option detail. Expert refinement improved accuracy, enabling efficient creation and enhancement of question structures. Responses from medical students showed the efficiency of the questions. 


Conclusions:
This research advances MCQ-based medical education assessment. Through medical educator-LMM collaboration, pilot model prompts showcase enhanced MCQs assessing medical knowledge and clinical reasoning. Preliminary findings encourage further implementation. 


Take-home Messages:
Collaborative MCQ creation with LLM enhances assessment efficiency, and quality, reflecting clinical accuracy and promoting the application of knowledge in medical education. LLM alone is inadequate for MCQs; educator expertise is vital for accurate and comprehensive question development. 



References (maximum three) 

Campbell DE. How to write good multiple-choice questions. J Paediatr Child Health. 2011 Jun;47(6):322-5. doi: 10.1111/j.1440-1754.2011.02115.x. Epub 2011 May 25. PMID: 21615597. 

Fozzard, N., Pearson, A., du Toit, E. et al. Analysis of MCQ and distractor use in a large first year Health Faculty Foundation Program: assessing the effects of changing from five to four options. BMC Med Educ 18, 252 (2018). https://doi.org/10.1186/s12909-018-1346-4 

Eysenbach G. The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers. JMIR Med Educ 2023;9:e46885. URL: https://mededu.jmir.org/2023/1/e46885. DOI: 10.2196/46885 

Speakers