ePoster
100% Page: /
Presentation Description
Rungroj Angwatcharaprakan1
Chaowaphon Ittiphanitphong1
1 Sawanpracharak Hospital
Chaowaphon Ittiphanitphong1
1 Sawanpracharak Hospital
Background:
In medical education, assessing students' knowledge application traditionally relies on methods like Multiple-Choice Questions (MCQs). The emergence of Large Language Models (LLMs), specifically ChatGPT, can enhance medical educators' ability to formulate higher-volume of quality MCQs. However, concerns exist about LLM capabilities, especially data accuracy and credibility. Validation by medical experts is crucial for data integrity. This research proposes an approach that combines medical educators' expertise with LLM to enhance MCQ-based assessments.
In medical education, assessing students' knowledge application traditionally relies on methods like Multiple-Choice Questions (MCQs). The emergence of Large Language Models (LLMs), specifically ChatGPT, can enhance medical educators' ability to formulate higher-volume of quality MCQs. However, concerns exist about LLM capabilities, especially data accuracy and credibility. Validation by medical experts is crucial for data integrity. This research proposes an approach that combines medical educators' expertise with LLM to enhance MCQ-based assessments.
Summary of Work:
Refining prompts to communicate with LLM for MCQs creation; MCQs characterized by the application of knowledge style, with coherent key options and distractors in terms of sentence length and category; adaptability of question topics, and accommodating queries for diagnosis, differential diagnosis, further investigations, or management; cross- sectional study, gathering and analyzing clinical-level medical students' responses to forming the basis of a model for assessment enhancement.
Refining prompts to communicate with LLM for MCQs creation; MCQs characterized by the application of knowledge style, with coherent key options and distractors in terms of sentence length and category; adaptability of question topics, and accommodating queries for diagnosis, differential diagnosis, further investigations, or management; cross- sectional study, gathering and analyzing clinical-level medical students' responses to forming the basis of a model for assessment enhancement.
Results:
Our pilot study yielded dual outcomes. Firstly, collaborative MCQs creation exhibited medical educator-LMM synergy, generating complex, contextually relevant questions and options. Secondly, student responses highlighted the optimum acceptability index and efficiency of distractors, indicating assessment efficacy enhancement.
Our pilot study yielded dual outcomes. Firstly, collaborative MCQs creation exhibited medical educator-LMM synergy, generating complex, contextually relevant questions and options. Secondly, student responses highlighted the optimum acceptability index and efficiency of distractors, indicating assessment efficacy enhancement.
Discussion:
LLM-generated MCQs displayed notable clinical vignette accuracy, but unclear option detail. Expert refinement improved accuracy, enabling efficient creation and enhancement of question structures. Responses from medical students showed the efficiency of the questions.
LLM-generated MCQs displayed notable clinical vignette accuracy, but unclear option detail. Expert refinement improved accuracy, enabling efficient creation and enhancement of question structures. Responses from medical students showed the efficiency of the questions.
Conclusions:
This research advances MCQ-based medical education assessment. Through medical educator-LMM collaboration, pilot model prompts showcase enhanced MCQs assessing medical knowledge and clinical reasoning. Preliminary findings encourage further implementation.
This research advances MCQ-based medical education assessment. Through medical educator-LMM collaboration, pilot model prompts showcase enhanced MCQs assessing medical knowledge and clinical reasoning. Preliminary findings encourage further implementation.
Take-home Messages:
Collaborative MCQ creation with LLM enhances assessment efficiency, and quality, reflecting clinical accuracy and promoting the application of knowledge in medical education. LLM alone is inadequate for MCQs; educator expertise is vital for accurate and comprehensive question development.
Collaborative MCQ creation with LLM enhances assessment efficiency, and quality, reflecting clinical accuracy and promoting the application of knowledge in medical education. LLM alone is inadequate for MCQs; educator expertise is vital for accurate and comprehensive question development.
References (maximum three)
Campbell DE. How to write good multiple-choice questions. J Paediatr Child Health. 2011 Jun;47(6):322-5. doi: 10.1111/j.1440-1754.2011.02115.x. Epub 2011 May 25. PMID: 21615597.
Fozzard, N., Pearson, A., du Toit, E. et al. Analysis of MCQ and distractor use in a large first year Health Faculty Foundation Program: assessing the effects of changing from five to four options. BMC Med Educ 18, 252 (2018). https://doi.org/10.1186/s12909-018-1346-4
Eysenbach G. The Role of ChatGPT, Generative Language Models, and Artificial Intelligence in Medical Education: A Conversation With ChatGPT and a Call for Papers. JMIR Med Educ 2023;9:e46885. URL: https://mededu.jmir.org/2023/1/e46885. DOI: 10.2196/46885