Syed Latifi¹
Mark Healy¹
1 Weill Cornell Medicine-Qatar

Recent advances in artificial intelligence (AI) have opened new avenues for educators. One such advancement is Generative AI. This is an emerging field which utilizes algorithms to generate content of varying formats (i.e., text, image, audio, video). Such technologies have the potential to support educators with various tasks, such as lesson planning, content creation, assessment generation and blueprinting, lecture review summaries, etc. 

One such branch of generative AI, large language models (LLMs), for example, can benefit both students and educators. Students can use LLMs to summarize lengthy texts, in the form of conversational dialogue (akin to a virtual study partner), while educators can use it as a tool to generate draft assessments. Automation (or partial automation) of some tasks, such as those outlined above, in learning and teaching are important because they carve out more time for educators to refine and enhance the students’ learning experience [1,2]. 

Session outline: 

This presentation is meant to present, discuss, and co-learn the challenges and opportunities of using Generative AI for medical education. The session will provide a grounding for educators on the capabilities, opportunities and challenges presented by Generative AI within medical education. Educators will be able to make more informed decisions on possible use-cases for Generative AI in the instructional design, assessments, and administrative tasks [2,3]. They will also be more cognizant of how to guide students to use such technologies effectively and responsibly. 

Who should attend: 
Anyone with an interest in the application of Generative AI in medical education. 

References (maximum three) 

1. Cooper, A., & Rodman, A. (2023). AI and Medical Education-A 21st-Century Pandora's Box. The New England journal of medicine. 

Nora AL-Shawee¹
Daire Lee¹, Ciaran Reinhardt¹, Shohdan Osman¹, Emma Farrell¹, Vanessa Farnan¹, Mohamed Elhassan¹, Suzanne Roche¹ and Muirne Spooner¹,
1 RCSI

Background:
Writing high-quality Multiple Choice Questions (MCQs) is time and resource- intensive, abergaing 1.5 hours per item (1). We studied the feasibility of ChatGPT in item- writing (IW). 

Summary of work:
This is a three-part study. Part 1: Item-writers (IW) instructed ChatGPT in the generation of 120 NBME-style MCQs for a final-year medicine (FYM) exam, after which they participated in an evaluation of the process. FYM will participate in the examination of traditional IW items and AI items in October. Part 2 will compare psychometrics on IW and AI items. Part 3 is student evaluation of the items. 

Results:
AI-generated items took an average of 13 minutes to create, from initial instruction to ChatGPT to completing edits. Items aligned with learning outcomes in 70% of cases. Editing was required in the vignette (70%), distractors (0.11%) and correct answers (0.3%). Distractors and correct answer required fewer edits ( 0.11 % and 0.3% respectively). Familiarity with the softwarewas required to use it. Ai-generated items were unable to add clinical images. absence of visual aids such as images and the potential lack of complexity in item tailored to FYM. 

Conclusion:
AI expedites IW time and creates good quality items. 

Discussion:
AI can alleviate IW barriers such as time constraints and challenges in developing a creative start (2).. 

Take-home messages / implications for further research or practice:
AI may optimise efficient item-writing, but requires some faculty development to potentiate its use. 

References (maximum three) 

1. CASE, SUSAN M.; HOLTZMAN, KATHY; RIPKEY, DOUGLAS R.. Developing an Item Pool for CBT: A Practical Comparison of Three Models of Item Writing. Academic Medicine 76(10):p S111-S113, October 2001. 

2. Karthikeyan S, O'Connor E, Hu W. Barriers and facilitators to writing quality items for medical school assessments - a scoping review. BMC Med Educ. 2019 May 2;19(1):123. doi: 10.1186/s12909-019-1544-8. PMID: 31046744; PMCID: PMC6498649 

Elizabeth Kachur¹
Simran Shamith², Carolyn Giordano2, Beverly Crawford³, Camille Lynch², Natasha Reddy², Indranil Chakrabarti², Tiffany Davis² and Dennis Novack²
1 Medical Education Development, Global Consulting
2 Drexel University College of Medicine
3 University of Pennsylvania

Background
Validity is one of the essential criteria that assures that the instrument measures what it is supposed to measure. Typically, validation includes literature searches, interviews, focus groups and expert reviews. However, with the advent of ChatGPT we have gained a new tool. 

Summary of Work
A project related to anti-racism training in health professions education resulted in two instruments: 1) An anti-racism learning environment scale which was closely fashioned after the Johns Hopkins Learning Environment Scale (JHLES) to later also become a subscale. 2) A reflection and attitude survey that is linked to a newly developed Allyship OSCE station. Both survey drafts were analyzed multiple times with ChatGPT, Versions 3.5 (free) and 4.0 (paid), using the same prompts. The results of the first survey were also compared with a subsequent student focus group. 

Results
A prompt-by-prompt comparison between the different ChatGPT administrations showed significant content similarities. However, the way the answers were structured differed. Each time the results appeared on the screen almost instantly. While the ChatGPT provided many more detailed analyses and recommendations, the student focus group offered more information about context-specific language and meanings. 

Discussion
This technology can save much time and manpower to make an instrument ready for use. However, at some point it will be important to consult with content experts and target groups to address local terminology interpretations. 

Conclusion
While it should not be the only method used for validating and editing a survey, ChatGPT will save time and human resources while making unique contributions to the instrument development 

Take-home messages

References (maximum three)