Terry Judd
Lachlan McOmish¹, Simone Elliott1¹, David Swanson¹,  and Anna Ryan¹,
1 The University of Melbourne 

Background
Assessment item development is a key aspect of designing effective medical and health professional curricula. Multiple-choice questions (MCQs) are commonly used to evaluate students' knowledge and critical thinking abilities. However, crafting high-quality MCQs can be time-consuming and challenging for educators. Generative language models, and ChatGPT in particular, have gained considerable notoriety for their ability to pass MCQ-based exams such as the USMLE (Gilson et. al., 2023), but can also be leveraged to create assessment items (Biswas, 2023; Agarwal, Sharma & Goswami, 2023). This workshop introduces an innovative tool called the 'Item Writer's Workbench' that harnesses and scaffolds the power of ChatGPT to assist in the development assessment items for application in health professional curricula. 

Importance for Research and Practice
Efficiently generating well-constructed assessment items is essential for maintaining the rigor and validity of educational assessments. The 'Item Writer's Workbench' addresses the need for a time-saving solution while ensuring the quality and relevance of MCQs. Leveraging ChatGPT's text generation capabilities, the tool enhances the item writing process by providing a user-friendly interface and a range of customizable templates optimized for various item types. 

This workshop presents an opportunity for health professional educators to explore and utilize the tool and to assess whether it, and the use of ChatGPT more generally, can enhance their item writing capacity and skills. 

Workshop Format 

The workshop will be interactive and hands-on, combining presentations, demonstrations, and practical exercises. Following a short presentation on the fundamentals of item design, and an introduction to the workbench tool, participants will have the opportunity to firstly explore its capabilities, learning how to create detailed and targeted query statements using its customisable templates and then apply their learning to develop draft items relevant to their own assessment contexts. The workshop will conclude with a discussion on best practices for utilising ChatGPT appropriately and effectively in assessment development processes. 

This workshop is designed for educators and curriculum and assessment developers involved in health professional education. The workshop is open to participants of all expertise levels, catering to both novice and experienced item writers. Educators new to assessment item writing will benefit from the tool's guidance and assistance in constructing MCQs, while seasoned practitioners can explore how the tool streamlines their existing workflows and enhances their item development strategies. 

By attending this workshop, participants will: 

· Gain familiarity with the 'Item Writer's Workbench' tool and its features (participants will continue to have access to an online version of the tool beyond the workshop). 

· Understand how to utilise customisable templates to generate detailed query statements for ChatGPT. 

· Experience the practical application of ChatGPT in generating draft assessment items for use in health professional curricula. 

· Explore best practices for integrating ChatGPT into their own item development processes. 

References (maximum three) 

Gilson, A., Safranek, C. W., Huang, T., Socrates, V., Chi, L., Taylor, R. A., & Chartash, D. (2023). How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment.JMIR Medical Education,9(1), e45312. 

Biswas, S. (2023). Passing is great: can ChatGPT conduct USMLE exams?.Annals of Biomedical Engineering, 1-2. 

Agarwal, M., Sharma, P., & Goswami, A. (2023). Analysing the Applicability of ChatGPT, Bard, and Bing to Generate Reasoning-Based Multiple-Choice Questions in Medical Physiology.Cureus,15(6).