KWANG CHIEN YEE¹
Ming Chao Wong¹, Saleem Ameen1 and Scott Wylie¹ 
1 University of Tasmania 

Generative artificial intelligence (AI) such as ChatGPT, using the large language model, is a disruptive technology that is affecting healthcare, especially medical education dramatically. Traditional medical school assessments such as essays and open book tests may no longer be useful, particularly given the success ChatGPT has demonstrated in answering medical examination questions, most recently achieving a high pass rate on the American Surgical Board examination. Furthermore, reflective essays can be generated by ChatGPT in full or in part, which often escape detection by anti-plagiarism software. 

A simplistic view would be to ban the use of AI completely in medical education assessment and attempt to use newer AI detection technologies. This will not, however, equip our students with the skills and knowledge required by the workplace, as AI is being integrated into clinical practice. While workplace assessments can reduce the impact of AI, they are expensive, time- consuming, and are introduced too late in the education process. 

We tested the performance of ChatGPT and have developed potential models that can be used in assessment, which incorporates the use of generative AI, while assessing the performance and competency of students. In this presentation, we will showcase some of our ChatGPT testings, and present our models for assessment. We propose that models for assessment need to take into consideration the process of delivering the output, rather than assessing the output exclusively itself. 

Given the potential pitfalls to using generative AI tools such as ChatGPT in healthcare, such as its capacity to generate erroneous text, we strongly believe that healthcare educators and students need to acquire knowledge about the basic science behind AI in order to appropriately utilise its potential. We have developed a preliminary framework for designing assessment in medical education for the AI era which will be presented for discussion. 

References (maximum three) 

Ameen, S, Wong MC, Yee KC. ChatGPT and the Era of Large Language Models in Healthcare – Is the Healthcare System Ready? in Print 

Abdulwhhab Abu Alamrain, Mary Adewunmi, Mahmoud Abu Al Amrain, Ming Chao Wong, Kwang Chien Yee. Exploring ChatGPT's potential in the context of colon cancer patient education: A comparative analysis. Journal of Medical and internet Research, 2023, http://doi.org/10.2196/preprints.51444