Chaya Prasad¹
Jesus Sanchez¹
1 Western University of Health Sciences 

Abstract 

Medical school curriculum emphasizes knowledge base rather than basic clinical skills. Research has demonstrated gaps between what residency program expectations and what newly graduated physicians can do. Incoming residents struggle with both clinical skills and soft skills. Graduating students report being ill-prepared to fulfill clinical and professional responsibilities. 

We created a residency boot camp (RBC) to identify graduating students’ strengths and weaknesses in clinical skills (e.g. writing prescription orders, admission/discharge/consult notes, handoffs, pages) and soft skills (e.g. wellness, time management, communication skills). Chairs of core specialties determined topics and speakers with input from residency program directors. A mandatory 1-day RBC was offered to graduating students. Faculty and residents offered a hybrid model with pre-recorded videos and live virtual panel discussions. Students voluntarily completed a post RBC survey assessing levels of confidence and competency in basic clinical skills and handling of commonly encountered clinical scenarios. 

186/347 participants completed the survey. Participants agreed that they were comfortable/confident post RBC, with: developing their own clinical questions (81.7%), triaging sick/not sick patients (80.7%), presentation of patients (79.0%), recognizing abnormal vital signs (88.7%), recognizing changes that require immediate attention (81.8%), requesting specialty consultations (78.5%), handling specialty consultations (70.4%), communicating effectively with patients/family (88.2%), incorporating feedback into daily practice (87.1%), clinical documentation (83.9%), patient sign-out (78.0%), managing common clinical scenarios (83.3%), caring for patients in the clinic (79.0%) and hospital (77.4%), when to seek senior help (76.3%), their own resiliency skills (79.6%) and about sources of help when feeling overwhelmed (78.5%). 75.3% of the students wanted to see more resident presentations and 67.2% of the students were satisfied with the pilot. Faculty identified clinical skills that they could help students develop in the 3rd/ 4th years of medical school. 

A mandatory RBC can be effective in assessing and filling in gaps in clinical and soft skills. 

References (maximum three) 

1. https://www.ama-assn.org/medical-students/preparing-residency/pre-residency-boot- camps-prep-med-school-grads-new-realities 

2. https://link.springer.com/article/10.1007/s40670-020-01124-7 

3. https://meridian.allenpress.com/jgme/article/11/2/214/10113/Assessing-Entrustable- Professional-Activities 

Sonia Saluja 

To address medical workforce shortages in regional Australia, the Regional Medical Pathway (RMP) was established as a partnership of two universities and two Hospital and Health service providers, enabling students to complete the entirety of their medical education and training in regional Australia, creating 'home-grown' doctors. Studies indicate that extended periods of rural immersion during medical school are a strong predictor of subsequent rural practice. 1,2 The RMP aims to support regionally based students, allowing them to study, train and practice in their own backyard. A new medical program can be particularly exciting for stakeholders and the local community, however, there is often uncertainty and apprehension from potential applicants. This study examined the perceptions and experiences of the first RMP cohort via focus group interviews. Qualitative analysis of data was informed by published research articles. 3 Students reported diverse motivations for selecting the RMP, such as accessibility to study medicine closer to home, and the benefit of the partnership between the two universities. 44% of the first cohort were from regional or rural Australia. Participants spoke favourably of the new program, perceiving that they had an active voice in shaping the RMP, with the small cohort size facilitating a strong sense of community and greater access to teaching staff. However, they reported challenges relating to regional living, including limited access to public transport and housing availability. 

These findings suggest that accessibility to medical education may widen participation of regional students in studying medicine. Support for students relocating to the regional areas to address challenges they faced was an important consideration for a positive student experience. These findings have implications for universities seeking to establish new medical programs. Inclusive partnerships between organisations may widen participation of regional students in medical programs and improve representation of regional and rural students in medicine. 

References (maximum three) 

3. Burnard, P., Gill, P., Stewart, K., Treasure, E. and Chadwick, B., 2008. Analysing and presenting qualitative data. British dental journal, 204(8), pp.429-432. 

Edward Li¹
Annette Mercer² and Julie Willems²
1 Pearson VUE
2 Monash University

Background:
The UCAT examination was first administered in July 2006 in the United Kingdom and other countries. In 2019 the UCAT exam was used for the UCAT_ANZ examination, which is administered in Australia and New Zealand, as well as other countries, to support applications to undergraduate medical/dental schools in Australia and New Zealand. This study specifically focuses on the cognitive sections of the exam. 

Summary of work:
The cognitive sections were analysed using the Rasch model, with a common-item equating design employed to equate various forms to the base metric. The study examined the exam-level performance, including reliability and dimensionality, and item-level performance, such as item parameter drift and differential item functioning (DIF). 

Results:
The findings revealed that all four cognitive sections displayed high levels of reliability and unidimensionality, indicating their suitability for measuring cognitive abilities. At the item level, parameters remained stable, with minimal DIF observed across gender groups. 

Discussion:
To enhance the inclusivity of the exam, considerable efforts were made to diversify the exam content to encompass a wide range of currencies, names, geographies, and cultures. UCAT_ANZ results indicated a similarity in the functioning of test items and the overall exam compared to UCAT, with ANZ candidate performances demonstrating comparable characteristics to those of UCAT candidates. 

Conclusion:
It suggests that the cognitive sections of the exam measure equivalent constructs across the two testing populations. As a reliable instrument for candidate selection based on defined constructs, the UCAT_ANZ exhibits fairness and consistency in its assessment approach. 

Take-home messages / implications for further research or practice:
While the exam demonstrates equivalence in measuring cognitive abilities, future research should investigate the constructs' relevance in light of emerging technological advancements, such as the growing use of artificial intelligence (AI) in the medical field. 

References (maximum three) 

1. Patterson F, Knight A, Dowell J, Nicholson S, Cousans F, Cleland J. How effective are selection methods in medical education? A systematic review. Medical Education 2016; 50:1: 36 – 60. 

2. https://www.ucat.edu.au/ 

Clare Owen¹
Veronica Davids²
1 MSC
2 Medical Schools Council

Background
The Medical Schools Council (MSC) is working with UK medical schools to introduce a national applied knowledge test (AKT) that will form part of the Medical Licencing Assessment regulated by the General Medical Council (GMC). A governance system has been put in place to oversee the design and development of the assessment. 

Summary of work
Once the committee structure was in place the utilisation of the governance process was piloted in 2021/2022. As a result of this the sequencing of governance groups was adapted. Papers are created by a bespoke algorithm. 

Results 

The change in sequencing will be outlined in the presentation. 

Discussion
One of the biggest challenges identified through the first large-scale pilot of the AKT was ensuring the items in the exam are fit for purpose. The governance system was changed to ensure that as many experts as possible looked at the items before they were standard set. Both the Exam Construction Group and the AKT Exam Board are now allotted significant time to review items.


Conclusions

The item writing process is rigorous and each item is subject to review additional QA provides further insight into the items and the overall structure of individual exams.


Take-home messages / implications for further research or practice
Robust QA of items is essential in the delivery of a national exam as an effective governance and oversight process. Acting at scale requires a different approach to the delivery of exams in a single institution once a year. 

References (maximum three) 

MSC website - https://www.medschools.ac.uk/our-work/medical-licensing- assessment accessed August 2023 

KWANG CHIEN YEE¹
Ming Chao Wong¹, Saleem Ameen1 and Scott Wylie¹ 
1 University of Tasmania 

Generative artificial intelligence (AI) such as ChatGPT, using the large language model, is a disruptive technology that is affecting healthcare, especially medical education dramatically. Traditional medical school assessments such as essays and open book tests may no longer be useful, particularly given the success ChatGPT has demonstrated in answering medical examination questions, most recently achieving a high pass rate on the American Surgical Board examination. Furthermore, reflective essays can be generated by ChatGPT in full or in part, which often escape detection by anti-plagiarism software. 

A simplistic view would be to ban the use of AI completely in medical education assessment and attempt to use newer AI detection technologies. This will not, however, equip our students with the skills and knowledge required by the workplace, as AI is being integrated into clinical practice. While workplace assessments can reduce the impact of AI, they are expensive, time- consuming, and are introduced too late in the education process. 

We tested the performance of ChatGPT and have developed potential models that can be used in assessment, which incorporates the use of generative AI, while assessing the performance and competency of students. In this presentation, we will showcase some of our ChatGPT testings, and present our models for assessment. We propose that models for assessment need to take into consideration the process of delivering the output, rather than assessing the output exclusively itself. 

Given the potential pitfalls to using generative AI tools such as ChatGPT in healthcare, such as its capacity to generate erroneous text, we strongly believe that healthcare educators and students need to acquire knowledge about the basic science behind AI in order to appropriately utilise its potential. We have developed a preliminary framework for designing assessment in medical education for the AI era which will be presented for discussion. 

References (maximum three) 

Ameen, S, Wong MC, Yee KC. ChatGPT and the Era of Large Language Models in Healthcare – Is the Healthcare System Ready? in Print 

Abdulwhhab Abu Alamrain, Mary Adewunmi, Mahmoud Abu Al Amrain, Ming Chao Wong, Kwang Chien Yee. Exploring ChatGPT's potential in the context of colon cancer patient education: A comparative analysis. Journal of Medical and internet Research, 2023, http://doi.org/10.2196/preprints.51444 

Haydeé Parra Acosta¹
José López Loya¹ and Brenda Jazmín Amador Negrete¹
1 Autonomous University of Chihuahua 

Virtual education from a socio-formative approach, applies the evaluation by competencies of students in authentic or simulated scenarios (Parra et al., 2019). It is based on evidence that seeks to improve student performance (Hernández-Mosqueda et al., 2016). The competences are evaluated in action. Therefore, it is important to continuously monitor the progress of the students and support them in the face of difficulties in an assertive way (Tobón, 2013). The evaluation involves four processes: self-evaluation, co-evaluation, hetero-evaluation and meta- evaluation. One of the most widely used evaluation instruments are the rubrics that assess performance through levels of achievement: receptive, decisive, autonomous and strategic. However, despite the changes in medical education due to the health crisis, there is a lack of research on the evaluation of medical skills in virtual education. The objectives of this study are: 1) Identify the elements most valued by students and teachers regarding virtual education. 2) Determine if there are significant differences in the perceptions of students and teachers regarding virtual education. 3) Identify the aspects of virtual education that are related in a significant with in the evaluation by competences. Correlational cross-sectional study where two questionnaires were applied to 190 science and clinical students and 71 science and clinical teachers, which were previously validated with an Alfha of 0.991 and 0.994. In the results, he highlighted that the best value was that teachers favor critical and complex thinking and self- directed study in virtual education; which is significantly related to the evaluation by competences when they use the rubrics since they allow to identify areas of opportunity. Likewise, significant differences were found between teachers and students regarding the use of rubrics in the evaluation of competencies who consider that they are applied more are the teachers. 

References (maximum three) 

Hernández-Mosqueda, J. S., Tobón-Tobón, S., & Guerrero-Rosas, G. (2016). Hacia una evaluación integral del desempeño: las rúbricas socioformativas. Ra Ximhai, 12(6), 359-376. https://www.redalyc.org/pdf/461/46148194025.pdf 

Parra, H., López, J., González, E., Moriel, L., Vázquez, A., Y González, N. (2019). Las tecnologías del aprendizaje y del conocimiento (TAC) y la formación integral y humanista del médico. Investigación en educación médica, 8(31), 72-81. https://doi.org/10.22201/facmed.20075057e.2019.31.18128 

Tobón, S. T. (2013). Formación integral y competencias (Vol. 227). Editorial Macro. Tobón, S.