Presentation Description
Elizabeth Kachur1
Simran Shamith2, Carolyn Giordano2, Beverly Crawford3, Camille Lynch2, Natasha Reddy2, Indranil Chakrabarti2, Tiffany Davis2 and Dennis Novack2
1 Medical Education Development, Global Consulting
2 Drexel University College of Medicine
3 University of Pennsylvania
Simran Shamith2, Carolyn Giordano2, Beverly Crawford3, Camille Lynch2, Natasha Reddy2, Indranil Chakrabarti2, Tiffany Davis2 and Dennis Novack2
1 Medical Education Development, Global Consulting
2 Drexel University College of Medicine
3 University of Pennsylvania
Background
Validity is one of the essential criteria that assures that the instrument measures what it is supposed to measure. Typically, validation includes literature searches, interviews, focus groups and expert reviews. However, with the advent of ChatGPT we have gained a new tool.
Validity is one of the essential criteria that assures that the instrument measures what it is supposed to measure. Typically, validation includes literature searches, interviews, focus groups and expert reviews. However, with the advent of ChatGPT we have gained a new tool.
Summary of Work
A project related to anti-racism training in health professions education resulted in two instruments: 1) An anti-racism learning environment scale which was closely fashioned after the Johns Hopkins Learning Environment Scale (JHLES) to later also become a subscale. 2) A reflection and attitude survey that is linked to a newly developed Allyship OSCE station. Both survey drafts were analyzed multiple times with ChatGPT, Versions 3.5 (free) and 4.0 (paid), using the same prompts. The results of the first survey were also compared with a subsequent student focus group.
Results
A prompt-by-prompt comparison between the different ChatGPT administrations showed significant content similarities. However, the way the answers were structured differed. Each time the results appeared on the screen almost instantly. While the ChatGPT provided many more detailed analyses and recommendations, the student focus group offered more information about context-specific language and meanings.
Discussion
This technology can save much time and manpower to make an instrument ready for use. However, at some point it will be important to consult with content experts and target groups to address local terminology interpretations.
Conclusion
While it should not be the only method used for validating and editing a survey, ChatGPT will save time and human resources while making unique contributions to the instrument development
Take-home messages
- ChatGPT is a worthwhile tool for medical educators, researchers and program evaluators.
- Speed and resource savings are definite benefits, the limited understanding of local circumstances and meanings can be a drawback.
- Strategically applied ChatGPT will complement other forms of survey development.
References (maximum three)
- ChatGPT. https://chat.openai.com/ (accessed 8/10/2023)
- Shochet RB, Colbert-Getz JM, Wright SM. The Johns Hopkins learning environment scale: measuring medical students' perceptions of the processes supporting professional formation. Acad Med. 2015 Jun;90(6):810-8. doi: 10.1097/ACM.0000000000000706. PMID: 25853689.
- Wiredu, John & Kumi, Moses & Ademola, Popoola & Museshaiyela, Percy. (2023). An investigation on the characteristics, abilities, constraints, and functions of artificial intelligence (ai): the age of ChatGPT as an essential ultramodern support tool. 13. 62614-62620. 10.37118/ijdr.26689.05.2023.