Presentation Description
Veronica Vele1
Mark Gierl2
1 Australian Dental Council
2 University of Alberta
Mark Gierl2
1 Australian Dental Council
2 University of Alberta
Background:
Building high-quality item banks is becoming increasingly difficult, given the constant need for new content. In the health sciences, this challenge is compounded by subject- matter experts (SMEs) having to juggle item development with teaching and clinical practice. Automated item generation (AIG) was piloted as a solution to increase the efficiency of item pool development for a dentistry licensing exam.
Building high-quality item banks is becoming increasingly difficult, given the constant need for new content. In the health sciences, this challenge is compounded by subject- matter experts (SMEs) having to juggle item development with teaching and clinical practice. Automated item generation (AIG) was piloted as a solution to increase the efficiency of item pool development for a dentistry licensing exam.
Summary of work:
A small group of SMEs received training in item modelling and learnt to create cognitive models. SMEs worked individually and in pairs to produce content. As they became familiar with the process, they developed increasingly complex cognitive models using images (e.g., radiographs) and writing testlets for clinical scenarios.
A small group of SMEs received training in item modelling and learnt to create cognitive models. SMEs worked individually and in pairs to produce content. As they became familiar with the process, they developed increasingly complex cognitive models using images (e.g., radiographs) and writing testlets for clinical scenarios.
Results:
By the end of the pilot, SMEs created 3 clinical scenarios with 12 distinct models, resulting in thousands of new items and testlet combinations. A sample of the generated items was reviewed by independent SMEs. These aligned with the examination blueprint and were found to be comparable to items created using traditional methods.
By the end of the pilot, SMEs created 3 clinical scenarios with 12 distinct models, resulting in thousands of new items and testlet combinations. A sample of the generated items was reviewed by independent SMEs. These aligned with the examination blueprint and were found to be comparable to items created using traditional methods.
Discussion:
The power of AIG lies in the ability to produce large numbers of items with minimal effort. However, changes to the item development process take time. SMEs’ experiences of what does and doesn't work shift and evolve as they revisit traditional methods and embrace new ways of content creation.
The power of AIG lies in the ability to produce large numbers of items with minimal effort. However, changes to the item development process take time. SMEs’ experiences of what does and doesn't work shift and evolve as they revisit traditional methods and embrace new ways of content creation.
Conclusions:
The pilot demonstrated that to gain the acceptance of the dental profession, items generated via AIG need to meet the same standards as those produced through traditional item development processes, and SMEs need to be involved and onboarded early, especially given the high-stakes nature of licensing exams.
The pilot demonstrated that to gain the acceptance of the dental profession, items generated via AIG need to meet the same standards as those produced through traditional item development processes, and SMEs need to be involved and onboarded early, especially given the high-stakes nature of licensing exams.
Take-home messages:
AIG can be used to create high-quality items using the suggestions and expertise of SMEs. The training and on-going support of SMEs is key to the success of AIG as an alternative item development method.
AIG can be used to create high-quality items using the suggestions and expertise of SMEs. The training and on-going support of SMEs is key to the success of AIG as an alternative item development method.
References (maximum three)
Gierl, M. J., Lai, H., & Tanygin, V. (2021). Advanced methods in automatic item generation. New York: Routledge.
Gierl, M. J., Swygert, K., Matovinovic, D., Kulesher, A., & Lai, H. (2022). Three sources of validation evidence needed to evaluate the quality of generated test items for medical licensure. Teaching and Learning in Medicine. https://doi.org/10.1080/ 10401334.2022.2119569.
Lai, H., Gierl, M. J., Byrne, B. E., Spielman, A., & Waldschmidt, D. (2016). Three modeling applications to promote automatic item generation for examinations in dentistry. Journal of Dental Education, 80, 339-347.