ePoster
100% Page: /
Presentation Description
Kent Hecker1
Richard Feinberg2, Fen Fan2, Courtney Vengrin3, Janine Hawley3 and Raja Subhiyah2
1 International Council for Veterinary Assessment/University of Calgary
2 NBME
3 International Council for Veterinary Assessment
Richard Feinberg2, Fen Fan2, Courtney Vengrin3, Janine Hawley3 and Raja Subhiyah2
1 International Council for Veterinary Assessment/University of Calgary
2 NBME
3 International Council for Veterinary Assessment
Background:
In health professions education, progress tests tend to be blueprinted for a specific milestone (e.g., competency at time of graduation) and used primarily for formative purposes to provide reinforcement-based learning within the cognitive domain over time (1). Defining a tests blueprint is a critical validity requirement in ensuring proper alignment between content and the intended construct. Traditionally, a test blueprint is informed by a practice analysis, but this process is often very time consuming, expensive, and includes content that may be out of scope for an assessment composed of multiple-choice questions. To address this problem, the National Board of Medical Examiners proposed a rapid blueprinting process (RBP) as a cost- effective and timely alternative (2,3).
In health professions education, progress tests tend to be blueprinted for a specific milestone (e.g., competency at time of graduation) and used primarily for formative purposes to provide reinforcement-based learning within the cognitive domain over time (1). Defining a tests blueprint is a critical validity requirement in ensuring proper alignment between content and the intended construct. Traditionally, a test blueprint is informed by a practice analysis, but this process is often very time consuming, expensive, and includes content that may be out of scope for an assessment composed of multiple-choice questions. To address this problem, the National Board of Medical Examiners proposed a rapid blueprinting process (RBP) as a cost- effective and timely alternative (2,3).
Summary:
The RBP consists of two phases. During phase 1 an expert panel representing various veterinary stakeholder groups participated in a 2-day meeting to create a draft blueprint by discussing expected competencies, outlining major content areas, assigning weights, and defining subtopics relevant for the progress test. During phase 2 the draft blueprint was packaged within a survey and communicated to a broader stakeholder community to review and suggest minor modifications.
The RBP consists of two phases. During phase 1 an expert panel representing various veterinary stakeholder groups participated in a 2-day meeting to create a draft blueprint by discussing expected competencies, outlining major content areas, assigning weights, and defining subtopics relevant for the progress test. During phase 2 the draft blueprint was packaged within a survey and communicated to a broader stakeholder community to review and suggest minor modifications.
Results:
In Phase 1 the expert panel identified 5 topic areas and weights. One hundred and thirty seven stakeholders covering different roles and primary areas of focus were then surveyed. 57% responded. 71% agreed that the content domains were appropriate and while 50.6% felt the weightings should be changed the mean weights provide by survey respondents were within 1 or 2 percentage points of what was proposed.
In Phase 1 the expert panel identified 5 topic areas and weights. One hundred and thirty seven stakeholders covering different roles and primary areas of focus were then surveyed. 57% responded. 71% agreed that the content domains were appropriate and while 50.6% felt the weightings should be changed the mean weights provide by survey respondents were within 1 or 2 percentage points of what was proposed.
Discussion:
Rapid blueprinting provided a practical framework to identify content areas and their relative importance from across academic veterinary medicine.
Rapid blueprinting provided a practical framework to identify content areas and their relative importance from across academic veterinary medicine.
Conclusion:
Rapid blueprinting provided a systematic and practical framework to collect validity evidence that can be applied to progress tests and other lower-stakes assessments.
Rapid blueprinting provided a systematic and practical framework to collect validity evidence that can be applied to progress tests and other lower-stakes assessments.
References (maximum three)
1.Dion V, St-Onge C, Bartman I, Touchie C, Pugh D. Written-Based Progress Testing: A Scoping Review. Acad Med. 2022 May 1;97(5):747-757. doi: 10.1097/ACM.0000000000004507. Epub 2022 Apr 27. PMID: 34753858.
2.Clauser, A.; Subhiyah, R., Martin, D. F.; Guernsey, J. A Fresh Perspective: Examination Blueprint Development. 2017. ABMS Conference.
3.Subhiyah, R., Clauser, A.; Park, Y. S.; Martin, D. F.; and Labovitz, A. J. Rapid Blueprinting: An Efficient and Effective Method for Designing Content of Assessments. (Submitted to NEJM)