ePoster
100% Page: /
Presentation Description
Richard Arnett1
Muríosa Prendergast1
1 RCSI University of Medicine & Health Sciences
Muríosa Prendergast1
1 RCSI University of Medicine & Health Sciences
1. Background
A key component of the new RCSI undergraduate medical curriculum is the use of Progress Testing. RCSI Progress Testing involves 4 diets per year each consisting of 160 SBAs with 3 options plus a ‘Don’t Know’ option. Penalty scoring is applied: +1 (correct), -0.5 (incorrect), and 0 (Don’t Know).
2. Summary of work
Students in the new curriculum are in Year 2 but eventually, Progress Testing will be happening simultaneously throughout the programme. The methodology for setting standards needs to balance the need for robustness and transparency, the use of penalty scoring, and the variety and workload of staff involved.
3. Results
We are using a modified Bookmark methodology. Response data are analysed under a Rasch model to produce an Ordered Item Booklet (OIB) in which questions are presented in order of increasing difficulty. Each question is accompanied by additional information for each cohort taking the test (including mean score, discrimination, estimated total score, and pass rates). Judges identify the point in the OIB where the probability of a Minimally Competent Candidate (MCC) drops below 50% (for each cohort). The group cut score is the mean ability level of these estimates. To account for penalty scoring, an ‘allowance’ is subtracted from the cut score. This allowance is based on a percentage of the total test items and decreases in each year of the programme.
4. Discussion
Incorporating negative marking into the standard-setting process has been challenging & staff have required additional support.
5. Conclusions
Progress Testing is currently confined to the first two years but the initial experience suggests this methodology should scale as additional cohorts come online. Applying a decreasing penalty allowance to the cut score allows some level of educated guessing.
6. Take-home messages/implications for further research or practice
Planning & ongoing communication are crucial.
References (maximum three)
Baldwin P. A Problem with the Bookmark Procedure’s Correction for Guessing. Educ Meas: Issues Pr. 2021;40(2):7–15
Karantonis A, Sireci SG. The Bookmark Standard-Setting Method: A Literature Review. Educational Measurement: Issues and Practice. 25(1):4–12