Presentation Description
Daniel Zahra1
Louise Belfield2
1 Peninsula Medical School
2 Brunel Medical School
Louise Belfield2
1 Peninsula Medical School
2 Brunel Medical School
Background:
Whilst standard setting assessments using the Angoff method is widely used, and often seen as a ‘gold standard’ for criterion-based pass marks, a steadily growing body of work (e.g. Burr et al 2017; 2022) has questioned both the underlying assumptions (what is a borderline candidate, or minimal competence, or difficulty?) and the practical implementation of the method (how many judges are needed, should judgements be proportional or binary?) These are all important questions, but ones which often overlook the social aspect of the Angoff process; from trying to define the criteria, to conceptualising the borderline candidate; from judge’s experiences to the group dynamics in moderation.
Whilst standard setting assessments using the Angoff method is widely used, and often seen as a ‘gold standard’ for criterion-based pass marks, a steadily growing body of work (e.g. Burr et al 2017; 2022) has questioned both the underlying assumptions (what is a borderline candidate, or minimal competence, or difficulty?) and the practical implementation of the method (how many judges are needed, should judgements be proportional or binary?) These are all important questions, but ones which often overlook the social aspect of the Angoff process; from trying to define the criteria, to conceptualising the borderline candidate; from judge’s experiences to the group dynamics in moderation.
Summary and Findings:
This work analyses variation in individual, pre-moderation and collectively discussed post-moderation Angoff judgements across the last ten years of single- best answer multiple-choice applied medical and dental knowledge in our medical and dental schools. We consider the number of items changed and direction of change not only as a function of previously studied variables such as number of judges and types of judgement, but also in relation to the expertise of each judge, and their seniority in the school structure.
This work analyses variation in individual, pre-moderation and collectively discussed post-moderation Angoff judgements across the last ten years of single- best answer multiple-choice applied medical and dental knowledge in our medical and dental schools. We consider the number of items changed and direction of change not only as a function of previously studied variables such as number of judges and types of judgement, but also in relation to the expertise of each judge, and their seniority in the school structure.
Discussion:
Findings are discussed in relation to managing these factors and their impact, their relationship to the assumptions of the Angoff standard setting method, and how to incorporate them effectively into staff training to raise awareness and promote reflection on the personal and social aspects of standard setting, working towards shared consensus and understanding of the required standards.
Findings are discussed in relation to managing these factors and their impact, their relationship to the assumptions of the Angoff standard setting method, and how to incorporate them effectively into staff training to raise awareness and promote reflection on the personal and social aspects of standard setting, working towards shared consensus and understanding of the required standards.
Take-Home Points:
There is no ‘gold-standard’ for standard setting, and social factors are as important to consider as pedagogic ones, especially where group deliberation is a part of the method. However, these factors can enhance the process and increase its value when acknowledged and incorporated appropriately.
There is no ‘gold-standard’ for standard setting, and social factors are as important to consider as pedagogic ones, especially where group deliberation is a part of the method. However, these factors can enhance the process and increase its value when acknowledged and incorporated appropriately.
References (maximum three)
Burr, S., Martin, T., Edwards, J., Ferguson, C., Gilbert, K., Gray, C., Hill, A., Hosking, J., Johnstone, K., Kisielewska, J., Milsom, C., Moyes, S., Rigby-Jones, A., Robinson, I., Toms, N., Watson, H., & Zahra, D. (2021) Standard setting anchor statements: a double cross-over trial of two different methods. MedEdPublish 10(1) Article 32
Burr, S.A., Zahra, D., Cookson, J., Salih, V.M., Gabe-Thomas, .E, & Robinson, I.M. (2017) Angoff anchor statements: setting a flawed gold standard?. MedEdPublish, 6(3) Article 53.