Presentation Description
Rajneesh Kaur1
Richmond Jeremy, Sally Middleton and Joanne Hart1
1 University of Sydney, School of Medicine
Richmond Jeremy, Sally Middleton and Joanne Hart1
1 University of Sydney, School of Medicine
Background
Each year ~300 mandatory research projects are undertaken by Doctor of Medicine (MD) degree students at the University of Sydney marked by ~ 150 academics and affiliates. Double marking, assessment rubrics and moderation of marking are used to enhance the objectivity and fairness of marking of written assessments. This study examined the impact of these quality assurance measures in the assessment of MD research projects.
Summary of work
We compared first and second assessors’ marks for marking of 801 research project final reports from 2021 and 2023 MD cohorts. Statistical analysis included calculating intraclass correlation coefficient, creating Bland-Altman plots and regression for proportional bias to assess agreement in mark. Consistency of examiner feedback comments was assessed through qualitative analysis, comparing the comments though thematic analysis.
Summary of results
Moderate intraclass correlation coefficient (ICC) was seen for the total mark (0.508; 95% CI: 0.411-0.588) with low to moderate ICC was seen for individual components of the report. Bland-Altman plots and qualitative findings further supported these results as there was a low agreement and lack of uniformity in examiner comments.
Discussion and conclusions
Final marks were awarded by averaging the marks of first and second examiners, addressing the issues around low correlation. Whenever a large inconsistency (a difference of >15 marks between both examiners) was noted between marks awarded by examiners one and two, a third examiner was engaged to help resolve differences. Given a large and changing pool of markers, a calibration process was not possible.
Take home messages / implications for further research and practice
Examiner training and engaging expert in the field as marker are recommended to further improve the quality of written assessment marking. Our next phase involves investigating the perspectives of both students and examiners regarding the assessment methods employedto ensure quality of marking.
References (maximum three)
ØBennett J. Second-marking and the academic interpretative community: ensuring reliability, consistency and objectivity between markers. Investigations in Teaching and Learning. 2005; 4(1): 80-86.
ØGiavarina D. Understanding Bland Altman analysis. Biochem Med (Zagreb). 2015 Jun 5;25(2):141-51. doi: 10.11613/BM.2015.015. PMID: 26110027;
ØRone-Adams S, Naylor S. Examination of the Inter-Rater Agreement among Faculty Marking a Research Proposal on an Undergraduate Health Course. Internet Journal of Allied Health Sciences and Practice. 2009. DOI:10.46743/1540-580X/2009.1267