Skip to main content
Ottawa 2024
Times are shown in your local time zone GMT

“Assessment Mixology:” Is assessing reflective practice an art or a science? Or is it both?                                       

Oral Presentation
Edit Your Submission
Edit

Oral Presentation

4:15 pm

27 February 2024

M217

Reflection in and on assessment

Presentation Description

Vicki Jones1
Sharmyn Turner1 and Nathan Ryckman1
1 University of Auckland 



Background 
Critical reflection is central to professionalism (MCNZ, 2021). In years four and five (Phase 2) of the medical programme, students' reflective practice skills are assessed using a student-led written portfolio. Portfolios are marked and moderated using a descriptive rubric with eight (numerically graded) criteria, 


Summary of work 
To test the reliability and defensibility of grading, the Phase 2 results data for 2021-2022 was statistically analysed. Outcomes were compared to a ‘blind’ benchmarking exercise that utilised the current marking and moderating process. 


Results 
Despite numerical grades values for ‘pass’ and ‘distinction’ outcomes overlapping, the numerical grades are significantly higher for ‘distinction’ compared to ‘pass’ outcomes. Further, the distribution of ‘distinction’ outcomes became less spread over time as assessors gain experience in utilising the qualitative assessment rubric. 


Discussion 
Summative assessment works to engage students’ reflective practice skills (Driessen et al., 2005). However, setting rigid assessment cut-points where qualitative assessment tools are used can serve to diminish the richness of the assessed content. Delineating between ‘pass’ and ‘distinction’ grades requires assessor flexibility and may not always be a direct reflection of a derived numerical assessment outcome (Biggs, 1992). 


Conclusion 
There appears to be little statistical basis to support a particular cut-point for a ‘distinction’ over a ‘pass’. This may be desirable considering the qualitative nature of the assessment. Rather than adhering to a rigid numerical structure to delineate between ‘pass’ and ‘distinction’, the PPS markers and moderator are encouraged to base their outcomes on the overall synthesis of quality and depth observed in each portfolio. Harnessing the experience of a stable marking team is important to optimise assessment reliability. 


Implications for further practice 
A revised rubric incorporating a te ao Māori perspective is in development. The final version will be analysed and benchmarked to ensure future grading remains robust and defensible. 



References (maximum three) 

Biggs, J. B. (1992). A qualitative approach to grading students. Herdsa News, 14(3), 3-6. 

Driessen, E.W., van Tartwijk, J., Overeem, K., Vermunt, J.D., ven der Vleuten, C.P.M. (2005). Conditions for successful reflective use of portfolios in undergraduate medical education. Medical Education, 39, 1230 – 1235. https://doi.org/10.1111/j.1365-2929.2005.02337.x 

Medical Council of New Zealand. (2021). Good Medical Practice. https://www.mcnz.org.nz/assets/standards/b3ad8bfba4/Good-Medical-Practice.pdf 

Speakers