Daniel Zahra¹
Gergo Pinter¹ and Louise Belfield²
1 Peninsula Medical School
2 Brunel Medical School

Background: The global healthcare workforce is moving away from siloed practice to multi- and inter-disciplinary teams, requiring programmes to place greater emphasis on competency and application over demonstrating discrete knowledge. New and aspiring doctors now also hail from a wider range of backgrounds than ever before, reflecting ever greater access to medical education. Yet assessment has, in most cases, been slow to reflect these changes, relying on methods of design, criteria and standard setting, and performance aggregation that have changed little from those designed largely by and for generations past. 

 

Importance: It is important to realise the impact that assessment has on learning, and curriculum design, and ensure that it is used as a tool to support and drive development of those qualities we wish to instil and champion as educators, and those currently required by our graduates. We need to be mindful of the types of skills assessments can – and cannot – measure, and create them in such a way as to allow all candidates to demonstrate their knowledge. Whilst traditional methods serve particular purposes, more can be done to make assessments accessible, equitable, and beneficial to all involved; students and educators alike. This also entails an awareness of over-assessment, and an obligation to use assessment more effectively, rather than use additional assessments or increasingly complex statistics to compensate for limitations in assessment, curricula, or regulatory design.

 

Format: The first part of the workshop will summarise the contextual changes necessitating a review of assessment practices, and involve group discussion of the commonplace considerations, leading into discussion of the broader, often unconsidered dimensions of assessment. These will be considered with respect to what attendees are familiar with, have encountered themselves, and the underlying cultural and systemic causes of assessment challenges. The second part will consider alternative approaches to the designing of a range of assessments, including but not limited to multiple choice formats, structured clinical examinations, and essay-based assessments, and how we can improve their accessibility, inclusivity, and overall assessment reliability and validity. We will evaluate the strengths and weaknesses of each, along with consideration of how assessment design, structure, and strategy needs to align with programme aims and other aspects of teaching, assessment, and the curriculum overall. The session will conclude with summary discussions of examples offered by the group, and what we’ve learnt about how assessments can be made more equitable, efficient, and aligned to programmatic aims, and how these learning points can be applied to the attendees’ own practice. 

 

Participant: Anyone with an interest in assessment, widening access, and education more generally should find something of use and interest in this workshop.

Level: ‘Intermediates’, with some knowledge of assessment practice would find this most useful, although we aim to provide introductions to concepts that are suitable for beginners, and provide new insights for advanced colleagues as well.

Message: Good assessments should allow everyone to excel, and provide useful information for improvement, directing future learning, and assuring institutions and regulators. They should also be suited to, and driven by, their aims and alignment to the teaching, desired learning, and learning outcomes, not by pre-existing methods or traditions.

 

 

 

Ali, K., Winter, J., Webb, O, and Zahra, D. (2022) Decolonisation of curricula in undergraduate dental education: an exploratory study. British Dental Journal, 233, 415–422

 

Burr, S.A., Kisielewska, J., Zahra, D., Hodgins, I., Robinson, I., Millin, P., Gale, T., Santos, N., Pêgo, J.M. (2022 preprint) Personalising knowledge assessments to remove compensation and thereby improve preparation for safe practice - developing content adaptive progress testing. Research Square. 

 

Watson, H., Dolley, M-K., Perwaiz, M., Saxelby, J., Bertone, G., Burr, S., Collett, T., Jeffery, R., and Zahra, D. (2022) ‘Everyone is trying to outcompete each other’ – A qualitative study of student attitudes to a novel peer-assessed undergraduate teamwork module. Federation of European Biochemical Societies: Open Bio

 

 

Maxim Morin¹
Viren Naik¹, Sean Gehring², Jon Dupre³ and Paul Glover³
1 Medical Council of Canada
2 The Royal College of Dentists of Canada
3 risr/

Background:
The delivery of high-stakes certification examinations is a critical aspect of many industries, including professional certifications and licensure. With the advent of technological advancements, multiple delivery modalities have emerged, such as remote proctoring, test centers, and pop-up test centers. This interactive workshop aims to compare the benefits, challenges and issues with these modalities using business case methodology and Van der Vleuten’s framework for evaluating the utility of assessment methods. 

Importance for Research and Practice: 
Research and practice in the field of high-stakes certification examinations must constantly evaluate the delivery modalities to ensure optimal outcomes.Thisworkshopaddressestheneedtoexaminetheimplications andrisksofdifferent delivery options, enabling participants to make informed decisions and implement efficient strategies in their respective organizations. 

Workshop Format and Participant Engagement Methods:
The workshop will utilize a combination of business case studies, group discussions, and interactive exercises to actively engage participants. Use cases representing different industries and scenarios will be presented, allowing participants to analyze and evaluate the implications and risks associated with remote proctoring, test centers, and pop-up test centers. Group discussions will facilitate knowledge sharing and the exploration of practical strategies. 

Target Participants:
This workshop is suitable for professionals involved in high-stakes certification examinations, including certification program managers, industry representatives, testing experts, policymakers, and individuals interested in assessment delivery. Participants will benefit from the multidimensional perspectives and diverse experiences in managing certification programs. 

Workshop Level:
This workshop is designed for intermediate-level participants who possess a foundational understanding of high-stakes certification examination processes. It is suitable for individuals seeking to deepen their knowledge and engage in practical discussions and decision-making related to delivery modalities. Intermediate-level knowledge of cost analysis and risk assessment is recommended. 

a) Enhanced understanding of the benefits, challenges and issues associated with different delivery modalities for high-stakes certification examinations. 

b) Identification of key factors influencing cost analysis and risk assessment in remote proctoring, test centers, and pop-up test centers. 

c) Comparison of business models and frameworks that can be used to evaluate the various implications and risks of each delivery modality. 

d) Identification of potential research gaps and areas for further exploration, such as long-term cost analysis, scalability, and technological advancements in delivery modalities. 

The workshop outcomes will enable participants to make informed decisions regarding the most suitable delivery modality for their certification programs. They will also contribute to the development of best practices and guidelines in the field of high-stakes certification examination delivery. 

References (maximum three) 

Morin, M., Alves, C., & De Champlain, A. (2021). The show must go on: Lessons learned from using remote proctoring in a high-stakes medical licensing exam program in response to severe disruption. Journal of Applied Testing Technology, 23, 15-35. 

Cherry, G., O'Leary, M., Naumenko, O., Kuan, L. A., & Waters, L. (2021). Do outcomes from high stakes examinations taken in test centres and via live remote proctoring differ?. Computers and Education Open, 2, 100061. 

Van Der Vleuten, C. P., & Schuwirth, L. W. (2005). Assessing professional competence: from methods to programmes. Medical education, 39(3), 309-317.