Jane Stanford¹
1 Advanced Paediatric Life Support 

“Of course, not everything that counts can be counted. Furthermore, not all outcomes of importance can be measured. We should resist the temptation to focus on the measurable at the expense of the important.”1 

Although Advanced Paediatric Life Support (APLS) and other Structured Resuscitation Training (SRT) programs receive widespread professional endorsement, studies have shown limited and short-term change in clinicians’ knowledge, skills and behaviour. This could be because SRT outcomes (knowledge, skills and an approach to care) are measured in isolation, which is not how the content of these programs is applied in the clinical context. 

Script Concordance Tests (SCTs) have been validated as a measure of knowledge and clinical reasoning following clinical placement training programs. However, SCTs have not been validated as an assessment tool for SRT programs. 

My Masters project was a validation study of an SCT for the APLS program. Guided by the frameworks of Messick and Kane, the study created and piloted an APLS SCT to collect qualitative and quantitative data for a validation argument. Despite small numbers, psychometric analysis indicated that the APLS SCT as designed showed potential to behave in a similar manner to SCTs created for other contexts. Larger studies with APLS learners will be required to further validate the SCT for the APLS context. However, the preliminary work indicated positive results. 

Since this study and continued implementation of programmatic assessment designs in other forms of training, the role of the SCT as a tool supporting assessment for learning is worthy of review for development of clinical reasoning in this setting. 

Aside from reporting the results of this study, this presentation outlines the application of test validation processes, of which psychometric analysis is only one component. 

References (maximum three) 

1. Wilkinson TJ. Outcomes from educational interventions [online]. Focus on Health Professional Education: A Multi-disciplinary Journal. 2016;17(1):I-III. 

2. Lubarsky S, Charlin B, Cook DA, Chalk C, van der Vleuten CP. Script concordance testing: a review of published validity evidence. Medical Education. 2011;45(4):329-38. 

3. Cook DA, Kuper A, Hatala R, Ginsburg S. When Assessment Data Are Words: Validity Evidence for Qualitative Educational Assessments. Academic Medicine. 2016;91(10):1359- 69.