Karen Fung¹
John Pugsley¹ and Salma Satchu¹
1 Pharmacy Examining Board of Canada 

At the onset of the COVID-19 pandemic, testing organizations developed and implemented innovative, yet defensible methodologies to ensure assessment processes would still take place. This was especially true for licensure/certification bodies in healthcare settings as the demands for healthcare professionals were high. With the adopted philosophy of “the show must go on” under the rapidly changing circumstances, psychometricians and testing organizations had to adapt their traditional processes and make use of current technology resources without jeopardizing the validity of exams. Standard setting is a process of establishing a cut score for an assessment. It is an important process involving subject matter experts (SMEs) to ensure levels of performance, or pass/fail decisions are accurately defined. Prior to COVID-19, most testing organizations conducted standard-setting in-person due to the need for engaged, in- depth discussions involving SMEs. In this presentation, the Pharmacy Examining Board of Canada will discuss the transition of in-person standard setting to a virtual format for performance-based exams (i.e., OSCE). The considerations in planning, adaptations made, along with participants’ feedback will be presented from a psychometric perspective. Although psychometric analyses such as generalizability theory demonstrated that the virtual standard setting is comparable to the in-person format, the in-person format is still preferable for various reasons. The decision to revert to the in-person format will also be discussed. 

References (maximum three) 

Cizek, G. J., & Bunch, M. B. (2007). Standard setting. Sage Publications. 

Cizek, G. J., Bunch, M. B., & Koons, H. (2004). Setting performance standards: Contemporary methods. An NCME Instructional Module.