Presentation Description
Angelina Lim
Daniel Malone, Sunanthiny Krishnan, Simon Furletti and Mahbub Sarkar
Daniel Malone, Sunanthiny Krishnan, Simon Furletti and Mahbub Sarkar
Background:
Although a thoughtfully designed Objective Structured Clinical Examination (OSCE) is a robust and valid assessment tool, debate exists about the effectiveness of OSCEs to authentically mirror real-life scenarios. Limited studies have explored the extrapolation inference in Kane Validity’s Framework.(1)
Although a thoughtfully designed Objective Structured Clinical Examination (OSCE) is a robust and valid assessment tool, debate exists about the effectiveness of OSCEs to authentically mirror real-life scenarios. Limited studies have explored the extrapolation inference in Kane Validity’s Framework.(1)
Summary of work:
A sequential mixed methods approach was used. Mystery shoppers visited pharmacy students on their community pharmacy placement and simulated the same case scenario students were given a recent infectious diseases OSCE. Students were marked with the same rubrics and these marks were compared with their OSCE score. The mystery shopping visit was then revealed to all the students and all students were asked to participate in a semi- structured interview.
A sequential mixed methods approach was used. Mystery shoppers visited pharmacy students on their community pharmacy placement and simulated the same case scenario students were given a recent infectious diseases OSCE. Students were marked with the same rubrics and these marks were compared with their OSCE score. The mystery shopping visit was then revealed to all the students and all students were asked to participate in a semi- structured interview.
Results:
Overall, 92 mystery shopper (Work Based Assessment (WBA)) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was 39.2% lower compared to the OSCE score (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage a real-life patient.
Overall, 92 mystery shopper (Work Based Assessment (WBA)) visits with students were conducted and 36 follow-up interviews were completed. The median WBA score was 39.2% lower compared to the OSCE score (p < 0.001). Interviews revealed students knew they did not perform as well in the WBA compared to their OSCE, but reflected that they still need OSCEs to prepare them to manage a real-life patient.
Discussion:
Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as oppose to an unfamiliar workplace.
Many students related their performance to how they perceived their role in OSCEs versus WBAs, and that OSCEs allowed them more autonomy to manage the patient as oppose to an unfamiliar workplace.
Conclusions:
OSCEs performance did not correlate to real life performance in this instance; however, students still valued having an OSCE before a WBA.
OSCEs performance did not correlate to real life performance in this instance; however, students still valued having an OSCE before a WBA.
Take-home messages:
Students scored lower on placement than in OSCEs even though they reflected that it is easier to manage a patient in real-life; outlining challenges to replicate a real- life pharmacy situation in an OSCE, and challenges to testing extrapolation. Whilst OSCEs are useful for testing process type skills, clinical problem solving may be best assessed in a workplace environment.
Students scored lower on placement than in OSCEs even though they reflected that it is easier to manage a patient in real-life; outlining challenges to replicate a real- life pharmacy situation in an OSCE, and challenges to testing extrapolation. Whilst OSCEs are useful for testing process type skills, clinical problem solving may be best assessed in a workplace environment.
References (maximum three)
Kane, M. T. (1992). An argument-based approach to validity. Psychol Bull, 112, 527-535.