Ottawa 2024

Times are shown in your local time zone GMT

Topics in quality assurance of assessments

Workshop

10:30 am

26 February 2024

M216

Themes

Theme 8: Evaluation

Session Program

10:30 am

Economic analyses in health professions training and assessment. How to get started.

Jonathan Foo - Lecturer - Monash University, David A Cook, Jennifer Cleland , Stephen Maloney, You You

Jonathan Foo¹
Jennifer Cleland², David A Cook³, You You⁴ and Stephen Maloney¹
1 Monash University
2 Nanyang Technological University
3 Mayo Clinic College of Medicine and Science
4 Peking University

1. Background

In this workshop, we will introduce health professions education researchers and leaders to basic economic concepts and research methodologies.

There are limited resources (e.g. time, staffing, money) for health professional training and assessment. As such, choices must be made between alternative options. An evidence-based approach for allocating resources requires the understanding and application of economic as well as educational concepts. Economic analyses inform implementation, and common uses include: measuring how much an assessment costs and where it might be made more efficient, identifying which assessment alternatives are more cost-effective, and quantifying the economic benefits of educational practices.

2. Why is the topic important for research and practice?

All educators grapple with the problem of implementing programs with finite resources. Economic research provides data to enable best use of available resources, inform balancing the cost of assessment against the value it generates, and support justification of the business case for programs when engaging with educational funders. Content for this workshop will draw on published guidance produced by the facilitator team [1], in particular AMEE Guide 123 [2] and AMEE Guide 139 [3].

3. Workshop format, including participant engagement methods

In this workshop, participants will be supported by facilitators to develop a written outline for their own real or hypothetical economic research study. The format will involve:

Introduction to economic cost, including common economic questions (didactic).
Discuss an economic problem (their own, or one of ours) and develop a focused economic research question (think-pair-share).
Introduction to economic study designs (didactic).
Identify an economic study design to answer their question (think-pair-share).
Introduction to methods for measuring costs and outcomes (didactic).
Develop a plan for measuring costs and outcomes related to their question (active, individual work).
Discussion of key literature and further readings (handout).

4. Who should participate?
Anyone with an interest in designing “cost conscious” education (i.e. activities that make best use of available resources) should participate. We will assume a basic knowledge of general research methods for education or clinical areas. No prior experience or knowledge of economic research is expected.

5. Level of workshop
Beginner to intermediate

6. Workshop outcomes
Upon completion of the workshop, participants will have developed an economic research study outline. Together with supporting resources provided, it is expected that participants will be able to plan and conduct a simple economic analysis, hold informed discussions with economic specialists, and undertake further learning on more advanced economic topics.

References (maximum three)

1. Cook, D. A., Stephenson, C. R., Wilkinson, J. M., Maloney, S., Baasch Thomas, B. L., Prokop, L. J., & Foo, J. (2022). Costs and Economic Impacts of Physician Continuous Professional Development: A Systematic Scoping Review. Academic Medicine, 97(1), 152– 161.
2. Maloney, S., Cook, D. A., Golub, R., Foo, J., Cleland, J., Rivers, G., Tolsgaard, M. G., Evans, D., Abdalla, M. E., & Walsh, K. (2019). AMEE guide no. 123 – How to read studies of educational costs. Medical Teacher, 41(5), 497-504.
3. Foo, J., Cook, D. A., Tolsgaard, M., Rivers, G., Cleland, J., Walsh, K., Abdalla, M. E., You, Y., Ilic, D., Golub, R., Levin, H., & Maloney, S. (2020). How to conduct cost and value analyses in health professions education: AMEE Guide No. 139. Medical Teacher, 43(9), 984- 998.

11:30 am

Utilising the Context, Input, Process, Product (CIPP) model for evaluating health professions programs

Sarah Meiklejohn - Lecturer - Monash University, Arunaz Kumar, Fiona Kent

Sarah Meiklejohn¹
Fiona Kent¹ and Arunaz Kumar²
1 Monash University
2 Monash University

1. Background

Curriculum change can be supported by frameworks in higher education however the impact of these frameworks in achieving their stated purposes remains unclear. An interprofessional education (IPE) framework and related curriculum were designed to develop requisite skills for collaborative care in health profession graduates. The utility and impact of this framework upon perceived work readiness for collaborative practice was evaluated using the context, input, process, product (CIPP) model at Monash University in the Faculty of Medicine, Nursing and Health Sciences in 2021 (Meiklejohn et al., 2022). This workshop will showcase how the CIPP model can be used to evaluate whole of faculty curriculum innovations (Stufflebeam, 2003). Participants will have the opportunity to workshop these evaluation principles in the context of their own program evaluations

2. Why is the topic important for research and / or practice?

Application of framework to health professional programs can evaluate the strengths and missing gaps in learning and educational practice. A systematic approach to learning, curriculum and integration in practice can be facilitated by introduction of validated frameworks. Training of staff and researchers on a pragmatic, easily applicable framework such as CIPP can improve their expertise in understanding what works and what needs to be changed.

3. Workshop format, including participant engagement methods

The workshop will commence with a brief presentation showcasing how the CIPP model was applied at Monash University to evaluate a whole of Faculty curriculum innovation.
Participants will then be guided through each stage of the model in small groups with facilitators to consider the application of CIPP within their small groups.
Participants will be encouraged to apply the principles to their own existing programs to explore its potential benefits for their programs.

4. Who should participate?

This workshop is recommended for academics and researchers looking to learn more about the CIPP evaluation framework and its applications within health professions education, including interprofessional education.
Educators, students and researchers who wish to expand their knowledge and understanding of how a health professional learning program can be benefitted by applying a framework.

5. Level of workshop (beginner / intermediate / advanced)
Intermediate

6. Take-home messages / workshop outcomes / implications for further research or practice

Appreciate the value of applying a systematic framework to health professional learning
Recognise the components of the CIPP evaluation method
Propose circumstances where the CIPP model may offer a useful evaluation framework

7. Maximum number of participants
We would recommend no more than 40 participants for this workshop.

References (maximum three)
Meiklejohn S, Anderson A, Brock T, Kumar A, Maddock B, Wright C, Walker L, Kent F. The utility of an interprofessional education framework and its impacts upon perceived readiness of graduates for collaborative practice. A multimethod evaluation using the context, input, process, product (CIPP) model. Nurse Educ Today. 2023 Feb;121:105707. doi: 10.1016/j.nedt.2023.105707. Epub 2023 Jan 5. PMID: 36640451.
Stufflebeam, D. (2003). The CIPP model of evaluation. In T. Kellaghan, D. Stufflebeam & L. Wingate (Eds.), Springer international handbooks of education: International handbook of educational evaluation.