Kirkpatrick model: how to assess a teaching program

The leadership of a medical course has just run a patient care simulation module for the fourth year for the first time. The students loved it: the form on the last day came back full of top marks and praise. At the faculty meeting, someone asks whether the module should become mandatory. Another professor raises her hand: "they liked it, but did they learn? And, if they learned, does that change anything in the clerkship?". Nobody has the answer, because nobody planned how to measure it.

This scene repeats itself in academic leagues, residency programs and continuing education programs. The Kirkpatrick model is the most used tool to organize this conversation: it separates what "working" means into four levels, from the satisfaction of those who took part to the impact on results. This guide explains the levels with examples from medical education, shows what reviews found when they applied the model, presents the most serious criticisms and proposes a practical way to design a program's evaluation before it begins.

Key points

• The Kirkpatrick model organizes the evaluation of a teaching program into four levels: reaction, learning, behavior and results.
• In medical education, most published evaluations stop at the first or second level; few measure change in practice and even fewer measure patient outcomes.
• High satisfaction does not guarantee learning, and learning on a test does not guarantee a change in behavior in the service.
• The main criticisms: the model looks only at results, assumes a causal chain between the levels and ignores the context and unplanned effects.
• The version called New World Kirkpatrick and models such as CIPP and the logic model help explain how and why a program works.
• The evaluation should be designed before the program, starting from the result you want to reach.

Summary

• What the Kirkpatrick model is
• The four levels in medical education
• What the reviews found
• The criticisms of the model
• Alternatives and complements
• How to design a program's evaluation
• Kirkpatrick in the daily life of those who teach
• How NEXO fits into a program's evaluation
• Frequently asked questions
• Checklist for a program's evaluation

What the Kirkpatrick model is

The model was proposed by Donald Kirkpatrick in the late 1950s to evaluate training in companies. The question was practical: how do you know whether a course offered to employees was worth the investment? The answer was to divide the evaluation into four levels, each more demanding than the previous one. First, how the participants reacted. Then, what they learned. Next, whether they changed their behavior at work. Finally, whether the organization obtained the expected results.

Medical education adopted the model with enthusiasm, because it offers a common vocabulary for a difficult question. When an article says it evaluated a course "at level two", any reader knows there was a measure of learning, and not only of satisfaction. The AMEE guide by Frye and Hemmer describes the Kirkpatrick model alongside other evaluation models and recalls that the choice depends on the complexity of the program and on what educators need to know.

An adaptation for medical education

In the tradition of evidence-based medical education, Belfield and colleagues proposed five levels of effectiveness: participation, reaction, learning, behavior and health results. When analyzing more than three hundred abstracts, they found little research on health outcomes and a lot of ambiguity about what counts as a measure of effectiveness. Many systematic reviews in the field still use some version of this scale, with level two divided into change in attitudes and change in knowledge or skills, and level four divided into change in the organization and benefit for patients.

Outcome evaluation and program evaluation

It is worth separating two types of question from the start. Outcome evaluation wants to know whether the program produced the expected effect. Program evaluation wants to know how and why the effect appeared or failed to appear, in what context and for whom. The Kirkpatrick model answers the first very well and the second poorly, and this difference explains most of the criticisms that appear below.

The four levels in medical education

Each level answers a different question and calls for different instruments. The table below summarizes the levels with typical examples from a clinical reasoning or simulation module.

Level	Question	Example of measure	Main limitation
1. Reaction	Did the participants like it and find it useful?	Satisfaction questionnaire at the last session	Liking is not learning
2. Learning	Was there a gain in knowledge, skill or attitude?	Pre and post test; skills station	A gain on the test may not reach the service
3. Behavior	Did practice in the service change?	Direct observation; chart review	Depends on context, supervisors and time
4. Results	Did patients or the institution benefit?	Quality and safety indicators	Many factors beyond the program

Level 1: reaction

Reaction is the easiest level to measure and the most measured. A short questionnaire at the end of the course reveals whether the content seemed relevant, whether the format worked and whether the participants would recommend the experience. This data has value: a course nobody can stand hardly teaches anything. The problem is stopping here. Satisfaction depends on factors such as the teacher's charisma, the schedule and expectations, and has no guaranteed relation to learning.

Level 2: learning

The second level asks whether there was a measurable gain in knowledge, skill or attitude. Pre and post tests, practical stations, assessment of written cases and attitude scales belong here. The quality of the instrument matters a great deal. In simulation-based assessment, the review by Ilgen and colleagues compared checklists and global rating scales and found a high correlation between the two types of instrument, with similar inter-rater reliability, which helps in choosing according to the task and the experience of the rater.

Level 3: behavior

The third level asks whether the person started doing something different at work. This is where assessment gets expensive: it requires observation in the service, chart review or reports from colleagues, and it needs time for the change to appear. It is also where context weighs most. A resident who learned to pause and review hypotheses may not use what was learned if the team makes no room for it.

Level 4: results

The fourth level asks whether patients, teams or the institution benefited. In medical education, this can mean fewer errors, better adherence to protocols or better clinical outcomes. It is the most desired level and the hardest to attribute to the program, because many factors change at the same time in a hospital.

What the reviews found

When researchers use the model to classify published studies, the picture repeats: plenty of assessment of reaction and learning, little of behavior and almost none of results.

Continuing education and clerkships

In a review of online continuing medical education, Curran and Fleet found that most assessments were based on participant satisfaction, with little research on change in practice and no study showing an effect on patient outcomes. In the review by Turkeshi and colleagues on undergraduate family medicine clerkships, 26 studies assessed level one, 26 level two and only 6 level three.

Spaced learning and digital narratives

The review by Phillips and colleagues on spaced learning in continuing education used the model to separate knowledge gains from changes in practice and patient outcomes, and showed how the effect becomes less certain as one moves up the levels. The review by Moreau and colleagues on digital narratives in health professions education included only studies with results at levels two or three; of the sixteen articles analyzed, fourteen dealt with learning and only two also investigated behavior change.

Simulation: a case apart

Simulation is the area with the most data at the higher levels. The meta-analysis by Cook and colleagues gathered 609 studies with more than 35,000 participants and found favorable effects of simulation compared with no intervention on knowledge, skills and behaviors, with moderate effects on patient outcomes. Even so, most studies used a single-group before-and-after design. And the review by Brydges and colleagues showed that the correlation between performance in simulation and patient outcomes in the service was much weaker than the correlation with the professionals' behaviors, which is a reminder that a good result at level two does not automatically become a result at level four.

The criticisms of the model

The model's popularity came with consistent criticism. It does not ask for the model to be abandoned, but for it to be used with awareness of what it measures and what it leaves out.

The levels do not form a causal ladder

The model is often read as a chain: a good reaction leads to learning, which leads to behavior, which leads to results. The review by Allen, Hay and Palermo lists this assumption of positive causal links between the levels as one of the main criticisms, alongside the exclusive focus on results, the idea that higher levels are always more important and the lack of attention to unintended effects.

It works for simple programs

Yardley and Dornan analyzed the use of the levels as a tool for critical appraisal of evidence and concluded that they carry so many implicit assumptions that they are only suitable for relatively simple instructional designs, short-term outcomes and beneficiaries who are not the learners themselves; conditions met by perhaps a fifth of medical education reviews. For the rest, hierarchical use of the levels adds little.

It ignores context

The same course can work in one hospital and fail in another, because of culture, workload or support from leadership. The original model does not ask about any of these factors. For someone coordinating a program, this means that a poor result at level three may have little to do with the quality of teaching and a lot to do with the environment the student returned to.

Higher levels are not always better

Another frequent criticism is the idea that a study at level four is automatically worth more than a study at level two. In practice, a well-designed study of learning, with a reliable instrument and a comparison group, can inform much more than a loose association between a course and a hospital indicator. The level says what was measured; the quality of the design says how much the result can be trusted. For those who read medical education articles, it is worth looking at the two things separately before drawing conclusions, and for those who write, it is worth describing the design with the same care used to describe the level reached.

Alternatives and complements

The response to the criticisms came from two directions: updating Kirkpatrick itself and combining the model with others that look at process and context.

New World Kirkpatrick

The version called New World Kirkpatrick reverses the planning order: it starts from the desired results and works back to the reaction. It also adds elements such as the critical behaviors expected to be seen, the factors that reinforce or hinder change and intermediate indicators. Moreau describes how this version tries to answer three central criticisms of the original model and invites medical educators to test it in complex programs.

Logic model and CIPP

The logic model draws the chain that links resources, activities, outputs and outcomes, and forces the team to make explicit why it believes the program will work. The CIPP model evaluates context, inputs, process and product, and is useful when the program is still being adjusted. The guide by Frye and Hemmer presents these models and the theories behind each one, from the reductionist view to complexity theory.

When to use each one

• Kirkpatrick: to communicate results in a standardized way and compare with the literature.
• New World Kirkpatrick: to plan a program from the desired result and track intermediate indicators.
• Logic model: to make the program's theory explicit and find weak links before starting.
• CIPP: for new or adjusting programs, when process and context matter as much as the result.
• Experimental designs: when the question is about effect and there are conditions for a comparison group.

How to design a program evaluation

An evaluation that works is planned together with the program, not after it. The script below works for an undergraduate module, an academic league activity or a residency program.

1. Start at the end. Define the result that would justify the effort: a behavior you want to see in the internship or an indicator for the service.
2. Make the program's theory explicit. Write down why you believe the activities lead to that result, and which conditions need to exist.
3. Choose one indicator per level. A short reaction questionnaire, a learning measure with a reliable instrument and a realistic way to observe behavior.
4. Plan the baseline. Measure before starting, or set up a comparison group.
5. Look for unplanned effects. Ask participants what changed that you did not expect, for better and for worse.
6. Use the data to adjust. The evaluation exists to improve the program, not only to justify its existence.

Example: a clinical reasoning module

Imagine an eight-week module with weekly clinical cases for the fourth year. At level one, a five-question questionnaire at the end. At level two, a test with written cases before and after, graded with a defined answer key. At level three, a review of a sample of written progress notes in the internship, looking for records of differential diagnosis. At level four, measuring patient outcomes may not be realistic, and it is fine to say so clearly. The guide on case-based learning helps design the module, and the text on feedback in clinical cases shows how to give results back to students.

Common mistakes

Measuring only satisfaction and calling it evaluation. Using a test put together in a hurry, without knowing whether it measures what you want. Expecting behavior change in two weeks. Attributing to the program an improvement that came from another change in the service. And, perhaps most common, deciding what to measure after the program has ended. To understand how to keep what was learned, the guide on retention of medical knowledge complements the discussion, and the text on active learning helps choose activities that tend to produce learning.

Kirkpatrick in the daily life of those who teach

The model often appears in articles and institutional reports, but it is also useful at smaller scales: an academic league, a clerkship, a weekly case discussion session. In these situations, nobody is going to set up a controlled study, and there is no need to. Just use the levels as a reminder of the questions worth asking.

In the academic league

Leagues organize classes, symposia and hands-on training with a lot of energy and little time to evaluate. A realistic path is to combine a reaction question at the end of each meeting with a small learning measure each semester, such as three short cases solved before and after a cycle of classes. If the league trains a skill, such as a sequence of care, it is worth observing a few members in simulation and recording what changed. That already puts the league above most published evaluations, which stop at satisfaction.

In the clerkship and the internship

The preceptor has privileged access to level three: they see the student in the service. A simple way to take advantage of this is to define, at the start of the clerkship, two or three behaviors expected by the end, such as presenting the case with a differential diagnosis or checking pending results before the shift handover. Over the weeks, the preceptor notes whether these behaviors appear. The record serves to give feedback to the student and, combined with those of other preceptors, to evaluate the clerkship itself.

In residency

Residency programs already collect a lot of data: performance assessments, procedure logs, service indicators. The challenge is linking that data to specific changes in the program. When a residency introduces a new activity, such as a weekly clinical reasoning session, it is worth choosing in advance a behavior indicator that is already collected and tracking the trend before and after, always remembering that other changes happen at the same time.

Talking about results with the students

Evaluating a program is also a teaching opportunity. Showing the class what was measured, what improved and what did not improve teaches, in practice, how to read educational evidence. And it opens space for the students themselves to suggest changes, which usually improves the program faster than any questionnaire.

How NEXO fits into program evaluation

NEXO can serve as a practice activity within a clinical reasoning program. The case of the day arrives in up to six clues, and with each guess the proximity in ICD-10 across five levels shows how far the hypothesis was from the diagnosis. The cases are fictional with a real clinical presentation. For those who design the evaluation, the question is the same as for any activity: at what level do you expect an effect and how will you measure it?

The guide what NEXO is explains the whole game, and the text on gamification in medical education summarizes what the evidence shows about games in teaching. How the modes are unlocked is on the NEXO Economy page.

If you coordinate a medical school, a student league or a residency and want to use NEXO with a class, learn about NEXO Closer. Researchers in medical education who want to design evaluation studies can propose projects through NEXO Researcher. Physicians who review cases take part through NEXO Curator, and those who create health content on social media can take part through NEXO Creators.

Frequently asked questions

What are the four levels of the Kirkpatrick model?

Reaction, learning, behavior and results. Each level answers a question: whether the participants liked it, whether they learned, whether they changed their practice and whether there was a benefit for patients or for the institution.

Do I need to measure all four levels in every program?

No. Choose the levels that make sense for the objective and for the available resources, and state clearly what was not measured. A short program can be well evaluated at levels one and two.

Does high satisfaction mean the course worked?

Not necessarily. Satisfaction shows that the experience was well received, but it does not guarantee learning or a change in practice.

What is the main criticism of the model?

The assumption that the levels form a causal chain and the focus on results without looking at the context and at unplanned effects. That is why the model works better combined with program evaluation approaches.

What is the New World Kirkpatrick?

An update that starts planning from the desired results and adds critical behaviors, reinforcement factors and intermediate indicators.

What model should I use instead of Kirkpatrick?

It depends on the question. The logic model and CIPP help to understand how and why a program works; experimental designs answer effect questions better. Many programs combine more than one.

Program evaluation checklist

• Define the result that would justify the program before planning the activities.
• Write the program theory: why the activities would lead to the result.
• Choose a realistic indicator for each level you are going to measure.
• Use instruments with evidence of validity to measure learning.
• Measure the baseline or combine a comparison group.
• Ask about unplanned effects.
• Report clearly what was not measured.
• Use the results to adjust the program for the next class.

Want a daily clinical reasoning activity for your class? Play the case of the day on NEXO.

References

1. Yardley S, Dornan T. Kirkpatrick's levels and education 'evidence'. Med Educ. 2012. PubMed 22150201
2. Allen LM, Hay M, Palermo C. Evaluation in health professions education-Is measuring outcomes enough?. Med Educ. 2022. PubMed 34463357
3. Moreau KA. Has the new Kirkpatrick generation built a better hammer for our evaluation toolbox?. Med Teach. 2017. PubMed 28649887
4. Curran VR, Fleet L. A review of evaluation outcomes of web-based continuing medical education. Med Educ. 2005. PubMed 15910431
5. Turkeshi E, Michels NR, Hendrickx K, Remmen R. Impact of family medicine clerkships in undergraduate medical education: a systematic review. BMJ Open. 2015. PubMed 26243553
6. Phillips JL, Heneka N, Bhattarai P, Fraser C, Shaw T. Effectiveness of the spaced education pedagogy for clinicians' continuing professional development: a systematic review. Med Educ. 2019. PubMed 31144348
7. Moreau KA, Eady K, Sikora L, Horsley T. Digital storytelling in health professions education: a systematic review. BMC Med Educ. 2018. PubMed 30200945
8. Frye AW, Hemmer PA. Program evaluation models and related theories: AMEE guide no. 67. Med Teach. 2012. PubMed 22515309
9. Belfield C, Thomas H, Bullock A, Eynon R, Wall D. Measuring effectiveness for best evidence medical education: a discussion. Med Teach. 2001. PubMed 11371293
10. Cook DA, Hatala R, Brydges R, Zendejas B, Szostek JH, Wang AT, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011. PubMed 21900138
11. Brydges R, Hatala R, Zendejas B, Erwin PJ, Cook DA. Linking simulation-based educational assessments and patient-related outcomes: a systematic review and meta-analysis. Acad Med. 2015. PubMed 25374041
12. Ilgen JS, Ma IW, Hatala R, Cook DA. A systematic review of validity evidence for checklists versus global rating scales in simulation-based assessment. Med Educ. 2015. PubMed 25626747