Kirkpatrick model: how to assess a teaching program
Table of contents
The clinical reasoning course ended, and the coordination asks for a report. The first idea is the usual one: a form on the last day, with scores from one to five for the instructor, the material and the schedule. The scores come back high, and everyone is satisfied. Still, nobody knows whether the class began to reason better, whether anything changed in the clerkship or whether any patient was seen differently because of the course.
NEXO is a daily medical diagnosis game: one clinical case a day, six progressive clues, and ICD-10 proximity feedback across five levels.
The question "did the course work?" has layers, and the Kirkpatrick model is the best-known way to separate them. This guide explains the four levels, shows what the reviews found when they applied the model to medical education, presents the most serious criticisms and proposes a roadmap for designing the evaluation of a course, a clerkship or a student league.
Key points
- The Kirkpatrick model organizes program evaluation into four levels: reaction, learning, behavior and results.
- In medical education reviews, most studies stay at the first two levels, and few reach behavior or the patient.
- A critical analysis concluded that the model works well only in simple designs, a condition met by about one fifth of the reviews in the field.
- The main criticisms: focus on the result without explaining why, the assumption that one level leads to the next and the idea that the highest level is worth more.
- A revised version, the New World Kirkpatrick, tries to answer some of these criticisms, but evaluation still requires questions about the process.
What the Kirkpatrick model is
An origin outside medicine
The model was created by Donald Kirkpatrick to evaluate training in industry and only later reached medical education (Yardley and Dornan, 2012). The central idea is simple: training can be evaluated in layers ever more distant from the classroom, from what the participant thought of it to the effect on the organization.
Today the model appears everywhere. A 2022 review described its use in the health professions as broad and growing, and showed that it entered research quality tools and review guidelines, such as MERSQI and BEME (Allen et al., 2022).
Evaluating the program, not just the student
Evaluating a program is different from evaluating a student. The exam measures what a person knows; program evaluation asks whether the set of classes, clerkships, materials and activities produced the intended effect, and why. The distinction matters because a program can pass every student and still fail at the goal that justified it.
When evaluating a tool is another conversation
Evaluating an app or a game, as the guide on gamification in medical education does, looks at an isolated piece. The Kirkpatrick model is more often used for the whole course, in which the tool is only one of the ingredients.
The four levels in medical education
Level 1: reaction
What the participants thought: satisfaction, perceived relevance, willingness to recommend. It is the easiest level to measure, with a form on the last day. It is also the one that says least about learning, because enjoying a class and learning from it are different things.
Level 2: learning
What changed in knowledge, skill or attitude. Here come pre and post tests, practical stations and attitude questionnaires. In a clinical reasoning course, for example, it would mean comparing performance on written cases at the start and at the end, with the same type of case and the same scoring criteria.
Level 3: behavior
What the participant began to do differently at work. It is harder to measure because it requires observing practice: chart records, direct observation, assessment by preceptors. The guide on patient encounter simulation shows observation instruments that can serve here.
Level 4: results
The effect on patients, services or system indicators. It is the level most cited as ideal and the rarest to reach, because many other things change between the course and the patient.
.jpg){kind=link}
What the reviews found
Lots of reaction, few patients
A review of continuing medical education over the internet classified the outcomes assessed by the Kirkpatrick levels and concluded that most of the research was based on data on participant satisfaction (Curran and Fleet, 2005). There was little research showing a change in clinical practice, and no study showed an effect on patient or health outcomes (Curran and Fleet, 2005).
The pattern repeated itself in undergraduate education. A systematic review of family medicine clerkships included 64 articles: 26 studies assessed at level 1, 26 at level 2 and 6 at level 3 (Turkeshi et al., 2015). The authors considered the quality of the research limited and called for standardized and validated instruments (Turkeshi et al., 2015).
When assessment goes further
A review of spaced education for practicing physicians found 17 studies, with 2,701 professionals, and all were assessed using a modified version of the four levels (Phillips et al., 2019). Ten studies showed a gain in knowledge, seven reported a change in clinical behavior and only two reported improvement in patient outcomes (Phillips et al., 2019).
The funnel design is typical: many studies at the start, few at the end. The explanation is partly practical, because measuring patient outcomes requires time, a sample and access to data that a course rarely has.
An example with the revised version
A review of digital narratives in teaching the health professions used the New World Kirkpatrick and included only studies with an outcome at levels 2 or 3 (Moreau et al., 2018). Of 16 articles with data, 14 dealt with learning and only two also measured a change in behavior (Moreau et al., 2018). Even with the revised model, level 3 remained rare.
The criticisms of the model
It works for simple designs
A critical analysis concluded that the levels carry so many implicit assumptions that they only work well in relatively simple teaching designs, with short-term results and beneficiaries that go beyond the students (Yardley and Dornan, 2012). According to the authors, these conditions are met by perhaps one fifth of evidence reviews in medical education (Yardley and Dornan, 2012).
Outside these conditions, using the levels as a hierarchy of quality adds little, and the quality of a piece of evidence depends as much on the use one wants to make of it as on any objective measure (Yardley and Dornan, 2012).
Four underlying problems
The 2022 review summarized the main criticisms of the model (Allen et al., 2022):
- The focus is the result, and the factors that influence the result are left out.
- A positive causal link between the levels is assumed, as if satisfaction led to learning and learning to a change in practice.
- The highest level is assumed to be the most important.
- The unintended effects of the program remain invisible.
The authors add that the use of the model by tools such as MERSQI and by organizations such as BEME and the WHO feeds the myth that it is the gold standard for evaluating programs (Allen et al., 2022). Their proposal is to move from measuring results to investigating how and why they happen (Allen et al., 2022).
The New World Kirkpatrick
In response to these criticisms came the New World Kirkpatrick, a revised version for complex settings (Moreau, 2017). The article that presents it highlights three criticisms of the original model and discusses how the new version tries to answer them, inviting teachers to test it and to investigate its strengths and weaknesses (Moreau, 2017).
In practice, the new version suggests starting from the desired result and designing the program backwards, as well as looking at what happens between the levels. It improves the map, but the question about the process is still up to whoever does the assessment.
How to design a program assessment
Start with the question
Before choosing an instrument, define what the program promises. A clinical reasoning course can promise that the class will build more complete lists of hypotheses, as described in the guide to differential diagnosis. That promise indicates the level that matters most and the type of data needed.
Choose one indicator per level
A minimal design can have one indicator per level:
| Level | Question | Example of indicator |
|---|---|---|
| Reaction | Did the class find the course relevant? | Short questionnaire at the end of each module |
| Learning | Did reasoning improve? | Cases written before and after, with the same scoring rubric |
| Behavior | Did practice change? | Observation by preceptors or review of records during the clerkship |
| Results | Did anything change for the service or the patient? | Indicator chosen together with the service, when there is access |
Level 4 can remain a long-term goal. Better to state this in the report than to invent a link that the data do not support.
Also ask how and why
To answer the criticism of focusing only on the result, add questions about the process (Allen et al., 2022): who actually took part, which parts worked, what got in the way, what effects nobody expected. Short interviews and focus groups serve this purpose.
Use feedback as data
The open comments on a form are process data. Reading these responses methodically, as the guide to feedback in clinical cases suggests, helps explain why one level rose and another stayed flat.
Follow up over time
One measurement on the last day shows the immediate effect. A second measurement months later shows what remained, a point linked to retention of medical knowledge. Programs of active learning and case-based learning benefit from this follow-up, because the intended effect usually shows up in the clerkship, not in the classroom.
Performance data and research at NEXO
NEXO records information that can serve as process indicators in an evaluation: the distribution of correct answers by clue shows at what point each player usually reaches the diagnosis, and the ICD-10 proximity in five levels shows how far each guess landed. The case of the day arrives in up to six clues, and the group ranking allows a class to be followed. These data describe performance in the game; what they say about practice is a research question.
The cases are fictional with a real clinical presentation, and the guide what NEXO is explains the whole match. How the modes are unlocked is on the NEXO economy page.
Researchers in medical education who want to study the game with their own evaluation design can propose studies through NEXO Researcher. If you coordinate a medical school, a student league or a residency program, get to know NEXO Closer. Physicians who review cases take part through NEXO Curator, and those who create health content on social media can take part through NEXO Creators.
Frequently asked questions
What is the Kirkpatrick model?
It is a model for evaluating training in four levels: reaction, learning, behavior and results.
What are the four Kirkpatrick levels?
Reaction (what the participants thought), learning (what they learned), behavior (what changed in practice) and results (the effect on the service or the patient).
Is level 4 always the most important?
The 2022 review lists this assumption as one of the criticisms of the model (Allen et al., 2022). The most important level depends on the program's question.
What is the New World Kirkpatrick?
A revised version of the model that tries to answer the criticisms of the original and deal with complex teaching environments (Moreau, 2017).
Why do so many studies stay at levels 1 and 2?
Because measuring behavior and outcome requires observation in practice, more time and access to data that many courses still fail to plan for.
Can NEXO be part of an evaluation?
Performance data in the game can serve as a process indicator, and study proposals go through NEXO Researcher.
Program evaluation checklist
- Write the promise of the program in one sentence.
- Choose the level that answers that promise.
- Define one indicator for each level you will measure.
- Measure before and after, with the same criterion.
- Ask how and why, beyond how much.
- Look for unexpected effects, good and bad.
- Repeat the measurement months later.
Want to see reasoning in data? Play the case of the day on NEXO.
References
- Yardley S, Dornan T. Kirkpatrick's levels and education 'evidence'. Med Educ. 2012. https://pubmed.ncbi.nlm.nih.gov/22150201/
- Allen LM, Hay M, Palermo C. Evaluation in health professions education-Is measuring outcomes enough? Med Educ. 2022. https://pubmed.ncbi.nlm.nih.gov/34463357/
- Moreau KA. Has the new Kirkpatrick generation built a better hammer for our evaluation toolbox? Med Teach. 2017. https://pubmed.ncbi.nlm.nih.gov/28649887/
- Curran VR, Fleet L. A review of evaluation outcomes of web-based continuing medical education. Med Educ. 2005. https://pubmed.ncbi.nlm.nih.gov/15910431/
- Turkeshi E, Michels NR, Hendrickx K, Remmen R. Impact of family medicine clerkships in undergraduate medical education: a systematic review. BMJ Open. 2015. https://pubmed.ncbi.nlm.nih.gov/26243553/
- Phillips JL, Heneka N, Bhattarai P, Fraser C, Shaw T. Effectiveness of the spaced education pedagogy for clinicians' continuing professional development: a systematic review. Med Educ. 2019. https://pubmed.ncbi.nlm.nih.gov/31144348/
- Moreau KA, Eady K, Sikora L, Horsley T. Digital storytelling in health professions education: a systematic review. BMC Med Educ. 2018. https://pubmed.ncbi.nlm.nih.gov/30200945/