Error log: how to find your study gaps

After getting a case wrong, the first reaction is usually to look at the right answer and move on. Two questions pay off more: what was missing and why. The first shows the content; the second shows the type of gap, and that is the one that tells you what to study next.

NEXO is a daily medical diagnosis game: one clinical case a day, six progressive clues, and ICD-10 proximity feedback across five levels.

This text is a guide to an error notebook for medical students: why it is worth looking at mistakes, how to record them, how to separate the type of error, how to review in a cycle and how to read which clue in the case tripped you up. The effect of testing yourself is in the text on reviewing by playing, and how to turn mistakes into spaced cards, in the guide on spaced repetition with Anki.

Main points

• Trying to answer and getting it wrong, followed by the right answer, helps you learn more than just reading the answer.
• The feedback after an error is what weighs most; uncertain correct answers also deserve review.
• Self-assessment without data usually fails; the notebook replaces impression with a record.
• Separating the type of error, such as lack of knowledge, reasoning or attention, points to what to do next.
• Reviewing the notebook in a cycle, testing again, closes the gap; rereading the notebook helps less.
• In NEXO, the up to six clues and the proximity in ICD-10 show at what point in the case the reasoning changed.

Contents

• Why look at mistakes
• Recording
• Type of error: knowledge, reasoning, attention
• Reviewing in a cycle
• Reading where you went wrong in the clues
• Frequently asked questions
• Error notebook checklist

Why look at mistakes

The error notebook starts from a simple idea: a mistake is data. It shows where the gap is more precisely than the feeling that a topic "is weak". The studies below did not test error notebooks; they show why getting it wrong and correcting helps, and why trusting only an impression is risky.

Trying and getting it wrong also teaches

In six experiments, participants tried to answer before seeing the answer or read question and answer together (Kornell et al., 2009). The questions were made so that no one could answer them, such as general knowledge questions invented by the researchers (Kornell et al., 2009). In other experiments, participants saw a word and tried to guess a not very obvious associated word, and the rare correct answers were excluded from the analysis (Kornell et al., 2009). In both types of material, unsuccessful attempts improved later learning (Kornell et al., 2009). In the test condition, the person tried to answer before seeing the answer; in the reading condition, they saw both together (Kornell et al., 2009). The authors suggest that taking difficult tests, instead of avoiding errors, may be one of the keys to learning (Kornell et al., 2009).

For those who study with cases, the message works as a practical suggestion: treat a mistake in a game or a question as part of studying, as long as you see the answer and understand why.

Feedback weighs more after an error

In an experiment with 258 people learning word pairs in an unknown language, giving the right answer after a wrong answer increased retention on the final test, 1 week later, by 494% (Pashler et al., 2005). Participants saw the material 2 times and were then tested on each item several times, with or without feedback, depending on the group (Pashler et al., 2005). Feedback with the right answer also improved performance during the learning session itself (Pashler et al., 2005). Feedback after correct answers, however, made little difference, even when the person was not confident (Pashler et al., 2005). Another experiment, already mentioned in the text on study routine, found a benefit of feedback on correct answers made with little confidence. For the notebook, as a practical suggestion: errors are the priority, and uncertain correct answers deserve at least a look. The type of feedback also weighs, as discussed in the text on clinical reasoning games.

Errors made with high confidence

In a memory study, the errors made with the most confidence were the ones most corrected on a new test, contrary to what the authors expected (Butterfield and Metcalfe, 2001). It is a laboratory result, with simple material: in it, the error that surprised was the one most corrected afterward. What the research says about getting it wrong and correcting also appears in the text on case-based learning.

Self-assessment fails

In a systematic review with physicians and residents, of 20 comparisons between self-assessment and external assessment, 13 showed a weak, null or inverse relationship, and 7 showed a positive association (Davis et al., 2006). The search found 725 articles, of which 17 met all the criteria, with widely varying methods and rigor (Davis et al., 2006). Several studies found the worst accuracy among the least skilled and the most confident (Davis et al., 2006). The authors conclude that physicians have a limited ability to self-assess accurately and suggest giving more weight to external assessment (Davis et al., 2006).

The review excluded studies with students, so it does not directly measure those in undergraduate training (Davis et al., 2006). Even so, it helps to understand the role of the notebook: it replaces the question "do I think I know?" with data about what you got wrong. The relationship between confidence and correctness in students is in the text about clinical reasoning games.

Log

Logging means turning each error into a line you can reread, test and count. The form matters less than the consistency: a paper notebook, a spreadsheet or a notes file all work.

What to write down

As a practical suggestion, each line of the notebook can have these columns:

Column	What to write
Date and source	Case, question, exam or shift
What you answered	The diagnosis or answer you gave
What it was	The right answer
Where it turned	The clue, the data or the word in the question that should have changed your answer
Type of error	Knowledge, reasoning or attention
Confidence	High, medium or low, before seeing the answer
Next step	What to study, the card to create or the question to test later
Reviews	The dates you tested again and the result

The "Where it turned" column is the most useful and the hardest. It forces you to go back to the case and find the point where the reasoning should have changed. The "Confidence" column shows, over time, whether you get more things wrong when you are sure or when you are in doubt.

Three example lines

As a practical suggestion, see how notebook lines look, without the clinical content, just the structure:

Source	Where it turned	Type	Confidence	Next step
Daily case, cardiovascular area	The clue with the test that separated two diseases of the same group	Reasoning	High	Redo the case listing findings for and against each hypothesis
Exam question	A piece of data in the question you did not read	Attention	Medium	Reread long questions marking the key data before the options
Discussion during the rotation	A mechanism you could not explain	Knowledge	Low	Study the mechanism and write a question about it

Notice that the same disease can appear in lines of different types. What the notebook keeps is the reason for the error, not just the topic.

Writing helps you reflect

The AMEE guide on reflection in medical education lists several ways to promote reflection, from written reflective diaries to critical incident reports (Sandars, 2009). The guide highlights guided reflection, with a mentor or tutor who questions assumptions, and the role of feedback in strengthening reflection (Sandars, 2009). According to the guide, the choice of format depends on the goal, but also on who is reflecting, because each person has a preferred style (Sandars, 2009). The same guide acknowledges that there is little research evidence that reflection improves the quality of care (Sandars, 2009).

A systematic review on reflection in the health professions reached a similar conclusion: the evidence supporting reflection interventions in the curriculum remains largely theoretical (Mann et al., 2009). The authors also note that the literature is spread across several fields and that it is not clear which approaches are effective or have impact (Mann et al., 2009). The error notebook, therefore, is a recommended practice based on principles, not a method with its own trial. If you research this type of question, NEXO Researcher is the program for proposing studies with NEXO.

A little, always

As a practical suggestion, write it down the same day, while the case is fresh, and one line per error. A notebook that is too long becomes a pile nobody rereads. If time is short, log only the errors that surprised you and the correct answers you guessed.

Type of error: knowledge, reasoning, attention

Not every mistake calls for the same response. Whoever did not know the content needs to study; whoever knew it and did not use it needs to train reasoning; whoever read it wrong needs to change the way they read. Separating the type of mistake is what turns the notebook into a map, and not just a list.

Three categories for the notebook

As a practical suggestion, use three simple categories. This division comes from the NEXO Editorial, designed for the notebook; the studies cited further below use their own categories.

Type	Mark in the notebook	What to do afterwards
Knowledge	You did not know the fact, the disease or the mechanism	Study the topic and create a question to test yourself later
Reasoning	You knew it, but did not connect the fact to the diagnosis, or stopped at the first hypothesis	Redo the case justifying each hypothesis with findings for and against
Attention	You read it wrong, skipped a fact or swapped a term	Change the way you read: underline key facts, reread the question before answering

These categories are a simplification for personal use. The studies with students use more detailed classifications, and they help refine what each category hides.

Attention errors

An attention error is easy to dismiss as a "silly mistake". As a practical suggestion, when the notebook line is about attention, write down exactly what you failed to see: a number, a negation in the question, a fact from the history, a clue read in a hurry. Over time, the notebook shows whether there is a pattern, such as skipping facts at the end of long questions, and the pattern suggests the change: read the whole question before the options, or reread the clues before making a guess.

What studies find in students

In a study with 88 students who solved cases with dyspnea on an electronic platform, the researchers identified eight causes of diagnostic error (Braun et al., 2017). The most common were lack of diagnostic skill, at 24%, and inadequate knowledge base, at 16% (Braun et al., 2017). Wrong generation of the case context, at 15%, and premature closure, at 10%, also weighed in (Braun et al., 2017). The causes varied from case to case (Braun et al., 2017). In all, there were 704 resolutions, and after each one the students stated the diagnosis and explained how they reached it (Braun et al., 2017). The "Where it went" column of the notebook asks for the same exercise: explain how you reached the answer.

The authors conclude that insufficient skill and knowledge, along with wrong generation of the context, are the main problems in students' reasoning (Braun et al., 2017). What is known about premature closure and biases is in the text about differential diagnosis.

Knowing is not enough

In a study with 21 students who solved cases thinking out loud, the use of different categories of diagnostic knowledge was not associated with more correct answers (Kiesewetter et al., 2016). The authors conclude that, to solve the case, cognitive actions, such as what the student does with the information, seem more important than the categories of knowledge (Kiesewetter et al., 2016). In the same study, more metacognitive activity, that is, thinking about one's own reasoning, went together with applying more conceptual knowledge and with more time on the case (Kiesewetter et al., 2016). The authors raise the hypothesis that this reduces premature closure (Kiesewetter et al., 2016). It is a small study. As a practical suggestion, a mistake classified as "reasoning" calls for training with cases, in addition to reading.

Getting it right for the wrong reason

In a study with 148 students and internal medicine cases, the researchers analyzed the justifications for correct diagnoses: the correct diagnosis appeared 1,135 times in 2,080 processes (Braun et al., 2019). Among the correct answers, 92% had a correct justification, 7% had a wrong justification and 1% was a guess (Braun et al., 2019). The wrong justifications came mainly from lack of knowledge of pathophysiology, at 50%, and lack of diagnostic skill, at 30% (Braun et al., 2019).

The authors summarize that about 1 in every 14 correct diagnoses rested on a wrong explanation (Braun et al., 2019). For the notebook, as a practical suggestion, also record the correct answers where you cannot explain why.

A taxonomy for teachers

A group of clinical teachers built, in action research cycles, a guide with five categories of clinical reasoning difficulty and indicators for each one (Audétat et al., 2013). There were 8 reflective cycles with the group of teachers, testing the ideas in their own clinical teaching (Audétat et al., 2013). The indicators can change depending on the type of supervision (Audétat et al., 2013). The goal is to help the preceptor identify the specific difficulty and offer a tailored correction (Audétat et al., 2013). For the student, the idea is the same as the notebook: give a name to the type of difficulty before deciding what to do. Taking the notebook to the conversation with the preceptor combines your record with the outside view.

The notebook in the clerkship

In the clerkship, a good part of the errors happens in front of the patient and the preceptor. As a practical suggestion, write down at the end of the day what the preceptor corrected and why, in the same format as the notebook. How to take doubts from the bedside to study is in the text about how to study in the clerkship, and how to present the case and hear the correction, in the text about clinical reasoning in the clerkship.

Reviewing in a cycle

A notebook that is only written in becomes a dead archive. The value is in returning to it by testing, not by rereading.

Test the error, not reread the error

As a practical suggestion, turn the "Next step" column into a question. During review, cover the answer and try to answer before looking. Testing yields more than rereading, as the text about reviewing by playing shows. If the gap is facts you need to keep for months, take the question to a card, as the guide on spaced repetition describes.

A possible cycle

As a practical suggestion, the cycle below can be adjusted to your stage and your time:

Moment	What to do with the notebook
On the day of the error	Record the line and read the explanation
A few days later	Test the week's questions again, without looking
At the end of the week	Count the types of error and choose one topic to study
Weeks later	Redo similar cases on the topic you missed most
Before the exam	Test the lines you missed more than once

Count the errors

After a few weeks of the notebook, count. How many errors of knowledge, reasoning and attention? In which areas? With what confidence? As a practical suggestion, use the count to decide the next week's study: many knowledge errors in one area call for studying the topic; many reasoning errors call for more cases; many attention errors call for changing the way you read.

Before the exam

As a practical suggestion, in the weeks before the exam, the notebook becomes the review script: start with the lines that remain open, then the areas with the most errors, and finally the correct answers by guessing. Testing the lines, instead of rereading, follows the principle described in the text about reviewing by playing.

Closing the line

A line only closes when you get the question right, days later, and can explain why. If you miss again, the line remains open and returns in the next cycle. If it is a card you always miss, the guide on spaced repetition shows what to do.

Reading where you went wrong in the clues

In a case that arrives little by little, the error has a place: the clue where you could have changed your mind and did not, or the clue where you changed to the wrong side. Finding that place is what separates "I missed the case" from "I did not connect the fever to the murmur", and it is that second sentence that goes into the notebook.

Up to six clues, with feedback on each guess

In NEXO, each case arrives in up to six clues, one at a time, and each guess shows the proximity in ICD-10 at five levels. On the third clue the area of the case appears and, on the fifth, the ICD range. This feedback on each guess leaves a trail: on which clue you came close, on which clue you moved away and on which clue you got it right.

What to look at at the end of the case

As a practical suggestion, at the end of each case, answer three questions and write them in the notebook:

1. On which clue should I have changed my mind? This one goes in the "Where it turned" column.
2. Was I close in ICD-10? Reaching the right group and missing the disease suggests a gap in differentiation; being far suggests that the reasoning started from the wrong area.
3. Why did I not change? I did not know what the clue meant, I knew and did not connect it, or I did not read carefully?

The explanation at the end of the case closes the cycle of testing and correcting. How to turn the clue that made you change your mind into a study plan is in the text about reviewing by playing.

Daily, Archive and your notebook

The Daily mode brings one case per day, the same for everyone, and renews at midnight: a new possible line in the notebook every day. The Archive gathers the previous daily cases, good for going back to an area where the notebook shows many errors. The streak records the days you played, and the achievements mark the path. To understand the limits of each mode, see the NEXO economy page. To turn the point you missed into a card, the Anki button in NEXO is described in the spaced repetition guide.

Discussing the error with the class

Since the Daily case is the same for everyone, the class can compare where each person turned. The AMEE guide recalls that guided reflection, with someone who questions the assumptions, is important (Sandars, 2009). Explaining your own error out loud to a colleague is a simple way to do this.

To research, bring to the class and create

If you research learning and want to study errors and gaps with NEXO, check out NEXO Researcher. To bring NEXO to the medical school, the student league or the residency, there is NEXO Closer. Doctors who review the cases of their specialty take part through NEXO Curator, and those who create health content on social media can take part through NEXO Creators.

Common mistakes with the notebook

• Writing down only the right answer: without the "why", the line shows the content, but not the type of gap.
• Rereading instead of testing: trying to answer before seeing the answer helps more (Kornell et al., 2009).
• Ignoring the correct guesses: part of the correct diagnoses relies on wrong reasoning (Braun et al., 2019).
• Trusting the impression that you already know: self-assessment is usually not very accurate (Davis et al., 2006).
• Classifying everything as "lack of study": diagnostic skill and case context also matter (Braun et al., 2017).
• Writing too much: one line per error, on the same day, is easier to keep up.

Frequently asked questions

What is an error notebook?

It is a record of the errors you make in cases, questions and exams, with what you answered, what it was, why you got it wrong and what to do next. It serves to find study gaps and to test again what you got wrong.

Does an error notebook work?

The notebook itself has not yet been tested in a trial. What studies show is that trying to answer and then seeing the answer helps learning (Kornell et al., 2009) and that feedback after an error greatly increases retention (Pashler et al., 2005). The notebook is an organized way to apply this.

Error notebook or Anki?

The two complement each other. The notebook keeps the reason for the error and helps decide what to study; the card keeps a fact accessible over the months. As a practical suggestion, the notebook line comes first and, when the gap is a fact you need to remember, it becomes a card, as described in the guide on spaced repetition.

How much time should you spend on the notebook?

The ideal time still has no good evidence. As a practical suggestion, keep one line per error, written on the same day, and save the review for the testing moments of the week.

Paper or digital?

It does not matter. What matters is having the right columns and going back to the notebook to test. A spreadsheet makes it easier to count the types of error.

Should I write down the correct answers?

The correct guesses and the correct answers you cannot explain, yes. In a study with students, 7% of the correct diagnoses had wrong reasoning (Braun et al., 2019).

How do you know if the error was knowledge or reasoning?

Ask yourself if you knew the content. If you did not know it, it is knowledge. If you knew it and did not use it, or stopped too early, it is reasoning. In studies with students, lack of diagnostic skill and lack of knowledge are among the most common causes of error (Braun et al., 2017).

Is there evidence for reflecting on errors?

The evidence on reflection in health education is still largely theoretical (Mann et al., 2009). Reflection is recommended as a practice, with the caveat that studies on the effect on quality of care are lacking (Sandars, 2009).

How does NEXO help find gaps?

Each case arrives with up to six clues, and each guess shows the proximity in ICD-10. This lets you see at which clue you changed your mind and whether you stayed close to the right group of diseases. The Daily mode gives one new case per day, and the Archive lets you return to the areas where you make the most mistakes.

Does the notebook work in the basic cycle?

It does. As a practical suggestion, in the basic cycle use the same columns for mechanisms and pathways you could not explain, with a question as the next step. In the clinical cycle and the internship, the notebook also receives errors from cases and from the differential. How to link mechanism and symptom is in the text about from the basic cycle to the clinic.

How often should you review the notebook?

The ideal interval for the notebook still lacks good evidence. Testing again after a few days and returning to the topic over the weeks follows the principle of spaced repetition, described in the guide on spaced repetition.

Checklist for the error notebook

1. Do you write down the error on the same day?
2. Do you record where you turned, the clue or the data that should have changed the answer?
3. Do you classify the type of error: knowledge, reasoning or attention?
4. Do you note your confidence before seeing the answer?
5. Do you record the correct guesses and the ones you cannot explain?
6. Do you turn each line into a question to test later?
7. Do you test again instead of just rereading?
8. Do you count the types of error and choose the week's study from that?
9. Do you close the line only when you get it right and explain it?
10. Do you take the notebook to the conversation with the preceptor or the class?

References

1. Kornell N, Hays MJ, Bjork RA. Unsuccessful retrieval attempts enhance subsequent learning. J Exp Psychol Learn Mem Cogn. 2009. https://pubmed.ncbi.nlm.nih.gov/19586265/
2. Pashler H, Cepeda NJ, Wixted JT, Rohrer D. Study on when feedback helps learn words (original title in English on PubMed). J Exp Psychol Learn Mem Cogn. 2005. https://pubmed.ncbi.nlm.nih.gov/15641900/
3. Butterfield B, Metcalfe J. Errors committed with high confidence are hypercorrected. J Exp Psychol Learn Mem Cogn. 2001. https://pubmed.ncbi.nlm.nih.gov/11713883/
4. Davis DA, Mazmanian PE, Fordis M, et al. Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA. 2006. https://pubmed.ncbi.nlm.nih.gov/16954489/
5. Sandars J. The use of reflection in medical education: AMEE Guide No. 44. Med Teach. 2009. https://pubmed.ncbi.nlm.nih.gov/19811204/
6. Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. Adv Health Sci Educ Theory Pract. 2009. https://pubmed.ncbi.nlm.nih.gov/18034364/
7. Braun LT, Zwaan L, Kiesewetter J, Fischer MR, Schmidmaier R. Diagnostic errors by medical students: results of a prospective qualitative study. BMC Med Educ. 2017. https://pubmed.ncbi.nlm.nih.gov/29121903/
8. Kiesewetter J, Ebersbach R, Tsalas N, et al. Knowledge is not enough to solve the problems: The role of diagnostic knowledge in clinical reasoning activities. BMC Med Educ. 2016. https://pubmed.ncbi.nlm.nih.gov/27881121/
9. Braun LT, Borrmann KF, Lottspeich C, et al. Guessing right: whether and how medical students give incorrect reasons for their correct diagnoses. GMS J Med Educ. 2019. https://pubmed.ncbi.nlm.nih.gov/31844657/
10. Audétat MC, Laurin S, Sanche G, et al. Clinical reasoning difficulties: a taxonomy for clinical teachers. Med Teach. 2013. https://pubmed.ncbi.nlm.nih.gov/23228082/