Diagnostic confidence calibration: when to answer

There is a moment in every clinical case when you decide you know enough. Sometimes that feeling comes early and is right. Other times it comes early and is wrong, or it takes so long that you order tests you did not need. The underlying question is the same: does your confidence match your accuracy?

NEXO is a daily medical diagnosis game: one clinical case a day, six progressive clues, and ICD-10 proximity feedback across five levels.

This guide explains what diagnostic calibration is, what studies find about confidence, time and amount of information, and how to use the number of data points you needed to answer as a picture of your doubt.

Key points

• Calibration is the distance between how much you trust and how much you get right.
• Answering fast is not a synonym for getting it wrong: in several studies, accuracy goes together with a fast answer.
• Seeking the right data matters more than seeking a lot of data.
• The feeling of certainty can last longer than knowledge, and that leaves the student overconfident over time.
• Recording when you answered and checking the result helps you see your own pattern.

Summary

• What diagnostic calibration is
• Time, speed and accuracy
• How much data you need
• Why calibration is hard to improve
• How to train calibration
• The clue in which you answer in NEXO
• Frequently asked questions
• Calibration checklist

What diagnostic calibration is

Confidence and accuracy are different measures

Calibration is how much your confidence in a diagnosis matches the real chance of it being right. Someone who says they are very sure and gets it right rarely is overconfident; someone who gets it right often and doubts everything is underconfident. Both mistakes matter: the first closes the investigation early, the second spends time and tests. The guide on diagnostic uncertainty shows why these two measures separate so much.

Overconfidence and underconfidence in students

In a general surgery exam with students about to graduate in Argentina, about 12% showed significant overconfidence relative to their own accuracy (Borracci and Arribalzaga, 2018). Underconfidence was less common, in 8.3% of the students, and appeared more among those who performed better (Borracci and Arribalzaga, 2018). The best students also showed less overconfidence, which suggests that knowledge and calibration grow together (Borracci and Arribalzaga, 2018).

Time, speed and accuracy

Those who get it right tend to answer faster

A common idea is that diagnostic error comes from haste. In a study with Canadian recent graduates right after the licensing exam, who diagnosed a series of written cases, the correlation between response time and accuracy was -0.54: the faster the answer, the higher the accuracy (Sherbino et al., 2012). The relationship held when the authors controlled for the knowledge measured in the licensing exam (Sherbino et al., 2012).

This does not mean that answering fast makes you get it right. It means that those who know the disease well recognize it early. Speed is a consequence of knowledge, and that is why a fast answer can be well calibrated.

Induced confidence barely changed accuracy

In an experiment with internal medicine residents, one group was led to believe that the cases were easy and the other that they were hard (Al-Maghrabi et al., 2024). The first group became more confident, with a mean of 0.75 versus 0.61 on a scale up to 1, but the time spent and the diagnostic accuracy differed little between the groups (Al-Maghrabi et al., 2024). The authors suggest studying overconfidence more as a personal trait than as something that changes with the situation (Al-Maghrabi et al., 2024).

How much data you need

The right data is worth more than a lot of data

In a study with family physicians facing difficult diagnostic problems, participants could ask for more information about the patient before answering (Kostopoulou et al., 2008). Asking for critical clues, the ones that separate the relevant differential diagnoses, predicted accuracy in most scenarios, and residents asked for more information than experienced physicians (Kostopoulou et al., 2008). Experience, in the same study, barely mattered for accuracy (Kostopoulou et al., 2008).

The lesson for calibration is direct. The useful question is "does the data I have separate my hypotheses?", not "do I already have enough data?". When the answer is yes, confidence has a basis. When it is no, one more well-chosen piece of data is worth more than the feeling of safety.

Answering early and answering late

Pattern	What it usually indicates	What to do
Answers early and gets it right	Known disease, quick recognition	Keep it; check whether the explanation matches your reasoning
Answers early and gets it wrong	Confidence ahead of knowledge	Ask which piece of data would separate the two hypotheses before answering
Answers late and gets it right	More doubt than necessary or a difficult case	Review at what point the answer was already clear
Answers late and gets it wrong	Gap in knowledge about the disease	Go back to the content before training more cases

Why calibration is hard to improve

Confidence lasts longer than knowledge

In a study with students learning to read ankle X-rays, each answer came with a degree of certainty, "certain" or "probably", and with immediate feedback (Pusic et al., 2015). Two weeks later, the average accuracy fell from 59% to 53%, while the frequency of "certain" answers stayed stable, which left the students relatively overconfident (Pusic et al., 2015). The authors conclude that the ability to monitor one's own performance needs to be measured separately, because it does not follow clinical skill (Pusic et al., 2015).

Feedback helps, but with limits

In a trial with family medicine residents, seeing the correct diagnosis after each difficult case barely improved calibration compared with the group that received no feedback (Kuhn et al., 2022). In the cases they got right, the group that received feedback ended up less calibrated, and the authors suggest that, after seeing their own mistakes, the residents lowered their confidence even when they were right (Kuhn et al., 2022).

These results show that calibrating is not just receiving the answer. It is looking at the pattern across many cases: in which situations you tend to be right when you feel confident and in which you tend to fool yourself. The guide on tracking your progress offers a way to build this record.

How to train calibration

Before answering

1. State your confidence on a number from 0 to 10 before seeing the answer.
2. Name the second hypothesis and the piece of data that would rule it out.
3. Ask whether the missing data changes the decision; if it does not, answer.

Signs that you answered too early

• You got the area right, but not the disease, and you cannot say what separates them.
• Your hypothesis explains the main complaint, but leaves out a finding from the physical exam.
• Your confidence was high and the error went to another ICD-10 chapter.

After the result

Write down your confidence and the result side by side. After a few weeks, separate the cases in which you were very confident and see how many you got right. That is the picture of your calibration, and it changes as your knowledge grows.

The clue at which you answer in NEXO

In NEXO, each case opens up to six clues, and the player chooses when to risk the diagnosis. The score for a correct answer falls with each clue revealed, and this makes the balance between answering early and waiting for one more piece of data visible. Each guess shows the proximity in ICD-10 across five levels, from Exact ICD to another chapter.

One possible reading, which is our inference and not a measurement of the game, is to treat the clue at which you answer as a picture of your confidence in that case. Answering at the first clue suggests confidence; waiting until the last suggests doubt. Crossing that clue with the result shows where your confidence had a basis. On the home screen, the distribution of correct answers by clue gathers this pattern across cases.

If you research diagnostic calibration and want to study the topic with NEXO, take a look at NEXO Researcher. To bring NEXO to your medical school, student league or residency program, there is NEXO Closer. Doctors who review cases in their specialty take part through NEXO Curator, and those who create health content on social media can take part through NEXO Creators.

Frequently asked questions

What is diagnostic calibration?

It is the match between confidence in a diagnosis and the real chance that it is right. A well-calibrated person is often right when they feel sure and doubts when they have reason to doubt.

Does answering fast increase error?

Studies indicate that not always. In a study with recent graduates, the fastest answers were the most correct, even controlling for knowledge (Sherbino et al., 2012).

Does seeing the right answer improve calibration?

It can help a little. In residents facing difficult cases, seeing the correct diagnosis after each case changed calibration little (Kuhn et al., 2022). Looking at the pattern across many cases tends to be more useful.

Is overconfidence common in students?

In an exam with final-year students, about 12% showed significant overconfidence, and the best students had less overconfidence (Borracci and Arribalzaga, 2018).

How does NEXO show my confidence?

NEXO records on which clue you got each case right and gathers this pattern in the distribution of correct answers by clue on the home screen. Reading this clue as a sign of confidence is a useful inference for study. To learn about the game, read what NEXO is.

Calibration checklist

1. Give a confidence score before each answer.
2. Name the second hypothesis and the data that would rule it out.
3. Seek the data that separates, not just any data.
4. Note confidence and result side by side.
5. Review high-confidence cases every week.
6. Distrust old certainty in topics you have not reviewed for a while.
7. Look at the pattern, not an isolated case.
8. Go back to the content when you miss with high confidence.

Want to see on which clue you usually get it right? Play the case of the day on NEXO.

References

1. Borracci RA, Arribalzaga EB. The Incidence of Overconfidence and Underconfidence Effects in Medical Student Examinations. J Surg Educ. 2018. https://pubmed.ncbi.nlm.nih.gov/29397355/
2. Sherbino J, Dore KL, Wood TJ, et al. The relationship between response time and diagnostic accuracy. Acad Med. 2012. https://pubmed.ncbi.nlm.nih.gov/22534592/
3. Al-Maghrabi M, Mamede S, Schmidt HG, et al. Overconfidence, Time-on-Task, and Medical Errors: Is There a Relationship? Adv Med Educ Pract. 2024. https://pubmed.ncbi.nlm.nih.gov/38410282/
4. Kostopoulou O, Oudhoff J, Nath R, et al. Predictors of diagnostic accuracy and safe management in difficult diagnostic problems in family medicine. Med Decis Making. 2008. https://pubmed.ncbi.nlm.nih.gov/18556634/
5. Pusic MV, Chiaramonte R, Gladding S, Andrews JS, Pecaric MR, Boutis K. Accuracy of self-monitoring during learning of radiograph interpretation. Med Educ. 2015. https://pubmed.ncbi.nlm.nih.gov/26152495/
6. Kuhn J, van den Berg P, Mamede S, Zwaan L, Bindels P, van Gog T. Improving medical residents' self-assessment of their diagnostic accuracy: does feedback help? Adv Health Sci Educ Theory Pract. 2022. https://pubmed.ncbi.nlm.nih.gov/34739632/