AI in clinical reasoning

You finish reading a difficult case, open an artificial intelligence assistant, paste the story and receive in seconds a list of ten well-explained diagnoses. It seems like the end of the problem. A randomized clinical trial with 50 doctors showed something less obvious: having a language model available, in addition to the usual resources, did not significantly improve the doctors' performance in diagnostic reasoning, although the model alone scored higher than the doctors (Goh et al., 2024).

NEXO is a daily medical diagnosis game: one clinical case a day, six progressive clues, and ICD-10 proximity feedback across five levels.

This text summarizes what studies show about artificial intelligence in clinical reasoning, what risks appear when it enters the routine and how to study with AI alongside without failing to develop your own reasoning.

Main points

• Language models already perform well on tests and vignettes, but having the tool available does not guarantee that the doctor reasons better.
• In some studies, doctors with AI support arrived at more complete differential diagnosis lists.
• Automation bias, the tendency to trust the machine too much, is a known risk, and there are ways to reduce it.
• An observational study suggests that continuous exposure to AI can worsen performance when it is not available.
• For those who study, order matters: first your reasoning, then the comparison with the tool.

Summary

• What AI can already do in diagnosis
• AI in the doctor's hands: what changes
• The risks: automation bias and loss of skill
• How to study clinical reasoning with AI alongside
• Where training without help remains essential
• Frequently asked questions
• Checklist for using AI in clinical reasoning

What AI can already do in diagnosis

The idea of using computers to help with diagnosis is old, and the quality of the tools has changed a lot in a few years. It is worth looking at the evolution to understand what the current numbers mean.

Before language models

In 2015, a study tested 23 online symptom checkers with 45 standardized vignettes (Semigran et al., 2015). The tools gave the correct diagnosis first in 34% of evaluations and included it among the first 20 in 58% (Semigran et al., 2015). In triage, the advice tended to be too cautious, sending people to seek care in situations where self-care was reasonable (Semigran et al., 2015).

Language models on exams

In 2023, a study evaluated language models on a set of medical question bases, and one of the versions reached 67.6% accuracy on questions in the style of the United States medical licensing exam (Singhal et al., 2023). The human evaluation of the answers, however, revealed important gaps, and even the version adjusted for medicine remained below the clinicians (Singhal et al., 2023).

The authors highlight the importance of evaluating answers on various axes, such as factuality, comprehension, reasoning, possibility of harm and bias (Singhal et al., 2023). Getting an exam alternative right is different from reasoning well about a patient.

Differential diagnosis in difficult cases

In 2025, a model optimized for diagnostic reasoning was tested on 302 challenging case reports published in the literature, evaluated by 20 clinicians (McDuff et al., 2025). Alone, the model included the correct diagnosis among the first ten hypotheses 59.1% of the time, against 33.6% of clinicians without help (McDuff et al., 2025).

Another study by the same group compared a diagnostic conversation system with primary care doctors in text consultations with trained actors (Tu et al., 2025). The system had higher diagnostic accuracy and was better on most evaluated axes, but the authors ask for caution: text chat consultation is not the usual way of providing care, and more research is needed before bringing the system into practice (Tu et al., 2025).

Where AI appears in health

Diagnosis is only one of the fronts. The World Health Organization guidance on large multimodal models describes five major applications in health (WHO, 2024):

• diagnosis and clinical care, such as answering written patient questions;
• patient-guided use, such as investigating symptoms and treatments;
• administrative tasks, such as documenting and summarizing consultations in the electronic health record;
• medical and nursing education, including simulated encounters with patients;
• scientific research and drug development.

For the student, this means AI will appear at various points in training, from summarizing an article to simulating a consultation. In each of them, the question is the same: is the tool doing the work for me or helping me do it better?

How to read these results

Type of study	What it measures	What it does not measure
Multiple-choice exams	Knowledge and pattern recognition in closed questions	Data collection, physical examination, patient context
Vignettes and case reports	Ability to build a differential from a ready-made text	The task of deciding which data to seek
Text-simulated consultations	Conversation and reasoning in a controlled format	In-person care, with limited time and resources
Trials with physicians	The effect of giving the tool to those who provide care	The effect on patients, except when the outcome is clinical

Table source: synthesis by the Editorial from the designs of Singhal et al., 2023, McDuff et al., 2025, Tu et al., 2025 and Goh et al., 2024.

The combined reading is that AI does well on tasks in which the data already arrive organized. The work of deciding what to ask, what to examine and what to order remains with those who provide care.

AI in the physician's hands: what changes

Knowing that the tool gets it right on its own is one thing. Knowing whether the physician with the tool decides better is another, and it is the question that matters for practice.

The randomized trial with physicians

In the 2024 trial, family medicine, internal medicine and emergency physicians were randomly assigned to use a language model along with usual resources or usual resources alone, with up to six vignettes in one hour (Goh et al., 2024). The primary outcome assessed the accuracy of the differential, the choice of factors for and against each hypothesis and the next steps of the investigation (Goh et al., 2024).

The median score was 76% in the group with the model and 74% in the group without it, a difference without statistical significance (Goh et al., 2024). The model alone scored 16 percentage points above the usual resources group, and the authors conclude that both the technology and the way professionals work with it need to be developed (Goh et al., 2024).

When the help expanded the differential

In the case report study, clinicians with model support reached more complete differential diagnosis lists and got more right among the first ten hypotheses (51.7%) than clinicians without help (36.1%) and those who used internet search (44.4%) (McDuff et al., 2025). In that study, all clinicians first made a differential without help and only then used the tool (McDuff et al., 2025).

What explains the difference between the studies

The designs are different, and the numbers cannot be compared directly. But one point stands out: having the tool is not the same as knowing how to use it. The 2024 trial suggests that, without training and without a defined way of working with AI, the physician may not take advantage of what it offers. The 2025 study suggests that thinking first and consulting afterward is a way of using it that expands the list of hypotheses.

The risks: automation bias and loss of skill

Every tool that helps decide brings a risk along with it: that of deciding for you. This risk has a name and was studied long before language models.

Automation bias

A systematic review defined automation bias as the tendency to trust automated systems too much and gathered 74 studies on the topic (Goddard et al., 2012). The authors recall that, although decision support systems improve overall performance in most research, the new errors they can introduce are often not recognized (Goddard et al., 2012).

Factors that influence the bias	What helps reduce it
Trust in the tool	Training
Cognitive style and experience with the task	Reinforcing that responsibility belongs to whoever decides
Workload	Showing updated confidence levels alongside the answer
Complex task and little time	Offer information instead of a ready recommendation

Source of the table: mediators and forms of mitigation described by Goddard et al., 2012.

The World Health Organization also cites automation bias among the risks of large multimodal models, alongside false, inaccurate, biased or incomplete answers (WHO, 2024). The same guidance lists medical education, with simulated patient encounters, among the possible applications of these models (WHO, 2024).

Loss of skill

An observational study at four endoscopy centers in Poland compared colonoscopies performed without AI before and after the introduction of a polyp detection tool (Budzyń et al., 2025). The adenoma detection rate in exams without AI fell from 28.4% to 22.4% after exposure to the tool (Budzyń et al., 2025).

The authors interpret that continuous exposure to AI may have a negative effect on performance when it is not available (Budzyń et al., 2025). It is an observational study, on a visual task and in one specialty, and it does not prove the same effect on diagnostic reasoning. But the warning holds for those building their own foundation: the skill you do not practice tends to weaken.

The risk for those still learning

For the student, the biggest risk is skipping the step that builds reasoning. When the list of hypotheses arrives ready, you stop training exactly what a text about clinical reasoning in the clinical cycle describes as the core of the skill: gathering data, generating hypotheses and testing them. And without a foundation of your own, it becomes hard to notice when the tool is wrong.

How to study clinical reasoning with AI alongside

The useful question is not whether you should use AI, but at what moment and for what. The order in which you use the tool changes what you learn.

First you, then the tool

The design of the case report study, in which each clinician made a differential on their own before consulting the tool, is a good model for studying (McDuff et al., 2025). In practice:

1. Read the case and write your summary in one sentence.
2. Build your differential with three to five hypotheses, in order.
3. Write one piece of data for and one against each hypothesis.
4. Only then consult the tool.
5. Compare: which hypothesis did it bring that you did not have? Did it ignore any data you considered important?

With this script, AI becomes feedback on your reasoning, not a substitute for it. It is the same logic of testing yourself before looking at the answer, explained in the text about reviewing by playing.

Use AI to ask, not only to answer

Some ways of using it tend to train reasoning more:

• Ask for the opposing argument. "Which data in this case weakens my main hypothesis?"
• Ask for the diagnosis that cannot be missed. "Which serious cause fits these findings and I left out?"
• Ask for the next question. "Which history question would separate the first two hypotheses?"
• Ask for an explanation of the mechanism. "Why does this disease cause this finding?"

In all cases, check the answer in a reliable source. Models can produce false or incomplete statements (WHO, 2024), and the fluency of the text does not guarantee that it is right.

Feedback on what you write

AI is also being studied as a way to give feedback on written reasoning. A study at two American institutions developed models to assess the documentation of clinical reasoning in residents' admission notes, and the authors conclude that this type of tool can expand feedback on reasoning (Schaye et al., 2025). Performance varied between models, and some performed insufficiently for implementation (Schaye et al., 2025).

A commented example

See how the script works in a study case. The case is a teaching vignette from the Editorial, and the goal is to show the method, not to teach management.

The case. A 34-year-old woman with fatigue for two months, weight gain and constipation. She reports feeling colder than her coworkers.

Step 1, your summary. "Young woman with fatigue, weight gain, constipation and cold intolerance for two months."

Step 2, your differential. You list hypothyroidism first, then depression, anemia and medication effect, and mark what favors each one. Notice that cold intolerance weighs in favor of the first hypothesis.

Step 3, the consultation. You ask the tool for a differential for the same summary. It brings a similar list and adds sleep apnea and pregnancy, which you had not considered.

Step 4, the comparison. The question is not who got it right, but why its list had items yours did not. Pregnancy, for example, is a hypothesis you ask about in every woman of childbearing age, and forgetting it is a habit error, not a knowledge error.

Step 5, the source. You check in a reliable reference which tests would help separate the hypotheses and note down what you learned. The record goes into your error notebook, along with the habit that was missing.

In the end, the tool served to show a gap in your reasoning, not to think in your place. It is this difference that the method protects.

Common mistakes when studying with AI

• Consulting before thinking. The ready-made list takes away from you the part of the exercise that builds reasoning.
• Accepting the first answer. A fluent explanation seems right even when it is not.
• Confusing knowledge with reasoning. Knowing the list of causes of a symptom is different from choosing the most likely one for a patient.
• Studying only with closed questions. Models do well on multiple choice; care starts with an open complaint.
• Forgetting the local context. The epidemiology, the protocols and the resources of your service change the decision, and the tool does not always take that into account.

Practical precautions

• Protect the data. Never paste a name, document or any information that identifies a person into a public tool.
• Check the source. Ask for references and open each one; models can cite articles that do not exist.
• Follow the rules of your service. Schools and hospitals may have their own rules about the use of AI.
• Record your decision. If the AI helped you remember a hypothesis, the decision and the justification are still yours.

Ethics and transparency

Using AI in study also involves honesty. If a college assignment or a case discussion had help from a tool, say so, just as you would cite a book. In care, responsibility for the decision belongs to the person providing care, and that is why the review on automation bias points to strengthening individual responsibility as one of the ways to reduce overconfidence (Goddard et al., 2012).

Where practice without help remains essential

Even if AI becomes common in care, the doctor needs their own reasoning to know when to trust and when to doubt. This reasoning is built with practice without help.

What only practice builds

Knowing which data to seek, noticing when a finding does not fit, feeling that a case is more serious than it seems: these skills are built with many cases and feedback. The text on differential diagnosis with clues shows how to train the generation and testing of hypotheses, and the one on clinical reasoning games compares training formats.

Calibrating trust in the tool

Trusting AI to the right degree requires knowing, from your own experience, what good reasoning looks like. Those who have solved many cases without help notice when the tool's answer ignores an important piece of data, when the main hypothesis does not explain a finding or when the list seems too generic. Those who skipped this step tend to accept the first well-written answer. That is why practice without help and the use of AI do not compete: the first is what makes the second safe.

Training on NEXO

On NEXO, the case arrives in up to six clues, one at a time, and you give your guess by the name of the diagnosis or by the ICD-10 code, with no one suggesting the answer. With each wrong guess, the game shows the proximity in five levels, from different chapter to Exact ICD, and opens the next clue. It is a training of your own reasoning: deciding with incomplete information and adjusting the hypothesis with the feedback.

The Daily brings one case per day, the same for everyone, and the Archive gathers the previous daily cases. Multimedia adds image, audio and other tests, and Continents changes the background epidemiology. Adrenaline brings time into the equation. How each mode is unlocked is on the NEXO economy page. After the case, if you want, compare your reasoning with that of an AI tool following the script above.

Saving, discussing and researching

The Anki button creates the card straight from NEXO and opens it in the Anki app, useful for saving the data point you did not value. The Group tab of the ranking, which brings together invited colleagues, lets you discuss with your class which clue each person got right. If you coordinate a medical school, a league or a residency and want to use cases like this in teaching, get to know NEXO Closer. Researchers who study AI and clinical reasoning can propose studies through NEXO Researcher, and physicians who want to review cases in their specialty can take part through NEXO Curator. Those who produce medical content can get to know NEXO Creators.

Frequently asked questions

Does artificial intelligence make diagnoses better than physicians?

In some studies with vignettes and case reports, the model on its own scored higher than physicians (Goh et al., 2024; McDuff et al., 2025). These studies use data already organized as text and do not measure data collection or in-person care.

Does using AI improve a physician's reasoning?

It depends on how it is used. In a randomized trial, the availability of the tool did not significantly improve physicians' reasoning (Goh et al., 2024). In another study, in which clinicians thought first and consulted afterward, the differentials were more complete (McDuff et al., 2025).

What is automation bias?

It is the tendency to trust automated systems too much. A systematic review identified factors that influence it, such as workload and little time, and ways to reduce it, such as training and reinforcing the responsibility of the person who decides (Goddard et al., 2012).

Can using AI make a physician lose skill?

An observational study in colonoscopy found a drop in the adenoma detection rate in exams without AI after physicians began using the tool (Budzyń et al., 2025). The result applies to that task, but it reinforces the importance of keeping up practice without help.

As a student, can I use AI to study cases?

You can, with method: make your differential first, consult afterward and compare. Check every piece of information in a reliable source and never paste data that identifies people. The WHO warns that these models can produce false or incomplete answers (WHO, 2024).

What else is AI used for in health?

The WHO describes applications in diagnosis and clinical care, use by patients themselves, administrative tasks such as documenting consultations, medical education and scientific research (WHO, 2024).

How does NEXO help with this training?

In NEXO, you solve the case with up to six clues and receive the proximity of your guess in ICD-10, training your own reasoning with feedback on every attempt. To get to know the game, read what NEXO is.

Checklist for using AI in clinical reasoning

1. Make your differential first before opening any tool.
2. Write down data for and against each hypothesis.
3. Consult AI as a second opinion, not as a first answer.
4. Ask for the opposing argument and the serious cause that may have been left out.
5. Check every piece of information in a reliable source.
6. Open the references that the tool cites.
7. Protect the data: nothing that identifies a person goes into the tool.
8. Be more suspicious when you are tired or in a hurry.
9. Keep training without help so you do not lose the basics.
10. Own the decision: the justification is yours.

Want to train your reasoning without help today? Play the case of the day on NEXO.

References

1. Goh E, Gallo R, Hom J, et al. Large Language Model Influence on Diagnostic Reasoning: A Randomized Clinical Trial. JAMA Netw Open. 2024. https://pubmed.ncbi.nlm.nih.gov/39466245/
2. McDuff D, Schaekermann M, Tu T, et al. Towards accurate differential diagnosis with large language models. Nature. 2025. https://pubmed.ncbi.nlm.nih.gov/40205049/
3. Tu T, Schaekermann M, Palepu A, et al. Towards conversational diagnostic artificial intelligence. Nature. 2025. https://pubmed.ncbi.nlm.nih.gov/40205050/
4. Singhal K, Azizi S, Tu T, et al. Large language models encode clinical knowledge. Nature. 2023. https://pubmed.ncbi.nlm.nih.gov/37438534/
5. Semigran HL, Linder JA, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ. 2015. https://pubmed.ncbi.nlm.nih.gov/26157077/
6. Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. J Am Med Inform Assoc. 2012. https://pubmed.ncbi.nlm.nih.gov/21685142/
7. Budzyń K, Romańczyk M, Kitala D, et al. Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy: a multicentre, observational study. Lancet Gastroenterol Hepatol. 2025. https://pubmed.ncbi.nlm.nih.gov/40816301/
8. World Health Organization. WHO releases AI ethics and governance guidance for large multi-modal models. 2024. https://www.who.int/news/item/18-01-2024-who-releases-ai-ethics-and-governance-guidance-for-large-multi-modal-models
9. Schaye V, DiTullio D, Guzman BV, et al. Large Language Model-Based Assessment of Clinical Reasoning Documentation in the Electronic Health Record Across Two Institutions: Development and Validation Study. J Med Internet Res. 2025. https://pubmed.ncbi.nlm.nih.gov/40117575/