Automation bias: trusting the machine too much

It is three in the morning in a busy emergency department. A second-year resident reviews a head CT while the support system flags, in red, a suspicious area in the frontal lobe. He zooms in, agrees with the flag, and moves on to the next patient. Two hours later, the on-call radiologist looks at the same scan and finds nothing in the flagged area, but sees a small bleed on the other side, which the system did not point out. The scene is fictional and was written for teaching.

This text is about automation bias, the tendency to trust the output of an automated system more than one's own clinical signals. The aim is to look at what studies show about electronic prescribing, imaging, pathology and endoscopy, and at what each finding teaches about reasoning. If you play the case of the day on NEXO, you will recognize the challenge: each new clue asks you to weigh the information, not to accept it just because it appeared on the screen.

Key points

• Automation bias is the tendency to trust the output of automated tools more than one's own judgment or the opinion of other people.
• It produces two kinds of error: failing to act because the system did not alert, and following wrong guidance from the system.
• In pathology, the same aid that improved accuracy when the suggestion was right reduced accuracy when it was wrong.
• Readers of all experience levels were influenced by false-positive AI findings on MR angiography.
• Students helped by a chatbot improved on average, but did not distinguish well between helpful and harmful influence.
• Recent reviews list automation bias among the most recurring themes in studies of generative AI.

What automation bias is

The concept came from aviation. Studies from the 1980s and 1990s and analyses of incident reports showed that pilots often failed to monitor important flight indicators or to disengage the autopilot when it failed (Sujan et al., 2019). The phenomenon was named automation bias, or automation-induced complacency, and is an example of poor decision-making due to excessive trust in the machine.

In medicine, the definition is similar. Automation bias is the human tendency to rely on the outputs of automated tools instead of non-automated signals, such as personal judgment or the opinion of other people (Bittencourt, 2025). A review on radiology sums up the idea in one line: it is the systematic tendency to accept AI suggestions even when they are wrong (Kocak and Cuocolo, 2026).

Automation bias and complacency

The same author separates two related concepts. Complacency is undue trust that leads to less vigilance and to uncritical acceptance of the system's output (Bittencourt, 2025). In practice, the two go together. Those who trust too much check less, and those who check less tend to accept what appears.

In digestive endoscopy, automation bias was described as the propensity to disengage cognitively from tasks handed to an external technology, with loss of situational awareness (Campion et al., 2024). The expression helps make sense of the on-call scene. The resident looked carefully at the flagged area, but stopped scanning the rest of the scan with the same attention.

Omission and commission

Automation bias can produce errors of omission, when a person fails to take a necessary action because the system did not warn them, and errors of commission, when a person follows inadequate guidance from the system (Sujan et al., 2019). The on-call scene has both. Accepting the wrong flag in the frontal lobe is commission. Letting the bleed the system did not point out go unnoticed is omission.

This split is useful for reasoning because each error calls for a different defense. Against commission, the defense is to check the suggestion before accepting it. Against omission, the defense is to keep the search complete even when the system stays silent. The text on diagnostic uncertainty shows how to tolerate doubt without stopping the investigation too early.

The opposite side: aversion to the algorithm

The opposite error also exists. Aversion to the algorithm is ignoring the AI output out of distrust (Kocak and Cuocolo, 2026). In endoscopy, when the user sees the algorithm make a mistake, they become much more likely to disregard the system's subsequent guidance (Campion et al., 2024). Between one extreme and the other lies calibrated trust, which is what the authors advocate: knowing when the tool usually gets it right and when it usually fails.

What the studies show in prescribing and imaging

The most direct evidence comes from experiments in which researchers control the quality of the suggestion. This makes it possible to see what happens to the professional when the machine is right and when it is wrong, something rarely observable in routine practice.

Electronic prescribing

An experiment with electronic prescribing varied the quality of decision support, with correct, incorrect or absent support, and task complexity, with and without interruption (Lyell et al., 2017). Compared with scenarios without support, correct support reduced omission errors, that is, failures to notice prescribing errors. Incorrect support increased these errors. Participants also made commission errors, accepting false-positive alerts, at all complexity levels tested.

The study design shows the core of the problem. The same tool that helps when it is right gets in the way when it is wrong, and users have trouble noticing the difference at the moment of decision. The text on clinical decision-making discusses how to weigh new information by what it changes in management.

MR angiography and cerebral aneurysms

A study in brain MR angiography assembled a set of exams in which half had at least one false-positive AI finding (Kim et al., 2025). False-positive results significantly increased aneurysm suspicion among readers. The less experienced recommended more intensive follow-up exams when they saw an AI false positive, with p = 0.005 (Kim et al., 2025).

The same study measured reading time. With AI assistance, inexperienced readers took an average of 164.1 seconds versus 228.2 without assistance, and the highly experienced took 117.9 versus 153.5 (Kim et al., 2025). The speed gain appeared at all levels. The authors concluded that radiology readers are susceptible to automation bias when faced with false-positive AI findings.

For reasoning, it is worth recalling a detail the article itself highlights: in screenings with many normal exams, even very precise tests produce more false positives than true positives, the so-called false-positive paradox (Kim et al., 2025). The text on pretest probability explains why this happens.

Mammography screening

A simulation with a large database of mammograms from Victoria, Australia, tested five ways of integrating AI into screening and analyzed the effect of human-machine interaction (Frazer et al., 2024). Used as a second reader or as a high-confidence filter, AI improved results and reduced the volume of human readings. Automation bias worsened performance in arrangements with multiple readers and improved the performance of single readers (Frazer et al., 2024). The effect depends on how the tool enters the workflow.

What the studies show in pathology, endoscopy and radiotherapy

The findings in imaging repeat in other areas that depend on visual recognition. In each one, the question is the same: what happens to the professional's performance when the suggestion is wrong?

Liver pathology

A deep learning assistant was tested in liver cancer classification by 11 pathologists with different levels of experience (Kiani et al., 2020). The assistant alone performed well on validation and test slides. Even so, use of the tool did not change the group's mean accuracy, and only improved the accuracy of a subgroup of nine pathologists with well-defined experience levels.

The most important finding came from the per-case analysis. When the model's prediction was correct, the help greatly increased accuracy, with an odds ratio of 4.289 (Kiani et al., 2020). When the prediction was wrong, the help reduced accuracy, with an odds ratio of 0.253, and both effects appeared at all levels of experience and difficulty (Kiani et al., 2020). The authors ask that the risks of assistance be considered already in the design and testing of these tools.

Digestive endoscopy

In colonoscopy, computer-aided detection systems were prioritized to improve the adenoma detection rate (Campion et al., 2024). The first results were encouraging, but some real-world studies showed smaller effects or no effect, plus more unnecessary removal of polyps without malignant potential. The authors suggest that factors in the human-AI interaction help explain the difference between experiment and real life.

The same paper notes that these systems also trigger false alarms, on average 2.4 per minute of scope withdrawal, and that the effect of these repetitions on endoscopist fatigue still needs to be studied (Campion et al., 2024). The tool that should ease attention ends up competing for it.

Prostate radiotherapy

In radiotherapy, a study compared manual contouring of prostate structures with contouring done by AI and adjusted by physicians (Arjmandi et al., 2026). The AI help improved inter-observer and intra-observer agreement. Even so, the adjusted contour did not always outperform the manual one, and the authors point to automation bias or selective acceptance as a possible influence on the adjustments. The conclusion calls for structured quality assurance and human supervision.

Area	Design	Finding about bias	Source
Electronic prescribing	Experiment with 120 participants	Incorrect support increased omission errors; false alerts were accepted	Lyell et al., 2017
Liver pathology	11 pathologists with and without assistant	Wrong suggestion reduced accuracy at all levels	Kiani et al., 2020
Brain MR angiography	Half of the exams with false positive	More aneurysm suspicion and faster reading	Kim et al., 2025
Mammography	Simulation with five setups	Bias worsened several readers and helped single readers	Frazer et al., 2024
Prostate radiotherapy	Manual and adjusted contouring	Adjustment did not always outperform the manual one	Arjmandi et al., 2026
Medical education	185 students with chatbot	Helpful and harmful influence seemed equal to the students	Hayden et al., 2026

Automation bias in medical training

Today's students will work with AI throughout their careers, and so the way they learn to use it matters. A pilot study with second-year students set up a two-part exercise: a class on AI capabilities and limits and a clinical case in which students ranked the differential diagnosis before and after consulting a chatbot (Hayden et al., 2026).

The class covered hallucinations, bias, privacy and the lack of genuine clinical reasoning, and the case ended with a discussion led by teachers (Hayden et al., 2026). Among 185 students, AI use was associated with more diagnostic accuracy, with the greatest benefit among those who had done worse before the consultation.

What the students noticed

The most interesting finding is in perception. The 65 students who improved and the 26 who got worse after using the AI rated its influence higher than the students whose accuracy did not change (Hayden et al., 2026). The perception of influence tracked whether the AI had changed the order of the hypotheses, whether the change was good or bad. The authors concluded that the students did not reliably tell useful influence from harmful influence, a pattern consistent with automation bias.

This result connects with what is known about training with cases. The text on learning based on clinical cases shows why the discussion after the case matters as much as the case. And training with differential diagnosis and clues helps you notice when new information has changed the list for good reasons.

A systematic review on generative AI found concern about loss of skill among the recurring themes, alongside automation bias (Al-Anezi, 2026), and the text on use of artificial intelligence in clinical reasoning discusses how to fit these tools into study.

Why we trust the machine too much

The reviewed articles offer some explanations that complement each other. The first is how AI is evaluated. Most performance studies use metrics such as sensitivity, specificity, accuracy and area under the ROC curve, and these numbers describe the algorithm alone, not the pair formed by doctor and machine (Bittencourt, 2025). Anyone who reads an article with high accuracy tends to carry that confidence over into daily use.

The second explanation lies in how the models themselves work. Machine learning systems can misread a relevant change in context or data and keep making wrong predictions with confidence when they receive cases different from those they saw in training (Challen et al., 2019). The same authors note that the clinician changes behavior in the face of a potentially serious outcome, even if that reduces apparent accuracy, and that the machine needs to imitate that caution.

The third explanation is time pressure. AI made reading faster at every level of experience in the MR angiography study (Kim et al., 2025), and the review on loss of skill describes shifts in correct diagnosis under time limits (Heudel et al., 2026). Speed and checking compete for the same attention.

When the problem is subtle or rare

A review on evaluating AI in decision support describes automation bias as a risk with dangerous consequences when the system errs, fails, or when the patient's problem is subtly unique (Magrabi et al., 2019). This is the case that worries diagnosis most. The rare disease, the atypical presentation and the patient outside the training profile are precisely the moments when the machine has the least basis and the doctor's doubt matters most.

The authors argue for rigorous evaluation before and during use, because the behavior of algorithms can change over time and with context (Magrabi et al., 2019). Pretest probability is a good anchor here: an AI flag is a test, and its value depends on who the patient is.

How to protect yourself day to day

The reviews converge on some recommendations. To reduce complacency, interventions are needed on several fronts: general education about AI tools and specific training on when each tool works well or badly (Bittencourt, 2025). Institutions, in turn, need to define the context of use, protocols, workflows, training and certification.

In radiology, the recommendation is that professionals have AI literacy to recognize failure modes and calibrate confidence (Kocak and Cuocolo, 2026). The same authors note that AI can work as a second opinion or safety net, not as a replacement for reading. The review on generative AI adds the presence of a human in the decision loop, the explainability of systems, governance and retraining of professionals (Al-Anezi, 2026).

A routine for using the suggestion

The sequence below is didactic and was assembled from the recommendations cited above. It does not replace local protocols.

1. Form your hypothesis first. In the study with students, the order of the hypotheses was recorded before and after consulting the AI (Hayden et al., 2026).
2. Ask what the tool was made for. Specific training includes knowing when it tends to be right and when it tends to fail (Bittencourt, 2025).
3. Check the suggestion. A commission error is following wrong guidance (Sujan et al., 2019).
4. Finish the full search. An omission error is failing to act because the system did not warn you (Sujan et al., 2019).
5. Be more suspicious in the atypical case. The risk grows when the problem is subtly unique (Magrabi et al., 2019).
6. Record it when you disagree. Structured human supervision is part of quality assurance (Arjmandi et al., 2026).

Back to the shift

The resident in the scene did what many would do: he looked carefully at what the machine showed and checked less what it did not show. Following the steps above, he would have read the whole exam before opening the marking, then checked the indicated area and, finally, gone back to the rest of the skull. The text on how to organize reasoning in the medical record shows how to record disagreement in a way that is useful to whoever reads it later.

Errors like this are valuable material for morbidity and mortality sessions, in which the team discusses what happened without looking for someone to blame. And the text on telemedicine and reasoning without an in-person exam shows another scenario in which technology comes between the doctor and the patient.

How NEXO helps train this reasoning

In NEXO, the case of the day is fictional, with a real clinical presentation and up to six progressive clues, and you propose the diagnosis by searching for the ICD-10 code.

The limits of using the game are described on the NEXO Economy page. If you research the teaching of clinical reasoning, the page for Researchers explains how to get in touch. Professionals who write or review cases find information under Curator and Creators, and institutions can learn about the proposal on the partnerships page. Knowing how to find the right ICD-10 code helps record the diagnosis in a way others can use the data.

Frequently asked questions

Does automation bias appear only with artificial intelligence?

It was first described in aviation, with pilots who stopped monitoring flight indicators or stopped disengaging the autopilot in case of failure (Sujan et al., 2019).

Are experienced doctors also affected?

In pathology, the wrong AI suggestion reduced accuracy at all levels of experience (Kiani et al., 2020).

Does AI improve the performance of those who use it?

It improves when it is right, and the greatest benefit among students appeared in those who had done worse before consulting it (Hayden et al., 2026).

What is the best defense against automation bias?

Reviews point to general education about AI combined with specific training on when each tool is right or fails (Bittencourt, 2025).

Take-home messages

• Automation bias is trusting the machine's output more than your own judgment.
• It produces omission errors, when the system does not warn you, and commission errors, when the system gives wrong guidance.
• The same help that improves accuracy when it is right reduces accuracy when it is wrong.
• Experience helps little: readers and pathologists at all levels were influenced.
• Published accuracy describes the algorithm alone, not the doctor and machine together.
• The risk grows in the atypical case, in which the machine has less to go on.
• Forming the hypothesis before consulting the tool preserves your own reasoning.

Want to practice reasoning with progressive clues? Play the case of the day.

References

1. Al-Anezi FM. Generative Artificial Intelligence in Healthcare: Automation Bias, Deskilling, and Cognitive Implications - A Systematic Review. J Healthc Leadersh. 2026. PMID 42445869.
2. Arjmandi N, Sebzari AR, Molaei F, et al. Clinical validation of AI-assisted contouring in prostate radiation therapy treatment planning: Highlighting automation bias and the need for standardized quality assurance. J Appl Clin Med Phys. 2026. PMID 41414851.
3. Bittencourt MS. Will You Tell Them When They Are Wrong? Automation Bias and Artificial Intelligence in Medicine. Arq Bras Cardiol. 2025. PMID 40667888.
4. Campion JR, O'Connor DB, Lahiff C. Human-artificial intelligence interaction in gastrointestinal endoscopy. World J Gastrointest Endosc. 2024. PMID 38577646.
5. Challen R, Denny J, Pitt M, et al. Artificial intelligence, bias and clinical safety. BMJ Qual Saf. 2019. PMID 30636200.
6. Frazer HML, Peña-Solorzano CA, Kwok CF, et al. Comparison of AI-integrated pathways with human-AI interaction in population mammographic screening for breast cancer. Nat Commun. 2024. PMID 39214982.
7. Hayden R, Blanco M, Ramesh S, et al. Diagnostic reasoning with and without AI: automation bias in pre-clerkship medical students. BMC Med Educ. 2026. PMID 42482261.
8. Heudel PE, Crochet H, Filori Q, et al. Artificial intelligence in medicine: a scoping review of the risk of deskilling and loss of expertise among physicians. ESMO Real World Data Digit Oncol. 2026. PMID 41890350.
9. Kiani A, Uyumazturk B, Rajpurkar P, et al. Impact of a deep learning assistant on the histopathologic classification of liver cancer. NPJ Digit Med. 2020. PMID 32140566.
10. Kim SH, Schramm S, Riedel EO, et al. Automation bias in AI-assisted detection of cerebral aneurysms on time-of-flight MR angiography. Radiol Med. 2025. PMID 39939458.
11. Kocak B, Cuocolo R. Human-AI interaction and collaboration in radiology: from conceptual frameworks to responsible implementation. Diagn Interv Radiol. 2026. PMID 41582537.
12. Lyell D, Magrabi F, Raban MZ, et al. Automation bias in electronic prescribing. BMC Med Inform Decis Mak. 2017. PMID 28302112.
13. Magrabi F, Ammenwerth E, McNair JB, et al. Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications. Yearb Med Inform. 2019. PMID 31022752.
14. Sujan M, Furniss D, Grundy K, et al. Human factors challenges for the safe use of artificial intelligence in patient care. BMJ Health Care Inform. 2019. PMID 31780459.