Symptom checkers: what they get right and wrong
Table of contents
A 22-year-old student wakes up with belly pain that started near the navel and now seems more to the right. Before deciding whether to go to the emergency department, she opens an app on her phone, answers a series of questions and receives a list of possible causes and an urgency recommendation. The app suggests seeking care the same day. She hesitates, because the pain eased a little after a painkiller. The scene is fictional and was written for teaching.
This text is about symptom checkers, digital tools that help laypeople assess the urgency and possible causes of their own complaints. The aim is to look at what studies show about diagnostic and triage accuracy, about use by actual users and about the limits of the studies themselves. If you play the case of the day in NEXO, you will recognize the challenge: clues arrive gradually, and the order of hypotheses changes as they arrive.
Key points
- In 2015, 23 checkers listed the correct diagnosis first in about one third of assessments.
- Five years later, the average triage performance of the apps had not improved and they had become less cautious.
- Triage accuracy is usually higher than diagnostic accuracy.
- In 2020, the apps performed worse than laypeople in at least one of the two triage decisions tested.
- Emergency patients were able to use a checker and found the tool useful.
- Studies with vignettes can overestimate performance, because the apps tend to solve the easier cases.
What a symptom checker is
Symptom checkers are digital tools that help laypeople assess whether and where to seek care, and for which diagnoses, based on the symptoms and history the user reports (Schmieding and colleagues, 2022). Many use a conversation format, like a chatbot, and set out to give a rapid diagnosis and consistent triage (Morse and colleagues, 2020).
A review on ethics describes the two main functions. The first reports the conditions that could explain the symptoms. The second, triage, says whether help is needed, where, such as the family doctor or the emergency department, and with what urgency, such as within a few days (Müller and colleagues, 2022). The same authors note that mobile apps can use device features, such as location and camera, and even connect to wearable devices.
Why this topic matters for reasoning
The checker does automatically what the clinician does in the first conversation: it gathers complaints, asks what is missing and proposes a list of hypotheses with an urgency decision. Studying where it goes wrong helps show where reasoning itself can go wrong. The text on signs, symptoms and history shows how to listen to the account without losing what matters.
Who uses it
In a large health system in the United States, 26,646 assessments were made with a chatbot-format checker, most by women, with a mean age of 34.3 years (Morse and colleagues, 2020). Almost half of the assessments, 12,357, took place outside office business hours. The authors concluded that users represented the population served well, although they were younger and more often women, and that the triage recommendations were comparable to those of telephone triage lines staffed by nurses.
In the same service, the triage advice took the form of one of eight possible suggestions, grouped into low, medium and high urgency levels (Morse and colleagues, 2020). The authors note that these apps are expected to reduce unnecessary visits to clinics and emergency departments, and cite an earlier study in which completing an assessment on the internet reduced the urgency of the care patients intended to seek (Morse and colleagues, 2020). They also point out that little is still known about the combined recommendations of a checker used by a large population.
The time of use is striking. When almost half of the assessments happen outside office hours (Morse et al., 2020), the app fills exactly the space where the patient would have the fewest people to talk to. That is when a wrong urgency advice, whether too much or too little, weighs the most.
The 2015 reference study
The work that became a reference tested 23 checkers with 45 clinical vignettes, typed into the app as if they were patients (Semigran et al., 2015). The correct diagnosis appeared first in 34 out of every 100 assessments and among the first 20 suggestions in 58 out of every 100 (Semigran et al., 2015).
Triage was appropriate in 80 out of every 100 emergency vignettes, in 55 out of every 100 non-urgent vignettes and in 33 out of every 100 self-care vignettes (Semigran et al., 2015). Triage performance ranged from 33 to 78 out of every 100 across the apps (Semigran et al., 2015), and the authors concluded that there were shortcomings in both triage and diagnosis. The apps tended to send people to seek care even when self-care was enough, and the average hides the direction of the error. The text on artificial intelligence in clinical reasoning discusses the same caution with more recent tools.
Five years later
A team repeated the study in 2020, with 22 checkers and the same 45 vignettes, and compared the results with public data on triage done by laypeople without help (Schmieding et al., 2022). The 2020 apps were less risk-averse than the 2015 ones and let more than 40 out of every 100 emergencies pass. Few outperformed laypeople in the decision on whether emergency care was needed or in the decision on whether self-care was enough, and none outperformed laypeople in both.
The authors concluded that the average triage of the checkers had not improved in five years and had worsened in two uses: the advice on when emergency care is needed and on when no care is needed at the moment (Schmieding et al., 2022). The text on disaster triage shows how simple rules try to balance excess and lack of urgency in another context.
What the reviews found
A systematic review gathered 29 publications on checkers and digital triage services (Chambers et al., 2019). The evidence on patient safety was weak, and diagnostic accuracy was generally low. Triage done by algorithm was more risk-averse than triage done by health professionals. The authors also found very little evidence on whether patients followed the advice they received. That last point is central: correct advice only protects the patient if it is followed, and wrong advice only causes harm if it is followed. Measuring accuracy without measuring what the user does afterwards leaves half the question unanswered.
A more recent review found main diagnosis accuracy between 19 and 37.9 out of every 100 assessments and triage accuracy between 48.8 and 90.1 out of every 100 (Wallace et al., 2022). Triage usually did better than diagnosis. Results varied even when the data entered were the same, which shows that the way the tool is used also matters.
Comparison with family physicians
A study with 200 primary care vignettes compared eight apps with seven family physicians (Gilbert et al., 2020). Coverage, that is, the ability to give some answer for the vignette, ranged from 99.0 out of every 100 vignettes in Ada to 51.5 in Babylon (Gilbert et al., 2020). Among the first three suggestions, the physicians were right on average in 82.1 out of every 100 vignettes, Ada in 70.5 and Your.MD in 23.5 (Gilbert et al., 2020). On safe urgency advice, the physicians reached on average 97.0 out of every 100 (Gilbert et al., 2020).
This study has an important limitation: most of the authors worked at Ada Health, the company behind one of the apps evaluated (Gilbert et al., 2020). The fact does not invalidate the result, but it calls for careful reading. In any comparison between products, it is worth asking who did the study.
| Study | Design | Main finding | Source |
|---|---|---|---|
| 2015 reference | 23 apps, 45 vignettes | Correct diagnosis in first place in 34 out of 100 | Semigran et al., 2015 |
| Repeat in 2020 | 22 apps, same vignettes | More than 40 out of 100 emergencies missed | Schmieding et al., 2022 |
| Primary care | 8 apps and 7 doctors, 200 vignettes | Doctors were more often right among the first three suggestions | Gilbert et al., 2020 |
| Emergency | 40 patients seen, with Ada | Sensitivity close to that of doctors who saw the same data | Fraser et al., 2022 |
| Rheumatology | 600 patients in a randomized trial | Moderate accuracy for inflammatory rheumatic disease | Knitza et al., 2024 |
| Primary care and emergency | Prototype in 581 patients | Better triage accuracy than the patients themselves | Chan et al., 2021 |
Actual users, not vignettes
Most studies use simulated patients, and there is little evidence on safety and accuracy when the checker is used by actual users (Chan et al., 2021). Studies with actual users are therefore the most informative for anyone who wants to know what happens outside the lab.
In the emergency department
In a United States emergency department, 40 of the 50 invited patients completed the assessment with Ada and a usability survey (Fraser et al., 2022). Three independent emergency doctors saw the same symptoms typed by the patients and made their own diagnoses. Ada's sensitivity for at least one of the final diagnoses, considering its first five suggestions, was 70 out of 100, close to the doctors' average of 68.9 considering their first three (Fraser et al., 2022).
Some assessments had missing data or no clear diagnosis in the emergency department, so 30 of the 40 entered the diagnosis analysis and 37 entered the triage analysis (Fraser et al., 2022). The independent doctors also followed a fixed order: first they looked only at the typed symptoms, then at vital signs, and then they redid their three main hypotheses. Among them, the first hypothesis matched the final diagnosis in 14 of the 30 assessments analyzed, on average (Fraser et al., 2022).
The authors treated the result as preliminary evidence that a checker can have acceptable usability and accuracy for patients with various urgent conditions (Fraser et al., 2022). The small sample calls for caution, and the design shows a useful point for reasoning: even experienced doctors, seeing only what the patient typed, got the first hypothesis right less than half the time. The history written by the patient is a valuable clue, but an incomplete one.
In rheumatology
Inflammatory rheumatic diseases are often diagnosed late because of nonspecific symptoms and the shortage of rheumatologists (Knitza et al., 2024). A randomized trial at three centers included 600 new patients, of whom 214 were diagnosed with inflammatory rheumatic disease. Ada's first suggestion got the presence or absence of this disease right in 63 out of 100 patients, and the list of the first five in 58 out of 100 (Knitza et al., 2024).
An earlier study with 33 doctors and 132 vignettes found a different scenario (Gräf et al., 2022). Ada identified inflammatory rheumatic disease more often than the doctors, 70 versus 54 out of 100, and got the main diagnosis right in 54 versus 32 out of 100. Professional experience was not associated with accuracy. The authors themselves note that the doctors had only the basic history, with no physical examination or additional tests.
A Canadian prototype
A checker prototype was tested on 281 emergency patients and 300 family practice patients in Canada (Chan et al., 2021). Sensitivity for emergencies was 10 out of 10 visits. For urgencies, it was 73 out of 81 among hospital patients and 34 out of 35 among clinic patients. The prototype got triage right more often than the patients themselves, 73 versus 58 out of every 100 (Chan et al., 2021).
What patients do with the answer
A survey of 329 users of the Isabel checker showed the most common reasons for use (Meyer et al., 2020). The main one was understanding the cause of symptoms better, cited by 232 of 304 respondents, followed by deciding whether and where to seek care. Most reported receiving useful information, and half reported positive effects on their health.
Among respondents, 103 of 213 discussed the result with their physician, and in this group feeling that the professional was interested in what the app showed was more common than disinterest (Meyer et al., 2020). Patients who had already experienced a diagnostic error used the checker more to decide where to seek care. The authors concluded that, despite doubts about accuracy, many users found the tool useful for diagnosis.
The patient who arrives with the list ready
This finding has a practical consequence. The patient who arrives with a list of hypotheses from the app brings information about what worries them, and that concern can guide the conversation. The text on diagnostic uncertainty shows how to welcome the patient's doubt without abandoning your own reasoning, and the text on overdiagnosis recalls the risk of investigating too much.
Ethical questions
A review of the ethical and social challenges of these apps found controversial debates in the literature and a lack of a specific legal perspective and empirical data (Müller et al., 2022). The authors note that the phone is always close, always on, allows instant use and tends to store a lot of personal data, including health data. These features increase the tool's reach and also the care needed with privacy.
The same review highlights that the apps promise improvements at several levels: the user receives information about symptoms and guidance for navigating the health system (Müller et al., 2022). The authors call for empirical research with the people affected, such as users, family physicians and health system specialists, to understand how the tool changes the relationship between patient and service.
The limits of the studies themselves
A reanalysis of the public data from two vignette studies applied test theory metrics (Kopka et al., 2023). When triage was adjusted for the difficulty of each vignette, most apps changed position in the ranking. The published accuracy figures tended to overestimate the checkers' ability, because they tend to assess the easier cases selectively.
The authors also found many vignettes with very low or even negative correlation with the set, which suggests low-quality items (Kopka et al., 2023). The proposal is to control the difficulty of the vignettes that each app did or did not assess. For those who read these studies, the lesson is to ask which cases went into the count.
The questions below summarize what the studies cited in this text teach about reading these comparisons:
- Were the cases vignettes or patients seen in practice (Chan et al., 2021)?
- Was the result adjusted for the difficulty of the vignettes (Kopka et al., 2023)?
- How many vignettes did the app leave unanswered (Gilbert et al., 2020)?
- Do the authors have ties to any of the products tested (Gilbert et al., 2020)?
- Did the physicians compared have a physical exam or only the history (Gräf et al., 2022)?
- Was triage measured in both directions, over- and under-referral of urgency (Schmieding et al., 2022)?
Each question corresponds to a bias that can inflate or reduce performance. An app that leaves half the vignettes unanswered can look very accurate on the ones it answered. A physician without a physical exam is at an artificial disadvantage. And a triage that sends everyone to the emergency department gets every emergency right, but fails the people who only needed rest.
And generative AI chatbots
A study with 30 vignettes created by general practitioners tested ChatGPT-3 on common complaints (Hirosawa et al., 2023). The correct diagnosis appeared in the chatbot's list of ten hypotheses in 28 of the 30 vignettes. Physicians still did better on the list of five hypotheses, 98.3 versus 83.3 per 100, and on the main diagnosis, 93.3 versus 53.3 per 100 (Hirosawa et al., 2023). The text on clinical decision-making helps weigh such a list by what it changes in management, not by its length or how well it is written.
Back to the student
The student in the scene was advised to seek care the same day. Based on the studies above, app triage usually gets more right than diagnosis (Wallace et al., 2022), but the 2020 version missed many emergencies (Schmieding et al., 2022). Improvement with the painkiller is a clue the app may not weigh. A well-considered pretest probability, with pain that migrated to the right in a young person, points to the decision to go. The text on telemedicine and reasoning without an in-person exam discusses similar limits when the physical exam is missing.
How NEXO helps train this reasoning
In NEXO, the case of the day is fictional, with a real clinical presentation and up to six progressive clues, and you propose the diagnosis by searching for the ICD-10 code.
The limits of using the game are described on the NEXO Economy page. If you research the teaching of clinical reasoning, the Researcher page explains how to get in touch. Professionals who write or review cases find information on Curator and Creators, and institutions can learn about the proposal on the partnerships page. Training with differential diagnosis and clues and knowing how to find the right ICD-10 code complete this practice.
Frequently asked questions
Do symptom checkers get the diagnosis right?
In 2015, the correct diagnosis appeared first in 34 per 100 assessments and among the top 20 in 58 per 100 (Semigran et al., 2015).
Is app triage better than diagnosis?
Triage accuracy is usually higher than main diagnosis accuracy (Wallace et al., 2022).
Have apps improved over time?
A repeat in 2020 found average triage equal to that of 2015 and worse in two uses (Schmieding et al., 2022).
Can patients seen in the emergency department use these tools?
In an emergency department, 40 of 50 invited patients completed the assessment and the usability survey (Fraser et al., 2022).
How to read a study that compares apps?
It is worth checking who did the study and whether the result was adjusted for vignette difficulty (Kopka et al., 2023).
Take-home messages
- Symptom checkers suggest causes and a level of urgency based on what the user types.
- The correct diagnosis in first place appeared in about one third of assessments in 2015.
- Triage usually gets more right than diagnosis, but the 2020 apps missed many emergencies.
- Family physicians did better than the apps among the top three hypotheses.
- With actual users, some apps performed close to physicians given the same data.
- Studies with vignettes can overestimate accuracy when apps solve only the easy cases.
- The patient who arrives with the app's list brings a concern worth hearing.
Want to practice reasoning with progressive clues? Play the case of the day.
References
- Chambers D, Cantrell AJ, Johnson M, et al. Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review. BMJ Open. 2019. PMID 31375610.
- Chan F, Lai S, Pieterman M, et al. Performance of a new symptom checker in patient triage: Canadian cohort study. PLoS One. 2021. PMID 34852016.
- Fraser HSF, Cohan G, Koehler C, et al. Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study. JMIR Mhealth Uhealth. 2022. PMID 36121688.
- Gilbert S, Mehl A, et al. How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs. BMJ Open. 2020. PMID 33328258.
- Gräf M, Knitza J, Leipe J, et al. Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy. Rheumatol Int. 2022. PMID 36087130.
- Hirosawa T, Harada Y, Yokose M, et al. Diagnostic Accuracy of Differential-Diagnosis Lists Generated by Generative Pretrained Transformer 3 Chatbot for Clinical Vignettes with Common Chief Complaints: A Pilot Study. Int J Environ Res Public Health. 2023. PMID 36834073.
- Knitza J, Tascilar K, Fuchs F, et al. Diagnostic Accuracy of a Mobile AI-Based Symptom Checker and a Web-Based Self-Referral Tool in Rheumatology: Multicenter Randomized Controlled Trial. J Med Internet Res. 2024. PMID 39042425.
- Kopka M, Feufel MA, Berner ES, et al. How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective. Digit Health. 2023. PMID 37614591.
- Meyer AND, Giardina TD, Spitzmueller C, et al. Patient Perspectives on the Usefulness of an Artificial Intelligence-Assisted Symptom Checker: Cross-Sectional Survey Study. J Med Internet Res. 2020. PMID 32012052.
- Morse KE, Ostberg NP, Jones VG, et al. Use Characteristics and Triage Acuity of a Digital Symptom Checker in a Large Integrated Health System: Population-Based Descriptive Study. J Med Internet Res. 2020. PMID 33170799.
- Müller R, Klemmt M, Ehni HJ, et al. Ethical, legal, and social aspects of symptom checker applications: a scoping review. Med Health Care Philos. 2022. PMID 36181620.
- Schmieding ML, Kopka M, Schmidt K, et al. Triage Accuracy of Symptom Checker Apps: 5-Year Follow-up Evaluation. J Med Internet Res. 2022. PMID 35536633.
- Semigran HL, Linder JA, Gidengil C, et al. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ. 2015. PMID 26157077.
- Wallace W, Chan C, Chidambaram S, et al. The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. NPJ Digit Med. 2022. PMID 35977992.