In a fictional case, a 68-year-old man with diabetes and chronic kidney disease (CKD) comes to the consultation with a new number on the lab report: a 24 percent risk of needing dialysis or a transplant within five years. The following week, admitted for surgery, he appears on a list of "high risk of acute kidney injury" generated by an algorithm in the electronic health record. These are two predictions about the same kidney, made by very different tools.

The first comes from the Kidney Failure Risk Equation (KFRE), an equation with four variables that has already been validated in more than 30 countries (Tangri et al., 2016). The second comes from a machine learning model, the family of tools that today carries the label of artificial intelligence (AI). Both types promise the same thing: to warn before the kidney fails.

This text shows what each tool has already proven, what is still a promise and which questions to ask before trusting a risk number. The focus is on three ideas: discrimination, calibration and clinical impact.

Key points

• The four-variable KFRE uses age, sex, estimated glomerular filtration and urine albumin-to-creatinine ratio (Tangri et al., 2016).
• In 31 cohorts from more than 30 countries, the KFRE discriminated well who would reach kidney failure, but it overestimated the risk in some cohorts outside North America (Tangri et al., 2016).
• Discriminating well and being well calibrated are different properties, and individual use of a model requires both (Huang et al., 2020).
• A deep learning model predicted 55.8 percent of acute kidney injury episodes up to 48 hours in advance, at the cost of two false alerts for each true one (Tomašev et al., 2019).
• When taken to other hospitals, an acute kidney injury model tends to lose performance (Song et al., 2020).
• In randomized trials, automatic acute kidney injury alerts changed conduct, but did not improve patient outcomes (Wilson et al., 2021; Li et al., 2024).
• The WHO asks that ethics and human rights be at the center of the design and use of AI in health (WHO, 2021).

Summary

• Why predict the kidney before it fails
• The KFRE: four variables, discrimination and calibration
• The KFRE in primary care and screening
• Machine learning in AKI and external validation
• Alerts are not enough: what the trials show
• Ethics, equity and governance
• How to evaluate a model before trusting it
• How NEXO helps train this reasoning
• Frequently asked questions
• Take-home messages

Why predict the kidney before it fails

Two problems, two time scales

CKD evolves over years, and the clinical question is who will need dialysis or a transplant and when. CKD is increasing in incidence and prevalence worldwide, and identifying who is at risk of progression can improve nephrology care (Tangri et al., 2016). Staging the disease by filtration and albuminuria, as in the text on staging CKD according to KDIGO 2024, is already a simple way to estimate this risk.

Acute kidney injury (AKI) happens in hours or days. It complicates between 10 and 15 percent of hospital admissions and more than half of intensive care admissions, according to the review cited by Song and colleagues (Song et al., 2020). Those who survive an episode have a higher risk of cardiovascular disease and of progression to CKD (Song et al., 2020).

What a good prediction allows

In CKD, a reliable estimate helps decide who to refer to the nephrologist, when to prepare vascular access and how to talk with the patient about the future. In AKI, the expected gain is to act before creatinine rises: review nephrotoxic drugs, adjust fluids and monitor urine output. The text on drug-induced kidney injury shows how many of these decisions depend on recognizing the risk in time.

A prediction only has value if it changes some decision. This is the same logic as pretest probability: a number only helps when it shifts the probability to a side where the conduct changes.

The KFRE: four variables, discrimination and calibration

How the equation was born

The KFREs were developed in 3449 patients with CKD stages 3 to 5 referred to a nephrologist in Ontario, Canada, and validated in patients referred in British Columbia (Tangri et al., 2016). The preferred versions have four and eight variables, all based on age, sex and laboratory tests (Tangri et al., 2016).

The four-variable version needs only age, sex, estimated glomerular filtration rate and urine albumin-to-creatinine ratio (Tangri et al., 2016). Because all these data are already in the laboratory system, the risk can be calculated and reported automatically whenever the tests are done (Tangri et al., 2016).

Validation in 30 countries

To find out whether the equation worked outside Canada, the CKD Prognosis Consortium assembled 31 cohorts with 721,357 participants with CKD stages 3 to 5, in more than 30 countries on four continents (Tangri et al., 2016). Over a median follow-up of 4 years, there were 23,829 episodes of kidney failure, defined as starting dialysis or transplantation (Tangri et al., 2016).

Discrimination was excellent, with a C statistic of 0.90 at 2 years and 0.88 at 5 years, and was similar across subgroups of age, race and diabetes (Tangri et al., 2016). Equations refitted with data from all the cohorts combined brought no improvement (Tangri et al., 2016).

Regional adjustment

Calibration was adequate in the North American cohorts, but the original equation overestimated risk in some cohorts from other regions (Tangri et al., 2016). A calibration factor that reduced baseline risk by 32.9 percent at 2 years and by 16.5 percent at 5 years improved calibration in 12 of 15 non-North American cohorts at 2 years and in 10 of 13 at 5 years (Tangri et al., 2016).

The proportion of patients with predicted risk above 20 percent at 2 years ranged from 0.23 percent in an Okinawa cohort to 50 percent in the European CRIB cohort (Tangri et al., 2016). The same equation is therefore applied to populations with very different baseline risks, and that is where local calibration matters.

Ranking patients or getting the number right

Discrimination measures the ability to rank patients by risk; calibration measures the agreement between estimated risk and actual risk (Huang et al., 2020). A model can separate very well those at higher and lower risk and at the same time get the absolute value of the probability wrong.

The performance of clinical models is usually published with measures of discrimination, but use in individual predictions requires adequate calibration (Huang et al., 2020). For the patient in the case, the difference between 24 and 12 percent risk can change the conversation about vascular access, even if the order between him and other patients stays the same.

How calibration is measured

The tutorial by Huang and colleagues describes measures such as global calibration, which compares the expected number of events with the observed number, and the Brier score, which is the mean squared error between the actual outcome and the estimated probability (Huang et al., 2020). The authors note that the Hosmer-Lemeshow test is still widely used, despite known limitations (Huang et al., 2020).

An example from the tutorial itself shows the problem: community dementia risk models greatly overestimated incidence, and at a predicted risk of 40 percent the observed incidence was only 10 percent (Huang et al., 2020). An error of that order in a kidney model would lead to unnecessary referrals, tests and conversations.

A summary for reading articles

Property	Question it answers	How it usually appears	Example in nephrology
Discrimination	Does the model put higher-risk patients first?	C statistic or area under the ROC curve	KFRE with a C statistic of 0.90 at 2 years (Tangri et al., 2016)
Calibration	Does the predicted risk match the observed risk?	Global calibration, calibration curves, Brier score	Overestimation outside North America (Tangri et al., 2016)
External validation	Does performance hold up elsewhere?	Cohorts from other hospitals or countries	Drop in performance between hospitals (Song et al., 2020)
Clinical impact	Does using the model improve outcomes?	Randomized trial	AKI alerts without gain in outcomes (Wilson et al., 2021)

Reading a risk model with these four questions is similar to the calibration of confidence in the diagnosis: the question is whether the stated certainty matches the actual accuracy.

The KFRE in primary care and in triage

Validation in primary care

Most CKD is followed outside nephrology services, and the KFRE had little external validation in primary care (Major et al., 2019). A British study included 35,539 people with CKD in an adult population of 630,504 seen in primary care (Major et al., 2019).

In this cohort, 176 people reached kidney failure within 2 years and 429 within 5 years, and discrimination was excellent, with a C statistic of 0.932 at 2 years and 0.924 at 5 years (Major et al., 2019). The equation, however, overestimated risk in the lower-risk groups, and reducing the baseline risk improved calibration in those groups at the cost of some underestimation in the higher-risk ones (Major et al., 2019).

Risk-based referral criteria

The authors simulated a referral criterion based on a 5-year kidney failure risk of 5 percent or more by the KFRE, with or without an albumin-to-creatinine ratio of 70 mg/mmol or more (Major et al., 2019). Compared with the criteria in use, this threshold reduced the number of people eligible for referral who did not progress to kidney failure and referred earlier, at higher filtration, those who did progress (Major et al., 2019).

The authors themselves warn that the cohort came from a single region and that validation in more diverse populations is needed before implementation (Major et al., 2019).

The experience of a Canadian province

In Manitoba, Canada, the nephrology service began in 2012 using a 5-year risk threshold of 3 percent by the KFRE to triage new referrals of patients with CKD stages 3 to 5 (Hingwala et al., 2017). Referrals below this threshold, and outside criteria such as pregnancy or suspected glomerulonephritis, went back to primary care with recommendations (Hingwala et al., 2017). In pregnant women with CKD, kidney and obstetric risk follow their own logic, discussed in the text on pregnancy in chronic kidney disease.

The median waiting time fell from 230 days before triage to 58 days after, although the monthly number of referrals rose from 68 to 94 (Hingwala et al., 2017). The authors acknowledge that they did not follow patients classified as low risk over the long term (Hingwala et al., 2017).

Machine learning in AKI and external validation

What the deep learning model showed

In 2019, a group from DeepMind and the United States Department of Veterans Affairs trained a model on electronic records of 703,782 adults from 172 inpatient sites and 1062 outpatient clinics (Tomašev et al., 2019). At each moment of the admission, the model estimates the probability of AKI of any stage in the following 48 hours (Tomašev et al., 2019).

The model predicted 55.8 percent of AKI episodes during admission and 90.2 percent of the AKIs that later required dialysis, up to 48 hours in advance and with two false alerts for each true one (Tomašev et al., 2019). At this cutoff, the model would lead to a daily clinical assessment of 2.7 percent of inpatients (Tomašev et al., 2019).

The numbers that stay out of the headline

The area under the ROC curve was 92.1 percent, but the area under the precision-recall curve was 29.7 percent (Tomašev et al., 2019). The difference reflects the rarity of the event: when few patients have AKI, even a model that ranks well generates many false alarms.

The authors acknowledge that the population does not represent the world population: women were 6.38 percent of patients, and performance was lower in this group (Tomašev et al., 2019). Validating the system in a general population would require training and evaluating the model on other representative datasets (Tomašev et al., 2019).

Internal and external validation in three health systems

Churpek and colleagues tested a gradient boosting model on 495,971 adult hospitalizations in three US health systems (Churpek et al., 2020). The area under the curve for predicting stage 2 or higher AKI within 48 hours was 0.85 to 0.86 in all three, and the model identified stage 2 AKI a median of 27 to 39 hours before creatinine doubled (Churpek et al., 2020).

The authors note that they defined AKI by creatinine alone, because hourly urine output was not available for everyone, and that the ideal threshold between detection and false alarms still needs interventional trials (Churpek et al., 2020).

Why a model gets worse when it changes hospitals

Song and colleagues developed an AKI model in one health system and tested it in six independent systems in the PCORnet network (Song et al., 2020). The work showed that a drop in performance between sites is likely and attributed that drop to heterogeneity of risk factors across populations (Song et al., 2020).

The authors' review sums up the pattern in the literature: traditional AKI models had an area under the curve of 0.71 to 0.80 in derivation studies, 0.66 to 0.80 in internal validation and 0.65 to 0.71 in external validation (Song et al., 2020). The authors also recall that the DeepMind model was trained on records of patients who were mostly men and white (Song et al., 2020).

Data that change their name

Part of the loss comes from technical details. In one of the examples, a medication variable was missing at every external validation site because of differences in how drugs are recorded (Song et al., 2020). The authors proposed a measure to estimate, before deployment, how much a model should lose when it is transported (Song et al., 2020).

For those who use the tool, the lesson is practical: a model validated in another country or in another type of hospital needs to be tested with local data before it guides management. KFRE showed the same principle with the regional calibration factor (Tangri et al., 2016).

Alerts are not enough: what the trials show

A multicenter trial with an alert in the medical record

Wilson and colleagues randomized 6030 hospitalized adults with AKI, in six hospitals, to receive or not an alert in the electronic medical record with a set of suggested prescriptions (Wilson et al., 2021). The primary outcome combined AKI progression, dialysis or death within 14 days (Wilson et al., 2021).

The outcome occurred in 21.3 percent of the alert group and in 20.9 percent of the usual care group, a difference without statistical significance (Wilson et al., 2021). In the two non-university hospitals, however, the alert was associated with more poor outcomes and more deaths, 15.6 percent against 8.6 percent (Wilson et al., 2021).

Why the alert can get in the way

The authors assessed whether alert fatigue would explain the worsening and did not find that association; they also found no single explanation for the possible harm (Wilson et al., 2021). They conclude that the heterogeneity of the effect across hospitals should lead to a reassessment of the AKI alert systems in use (Wilson et al., 2021). The text on alert fatigue discusses why too many warnings lose their force.

A trial with an alert on the cell phone

In a university hospital in Nanjing, China, Li and colleagues randomized 2208 adults with AKI to receive or not an alert sent to the doctors' cell phones, together with a package of general care measures (Li et al., 2024). The maximum change in estimated filtration over 7 days was similar in both groups, and so were the patient-centered outcomes, such as death and dialysis (Li et al., 2024).

The alert changed behavior: the alerted group received more intravenous fluids, fewer nonsteroidal anti-inflammatory drugs (5.0 against 11.0 percent) and had more records of AKI at discharge (Li et al., 2024). The authors suggest combining the alert with high-quality interventions in future studies (Li et al., 2024).

What these trials teach

Predicting or detecting AKI is only the first step; the benefit depends on what the team does with the warning. A model with a good area under the curve, tied to a weak clinical response, may change nothing for the patient. The same holds for other emergencies recognized by scores, as in sepsis in nursing triage.

Ethics, equity and governance

The WHO guidance

In 2021, the WHO published the guidance Ethics and governance of artificial intelligence for health, the result of consultations with an international panel of experts in ethics, digital technology, law and human rights and with ministries of health (WHO, 2021). The document describes ethical risks and proposes six consensus principles so that AI benefits the public in all countries (WHO, 2021).

The WHO statement warns against overestimating the benefits of AI at the expense of investments essential to universal health coverage (WHO, 2021 statement). It also recalls that systems trained mostly on data from high-income countries may perform poorly in middle- and low-income settings (WHO, 2021 statement).

The position of the nephrology society

In 2026, a working group of the American Society of Nephrology published principles for the responsible use of AI in kidney care: prioritize patient benefit, ensure clinical oversight and encourage innovation in diseases with a large burden (Tangri et al., 2026). The starting point is that the physician always remains in the decision loop (Tangri et al., 2026).

The document points to data quality, equity, transparency and integration into the clinical workflow as challenges (Tangri et al., 2026). In kidney transplantation, the authors cite smaller, fragmented databases and the risk of bias as important barriers (Tangri et al., 2026).

The risk of trusting too much

When a number appears on the screen, there is a temptation to accept it without question. The text on automation bias explains how excessive trust in automated systems leads to errors, and the text on the use of artificial intelligence in clinical reasoning discusses the role of AI more broadly.

How to assess a model before trusting it

Questions for the article

Before using a kidney risk model, it is worth asking five questions of the article that describes it:

1. In what population was it developed and who was left out? The DeepMind model, for example, had few women (Tomašev et al., 2019).
2. Was there external validation in another population? Areas under the curve usually drop in external validation (Song et al., 2020).
3. Was calibration reported, and not only discrimination? Individual use requires both (Huang et al., 2020).
4. How many false alerts does the model generate per correct one? In the DeepMind model, there were two false for each true one (Tomašev et al., 2019).
5. Did any trial show benefit for the patient? The AKI alert trials showed a change in management, but not in outcome (Wilson et al., 2021; Li et al., 2024).

Questions for the service

In the service, the questions change. Do the local data have the same format as the development data? Differences in how medications are recorded have already knocked out entire variables when transferring between hospitals (Song et al., 2020). Who will respond to the alert, with what management and in how much time? Recording the calculated risk and the decision made in the medical record, as in the text on how to organize clinical reasoning in the medical record, helps audit the use later.

Back to the case

For the 68-year-old man, the 24 percent risk over 5 years by KFRE comes from an equation with good discrimination in many countries, but whose calibration needs to be checked in the local population (Tangri et al., 2016). In a service that used the 5 percent threshold simulated in the British study, he would already be referred to the nephrologist (Major et al., 2019).

The AKI alert during hospitalization deserves attention, but the benefit depends on the clinical response, since the alert alone changed management without improving outcomes (Li et al., 2024). If he also has CKD anemia, the text on anemia in CKD completes the assessment. Dealing with the uncertainty that remains, as discussed in diagnostic uncertainty, is part of the mature use of any model, and clinical decision-making remains with the physician and the patient.

How NEXO helps train this reasoning

In NEXO, the case of the day is fictional, with a real clinical presentation and up to six progressive clues, and you propose the diagnosis by finding the code in ICD-10. Reading a risk model requires the same exercise: weigh each new piece of information, revise the probability and decide when it is enough to act, as in training differential diagnosis with clues.

The limits on using the game are described on the NEXO Economy page. If you research the teaching of clinical reasoning, the page for Researchers explains how to get in touch. Professionals who write or review cases find information under Curator and Creators, and institutions can learn about the proposal on the partnerships page. Finding the right ICD-10 code completes this training.

Frequently asked questions

What is the KFRE?

It is an equation that estimates the risk of kidney failure, defined as dialysis or transplant, at 2 and 5 years; the four-variable version uses age, sex, estimated filtration and the albumin-to-creatinine ratio (Tangri et al., 2016).

Does the KFRE work in any country?

Discrimination was excellent in 31 cohorts from more than 30 countries, but some regions needed a calibration factor (Tangri et al., 2016).

What is the difference between discrimination and calibration?

Discrimination measures whether the model orders patients by risk; calibration measures whether the predicted risk matches the observed risk (Huang et al., 2020).

Can AI already predict acute kidney injury?

Machine learning models predicted part of the episodes up to 48 hours in advance, at the cost of false alerts (Tomašev et al., 2019; Churpek et al., 2020).

Do automatic AKI alerts improve outcomes?

In the randomized trials cited, the alerts changed management, but did not reduce progression, dialysis or death (Wilson et al., 2021; Li et al., 2024).

Can I use a model validated in another country?

With caution: performance tends to fall when the population changes, and local validation is recommended (Song et al., 2020).

Take-home messages

• Use the KFRE as an estimate of the risk of kidney failure in CKD stages 3 to 5, remembering that calibration may vary by region (Tangri et al., 2016).
• Require discrimination and calibration before using a model for individual decisions (Huang et al., 2020).
• Consider risk-based referral criteria, which reduced waits and low-risk referrals (Hingwala et al., 2017; Major et al., 2019).
• Ask how many false alerts the AKI model generates for each correct one (Tomašev et al., 2019).
• Test the model with local data before deploying it (Song et al., 2020).
• Tie every alert to a defined clinical response and measure the effect on outcomes (Wilson et al., 2021).
• Keep the physician in the decision loop and assess equity and transparency (Tangri et al., 2026; WHO, 2021).

Milestone inventory

• 2016: the KFRE is validated in 31 cohorts from more than 30 countries (Tangri et al., 2016).
• 2017: Manitoba reports screening referrals with the KFRE (Hingwala et al., 2017).
• 2019: the KFRE is validated in UK primary care (Major et al., 2019).
• 2019: a deep learning model predicts AKI up to 48 hours in advance (Tomašev et al., 2019).
• 2020: studies show external validation in three systems and loss of performance between hospitals (Churpek et al., 2020; Song et al., 2020).
• 2021: WHO publishes its guidance on the ethics and governance of AI in health (WHO, 2021).
• 2021: a multicenter trial of AKI alerts shows no benefit in outcomes (Wilson et al., 2021).
• 2026: the American Society of Nephrology publishes principles for the responsible use of AI (Tangri et al., 2026).

References

1. World Health Organization. Ethics and governance of artificial intelligence for health: WHO guidance. Geneva: WHO; 2021. ISBN 9789240029200. Full text.
2. World Health Organization. WHO issues first global report on Artificial Intelligence (AI) in health and six guiding principles for its design and use. Press release, 28 Jun. 2021. Full text.
3. Tangri N, Grams ME, Levey AS, et al. Multinational Assessment of Accuracy of Equations for Predicting Risk of Kidney Failure: A Meta-analysis. JAMA. 2016;315(2):164-174. PMID 26757465. DOI 10.1001/jama.2015.18202.
4. Major RW, Shepherd D, Medcalf JF, et al. The Kidney Failure Risk Equation for prediction of end stage renal disease in UK primary care: An external validation and clinical impact projection cohort study. PLoS Med. 2019;16(11):e1002955. PMID 31693662. DOI 10.1371/journal.pmed.1002955.
5. Hingwala J, Wojciechowski P, Hiebert B, et al. Risk-Based Triage for Nephrology Referrals Using the Kidney Failure Risk Equation. Can J Kidney Health Dis. 2017;4:2054358117722782. PMID 28835850. DOI 10.1177/2054358117722782.
6. Tomašev N, Glorot X, Rae JW, et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature. 2019;572(7767):116-119. PMID 31367026. DOI 10.1038/s41586-019-1390-1.
7. Churpek MM, Carey KA, Edelson DP, et al. Internal and External Validation of a Machine Learning Risk Score for Acute Kidney Injury. JAMA Netw Open. 2020;3(8):e2012892. PMID 32780123. DOI 10.1001/jamanetworkopen.2020.12892.
8. Song X, Yu ASL, Kellum JA, et al. Cross-site transportability of an explainable artificial intelligence model for acute kidney injury prediction. Nat Commun. 2020;11(1):5668. PMID 33168827. DOI 10.1038/s41467-020-19551-w.
9. Wilson FP, Martin M, Yamamoto Y, et al. Electronic health record alerts for acute kidney injury: multicenter, randomized clinical trial. BMJ. 2021;372:m4786. PMID 33461986. DOI 10.1136/bmj.m4786.
10. Li T, Wu B, Li L, et al. Automated Electronic Alert for the Care and Outcomes of Adults With Acute Kidney Injury: A Randomized Clinical Trial. JAMA Netw Open. 2024;7(1):e2351710. PMID 38241047. DOI 10.1001/jamanetworkopen.2023.51710.
11. Huang Y, Li W, Macheret F, et al. A tutorial on calibration measurements and calibration models for clinical prediction models. J Am Med Inform Assoc. 2020;27(4):621-633. PMID 32106284. DOI 10.1093/jamia/ocz228.
12. Tangri N, Cheungpasitporn W, Crittenden SD, et al. Responsible Use of Artificial Intelligence to Improve Kidney Care: A Statement from the American Society of Nephrology. J Am Soc Nephrol. 2026;37:881-890. PMID 41201255. DOI 10.1681/asn.0000000929.

Play the case of the day on NEXO.