Critical appraisal of articles for medical students

At the study group meeting, a colleague shares the headline: "new treatment halves mortality". The abstract looks convincing, the journal is well known, and someone is already asking why the hospital has not adopted the approach yet. You open the full text. The study was stopped early, had few events, and the primary outcome changed between registration and publication. None of that was in the abstract. None of that was in the headline.

Critical appraisal of articles is the skill of looking at a study and asking whether it is reliable, what it actually showed and whether the result applies to your patient. For the medical student, it is the bridge between research and clinical decisions, and also a protection against overconfidence in any publication. This guide explains why to read critically, how a three-question script works, which study designs have which weaknesses, which biases and reporting pitfalls appear most often, what the evidence shows about teaching this skill and how to practice it day to day.

Key points

• Critical appraisal means assessing the validity, the results and the applicability of a study before using it to decide.
• Students nearing the end of the course often master only part of the concepts of evidence-based medicine, especially the quantitative ones.
• Each study design best answers one type of question and has characteristic weaknesses.
• Abstracts can distort results: biased presentation appears often in trials without a significant difference in the primary outcome.
• Trials stopped early for benefit tend to overestimate the treatment effect, especially when they have few events.
• Teaching critical appraisal improves knowledge and attitudes, but there is little evidence of transfer to clinical practice.

Contents

• Why to read critically
• A three-question script
• Study designs and their weaknesses
• Reading diagnostic studies
• Biases every student should recognize
• Reporting and interpretation pitfalls
• How to learn critical appraisal
• How NEXO connects the article to the case
• Frequently asked questions
• Critical appraisal checklist

Why to read critically

Evidence-based medicine was defined by Sackett and colleagues as the conscientious, explicit and judicious use of the best available evidence in decisions about the care of each patient, integrating that evidence with clinical experience and the patient's values. The word "judicious" is the heart of critical appraisal. It is not enough to know that a study exists; you need to know whether it deserves trust and whether it answers the question you have before a concrete patient.

Not everything published is true

In a widely cited essay, Ioannidis argued that the probability of a research finding being true falls when studies are small, when effects are small, when there is great flexibility in design, definitions and analyses, when there are financial or other interests and when many teams compete for statistical significance. The text is not an invitation to cynicism, but a reminder that the mere fact that a result is published does not make it reliable.

What students know

In a cross-sectional study with third- and fourth-year students, Smith and colleagues applied the Fresno test of evidence-based medicine and found a mean performance of 47.2%; the questions on number needed to treat (16.9% correct) and on diagnostic studies (5.0%) were the hardest. In other words, students close to graduation mastered about half of the concepts, and precisely the most quantitative ones were the weakest. This is a common gap and entirely correctable with practice.

Critical appraisal is clinical reasoning

Reading an article critically uses the same skills as interpreting a test: understanding what the result means, what the chance is that it is wrong and whether it changes the probability of what you are thinking. The guide on pretest probability shows this logic applied to tests, and the text on clinical decision-making shows how to bring together evidence, context and patient preferences.

A script in three questions

There are many critical appraisal scripts, with versions specific to each study design. To start, three questions cover the essentials and work for any article.

1. Are the results valid? Is the design suited to the question? Was there randomization and allocation concealment? Were the groups comparable at the start? Were losses to follow-up small and balanced? Did the assessors know which group each participant belonged to?
2. What are the results? What is the size of the effect, not just whether it is statistically significant? What is the confidence interval? What is the effect in absolute terms, such as risk difference or number needed to treat?
3. Do the results apply to my patient? Do the study participants resemble my patient? Do the outcomes measured matter to them? Do the benefits outweigh the risks, costs and effort?

Start with the study question

Before the three questions, identify the question the study tried to answer, in a structured format: who the participants are, what the intervention or exposure was, what it was compared with and what the outcome was. Many confusions in reading come from not noticing that the study answered a different question from the one you had.

Relative effect and absolute effect

A halving of relative risk looks enormous. If the risk was two in every hundred people and fell to one in every hundred, the absolute difference is one person in every hundred, and you need to treat one hundred people to prevent one event. Both numbers are correct, but they tell very different stories. Critical reading asks you to always look for the absolute effect and compare it with the harms of treatment.

Quality of evidence

The GRADE system, described by Guyatt and colleagues, proposes an explicit way to grade the quality of a body of evidence and the strength of a recommendation. Even if you are not going to apply GRADE formally, the logic is useful: randomized trials start at high quality and observational studies at low quality, and confidence rises or falls according to risk of bias, inconsistency, imprecision, indirect evidence and publication bias.

Study designs and their weak points

Each design answers one type of question better and fails in characteristic ways. Knowing these weak points saves time: you already know where to look for problems.

Design	Best for	Typical weak point	What to check first
Randomized clinical trial	Effect of treatments	Early stopping, losses, surrogate outcomes	Randomization, concealment, blinding, intention-to-treat analysis
Cohort	Prognosis and risk factors	Confounding and losses to follow-up	Comparability of groups and adjustment
Case control	Rare diseases and exposures	Recall bias and selection of controls	How cases and controls were chosen
Cross-sectional	Prevalence and association	Does not establish temporal sequence	Representativeness of the sample
Diagnostic accuracy study	Performance of tests	Inadequate patient spectrum and flawed reference standard	Who was tested and with which reference
Systematic review and meta-analysis	Synthesis of several studies	Quality of the included studies, heterogeneity, publication bias	Search, inclusion criteria and risk of bias assessment

Randomized trials

The randomized trial is the strongest design for assessing treatments, because randomization distributes in a balanced way the known and unknown factors that could influence the outcome. Even so, it can fail: poorly done allocation concealment, unbalanced losses, surrogate outcomes that do not reflect what matters to the patient and early stopping that inflates the effect.

Observational studies

Cohorts and case control studies are indispensable when randomization is not possible or not ethical, and for studying prognosis and risk factors. The big risk is confounding: a third variable that is associated with both the exposure and the outcome and creates an apparent association. Statistical adjustments help, but they only control what was measured.

Systematic reviews

A good systematic review defines the question, searches the literature broadly and reproducibly, assesses the risk of bias in each study and synthesizes the results. A meta-analysis of bad studies is still bad, and many small studies with positive results can reflect publication bias. So the question "which studies were included and what is their quality?" comes before looking at the combined result.

Reading diagnostic studies

For those who study clinical reasoning, diagnostic accuracy studies deserve special attention. They tell how much a test, a sign or a decision rule helps separate those who have a condition from those who do not. And, as the Fresno test showed, questions about diagnosis are precisely the ones students get wrong most often.

Sensitivity, specificity and likelihood ratios

Sensitivity is the proportion of people with the condition that the test identifies; specificity is the proportion of people without the condition that the test correctly classifies as negative. Likelihood ratios combine the two and tell how much a positive or negative result changes the probability of the hypothesis. The farther from one, the more useful the result. This is the direct link between the article and the patient: with the probability before the test and the likelihood ratio, you estimate the probability after the test.

The spectrum of patients

A test tried in very sick patients compared with healthy volunteers looks excellent, because the two groups are easy to tell apart. In the real world, the test is used in patients with an uncertain picture, and performance drops. When reading an accuracy study, ask who was tested: patients similar to the ones you see, with the same diagnostic question, or a group selected to make the result look better?

The reference standard

Every accuracy study compares the test with a reference standard, which defines who really has the condition. If the standard is flawed, or if not all participants undergo it, the result is distorted. It is also important that whoever interprets the test does not know the reference standard result, and vice versa. These questions seem technical, but they greatly change the confidence you can have in the final number.

From the number to the decision

A test with good accuracy is only useful if the result changes management. If the probability of the hypothesis is already very high or very low, no result will change the decision, and the test adds cost and risk without benefit. Thinking about this before ordering is a way to apply critical reading at the bedside.

Biases every student should recognize

Bias is a systematic error that skews the result in one direction. Unlike chance, it does not shrink with larger studies. Some types appear so often that they are worth knowing by name.

• Selection bias: the participants chosen do not represent the population of interest, or the groups compared differ from the start.
• Measurement bias: the outcome is measured differently between groups, often because the assessor knew who received the treatment.
• Confounding: a third variable explains the association between exposure and outcome.
• Loss to follow-up: participants who leave the study may differ from those who stay, and the result changes.
• Publication bias: studies with positive results are more likely to be published, and the literature becomes more optimistic than reality.
• Selective outcome reporting: the article highlights the outcomes that worked and sets aside the ones that did not.

Bias and chance

The confidence interval shows the uncertainty due to chance. It says nothing about bias. A huge study with a narrow interval can be systematically wrong if it has a design problem. That is why validity comes before results in the script: there is no point interpreting with precision a number that does not deserve trust.

The reader has biases too

Readers bring biases too. We tend to accept more easily the results that confirm what we already thought and to look for flaws in those that contradict us. A good practice is to apply the same script to all articles, including the ones you wish were right.

Reporting and interpretation traps

Even well-done studies can be presented in a way that exaggerates the results. Some traps appear so often that they deserve special attention.

Biased presentation in abstracts

In a sample of randomized trials published with a primary outcome that showed no statistically significant difference, Boutron and colleagues found biased presentation, a way of reporting that highlights a benefit the primary outcome did not show, in the abstract conclusions of 58.3% of the articles, and more than 40% of the articles had this distortion in at least two sections of the main text. Reading only the abstract is therefore risky. Go to the primary outcome declared in the methods and see what happened to it.

Trials stopped early

Trials that stop earlier than planned because of an apparent benefit draw a lot of attention. The systematic review by Montori and colleagues showed that these trials often fail to report important information about the decision to stop and present implausibly large effects, especially when the number of events is small. Then Bassler and colleagues compared trials stopped early with meta-analyses of trials not stopped early on the same questions and found larger effects in the stopped trials, with more marked differences in studies with fewer events; in 62% of the questions, the trials not stopped early showed no significant benefit.

Surrogate and composite outcomes

A treatment that improves a laboratory test does not always improve the patient's life. Surrogate outcomes, such as a laboratory measure in place of clinical events, can mislead. Composite outcomes, which combine serious and mild events, can hide the fact that the effect came only from the least important component. Always ask: which outcome changed, and does it matter to the patient?

Subgroups

Subgroup analyses generate hypotheses but rarely prove anything. When many subgroups are tested, some will show differences by chance. Be wary of conclusions based on a subgroup that was not planned in advance, especially when the overall result was negative. A convincing subgroup is usually defined before the study, has a plausible biological explanation and repeats in other independent studies.

How to learn critical reading

Critical reading is a skill, and skills are learned through deliberate practice and feedback. What the evidence shows about teaching helps in choosing formats.

What the evidence shows about teaching

The BEME review by Ahmadi and colleagues on teaching evidence-based medicine in undergraduate education included 27 studies and concluded that some strategies can improve knowledge, attitudes and skills, with no clear superiority of one method and no evidence of transfer to clinical practice; distance learning was as effective as traditional teaching in good-quality studies. The Cochrane review by Horsley and colleagues found only three studies on teaching critical appraisal to health professionals, none with care process or patient outcomes, and concluded that low-intensity interventions may bring modest gains. In a randomized trial with sixth-year students, Sasannia and colleagues observed a large effect on knowledge and confidence and a medium effect on attitudes and skills after training with multimedia and workshops.

Journal clubs

The journal club is the most traditional format for teaching critical reading. The systematic review by Deenadayalan and colleagues identified features of effective clubs: regular and scheduled meetings, mandatory attendance, a clear objective, a suitable time, a leader trained to choose the articles and lead the discussion, prior circulation of the texts, use of critical appraisal checklists and a final summary of the conclusions. In undergraduate education, the review by Bello and Grant showed that student satisfaction dropped when the session went over two hours, and that combining the club with classes, mentoring or appraisal checklists helped with acquiring knowledge and skills.

A plan for the student

1. Choose one article per week linked to a case you saw on your rotation.
2. Write the study question in a structured format before reading the results.
3. Use a checklist suited to the design and answer the three questions in writing.
4. Calculate the absolute effect and the number needed to treat or to harm.
5. Compare the abstract with the full text and note whether there is a difference between what was highlighted and the primary outcome.
6. Discuss it with someone in a journal club, in a student league or with your preceptor.

How NEXO connects the article to the case

Critical reading makes sense when it is tied to a patient. In NEXO, the case of the day arrives in up to six clues, and with each guess the proximity in ICD-10 across five levels shows how far your hypothesis was from the diagnosis. The cases are fictional with a real clinical presentation. A good exercise is, after playing, to pick a question the case raised, such as the value of a test or the probability of a hypothesis, and look for an article to answer it using the three questions in this guide.

The guide what NEXO is explains the whole game, and the text on how to train differential diagnosis with clues helps turn the case into questions. How the modes are unlocked is on the NEXO Economy page.

If you coordinate a medical school, a student league or a residency and want to use NEXO with a class, see NEXO Closer. Researchers in medical education can propose studies through NEXO Researcher. Physicians who review cases take part through NEXO Curator, and those who create health content on social media can take part through NEXO Creators.

Frequently asked questions

What is critical reading of articles?

It is the process of assessing the validity, the results and the applicability of a study systematically, before using it to make clinical decisions.

Do I need to know advanced statistics?

No. Basic concepts such as relative and absolute risk, number needed to treat, confidence interval, sensitivity and specificity are enough to assess most clinical articles.

Can I trust the article abstract?

With care. Studies show that abstracts of trials with no significant difference in the primary outcome often highlight benefits that the primary outcome did not show. Always check the methods and the full results.

Which study design is the best?

It depends on the question. Randomized trials are the strongest for treatments; cohorts, for prognosis; accuracy studies, for tests. Systematic reviews of good studies sit at the top when they exist.

Do journal clubs work?

Well-organized journal clubs improve knowledge and critical appraisal skills. They work best with regular meetings, a trained leader, prior reading, appraisal checklists and sessions of up to two hours.

Where should I start?

Pick an article tied to a case you have seen, write the study question, answer the three checklist questions and discuss it with someone. Repeat every week.

Critical reading checklist

• Write the study question: participants, intervention, comparison and outcome.
• Check whether the design suits the question.
• Assess randomization, allocation concealment, blinding and loss to follow-up.
• Look for the absolute effect and the confidence interval.
• Compare the abstract with the primary outcome stated in the methods.
• Be wary of trials stopped early, surrogate outcomes and unplanned subgroups.
• Ask whether the participants resemble your patient and whether the outcome matters to them.

Want to turn case questions into reading questions? Play the case of the day on NEXO.

References

1. Ahmadi SF, Baradaran HR, Ahmadi E. Effectiveness of teaching evidence-based medicine to undergraduate medical students: a BEME systematic review. Med Teach. 2015. PubMed 25401408
2. Horsley T, Hyde C, Santesso N, Parkes J, Milne R, Stewart R. Teaching critical appraisal skills in healthcare settings. Cochrane Database Syst Rev. 2011. PubMed 22071800
3. Sasannia S, Amini M, Moosavi M, Askarinejad A, Moghadami M, Ziaee H, et al. Critical appraisal skills training to undergraduate medical students: A Randomized Control Study. J Adv Med Educ Prof. 2022. PubMed 36310666
4. Bello JO, Grant P. A systematic review of the effectiveness of journal clubs in undergraduate medicine. Can Med Educ J. 2023. PubMed 37719396
5. Deenadayalan Y, Grimmer-Somers K, Prior M, Kumar S. How to run an effective journal club: a systematic review. J Eval Clin Pract. 2008. PubMed 19018924
6. Smith AB, Semler L, Rehman EA, Haddad ZG, Ahmadzadeh KL, Crellin SJ, et al. A Cross-Sectional Study of Medical Student Knowledge of Evidence-Based Medicine as Measured by the Fresno Test of Evidence-Based Medicine. J Emerg Med. 2016. PubMed 26961178
7. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005. PubMed 16060722
8. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ. 1996. PubMed 8555924
9. Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010. PubMed 20501928
10. Montori VM, Devereaux PJ, Adhikari NK, Burns KE, Eggert CH, Briel M, et al. Randomized trials stopped early for benefit: a systematic review. JAMA. 2005. PubMed 16264162
11. Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, et al. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008. PubMed 18436948
12. Bassler D, Briel M, Montori VM, Lane M, Glasziou P, Zhou Q, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010. PubMed 20332404