Review by playing: why testing helps memory

In a classic experiment about memory, students who reread a text several times came out more confident that they would remember it than those who tested themselves (Roediger and Karpicke, 2006). Days later, it was those who tested themselves who remembered more (Roediger and Karpicke, 2006). The feeling of knowing and actually knowing can go in opposite directions, and that is why the way you review matters as much as the time you spend reviewing.

NEXO is a daily medical diagnosis game: one clinical case a day, six progressive clues, and ICD-10 proximity feedback across five levels.

This text is about how to review Medicine by testing memory: what the testing effect is and what meta-analyses show, which test formats yield more, what is known about goals, points and sequence, and how NEXO fits into review. The principles of the test, miss and correct cycle are in the text about how to fit games into your routine, and their application to the basic cycle subjects is in the text about how to study Physiology, Biochemistry and Histology. Spaced repetition and Anki will have their own text.

Main points

• Testing yourself yields more delayed retention than rereading, even when rereading gives a greater feeling of mastery.
• The testing effect repeats in the classroom: in a meta-analysis with more than 48 thousand students, testing raised performance to an average level (Yang et al., 2021).
• Tests that require effortful recall give greater gains than recognition tests.
• Feedback after the test and spaced repetition expand the effect.
• Points, goals and sequence help when they bring the student back to the test; what teaches is the test with feedback.
• In NEXO, each case is a short test with immediate feedback through proximity in ICD-10.

Summary

• The testing effect
• What the meta-analyses show
• Test formats for reviewing
• Goals, points and sequence
• NEXO in review
• A week of review
• Frequently asked questions
• Review checklist

The testing effect

The testing effect is the memory gain that comes from trying to recall information, compared with studying it again. It changed the way the exam is seen: besides measuring what you know, the test helps you learn.

The experiment that became a reference

In two experiments, university students read prose texts and then took one or three free recall tests, without correction, or reread the text the same number of times (Roediger and Karpicke, 2006). The final test came 5 minutes, 2 days or 1 week later (Roediger and Karpicke, 2006).

With the final test 5 minutes later, rereading yielded more recall than testing (Roediger and Karpicke, 2006). In the delayed tests, the advantage reversed, and testing yielded much greater retention than rereading (Roediger and Karpicke, 2006). It is a laboratory study, with university students and texts, but the pattern has repeated in many studies since.

Why rereading deceives

The detail that matters most for anyone reviewing is in the confidence: those who reread thought they would remember more (Roediger and Karpicke, 2006). Rereading makes the text familiar, and familiarity feels like mastery. The test shows what you can produce without the text in front of you, which is what the exam and the patient will ask of you.

The effort of recalling

A meta-analysis on the testing effect found support for the role of effortful processing: recall tests, in which you produce the answer, gave greater benefits than recognition tests (Rowland, 2014). The same analysis found little support for the explanation that testing works only because it makes you think more about meaning, and suggests that other mechanisms linked to memory retrieval itself weigh in (Rowland, 2014).

For review, the consequence is practical: the effort of searching for the answer is part of the method. In the classic experiment, testing yielded more delayed retention than rereading, although rereading gave more confidence (Roediger and Karpicke, 2006).

Why this matters for clinical practice

A review applied to medical education recalls that research in assessment allows us to conclude that clinical expertise rests on a broad knowledge base and on memory networks that give easy access to that knowledge (Larsen et al., 2008). The authors suggest that learning through tests can strengthen that clinical knowledge (Larsen et al., 2008). In other words, reviewing by testing is training access to what you have already studied, and it is that access that the case and the exam demand. Feedback after the test helps correct mistakes and confirm correct answers (Larsen et al., 2008).

What the meta-analyses show

A single experiment can be an exception. Meta-analyses bring together dozens or hundreds of studies and show whether the effect holds outside the laboratory.

Practice tests against other forms of study

A meta-analysis on practice tests compared testing with conditions without testing, such as restudying, practicing in another way or doing distractor activities (Adesope et al., 2017). Practice tests were more beneficial than restudying and than all other comparison conditions (Adesope et al., 2017). The effect size varied with the characteristics of the test, the participants, the outcome and the study methods (Adesope et al., 2017).

In the classroom

A systematic review with meta-analysis gathered data from 48,478 students in 222 independent studies carried out in classrooms (Yang et al., 2021). Overall, testing raised academic performance to a medium degree, with an effect size g of 0.499 (Yang et al., 2021).

The effect varied with several factors, among them corrective feedback, the number of test repetitions, the match between the format of the practice test and that of the final exam, and how much the material tested coincided with what was required (Yang et al., 2021). The authors point to three explanations compatible with the data: additional exposure to the content, processing similar to what the exam demands and motivation (Yang et al., 2021).

In the health professions

In the BEME 48 systematic review, on learning through tests in the health professions, the 19 included studies reported 41 outcomes with enough data to calculate the effect (Green et al., 2018). All 7 transfer outcomes favored testing over studying, with standardized mean differences from 0.33 to 1.1 (Green et al., 2018). The authors conclude that the effect goes beyond knowledge measured in exams and reaches clinical applications (Green et al., 2018). The retention results of the same review are in the text about how to fit games into the routine.

With residents, six months later

In a randomized trial with residents, each participant took short-answer tests, with feedback, on one topic, and studied a review sheet on another: status epilepticus and myasthenia gravis (Larsen et al., 2009). Tests and study were repeated at intervals of about 2 weeks, and the final exam came around 6 months later (Larsen et al., 2009). The effect of repeated testing was large, with an effect size d of 0.91 (Larsen et al., 2009).

The design has a merit for those studying Medicine: the same resident was their own control, tested on one topic and rereading the other. The difference, therefore, comes from the form of reviewing, and not from who reviewed.

At school and in the classroom, with another definition

A systematic review of applied research in schools and classrooms coded 50 experiments, with 49 effect sizes and 5,374 participants (Agarwal et al., 2021). Most effects, 57%, showed a medium or large benefit of retrieval practice (Agarwal et al., 2021). The benefit appeared at various levels of education, content areas, intervals until the final exam and test formats (Agarwal et al., 2021). The authors recall a limitation: only 6% of the experiments were carried out outside Western, rich and industrialized countries (Agarwal et al., 2021).

What these reviews have in common

Source	Population	What it found
Roediger and Karpicke, 2006	University students, texts	Testing yielded more delayed retention; rereading gave more confidence (Roediger and Karpicke, 2006)
Rowland, 2014	Laboratory meta-analysis	Recalling with effort yielded more than recognizing (Rowland, 2014)
Adesope et al., 2017	Meta-analysis of practice tests	Testing surpassed restudying and the other comparisons (Adesope et al., 2017)
Yang et al., 2021	48,478 students in the classroom (Yang et al., 2021)	Average effect, g of 0.499 (Yang et al., 2021)
Green et al., 2018	Health professions	All 7 transfer outcomes favored testing (Green et al., 2018)
Agarwal et al., 2021	5,374 participants in schools and classrooms (Agarwal et al., 2021)	Average or large benefit in 57% of the effects (Agarwal et al., 2021)
Larsen et al., 2009	Residents, 6 months (Larsen et al., 2009)	Large effect, d of 0.91 (Larsen et al., 2009)

Test formats for reviewing

If testing helps, the next question is how to test. The studies give some clues.

Producing the answer

Recall tests give larger benefits than recognition tests (Rowland, 2014). In practice, this favors formats in which you write or say the answer before seeing options: open questions, cases with a free response, cards in which you answer before flipping.

Feedback after the test

A review applied to medical education sums up three recommendations: test often and spaced over time, use questions that require effortful recall and give feedback, which is crucial for learning from the test (Larsen et al., 2008). Corrective feedback also appears among the factors that modulate the effect in the classroom (Yang et al., 2021). What feedback does with correct answers given with little confidence is in the text about how to fit games into the routine.

Repeat and space

The number of test repetitions also modulates the effect (Yang et al., 2021). Testing once helps; returning to the same content at increasing intervals helps more, and it is the principle behind spaced repetition, the topic of the next text in this series.

Similar to the exam

The match between the format of the practice test and that of the final exam modulates the effect (Yang et al., 2021). For review, this suggests mixing formats: questions in the style of the exam you will take, and cases with a free response to train retrieval with more effort.

Exam questions and anxiety

In a study with 72 medical students at a school in the United States, test anxiety was a negative predictor of the licensing exam score, even controlling for other factors (Deng et al., 2015). The study is observational and does not show cause, but it reminds us that review is also the place to get used to answering under some pressure, in a low-risk setting.

Formats by goal

As a practical suggestion, one way to choose the format:

Goal	Format	Why
Lock in facts and numbers	Cards with the answer before flipping	Effortful recall yields more than recognition (Rowland, 2014)
Train for the exam	Questions in the exam format, with commentary	Format match and feedback modulate the effect (Yang et al., 2021)
Train diagnosis	Cases with a free response and feedback	Transfer to clinical applications (Green et al., 2018)
Review an area	Block of tests on the topic, repeated later	Test repetitions modulate the effect (Yang et al., 2021)

A review script

As a practical suggestion, four steps for each review session:

1. Try to remember before opening the material: write what you know about the topic, or answer the questions.
2. Check and correct: read the feedback for each answer, including the ones you got right.
3. Note what was missing: the mistake shows what to review.
4. Mark the next review: return to the same topic days later, testing again.

Clinical cases as a test

A clinical case with a free response is a recall test: you need to search your memory for what you know about signs, tests and diseases, and produce a name. Why the case counts as a test is in the text about how to fit games into the routine. For review, the case has an extra advantage: it covers several topics at once and forces you to choose between similar diseases, which comes close to what clinical practice demands. The transfer outcomes of the BEME 48 review, which favored testing, point in the same direction (Green et al., 2018).

Reviewing at each stage of the course

As a practical suggestion, the format changes with the stage:

Stage	Main format	What for
Basic cycle	Cause and effect questions and fact cards	Lock in mechanisms and vocabulary
Clinical cycle	Cases with a free response and questions	Link signs to diseases and train the differential
Clerkship	Questions in the exam format and short cases	Review for the exam in the gaps between rotations

How to review with little time in the internship is in how to study in the internship.

When review seems not to work

Some common situations and what to try, as a practical suggestion:

• "I got everything wrong again." Getting things wrong on a test is part of it; what matters is reading the feedback and returning to the topic days later. If the mistake repeats, the problem may be in studying the topic, and not in the review.
• "I get the questions right, but I forget in the case." Maybe you are recognizing, and not producing. Swap part of the questions for cases or open questions.
• "I don't have time to review everything." Prioritize the topics you got wrong and the ones that come up on the next exam. A short test per day already keeps the cycle going.
• "I get anxious on the questions." Start with short blocks and no timer; test anxiety was associated with lower grades in an observational study (Deng et al., 2015).

Goals, points and streak

Study games usually add elements to the test such as points, daily goals, day streak and ranking. What the studies say about them is in the text about gamification in medical education; here, the focus is how these elements connect to the testing effect.

The test at the center

In the systematic review on gamification in the health professions, no study reported negative effects of using gamification (van Gaalen et al., 2021). The same review points out that most studies did not have a well-defined control group, which limits the conclusions (van Gaalen et al., 2021). Almost all studies included assessment elements, often combined with challenge (van Gaalen et al., 2021).

An example of a spaced question game

In a trial with urologists, an email question game combined the testing effect with game mechanics: wrong questions came back in 12 days and right ones in 24, and each question was "retired" when the participant got it right twice in a row (Kerfoot and Baker, 2012). Receiving the questions grouped reduced performance, with an effect size d of 0.43 (Kerfoot and Baker, 2012). The trial compared sending rhythms, and not game against no game, so it shows how a spaced question game can work, and not that points or streak cause learning.

When game elements distract

An analysis on gamification in medical education points out that poorly applied game elements can undermine motivation, through the overjustification effect or the negative effects of competition (Rutledge et al., 2018). In the review, the warning sign is noticing that the goal became keeping the streak, and not learning from the test. Some questions help, as a practical suggestion:

• After the game, do you know what you got wrong? If so, the game is serving the review.
• Did you read the feedback, or just look at the score? The feedback is what teaches.
• Did the topic of the mistake return to your study? That is where the cycle closes.

Using the elements in your favor

Used as a reminder, points, goals and streak help you return to the test every day. The streak marks regularity; the daily goal sets a possible review size; the ranking with the class opens space to discuss the case. In all cases, what stays in memory comes from the test with feedback, and the game elements work so that it happens.

NEXO in review

In NEXO, each case is a short test with a free response: you write the diagnosis before seeing the answer. It is the recall format, and not recognition, that studies on the testing effect favor (Rowland, 2014).

Up to six clues, with feedback on every guess

Each case arrives in up to six clues, and each guess shows the proximity in ICD-10 at five levels. This immediate feedback tells you whether you are in the right group of diseases before getting it right, and the explanation at the end of the case closes the cycle of testing and correcting. On the third clue the area of the case appears and, on the fifth, the ICD range.

Daily and Archive

The Daily mode brings one case per day, the same for everyone, and renews at midnight: a short daily test, good for keeping review regular. The Archive gathers the previous daily cases, good for reviewing an area before the exam, with several tests in a row on the same topic.

Streak and achievements

The streak records the days you played, and the achievements mark the path. Used as a reminder, they help you come back to the test the next day. To understand the limits of each mode, see the NEXO Economy page.

From the case to Anki

When a case reveals a gap, the Anki button creates the card straight from NEXO, opens it in Anki, searches the cards you already have and, when the case has an image, takes the image to the card. How to build and review these cards is the topic of the text on spaced repetition.

The case error as a topic

The case ends with the diagnosis and the explanation, and that is where review pays off. As a practical suggestion, after each case, write in one line the clue where you changed your mind and the disease you confused with the right one. That line becomes a topic for the day's study and a question for the next test: days later, try to remember what separates the two diseases before looking at the note.

Over time, the notes show patterns: areas where you miss more, clues you tend to ignore, ICD-10 groups where you get close but do not get it right. These patterns are the map of the next review.

Reviewing with the class

The Daily case is the same for everyone, which allows reviewing in a group: each person plays alone and, afterward, the class discusses at which clue they changed their mind and which disease they confused with which. The discussion becomes a second test, now explaining the reasoning out loud. The group ranking helps track this collective review.

To research, bring to the class and create

If you research learning and want to propose studies with NEXO, meet NEXO Researcher. To bring NEXO to your school, league or residency, there is NEXO Closer. Doctors who review the cases in their specialty take part through NEXO Curator, and those who create health content on social media can take part through NEXO Creators.

A week of review

The plan below is a practical suggestion from the Editorial team, to adjust to your stage of the course.

Day	Review
Monday	Daily case and questions on the week's topic, with feedback
Tuesday	Daily case and free recall: write what you know about a topic before opening the material
Wednesday	Daily case and a block of Archive cases in the week's area
Thursday	Daily case and cards of the week's errors
Friday	Daily case and questions again on Monday's topic
Weekend	Review the error notes and mark the topics for the following week

How to adjust

In an exam week, increase the blocks of questions in the exam format, because format consistency weighs on the effect (Yang et al., 2021). In a heavy week, keep only the case of the day and a short review of errors: the short test, done regularly, is already review.

Common mistakes in review

• Rereading as the main review: it gives confidence, but yields less long-term retention (Roediger and Karpicke, 2006).
• Testing without reading the feedback: feedback is crucial to learn from the test (Larsen et al., 2008).
• Only recognizing, never producing: recall tests yield more than recognition tests (Rowland, 2014).
• Testing once and not coming back: repetitions modulate the effect (Yang et al., 2021).
• Measuring review by the score: what counts is what you got wrong and corrected.

Frequently asked questions

How to review Medicine efficiently?

Test your memory instead of rereading, read the feedback for each answer and come back to the same topic at intervals. In a meta-analysis with 222 classroom studies, testing raised performance by a medium degree (Yang et al., 2021).

What is the testing effect?

It is the memory gain that comes from trying to recall information, compared with studying it again. In a classic experiment, testing yielded more long-term retention than rereading (Roediger and Karpicke, 2006).

Does rereading the summary work?

Rereading gives a feeling of mastery and helps in an immediate test, but yields less retention days later than testing (Roediger and Karpicke, 2006). Use rereading after trying to recall, to check.

Multiple-choice questions or open questions?

Tests in which you produce the answer give greater benefits than recognition tests (Rowland, 2014). For the exam, it is also worth training in its format, because format consistency modulates the effect (Yang et al., 2021).

Does the testing effect apply to Medicine?

Yes. In the BEME 48 review, in the health professions, all 7 transfer outcomes favored testing (Green et al., 2018). With residents, the effect six months later was large, with d of 0.91 (Larsen et al., 2009).

Do points and streaks help learning?

They help you return to studying; what teaches is testing with feedback. Most gamification studies in the health professions did not have a well-defined control group (van Gaalen et al., 2021), and poorly applied elements can undermine motivation (Rutledge et al., 2018).

How many times should you test the same topic?

Studies do not give a single number, but the number of test repetitions modulates the effect (Yang et al., 2021). In the trial with residents, tests and study were repeated at intervals of about 2 weeks (Larsen et al., 2009).

Does the testing effect hold outside the laboratory?

Yes. A review of applied research in schools and classrooms found a medium or large benefit in 57% of the effects analyzed (Agarwal et al., 2021).

What about spaced repetition and Anki?

Repeating the test at intervals widens the effect, and it is the principle behind spaced repetition (Larsen et al., 2008). How to use Anki for this will be the topic of the next text in this series.

Does testing work from the basic cycle onward?

Yes. In schools and classrooms, retrieval practice improved learning at various levels of education and content areas (Agarwal et al., 2021). In the basic cycle, cause-and-effect questions and fact cards play this role; in the clinical cycle, cases carry more weight.

Is getting many answers wrong on a test a bad sign?

Getting answers wrong on a test is part of it. The test shows what you cannot yet produce, and the feedback after it is crucial for learning (Larsen et al., 2008). A corrected error becomes a study topic and a question on the next test.

Should you review right after class or days later?

Both have a place. In the classic experiment, rereading did better on a test 5 minutes later, but testing did better on the tests 2 days and 1 week later (Roediger and Karpicke, 2006). Since the goal is to remember on the exam and with the patient, it is worth testing and returning to the topic after a few days.

How does NEXO help with review?

Each case is a short test with a free response and feedback on each guess based on proximity in ICD-10. The Daily mode keeps daily review going, the Archive lets you review an area, and the Anki button takes the case's gap to a card.

Review checklist

1. Do you try to remember before opening the material?
2. Do you read the feedback for every answer, including the correct ones?
3. Do you produce the answer, and not just recognize it among options?
4. Do you return to the same topic days later, testing again?
5. Do you practice in the exam format when it is approaching?
6. Do you write down what you got wrong and bring the error into your studying?
7. Do you distrust the feeling that you already know it after rereading?
8. Do you use points and streaks as a reminder, and not as a goal?
9. Do you mix formats: questions, cases and cards?
10. Do you keep one short test a day, even in heavy weeks?

Want to practice today? Play the case of the day on NEXO.

References

1. Roediger HL 3rd, Karpicke JD. Test-enhanced learning: taking memory tests improves long-term retention. Psychol Sci. 2006. https://pubmed.ncbi.nlm.nih.gov/16507066/
2. Rowland CA. The effect of testing versus restudy on retention: a meta-analytic review of the testing effect. Psychol Bull. 2014. https://pubmed.ncbi.nlm.nih.gov/25150680/
3. Adesope OO, Trevisan DA, Sundararajan N. Rethinking the Use of Tests: A Meta-Analysis of Practice Testing. Rev Educ Res. 2017. https://doi.org/10.3102/0034654316689306
4. Agarwal PK, Nunes LD, Blunt JR. Retrieval Practice Consistently Benefits Student Learning: a Systematic Review of Applied Research in Schools and Classrooms. Educ Psychol Rev. 2021. https://doi.org/10.1007/s10648-021-09595-9
5. Yang C, Luo L, Vadillo MA, Yu R, Shanks DR. Testing (quizzing) boosts classroom learning: A systematic and meta-analytic review. Psychol Bull. 2021. https://pubmed.ncbi.nlm.nih.gov/33683913/
6. Green ML, Moeller JJ, Spak JM. Test-enhanced learning in health professions education: A systematic review: BEME Guide No. 48. Med Teach. 2018. https://pubmed.ncbi.nlm.nih.gov/29390949/
7. Larsen DP, Butler AC, Roediger HL 3rd. Test-enhanced learning in medical education. Med Educ. 2008. https://pubmed.ncbi.nlm.nih.gov/18823514/
8. Larsen DP, Butler AC, Roediger HL 3rd. Repeated testing improves long-term retention relative to repeated study: a randomised controlled trial. Med Educ. 2009. https://pubmed.ncbi.nlm.nih.gov/19930508/
9. Deng F, Gluckstein JA, Larsen DP. Student-directed retrieval practice is a predictor of medical licensing examination performance. Perspect Med Educ. 2015. https://pubmed.ncbi.nlm.nih.gov/26498443/
10. van Gaalen AEJ, Brouwer J, Schönrock-Adema J, et al. Gamification of health professions education: a systematic review. Adv Health Sci Educ Theory Pract. 2021. https://pubmed.ncbi.nlm.nih.gov/33128662/
11. Kerfoot BP, Baker H. An online spaced-education game for global continuing medical education: a randomized trial. Ann Surg. 2012. https://pubmed.ncbi.nlm.nih.gov/22664558/
12. Rutledge C, Walsh CM, Swinger N, et al. Gamification in Action: Theoretical and Practical Considerations for Medical Educators. Acad Med. 2018. https://pubmed.ncbi.nlm.nih.gov/29465450/