Tracking your progress in medical studies
In four studies with mood, grammar and logic tests, those in the bottom quartile, with a result near the 12th percentile, estimated they were in the 62nd percentile (Kruger and Dunning, 1999). The authors linked this difference to the difficulty of telling one's own correct answers from one's own mistakes (Kruger and Dunning, 1999). For those studying Medicine, the lesson is direct: the feeling of doing well is weak data, and measuring progress calls for numbers that come from outside your head.
NEXO is a daily medical diagnosis game: one clinical case a day, six progressive clues, and ICD-10 proximity feedback across five levels.
This text is about how to measure progress in your studies: what the research says about self-assessment by health students, how to compare confidence with correct answers, which metrics really help, how to separate process goals from outcome goals and how to read a mock exam. The part about recording and classifying each mistake is in the text about the error notebook.
Main points
- Self-assessment by health students had low to moderate validity in a classic review (Gordon, 1991).
- Medical students assess themselves with moderate accuracy and become more precise in the final years of the course (Blanch-Hartigan, 2011).
- Calibration is comparing confidence with correct answers, and it can be tracked as a metric.
- Accuracy by topic, mistakes by type and calibration say more than the overall score.
- Process goals at the start and outcome goals later performed best in a classic experiment (Zimmerman and Kitsantas, 1997).
- Mock exams and progress tests are useful external measures when read as a series, and not as an isolated score (Schuwirth and van der Vleuten, 2012).
Summary
- Why measure progress
- The limits of self-assessment
- Calibration: confidence and correct answers
- Metrics that help
- Process goals and outcome goals
- Mock exams as a measure
- A weekly dashboard
- Where NEXO comes in
- Frequently asked questions
- Checklist to track your progress
Why measure progress
The Medicine course is long, the volume of content is large and the big exams come months apart. Between one exam and the next, the reference closest at hand is your own impression, and it is precisely that which is most prone to error.
The impression of learning
Rereading a summary gives a feeling of mastery, because the text seems familiar. The text about the study routine explains why the feeling of knowing deceives you. Measuring progress means replacing that impression with answers: how many cardiology questions you got right this week, which type of mistake you made most, how much your confidence matched the result.
Measuring to decide
A measure is only worth it if it changes some decision. As a practical suggestion, before creating any metric, ask: if this number drops, what do I do differently? If the answer is clear, the metric goes on the dashboard. If the answer is "nothing", it can be left out without harm.
What this text measures
Here the focus is performance on questions and cases: correct answers, mistakes, confidence and evolution over time. Study hours and pages read appear as process goals, which have their place, but which measure effort and not learning.
The limits of self-assessment
Self-assessment is the first tool every student has, and research on it in the health professions is old.
A classic review
Gordon analyzed 18 articles published between 1970 and 1990 that compared the self-assessment of people in training, 14 of them with health students and residents, with the assessment of experts or objective tests (Gordon, 1991). The validity of self-assessment was low to moderate and did not improve over time in conventional training programs (Gordon, 1991).
Two findings from the review matter a lot for those who study. Self-assessment seemed linked to a general image the person has of themselves and changed little with grades, exam results or teacher assessments (Gordon, 1991). In 5 programs with explicit self-assessment goals and training for it, validity was moderate to high or improved over time (Gordon, 1991). In other words, assessing yourself is a skill that can be trained.
Three meta-analyses with medical students
Blanch-Hartigan gathered 35 articles on self-assessment by medical students in three meta-analyses, one for each way of measuring accuracy (Blanch-Hartigan, 2011). The students proved moderately capable of assessing their own performance and more precise in the final years of the course (Blanch-Hartigan, 2011).
As a group, they neither significantly overestimated nor underestimated, but they tended to overestimate more in encounters with standardized patients, based on communication, than in objective measures of knowledge (Blanch-Hartigan, 2011). Female students underestimated their own performance more than male students (Blanch-Hartigan, 2011).
| Study | What was analyzed | Main result |
|---|---|---|
| Gordon, 1991 | 18 articles (Gordon, 1991) | Low to moderate validity (Gordon, 1991) |
| Blanch-Hartigan, 2011 | 35 articles (Blanch-Hartigan, 2011) | Moderate accuracy, higher in the final years (Blanch-Hartigan, 2011) |
| Kruger and Dunning, 1999 | 4 studies outside health (Kruger and Dunning, 1999) | The lowest quartile greatly overestimated their own result (Kruger and Dunning, 1999) |
Those who know less are more wrong about themselves
In the study by Kruger and Dunning, improving the participants' skill also helped them recognize the limits of what they knew (Kruger and Dunning, 1999). For the student, this has an encouraging side: as you learn a topic, you also get better at noticing what is still missing in it. With new topics, it is worth distrusting your own impression more.
Rethinking what self-assessment is
Eva and Regehr proposed reformulating the very concept of self-assessment in the health professions and suggested a research agenda on the topic (Eva and Regehr, 2005). As an editorial reading, the practical consequence for those who study is to seek external sources of information, such as questions, cases and mock exams, instead of relying only on internal judgment. The review on the accuracy of physicians' self-assessment already mentioned in the error notebook goes in the same direction; it is in the section self-assessment fails.
Calibration: confidence and correctness
Calibration is how much your confidence follows your correctness. A well-calibrated student gets almost everything they mark as "I am sure" right and gets much of what they mark as a "guess" wrong. The studies on confidence and correctness in diagnosis are gathered in the section feedback and calibration of the text on clinical reasoning games.
Recording confidence before the answer
As a practical suggestion, when solving questions or cases, note your confidence before seeing the answer key, on three levels: high, medium and low. After a few weeks, separate the answers by level and look at the hit rate of each one. What matters is the pattern, not an isolated question.
How to read the result
| Pattern | Possible reading (Editorial suggestion) |
|---|---|
| High confidence and high correctness | Solid topic; keep in spaced review |
| High confidence and low correctness | Overconfidence; study priority |
| Low confidence and high correctness | Knowledge you still underestimate; test more to gain confidence |
| Low confidence and low correctness | Gap already noticed; study the basics |
The most important row in the table is the second. A mistake made with great certainty shows a point where self-assessment failed, and that is where correction pays off most. The handling of these errors in the notebook is in the text on the error notebook.
Calibration also improves
The programs that taught self-assessment explicitly had better results in Gordon's review (Gordon, 1991). As an editorial reading, recording confidence and comparing it with the answer key is a simple way to do this training on your own, with feedback every session.
Metrics that help
The overall score of a question list summarizes too much. Two people with the same percentage can have completely different gaps. Finer metrics show where to act.
Hit rate by topic
Separate the questions by major area and, within it, by topic: in cardiology, for example, heart failure, arrhythmias and valvular diseases. As a practical suggestion, track the hit rate of each topic over the weeks. The topic that stalls or drops is the candidate for the next study block, even if the overall score is good.
Errors by type
Counting errors by type shows whether the problem is lack of knowledge, reasoning or attention. The categories and how to record them are in type of error: knowledge, reasoning, attention. Here, the point is to use this count as a metric: if attention errors rise in a month of exams, the response is different from when knowledge errors rise.
Calibration by topic
The confidence and accuracy table can also be made by topic. Overconfidence concentrated in one topic usually indicates that it was studied only by rereading. As a practical suggestion, put that topic in a round of questions without consulting anything.
Trend, not the day
One bad week says little. Progress shows up in the series: the same metric measured the same way, repeated over time. As a practical suggestion, compare averages across several weeks, not one day with another.
| Metric | What it shows | What to do when it gets worse (Editorial suggestion) |
|---|---|---|
| Accuracy by topic | Where knowledge is solid or weak | Study the basics of the topic and test again |
| Errors by type | Whether the problem is knowledge, reasoning or attention | Adjust the method to the type of error |
| Calibration | Whether confidence matches accuracy | More questions without consulting anything on the topic |
| Progress on mock exams | Trend over the course | Review how time is distributed across areas |
Metrics that call for caution
Study hours, pages read and number of questions done are useful process goals, but they measure effort. They go on the dashboard alongside performance metrics, not in place of them. A week with many hours and stalled accuracy is a sign to change the method, as the active learning strategies show.
Process goals and outcome goals
A process goal describes what you will do: solve a question set every day, review the week's errors on Sunday. An outcome goal describes where you want to get: raise your accuracy in cardiology, pass the residency exam. Both have a place, and the order matters.
The darts experiment
Zimmerman and Kitsantas studied 90 high school students, all girls, learning to throw darts (Zimmerman and Kitsantas, 1997). Those who started with process goals and then moved to outcome goals outperformed those who stayed only with process goals, who in turn outperformed those who used only outcome goals (Zimmerman and Kitsantas, 1997).
The gain showed up in skill, in reactions to one's own performance, in confidence in one's own ability and in interest in the task (Zimmerman and Kitsantas, 1997). Self-recording, a way of noting down one's own performance, also improved skill and confidence (Zimmerman and Kitsantas, 1997).
What can be taken into Medicine
The experiment was with a motor skill, outside Medicine, and the result should be read with that caveat. As an editorial reading, the idea it suggests is simple: in a new topic, start with process goals, such as doing a block of questions per day; once the method is steady, start looking at the outcome, such as the topic's accuracy rate. Noting down your own performance, which is what the dashboard in this text proposes, connects with the effect of self-recording in the study.
Thirty-five years of research on goals
Locke and Latham summarized 35 years of research on goal setting, with the main findings, the mechanisms through which goals act, the factors that change their effect and the limits of the theory (Locke and Latham, 2002). It is a good read for anyone who wants to understand why some goals help more than others. The game side of goals, with points and streak, is in goals, points and streak.
Examples of goals
| Type | Example (Editorial suggestion) | How to measure |
|---|---|---|
| Process | Solve a block of cardiology questions per day | Days completed in the week |
| Process | Review the week's errors on Sunday | Review done or pending |
| Outcome | Raise accuracy in arrhythmias | Topic accuracy rate over the weeks |
| Outcome | Improve calibration in infectious diseases | Accuracy of answers with high confidence |
Mock exams as a measure
The mock exam is the external measure closest to the real test. Read well, it shows where you are and where you are going; read badly, it becomes just a score to celebrate or mourn.
What a mock exam can predict
Morrison and colleagues studied 12,224 students from the United States and Canada who took an official mock exam before their first attempt at Step 1, the first stage of the American medical licensing exam (Morrison et al., 2010). The mock exam score explained 67% of the variation in the exam score when used alone, and 69% when the interval between the tests was also taken into account (Morrison et al., 2010). Those who had a low mock exam score had a higher risk of failing on the first attempt (Morrison et al., 2010).
The authors conclude that the mock exam can give the student a realistic self-assessment of their preparation for the exam (Morrison et al., 2010). The finding holds for a specific mock exam, made by the same board as the test, and for that exam.
The progress test
The progress test is a form of longitudinal assessment: equivalent but different tests, applied several times throughout the course, whose results are combined to show each student's growth in knowledge (Schuwirth and van der Vleuten, 2012). The authors describe a positive effect on study behavior, by discouraging concentrated last-minute study, and good predictive validity (Schuwirth and van der Vleuten, 2012).
They also point out a technical concern: matching the difficulty of successive tests, so that the variation between one and another is not mistaken for progress (Schuwirth and van der Vleuten, 2012). The guide by Wrigley and colleagues notes that multiple-choice tests carry an error variation that challenges the validity and reliability of the results, and proposes a model to improve the quality of the progress test data (Wrigley et al., 2012).
How to read your mock exam
As a practical suggestion, read each mock exam in three layers:
- The overall score, compared with your previous mock exams, and not with the class average on a single day.
- The hit rate by area and by topic, to find what dropped or stayed flat.
- The errors by type and the confidence, to decide whether the next step is studying the basics, training reasoning or working on attention.
Mock exams from different sources have different difficulties. As an editorial reading, compare preferably mock exams from the same source, for the same reason that leads the progress test to match the difficulty of the tests.
A mock exam is also training
Taking the mock exam is already a way of testing yourself, and reviewing with feedback closes the cycle. Why testing yourself helps you remember is in the text about reviewing by playing.
A weekly panel
Putting the metrics in one place helps you see the trend. The panel can be a simple spreadsheet or a page in your notebook.
What goes in
As a practical suggestion, a weekly panel can have five columns: week, hit rate by topic in the areas you are studying, count of errors by type, hit rate of answers with high confidence and process goals met. One row per week is enough.
A ten-minute review
At the end of the week, look at the panel and answer three questions: which topic improved, which stayed flat and which type of error appeared most. The answer becomes the process goal for the following week. This short ritual fits into the routine described in the text about how to fit games into your study routine.
Common mistakes when measuring progress
- Measuring only the overall score. It hides weak topics behind strong ones.
- Comparing one day with another. Natural variation misleads; look at the series.
- Changing the yardstick every week. Metrics that change definition stop being comparable.
- Trusting the impression more than the number. Research on self-assessment shows why the impression fails (Gordon, 1991).
- Measuring effort as if it were learning. Hours and pages are process; the hit rate is the result.
- Comparing mock exams from different sources. The difficulty changes from one source to another.
Where NEXO comes in
Each NEXO case arrives in up to six clues, one at a time, and each guess receives proximity feedback on the ICD-10, from the chapter to the exact code. That is already a small measurement for each case: at which clue you reached the diagnosis and how close you got in the previous guesses.
At how many clues the diagnosis is reached
The home screen shows the distribution of your hits by clue: at how many clues the diagnosis usually is reached, as described in the NEXO what's new. As a practical suggestion, follow this distribution over the weeks; hitting with fewer clues is a sign of progress in reasoning, and hits that concentrate in the last clues indicate topics to review.
Consistency, level and achievements
The history of your streak appears as a calendar, and the level bar on the home screen shows the XP earned at the current level, both described in What's new at NEXO. Achievements appear as tracks, with the steps in order, what has already been completed and which one is next. These are good process measures: they show whether the habit is holding. The Ranking has the Global, Universities and Group tabs, for those who like to keep up with their class.
Bring the case to your dashboard
As a practical suggestion, when you miss a case or only get it right on the last clues, note the topic and the type of error on your weekly dashboard, and use the Archive to redo previous cases on the same topic. The Anki button takes the diagnosis you missed to a card, as shown in the guide on spaced repetition.
To research, bring to your class and create
If you research assessment and self-assessment in medical education and want to propose studies with NEXO, check out NEXO Researcher. To bring NEXO to your medical school, student league or residency, there is NEXO Closer. Doctors who review cases in their specialty take part through NEXO Curator, and those who create health content on social media can take part through NEXO Creators.
Frequently asked questions
How do you measure progress in medical studies?
With external measures, repeated the same way over time: hit rate by topic in questions and cases, count of errors by type, comparison between confidence and accuracy, and progress on mock exams from the same source. Personal impression helps little, because the self-assessment of health students had low to moderate validity in a classic review (Gordon, 1991).
Is the self-assessment of medical students reliable?
It is moderate. In Blanch-Hartigan's meta-analyses, students assessed themselves with moderate accuracy and improved in the final years of the course (Blanch-Hartigan, 2011). So it is worth combining self-assessment with data from questions and mock exams.
What is calibration in studying?
It is how closely your confidence follows your accuracy. To measure it, note your confidence before seeing the answer key and compare the hit rate of answers with high, medium and low confidence.
Process goal or outcome goal?
Both, in sequence. In the experiment by Zimmerman and Kitsantas, starting with process goals and then moving to outcome goals gave better performance than using only one of them (Zimmerman and Kitsantas, 1997). The study was with a motor skill, outside Medicine.
Does a mock exam predict the test result?
In a study with 12,224 students, the score on an official mock exam explained 67% of the variation in the Step 1 score, the first stage of the United States medical licensing exam (Morrison et al., 2010). The finding applies to that mock exam and that test; in other contexts the relationship may be different.
What is a progress test?
It is a longitudinal assessment, with equivalent tests applied several times throughout the course to show each student's growth in knowledge (Schuwirth and van der Vleuten, 2012).
How often should I look at my metrics?
A short review once a week is a good practical suggestion. Progress shows up in the series, so compare averages over several weeks instead of one day with the next.
How does NEXO help track progress?
The home screen shows at how many clues your diagnosis usually lands, the streak appears as a calendar and achievements appear as tracks, with the next step in view. Each case arrives in up to six clues, with proximity feedback on ICD-10 for each guess.
Checklist to track your progress
- Have I defined the metrics I will track and how I will measure each one?
- Do I note my confidence before seeing the answer key?
- Do I break down accuracy by topic, and not just by overall score?
- Do I count errors by type at the end of the week?
- Do I have process goals for new topics?
- Do I move to outcome goals when the method is holding?
- Do I compare mock exams from the same source over time?
- Do I read each mock exam in terms of score, topic and type of error?
- Do I look at the trend over several weeks, and not a single day?
- Have I decided what changes the following week based on the dashboard?
References
- Gordon MJ. A review of the validity and accuracy of self-assessments in health professions training. Acad Med. 1991. https://pubmed.ncbi.nlm.nih.gov/1750956/
- Eva KW, Regehr G. Self-assessment in the health professions: a reformulation and research agenda. Acad Med. 2005. https://pubmed.ncbi.nlm.nih.gov/16199457/
- Blanch-Hartigan D. Medical students' self-assessment of performance: results from three meta-analyses. Patient Educ Couns. 2011. https://pubmed.ncbi.nlm.nih.gov/20708898/
- Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one's own incompetence lead to inflated self-assessments. J Pers Soc Psychol. 1999. https://pubmed.ncbi.nlm.nih.gov/10626367/
- Zimmerman BJ, Kitsantas A. Developmental phases in self-regulation: shifting from process goals to outcome goals. J Educ Psychol. 1997. https://doi.org/10.1037/0022-0663.89.1.29
- Locke EA, Latham GP. Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. Am Psychol. 2002. https://pubmed.ncbi.nlm.nih.gov/12237980/
- Morrison CA, Ross LP, Fogle T, et al. Relationship between performance on an official basic sciences practice test and USMLE Step 1 in students from the United States and Canada (original title in English on PubMed). Acad Med. 2010. https://pubmed.ncbi.nlm.nih.gov/20881715/
- Schuwirth LW, van der Vleuten CP. The use of progress testing. Perspect Med Educ. 2012. https://pubmed.ncbi.nlm.nih.gov/23316456/
- Wrigley W, van der Vleuten CP, Freeman A, Muijtjens A. A systemic framework for the progress test: strengths, constraints and issues: AMEE Guide No. 71. Med Teach. 2012. https://pubmed.ncbi.nlm.nih.gov/22905655/