Observations and Trends After the Switch to USMLE Step 1 Pass/Fail: What Learning Specialists Are Saying

2024 Update: This blog was originally posted on April 19, 2022. In 2024, an additional subheading was added to reevaluate the advice given within the context of the 2024 USMLE Step 1 landscape.

One of the hottest topics among medical education learning specialists (MELS) is the behavior, motivation, and performance of medical students on USMLE Step 1 now that the exam has gone to pass/fail. Are students incentivized to study with the same intensity? Has stress increased or decreased in response to this change? How are students utilizing the recommended resources? These questions have been piquing the interest of MELS and medical students alike as the new data will inform what institutions can expect of these changes moving forward. 

A recent conversation with Dr. Thomas Smith, the executive director at the Center for Academic Excellence at MUSC, covered several important themes that should be strongly considered in preparation for Step 1. We have outlined them here for students and educators so that they can make more informed decisions in preparation for this exam.

General Observations: USMLE Step 1 Switch to Pass/Fail

Dr. Smith has noted some interesting observations so far with his students who most recently took USMLE Step 1 in February 2022, right after the switch to pass/fail. In that cohort of students, there was less time spent on focused review for Step 1. Compared to prior years, students are on average completing a lower overall percentage of their UWorld question bank and fewer cards in their Anki decks. Further, although students show the typical increase on successive NBME practice tests, their improvement is not quite at the rate previously seen. Dr. Smith notes a shift to more passive learning styles which has included an increase in reading or video-based review activities compared to active/retrieval-based study approaches such as Qbanks or flashcards. On average, students are starting their dedicated study a week or so later and have done notably less USMLE Step 1 specific work in the months immediately preceding the dedicated study period since the exam has become pass/fail. Generally, students scheduled their tests sooner in the available time frame to take the test, than they have previously. Whereas the last two cohorts scrambled to find later spots in Prometric centers all over their region, this year’s cohort often scheduled their Step 1 exam 2-3 weeks earlier.

Changing Role of Q-bank Self-Assessment

Dr. Smith suggests that the switch to pass/fail on USMLE Step 1 has created a significant problem in performance of self-assessment among medical students in the first cohorts to take the Step 1 exam without a numerical score.  Instead of a 3-digit numerical score that can be referenced against the performance of students in prior years, students now only receive a percentile rank score that they obtain from their UWorld Q-bank.  Prior to pass/fail scoring system of USMLE Step 1, a typical student would have completed the UWorld Q-bank more than one time, and often have completed the first pass well in advance of their exam. However, the most recent cohort of students are not using the UWorld Q-bank as intensively but their performance is being judged against those who went through the questions more than once.  Thus, students are reporting more anxiety because they are performing more poorly relative to those who had multiple attempts at the same set of questions.  

Dr. Smith also noted that at his school, medical students are reporting more extreme differences in their performance on the UWSA 1 and 2.  At his institution, these have historically been taken as one long 8 block practice test, 7-10 days before sitting for the actual USMLE Step 1 exam. His tracking of the students’ performance has shown that the gap is much larger between the scores on UWSA 1 and UWSA 2. Additionally, there was a sharp increase in the number of students who obtained a ‘failing’ score on the UWSA 2 practice exam, despite the student having had above-passing performance on the NBME practice exams and the UWSA 1.  The utility of the UWSA 2 when the Step 1 exam was scored was to help a student find weaknesses to address and avoid overconfidence.  In his view, in the new pass/fail environment, UWSA2 may be more indicative of UWorld Question Bank completion than readiness for passing USMLE Step 1. It may therefore generate unnecessary anxiety, particularly among those who have completed ~60% of the question bank or less.

In Dr. Smith’s opinion, UWorld seems to have been targeted toward those who want to score especially high on Step 1 – leading to a “high score” mentality when working with this resource. However, now that USMLE Step 1 has moved to pass/fail, students are developing a different mindset, one that he believes reduces the emphasis on a comprehensive review of this Qbank.

Dr. Smith suggests that there may be need for revision in how the UWorld Q-bank should be approached. They believe that the predicted scores on Qbanks and self-assessments should be revised to reflect the nature of the pass/fail environment more accurately. This could be achieved by altering the way in which self-assessments compute scores. One important revision may include altering the way it computes scores on the self-assessment, even if it retains the 3-digit format. An additional helpful change to UWorld would be the capability to sort questions based on tiers of difficulty. For those students who do not complete the entire question bank, such a change would allow students to target their completed questions toward material that, for example, 55% or more of medical students got correct. Such targeting may be more useful than randomly distributed difficulty in a pass/fail environment. Some question banks already provide such sorting mechanisms, which did not seem quite as important in the competitive 3-digit scoring environment. Companies that develop study resources have to take into account students’ new study habits rather than assume that what worked before to improve scores at the margins will still work for pass/fail. The balance has shifted from what will possibly appear on USMLE Step 1 to what will routinely be tested. 

– – –

We found this discussion with  Dr. Thomas Smith to be very insightful. With the change of USMLE Step 1 to pass/fail, the incentive for students to study as intensively as in prior years has clearly dropped.  The changes Dr. Smith has observed support this idea.  An interesting question will be whether the reduced time and intensity preparing for the Step 1 exam will have effects on student performance on their clinical rotations?  Additionally, with Step 1 becoming pass/fail, many assume greater emphasis will fall on Step 2 performance and NBME Shelf exam performance. Although these tests differ from Step 1 in their focus (Step 1: basic science of medicine & diagnosis; Step 2/Shelf: diagnosis & basic management), there is significant content overlap between them.  Therefore it will be interesting to see if there is a measurable effect on Step 2 and Shelf exam performance, with a significant concern that students who took Step 1 pass/fail will have weaker performance on Step 2 and Shelf exams as compared to the prior years who took Step 1 for a score.

2024 Update: Navigating the Impact of USMLE Step 1’s Transition to Pass/Fail

As some time has passed since originally posting this blog, we thought it may be time to reflect on the insights gained from this interview with Dr. Smith. We aimed to assess the accuracy of his predictions, and whether new insights can be gained from reviewing more recent data. Looking through the current landscape of USMLE Step 1, here are some key insights that we recommend med school students keep in mind in 2024:

Declining Step 1 Pass Rates

According to USMLE performance data, Step 1 pass rates have exhibited a consistent downward trend since the shift to pass/fail. This decline persisted throughout 2022, with ongoing decreases observed among first-time test takers from 95% in 2021, to 93% in 2022 and 92% in 2023.

The implication of this to students is that you may not be as prepared for Step 1 as you think, especially if this trend is set to continue in 2024 and on. We recommend approaching your Step 1 dedicated as if you are studying for a scored exam, rather than aiming just to pass. Make sure that you are fully prepared on all high-yield topics so that you can cruise through on test day!

Challenges in Assessing Readiness

One of the significant challenges arising from the transition to pass/fail grading is accurately gauging students’ readiness for Step 1, as noted by Dr. Smith in early 2022. The absence of a numerical score has added complexity to this assessment process for both medical schools and students. While some resources, such as the NBME practice exams, have adjusted how they report scores to reflect the pass/fail system, the continued decline in pass rates suggests that this adjustment may not sufficiently help students evaluate their readiness.

We recommend seeking guidance from trusted mentors or a Step 1 tutor to determine whether you are truly prepared for the exam. Personalized guidance from someone familiar with both you and the content is invaluable, not only for assessing readiness but also for guiding you toward the most effective preparation strategies for you.

Effects of the Pass/Fail System on Step 2 Scores

While both EMP experts and Dr. Smith predicted that Step 2 scores will suffer from this change, it seems that they have not been significantly affected. According to USMLE data, the Step 2 mean score has been gradually increasing over the past few years from 246 in 2020-2021 to 248 in 2022-2023. It’s important to note that while the scores have been increasing, the changes are relatively small and within the range of the standard deviation (15). This suggests that while there may be a slight upward trend, the performance on Step 2 CK remains relatively stable.