Observations and Trends After the Switch to USMLE Step 1 Pass/Fail: What Learning Specialists Are Saying

One of the hottest topics among medical education learning specialists (MELS) is the behavior, motivation, and performance of medical students on USMLE Step 1 now that the exam has gone to pass/fail. Are students incentivized to study with the same intensity? Has stress increased or decreased in response to this change? How are students utilizing the recommended resources? These questions have been piquing the interest of MELS and medical students alike as the new data will inform what institutions can expect of these changes moving forward. 

A recent conversation with Dr. Thomas Smith, the executive director at the Center for Academic Excellence at MUSC, covered several important themes that should be strongly considered in preparation for Step 1. We have outlined them here for students and educators so that they can make more informed decisions in preparation for this exam.

General Observations: USMLE Step 1 Switch to Pass/Fail

Dr. Smith has noted some interesting observations so far with his students who most recently took USMLE Step 1 in February 2022, right after the switch to pass/fail. In that cohort of students, there was less time spent on focused review for Step 1. Compared to prior years, students are on average completing a lower overall percentage of their UWorld question bank and fewer cards in their Anki decks. Further, although students show the typical increase on successive NBME practice tests, their improvement is not quite at the rate previously seen. Dr. Smith notes a shift to more passive learning styles which has included an increase in reading or video-based review activities compared to active/retrieval-based study approaches such as Qbanks or flashcards. On average, students are starting their dedicated study a week or so later and have done notably less USMLE Step 1 specific work in the months immediately preceding the dedicated study period since the exam has become pass/fail. Generally, students scheduled their tests sooner in the available time frame to take the test, than they have previously. Whereas the last two cohorts scrambled to find later spots in Prometric centers all over their region, this year’s cohort often scheduled their Step 1 exam 2-3 weeks earlier.

Changing Role of Q-bank Self-Assessment

Dr. Smith suggests that the switch to pass/fail on USMLE Step 1 has created a significant problem in performance of self-assessment among medical students in the first cohorts to take the Step 1 exam without a numerical score.  Instead of a 3-digit numerical score that can be referenced against the performance of students in prior years, students now only receive a percentile rank score that they obtain from their UWorld Q-bank.  Prior to pass/fail scoring system of USMLE Step 1, a typical student would have completed the UWorld Q-bank more than one time, and often have completed the first pass well in advance of their exam. However, the most recent cohort of students are not using the UWorld Q-bank as intensively but their performance is being judged against those who went through the questions more than once.  Thus, students are reporting more anxiety because they are performing more poorly relative to those who had multiple attempts at the same set of questions.  

Dr. Smith also noted that at his school, medical students are reporting more extreme differences in their performance on the UWSA 1 and 2.  At his institution, these have historically been taken as one long 8 block practice test, 7-10 days before sitting for the actual USMLE Step 1 exam. His tracking of the students’ performance has shown that the gap is much larger between the scores on UWSA 1 and UWSA 2. Additionally, there was a sharp increase in the number of students who obtained a ‘failing’ score on the UWSA 2 practice exam, despite the student having had above-passing performance on the NBME practice exams and the UWSA 1.  The utility of the UWSA 2 when the Step 1 exam was scored was to help a student find weaknesses to address and avoid overconfidence.  In his view, in the new pass/fail environment, UWSA2 may be more indicative of UWorld Question Bank completion than readiness for passing USMLE Step 1. It may therefore generate unnecessary anxiety, particularly among those who have completed ~60% of the question bank or less.

In Dr. Smith’s opinion, UWorld seems to have been targeted toward those who want to score especially high on Step 1 – leading to a “high score” mentality when working with this resource. However, now that USMLE Step 1 has moved to pass/fail, students are developing a different mindset, one that he believes reduces the emphasis on a comprehensive review of this Qbank.

Dr. Smith suggests that there may be need for revision in how the UWorld Q-bank should be approached. They believe that the predicted scores on Qbanks and self-assessments should be revised to reflect the nature of the pass/fail environment more accurately. This could be achieved by altering the way in which self-assessments compute scores. One important revision may include altering the way it computes scores on the self-assessment, even if it retains the 3-digit format. An additional helpful change to UWorld would be the capability to sort questions based on tiers of difficulty. For those students who do not complete the entire question bank, such a change would allow students to target their completed questions toward material that, for example, 55% or more of medical students got correct. Such targeting may be more useful than randomly distributed difficulty in a pass/fail environment. Some question banks already provide such sorting mechanisms, which did not seem quite as important in the competitive 3-digit scoring environment. Companies that develop study resources have to take into account students’ new study habits rather than assume that what worked before to improve scores at the margins will still work for pass/fail. The balance has shifted from what will possibly appear on USMLE Step 1 to what will routinely be tested. 

– – –

We found this discussion with  Dr. Thomas Smith to be very insightful. With the change of USMLE Step 1 to pass/fail, the incentive for students to study as intensively as in prior years has clearly dropped.  The changes Dr. Smith has observed support this idea.  An interesting question will be whether the reduced time and intensity preparing for the Step 1 exam will have effects on student performance on their clinical rotations?  Additionally, with Step 1 becoming pass/fail, many assume greater emphasis will fall on Step 2 performance and NBME Shelf exam performance. Although these tests differ from Step 1 in their focus (Step 1: basic science of medicine & diagnosis; Step 2/Shelf: diagnosis & basic management), there is significant content overlap between them.  Therefore it will be interesting to see if there is a measurable effect on Step 2 and Shelf exam performance, with a significant concern that students who took Step 1 pass/fail will have weaker performance on Step 2 and Shelf exams as compared to the prior years who took Step 1 for a score.