Reliability of the New NBME Forms in Predicting USMLE Step 1 Score

If you weren’t aware, the NBME will be retiring CBSSA Step 1 Forms 13, 15, 16, 17 and 19 on March 24th.  On March 25th they will release 3 new NBME Step 1 practice exams, Forms 20, 21, and 22. Later in the Spring, they will release forms 23 and 24.  Here we will attempt to address the issue of score prediction with the new exams.

Whenever we speak with students about preparing for the USMLE Step 1, we tell them that first we need to get a baseline performance and then we need to measure their performance at regular intervals. However, many students are afraid to check themselves because they don’t want to be disappointed.

There is an additional problem that comes up when discussing predicted score performance on the USMLE Step 1: Which NBME form is the best for predicting my actual score on the USMLE Step 1?  And now that many of the existing forms will be retired, and 5 new ones will replace them, the question becomes: How good will the new NBME forms be at predicting my actual score on the USMLE Step 1?   

Because the scoring system for the USMLE Step 1 (and by extension the NBME CBSSA) is essentially a black box, we cannot know for sure how predictive these exams will be. One possibility is that the NBME is retiring exam Form 19 because it has been a poor predictor of USMLE performance. Our tutors and our client data suggest that NBME Form 19 has underestimated actual USMLE scores by 10-15 points.  This finding is consistent with hundreds of comments in forums such as Reddit.  Since its release in Spring of 2017,  NBME CBSSA Form 19 has caused disappointment and trepidation for many students.  We believe that the NBME is aware of this issue and will attempt to ensure that predicted scores reported on the new NBME forms 20, 21, 22, 23, and 24 will be more accurate. However, we cannot know for sure until after we collect data from our students.

Recall that NBME Form 18 is NOT being retired, and this form is well accepted as an accurate (within ~5 points) predictor of actual USMLE performance.  As much as the NBME can seem like a soulless automaton, its exam writers are drawn from medical school faculty across the country who teach and mentor thousands of medical students every year.  Most of those faculty are MDs themselves, some even with children in medical school. They know the impact of the USMLE and the importance of accurate score prediction.

In our experience, all of the currently available forms, and even many of the long retired older forms, remain excellent predictors of USMLE Step 1 score performance. At the end of the day, medicine is medicine, and for the Step 1 student, not that much has actually changed over the years beyond a few additional medications. Note: this isn’t necessarily the case for Step 2 CK, which tests management and treatment guidelines that do change over time.

For USMLE Step 1 students, it can feel as though so much new information has been added.  More likely, as students have become more attuned to the information mentioned on the test, they have compiled ever greater lists of topics, terms, diseases, etc that they may have seen.  However, just because a particular rare disease was listed as an answer choice on a real Step question does NOT mean that the disease itself was (or will ever be) explicitly tested.

So, what is the bottom line?  

• We don’t know what the accuracy of the new NBME Forms will be.
• However, we expect that the new NBMEs will do a better job at score prediction than Form 19, which is being retired.  
• Tried and tested Form 18 will remain to anchor Step 1 score prediction.

In the end, using the old forms or the new forms solely for score prediction is a poor use of a valuable Step 1 learning resource.