Are NBME Practice Exams a True Reflection of Shelf Exam Difficulty?

In medical education, shelf exams evaluate students’ knowledge and clinical reasoning during core clinical clerkships. Administered by medical schools, often written by the National Board of Medical Examiners (NBME), these pivotal exams determine students’ readiness for clinical rotations and overall performance. Questions often arise about the correlation between NBME practice exams and the actual difficulty of shelf exams. In this blog post, we explore these differences, discuss average shelf exam scores, assess difficulty, and reflect on what constitutes a good shelf exam score, drawing from my personal experiences in medical school.

While my insights provide a valuable perspective, it’s important to note that published data on the correlation between practice shelf exam scores and true shelf exam scores is limited. For those interested in a deeper exploration of this topic, I invite you to read a relevant study that suggests a significant impact of practice exams on Step 1 scores, yet no established correlation for shelf exam scores.

Understanding the Difference Between NBME Practice Exams and Real Shelf Exams

First, understand that NBME shelf exams evaluate a student’s competency in a medical specialty at the end of core clerkships. The core clerkships that have a shelf exam are internal medicine, surgery, family, neurology, pediatrics, obstetrics and gynecology, and psychiatry. Additionally, students will often participate in various electives or supplemental rotations, such as radiology, emergency medicine, or dermatology. These electives often do not have a shelf exam associated with them. However, medical schools sometimes elect to generate in-house versions of a shelf exam for students to take.

While medical schools may create their shelf exams, most collaborate with the NBME, streamlining the creation and administration process. One reason medical schools choose this is its cost efficiency; they avoid spending time generating questions by collaborating with the NBME. This benefits students as the questions are from the same company that administers the USMLE Step 1 and Step 2.

NBME practice exams are designed to simulate the format and content of real shelf exams. This provides students with an opportunity to assess their knowledge and identify areas for improvement.  In my experience, these practice shelf exams are the best predictors of success on test day. However, there are some differences between NBME practice exams and real shelf exams that students should be aware of:

Content Variation

While NBME practice exams address crucial topics within a particular discipline, the content may not precisely reflect the questions encountered on the actual shelf exam. Genuine shelf exams might encompass questions demanding a more profound grasp of clinical concepts and the integration of knowledge across diverse disciplines. It’s nearly impossible to predict all the content tested on the authentic shelf exam, but the practice exams effectively provide a comprehensive preparation.

Question Format

The format of questions on NBME practice exams may differ slightly from that of real shelf exams. While both exams include multiple-choice questions, real shelf exams will incorporate clinical vignettes and case-based scenarios to assess students’ clinical reasoning skills. Sometimes, the format of these vignettes and scenarios may be slightly different than those on the practice shelf exam. However, in my experience, I find that the practice of exams do mirror the real exam. I believe that question banks such as UWorld and Amboss, while assessing relevant content, often differ significantly from the question style found in NBME shelf exams due to their frequently longer format.

Practice vs. Shelf Exam Difficulty Level

NBME practice exams are intended to provide a realistic assessment of students’ preparedness for shelf exams. However, the difficulty level of practice exams can vary, and students may find the real shelf exams to be either slightly easier or more challenging than their practice exams, depending on individual factors such as test-taking strategies and familiarity with the exam format. Students often state that particular practice exams are more difficult than others. While we have no objective data to support this, in our experience, we believe there may be a significant amount of variation in difficulty between each shelf exam. For instance, the medicine shelf exam is known to be particularly difficult.

Scoring Algorithm

The scoring algorithm used for NBME practice exams may differ from that of real shelf exams. Unfortunately, the exact way that these shelf exams are scored may differ from school to school. While practice exams provide a numerical score based on the number of correct answers, real shelf exams are often scaled to account for variations in question difficulty and ensure consistency across different administrations. These scaled numbers are then scaled again by each medical school in the context of their overall clinical grade. For example, students are often told that they must score in a certain percentile, rather than an objective number of correct questions, to obtain honors.

Average Shelf Exam Scores and Passing Difficulty:

The average shelf exam score varies among medical schools and may depend on factors such as institutional curriculum, student demographics, and faculty expectations. However, a general understanding of the average shelf exam score can provide insight into the difficulty level of these assessments.

On average, shelf exam scores typically range between 70% and 80%, as per our experience, though notable variations exist depending on the subject and institution. Medical schools often set their own passing benchmarks for shelf exams, necessitating students to attain a minimum score to showcase proficiency in the respective discipline.

Importantly, the average score and minimum passing score are often scaled, depending on when the examination is taken. Students who take their internal medicine shelf exam as their first shelf exam will undoubtedly score lower than students who take this shelf exam as their last shelf exam. This is because students are constantly accumulating more knowledge throughout their clinical training. As a result, medical schools will often increase the passing threshold throughout the clinical year for all shelf exams. From our experience, the minimum passing score for any shelf exam may lie around 50 to 70%, depending on when the shelf exam is taken, the specific subject being tested, and the particular medical school.

While shelf exams are challenging assessments, designed to evaluate students’ readiness for clinical rotations, passing these exams is achievable with adequate preparation and study strategies. Students who engage in active learning techniques, such as practice questions, review sessions, and clinical rotations, are better positioned to succeed on shelf exams. Additionally, seeking guidance from faculty members, utilizing resources such as textbooks and online modules, and participating in study groups can enhance students’ preparedness and confidence for shelf exams.

What Constitutes a Good Shelf Exam Score?

Determining what constitutes a good shelf exam score depends on various factors, including individual goals, institutional expectations, and future career aspirations. While there is no universally defined threshold for a good shelf exam score, several considerations can help students assess their performance. One way is to compare your shelf exam scores to institutional benchmarks established by their medical school. These benchmarks may vary across different disciplines and serve as a reference point for evaluating performance relative to peers. Often, clerkship directors will inform you of the minimum passing score, and score required to receive an honors, at the beginning of your rotation.

The Impact of Shelf Exam Performance on Residency Applications

Students pursuing competitive residency specialties (neurosurgery, dermatology, orthopedic surgery, etc.) may aim to obtain higher shelf exam scores to remain competitive in the residency application process. Residency programs often consider shelf exam scores as part of their selection criteria, alongside other academic achievements and clinical experiences. As a general reminder, you should try and obtain an honors on every rotation, but especially the rotation of which you ultimately intend to pursue residency training. Consistent performance and improvement on shelf exams over time are indicative of students’ mastery of clinical concepts and readiness for clinical practice. Students should strive for steady progress and identify areas for continued growth and development. It is not uncommon to see a student perform poorly at the beginning of the year. However, we hope they demonstrate improvement over time and should score higher on shelf exams at the end of their clinical year. Ultimately, a good shelf exam score reflects your effort, dedication, and understanding of clinical concepts. 

The correlation between NBME practice exams and real shelf exams varies based on factors like content, question format, difficulty, and scoring algorithm. While NBME practice exams are valuable for self-assessment, students must be aware of differences and adapt study strategies. An effective approach for shelf exams involves utilizing a question bank, a primary resource (textbook or videos), and NBME practice shelf exams.