Could ChatGPT Replace QBanks for USMLE Practice Questions?

If you are a medical student preparing for USMLE Step 1, 2 CK or 3, you may be wondering if there are alternative resources for USMLE-style practice questions beyond traditional question banks. One potential option that has gained attention recently is ChatGPT, a variant of the language generation model GPT-3. Could ChatGPT be used as a replacement for QBanks and create practice questions in USMLE preparation?

Before diving into whether ChatGPT could be a viable option for USMLE prep, it’s important to understand what ChatGPT is and how it works. ChatGPT is a variant of the GPT-3 language generation model developed by OpenAI. It is designed to generate human-like text in response to a given prompt. This means that when given a specific question or topic, ChatGPT can generate a coherent and informative response. Though it was not designed for medical uses specifically, it can be used to generate USMLE-style practice questions and for other uses that may be on interest for medical students, such as to draft a personal statement for medical school and residency applications, or to generate a custom study schedule.

So, how could ChatGPT be used for USMLE prep? One potential way would be to use ChatGPT to generate practice questions on a specific topic. For example, you could provide ChatGPT with a prompt such as “Generate 10 USMLE-style questions on cardiovascular physiology” and ChatGPT would generate a list of practice questions on the given topic.

We are curious about how this new AI technology could potentially replace QBanks for USMLE practice questions. So I went ahead and did that for you. I created an account with OpenAI and asked it to generate many different questions on a variety of topics. Some questions I had going into this exercise were: Does it accurately represent the style of USMLE questions? How is the difficulty level of the questions? Are the explanations in-depth? Is it user-friendly? Overall, will I feel confident using this model to supplement my preparation for the Step exams? As you follow this blog post, I will share more of my thoughts.

How ChatGPT Generates USMLE-Style Questions

As a language generation model, ChatGPT is able to generate text in a human-like manner. This means that when given a prompt, it can generate a coherent and informative response. In the case of USMLE prep, this means that ChatGPT can theoretically generate practice questions that are similar in style and format to those found on the actual USMLE exams. While the generated questions may not be identical to those found on the exam, they can provide a useful way to practice your knowledge and understanding of the material.

To generate USMLE-style questions, ChatGPT is trained on a large dataset of medical texts, including past USMLE exam questions and other medical literature. This allows the model to understand the language and structure of medical questions, as well as the content that is likely to be covered on the exam.

Why ChatGPT is Not a Replacement for Question Banks

Wouldn’t it be awesome if this model can be used to write practice questions? In addition to that, it can be beneficial as an interactive tool where you can ask follow-up questions to further enhance your understanding. While ChatGPT has the potential to generate practice questions on a variety of medical topics, it is NOT a replacement for traditional USMLE question banks at this time. There are several reasons why ChatGPT is not yet a viable replacement for traditional question banks. 

First and foremost, I found that my experience with generating USMLE-style practice questions left me quite frustrated… 

The difficulty level of the questions were not on-par with what I expected.

I started with asking ChatGPT to “generate 1 USMLE-style question on cardiovascular physiology” and it did what it was asked. It gave me a very basic 4-sentence question that reflected the USMLE Step 1 style. If I had to rate the difficulty of the question in comparison to traditional question banks, I would rate it 3/5. So I continued to ask the AI to generate 10 more USMLE-style questions. The results were all the same – it generated very basic first-order questions testing me on the fundamentals of cardiovascular physiology and whether or not I could identify the disease. As a tutor, I guess what I expected was more detailed questions testing knowledge of pathophysiology, hemodynamic profile, ECG, etc. I wanted ChatGPT to sprinkle in some tricky questions that would force me to use higher-level critical thinking. While the generated questions may be similar in style and content to those found on the USMLE, they do not provide the same level of detail and guidance as traditional question banks. This means that, while it could supplement with a few extra questions on a given topic. you would still need to rely on other resources to fully understand the material and prepare for the exam.

In order for ChatGPT to generate a question targeting a specific concept, you have to ask it to do so.

So when I asked it to “generate a more difficult USMLE-style question on cardiovascular physiology”, it provided me with a longer vignette with an added ECG component. If I specifically asked it to generate a question “testing knowledge of the hemodynamic profile of aortic stenosis”, then it would do so. If students don’t know these more nuanced and difficult concepts to begin with, they won’t know exactly what to ask ChatGPT to generate. Contrast this to the UWorld QBank, where human experts have created a bank of USMLE questions designed to test understanding of higher-order concepts and make such connections. From this perspective, ChatGPT is lacking in its ability to properly prepare someone to take the USMLE Step exams.

The program still has some kinks that it needs to work out.

I continued to ask ChatGPT to “generate a USMLE-style question that includes an image of the hemodynamic profile of aortic stenosis found on cardiac catheterization”. Instead of presenting me with an actual image, it inputted a line that read “[Insert image: Hemodynamic profile in aortic stenosis.png]”. Without the actual image, how can I learn to read these types of diagrams (which are important for the USMLE exams)? ChatGPT is a language model, and so it does not have the ability to generate images like this. OpenAI does have an image-generating AI called DALL-E, but until these two AIs are able to communicate to generate questions with associated images, or something similar, this will continue to be a glaring disadvantage when compared to traditional question banks.

Another fault of ChatGPT is that when I asked it to give me an explanation of the answer choices, sometimes it is incomplete. What I mean by incomplete is that it will just cut off abruptly mid-sentence. I will then have to follow up with another question to get it working again. This product is still being tested and this may be worked out in the future, but right now that aspect of it is not so user-friendly and can be quite frustrating…

The follow-up explanations are not thorough enough.

Even after adjusting to the program and phrasing my prompts so that ChatGPT could walk me through the reasoning step by step, I still found that it was still lacking in details and high-yield concepts. Again, this stems from the fact that the AI does not quite understand the nuances that USMLE test-writers will test medical students on. In addition, traditional question banks often come with additional resources such as explanations and rationales for each question, as well as additional practice questions on a variety of topics. These additional resources often come in the form of images, diagrams, flow-charts, and videos, and are extremely valuable – especially for those who are more visual learners. This can be particularly helpful in preparing for the USMLE, as it allows you to not only practice answering questions, but also develop a deeper understanding of the reasoning behind the answers. For USMLE prep, the quality of the explanation to the question and the associated resources are oftentimes more important than the question itself. This is probably a major reason why the UWorld Qbank seems to be consistently rated as the best for USMLE test preparation.  However, ChatGPT is lacking in these key features that promote learning and deep understanding.

Another major limitation of ChatGPT is that it is only as good as the data it is trained on.

This means that if the model was not trained on a particular topic or if the training data was incomplete or biased, the generated questions may not be accurate or representative of the material covered on the USMLE. ChatGPT may also occasionally generate incorrect information and has limited knowledge of the world and events after 2021. I encountered ChatGPT’s inaccuracy when I asked it to generate a question testing me on the treatment for Clostridioides difficile infection. It very confidently told me that IV vancomycin was the answer (over oral vancomycin), when this is not the case. This is especially dangerous because a student could easily be learning incorrect information if they are not checking the information provided!  

This limitation also holds when trying to use ChatGPT to provide an explanation for why the answer is correct. While you can follow up any question generated with ChatGPT with a prompt like “What is the correct answer, and why?” and ChatGPT will provide an answer, there is no guarantee that the information is accurate and comprehensive. Again, this means that you would still need to rely on other resources to fully understand the material and prepare for the exam, and to keep you from accidentally learning incorrect information.

Using ChatGPT in Conjunction with a Tutor or Mentor

While ChatGPT can be a useful tool for generating practice questions, it is not a replacement for traditional question banks or other study materials. It is important to continue using these resources and seeking guidance from a tutor or mentor in order to fully prepare for the USMLE.

However, using ChatGPT in conjunction with a tutor or mentor can be a helpful way to supplement your USMLE preparation. For example, you might use ChatGPT to generate practice questions on a specific topic, and then discuss the questions and answers with your tutor or mentor to get a better understanding of the material. In this case, your tutor or mentor can also be sure to double-check the accuracy of the question and answer choices generated.

This can be particularly useful if you are struggling to understand a particular concept or topic. By discussing the questions with a tutor or mentor, you can get additional clarification and guidance on the material.

A Few Final Thoughts

While ChatGPT may be a useful tool to supplement your USMLE preparation, it is NOT a replacement for traditional question banks such as UWorld. There are several limitations to consider when using ChatGPT for USMLE prep, including the fact that it cannot generate images to accompany questions, it may provide inaccurate or less in-depth information, and it does not link to additional resources to help you understand the question and it’s answer choices fully. In addition, the range of difficulty in questions that it generates is not great. Lastly, ChatGPT is definitely not as user-friendly as traditional question banks.

To fully prepare for the USMLE, it is important to continue using traditional question banks and seeking guidance from a tutor or mentor that you trust. While ChatGPT may be a helpful tool to supplement your preparation, it should not be your sole source of practice questions. If you are looking for professional guidance on medical exam tutoring, contact us to learn more today!