New Podcast Miniseries Answers Your Top Questions About Studying and Test-Taking In Med School
In our newest podcast miniseries, we dig into the most common questions we get asked about studying and board-style test-taking in medical school. In Part 1, we dig into a top question we hear all the time: “With regards to medical board exams, are good test-takers born, or are they made?” We examine the “nature versus nurture” argument to see if bad test-takers are born or made.
“When we talk about test-taking at this level, I’m not interested in test-taking tricks or deductive reasoning strategies. That stuff is all invalid in my book. Test-taking at this level should be about cleaning up the test-taker’s ability to interface with and show what they know on boards. Being a good test-taker means you consistently plug into a question and read it accurately, without adding or losing key information while drawing the correct inferences using the parts of what you know. In some manner, the good test-taker narrows the choices by eliminating options that are partially false and then choosing the safest remaining answer choice.”
Not a Good Test-Taker?
If you’re struggling with test-taking on the boards, check out our STATMed Boards Test-Taking Workshop to learn more.
Subscribe to the STATMed Podcast!
Transcript
– [Narrator] Welcome to the STATMed podcast, where we teach you how to study in med school and how to pass board style exams. Your host is Ryan Orwig, a learning specialist with more than a decade of experience working with med students and physicians. In this new mini series, Ryan focuses on answering top questions about studying and test-taking in med school and on medical boards.
– [Ryan] Good test-taking is good reading at the highest, most nuanced of levels. Good test-taking is where reading and thinking intersect in a clean and optimal manner. Reading is so much more than what the layperson thinks. They often think reading means can I read, yes or no? How fast can I read, and how much do I remember? But it is so much more nuanced than that.
Hey, everyone, Ryan Orwig here, with a solo Ask STATMed A Question episode, where I answer a single question about learning and test-taking in med school and on boards. Today’s question is, with regards to medical board exams, are good test-takers born, or are they made? This is asking if being a good test-taker at the medical board’s level is a “nature or nurture” type of thing. It’s asking if good test-takers are just born that way, or are they good because of some aspect of their educational experience.
This is a great question, and it taps into one of the things we are best at, and that’s helping bad test-takers at the medical board’s level become good at taking board-style exams. And I have been answering versions of this question for over 15 years. So, surprise, surprise, I have a lot of thoughts on this. I’m gonna break this up into two episodes, so here is part one.
First, let’s note this. When I talk about test-taking, I’m talking about medical board exams from the USMLE and COMLEX, through NBMEs and shelf exams, through specialty boards, like emergency medicine, internal medicine, ped, surgery, anesthesia, et cetera. This all also applies to the NAVLE and MAPLEX and other boards in related medical fields. And the insights and observations I’m making do not necessarily apply to other tests these students and doctors took earlier in their careers, like the SAT, ACT, MCAT, GRE, et cetera. These tests have different designs and create different and lesser cognitive burdens on the test-taker, and ultimately require different reading interface skills. So, let’s always keep that in mind.
So, first of all, let’s define what a good boards level test-taker is. The simplest way to put it is this. They can consistently show what they know on boards. They’re not making any unforced errors. So, that’s pretty straightforward. When we talk about test-taking at this level, I’m not interested in test-taking tricks or deductive reasoning strategies. That stuff is all invalid in my book. The psychometricians have scrubbed all validity from those kinds of strategies from the boards. Test-taking at this level should be about cleaning up the test-taker’s ability to interface with and show what they know on boards. Being a good test-taker means you consistently plug into a question and read it accurately, without adding or losing key information while drawing the correct inferences using the parts of what you know. In some manner, the good test-taker narrows the choices by eliminating options that are partially false, and then choosing the safest remaining answer choice. All the while, not losing or distorting any aspect of the question in their reading and thinking brain. So, here’s a key question bad test-takers have when they come to us. They say, “Do good test-takers just learn “how to be good test-takers “because they learned how to take tests “at their high school or some college course, “or they’re with the right review program?”
I don’t think this is the case. I’ve met with great test-takers and terrible test-takers from all walks of life, from rural Appalachian public schools to graduates from the best schools in the country. When I look under the hood of the excellent test-taker, it’s a closed system. That means it’s inaccessible, it’s unreadable. I can’t see the gears about what’s happening there. But when I look under the hood of the bad test-taker, I can see the gears. I can see the computer code. I can see where things are working and where they are breaking down. I can see the matrix of it, if that makes any kind of sense.
The urban legend is this. My friend’s roommate’s cousin went to this private school in Atlanta, and she’s a psych major and doesn’t know anything about medicine, but she can sit down and get 70 percent of the USMLE practice questions right. This is just so not true, and I’ve heard versions of this so many times over the years. If this were possible, don’t you think it would be a service that you could buy? Come on. But stories like this perpetuate the myth of the good test-taker, meaning a person who can hack and see inside of the test and get an elevated score by outsmarting the test design. This is actually super toxic, especially for the bad test-taker. If you think someone can divine the answers without the knowledge, you’re giving yourself reasons to aspire toward that when it doesn’t exist. And you’re perpetuating the notion that there is some game master plotting against you on the other side of the test design curtain.
Look, these questions are tricky, no doubt. They’re asking you to hold a ton of info in your head, draw specific inferences, and connect a variety of clues to encoded information in your memory, but they are not trick questions. If you think these questions are trick questions, you give yourself license to ignore any given clue that you want or to twist square pegs to fit into round holes, or to just disregard random clues, all of which will make you miss more questions than you should, and thus make you a bad test-taker.
So, that’s a roundabout way of saying I don’t think good test-takers come exclusively from elite programs that taught them how to take tests like this at an early age. So, to me, the good test-taker is looking through a clean window with a clear pane of glass and reading the scenario on the other side of the window. The bad test-taker is being asked to also read the same clinical scenario through a window, but the glass in their window is warped and splattered with mud. It makes reading the clinical scenario harder for them. But the person next to them’s like, “Come on, just look through the window, read the question.”
So, are good test-takers born this way, or did they evolve and grow into this? The answer is it depends. Mostly they are born that way, at least in some ways. And, in other ways, they grow the skills that become part of their operating systems. Good test-taking is good reading at the highest, most nuanced of levels. Good test-taking is where reading and thinking intersect in a clean and optimal manner. Everyone at this level is obviously literate, but it’s funny because reading is so much more than what the layperson thinks. They often think reading means can I read, yes or no? How fast can I read, and how much do I remember? But it is so much more nuanced than that. So, some good test-takers are going to be good test-takers no matter what, just based on their reading and thinking brains, sort of how they are just wired innately. Some others will be good if they have just a decent literacy-building background due to their innate gifts. Some will be good, despite issues with literacy background, for reasons that are harder to sort out when it comes to the reading brain. And some will be good test-takers, despite some issues that I’m gonna cite next.
I don’t know what the percentage of good to bad test-takers are in this field of medical boards, but I presume that the strong majority fall in the good to decent range. I wonder if, when we talk about bad test-takers at the medical boards level, are we talking about a small sliver of the pie, five percent, 10 percent, or does it go up to 15 or 20 percent? It’s hard to say. At any rate, some will be bad test-takers because of some built-in wiring challenges, like reading issues, or learning disorders, or ADHD, or issues with managing their emotions. Some will just not develop the right reading interface tools for unknown reasons.
Is that born, or is it made? It’s sort of a chicken or egg scenario, at the end of the day, and maybe doesn’t matter when it comes down to it at this point in people’s lives, because now it’s all about addressing the issue where it matters; at test-taking. But it is worth considering, on a case-by-case basis, to see if we can understand it. At this level, we are mainly looking at the way the testing issues manifest, and then we’re concerned with installing methods to override, or scaffold, or work around these very specific issues. A related issue we can run into is this. When people say, can you be a great test-taker earlier in life, like with the SAT, or MCAT, or in undergrad exams, or even classroom exams in the didactic years leading up to boards, and then turn out to be a bad test-taker on medical board exams? And the answer to that is: absolutely.
This can even happen from step one to step two, or from step two to step three, or it might not even manifest until residency or specialty boards. Because, here’s the thing, board exam-style questions are uniquely designed constructs that are unlike other computer-based multiple-choice exams. That’s a fact. And, like any unique text-based construct, it requires very specific reading strategies to unlock the way you optimally interface with it. This is what we teach. So, I think that leads to the next part.
What are some of the things that make bad test-takers exist? I’ll run through a few. Number one, one of the biggest reasons people are bad test-takers when it comes to medical boards, in my opinion, has to do with working memory. This is the aspect of cognition where we solve problems. It lasts very briefly, it’s very limited and will fill up fast and overwrite itself. So, it only lasts 30 seconds, give or take. The limited capacity can be thought of as seven plus or minus two, or maybe it’s four plus or minus one, depending on which model we use. For simplicity’s sake, let’s say one person might be able to hold onto nine items in working memory on the gifted side, while someone else might hold onto seven, sort of in the average, and someone else might only hold onto four or five on the low end, using the seven plus or minus two model.
So, as we read and work any board-style multiple-choice clinical vignette question, our working memories start to fill up. This experience will be invisibly different for the person with a robust working memory than it will be for someone with a limited or impaired working memory. If you can hold only five items, you start to lose key clues or connections as you work your way through the question and sort your way through the answer options. Incoming information overwrites existing items you’re trying to hold onto. This can make wrong answers appear to be right. This makes the test-taker with limited working memory look through that warped, muddy window, when the person beside them, with a robust working memory, is looking through the clear, clean window. That’s going to be a lot harder for one of these two test-takers.
My theory is that board-style questions unintentionally evolved thinking that all working memories were the same, that the higher the IQ then the higher the working memory. And since all doctors and veterinarians and other folks taking these high-end board exams must be “super smart,” then, hey, no problem. I mean, the idea that IQ and working memory were parallel was indeed what they thought back in the day. We now know this is not a static relationship, and you can be a genius and still have an absolutely impaired working memory. And I think many boards test-takers I meet who are super bright and super knowledgeable, but they just are wrestling with low or super low working memories has nothing to do with their ability to be an amazing physician or to know stuff, just the way that you’re interfacing with these questions.
So, under this pattern, are good test-takers born this way, or made this way through their environment and upbringing, and the evolution of their reading brain? I think this is more of a born this way argument unless we get into the weeds about whether or not you can grow working memory based on environment. So, broadly speaking, this is a hardware issue. And, if you have this robust working memory hardware, you are likely to be a better test-taker at the medical board’s level, and if you have a poor working memory, you are more likely to run into issues on board-style exams. And then you can say something like this. If you have ADHD, for example, that is, in part, an impairment of working memory, so that profile would be more likely to fall into this kind of test-taking issue. I’m painting with a broad brush here, so keep all that in mind.
Additionally, you can have a limited working memory that does not rise to anything diagnosable either, in my experience, and it might never become an issue, but it might have been a dormant issue that you could compensate for before, up to a point, and that you will compensate for later in your life and your career, but becomes a problem at some stage with boards test-taking. Why? Because the way these questions are built just puts such an incredible burden on working memory. So, I’ve not seen anything that makes me believe you can change working memory. I know there are people out there looking into this and doing work in this field. I, personally, have not seen anything that makes me think that it works in this regard, but it might be out there. Instead, the way I deal with this is by saying, “Okay, there’s a limitation on the working memory here,” so let’s install a meticulous, organized step-by-step approach for reading and working board style questions that limits the burden on working memory. This is absolutely something you can do, as long as you tie in the required behavior modifications since bad test-taking is also bad behavior.
Okay, area number two regarding good and bad test-taking involves whether you are using your partial knowledge to narrow and choose your answer, or if you are limiting yourself by relying on absolute knowledge, approaching the questions with a binary, do I know this or do I not mentality? Okay, good test-takers use parts of what they know to systematically rule out answer options that are partially false, because a little bit false is all false. Then, they choose the safest answer remaining, “safe,” in quotes here, meaning picking the option that most closely connects, on the one end, with the parts of what they know, and, on the other end, to the specific question being asked.
Many bad test-takers, on the other hand, often think they need to know everything to get a question right, and when they don’t know everything when they hit one little thing they don’t know, they push away from the thing they know and latch onto another option. This is one of the key patterns underlying the statement of I always narrow down to two and pick the wrong one. So, let’s try a quick scenario here with a binary test-taker who is only comfortable choosing the answer if they are 100 percent sure they know the answer. They wanna click on it, and say, “I know,” click, “This is a hundred percent right.” So, when they are not a hundred percent sure, bad things happen. And here’s the thing. More often than not, on boards, you’re gonna be not sure. So, we can’t allow that to happen.
So, in this example here, we have a pediatric resident. Let’s say his name is Joe. And he is working on a question about a young girl with an infection above her neck, probably an abscess. Now, Joe excels clinically, and he knows his stuff, but he is a bad test-taker, so how has he made it this far? His entire career thus far, he has compensated the way many bad test-takers compensate, with over knowledge, and that comes from overstudying. So, the whole way through, it’s been a real slog for Joe, because he does so well, but his scores never reflect his knowledge. And it hasn’t gotten him in trouble yet per se overly much, but he’s right on that edge, so we’re trying to fix it before he actually takes his peds boards, right? But I think it’s a really important thing to note, how does that bad test-taker get by at this level? And it’s just robust over knowledge. They are just muscling through questions. They’re not being nimble. They’re not being agile. They’re not being efficient. It’s just brute force, and that’s rough. That’s a rough thing to have to live with, but you gotta do what you gotta do, right? So, this binary mentality of thinking you need to know everything, it can push you even more into maintaining, and keeping, and growing a very binary mentality, and then getting frustrated when you don’t know everything about a question, and that’s what happens here with Joe. He knows diagnostically in this question that this is an upper airway infection, the key being that it’s above the neck. So, he understands the scenario but is thrown off by not knowing everything.
Let’s see if we can illustrate how his desire to know it all 100 percent actually makes him miss this question, a question he should get right 10 times out of 10. Number one, he is not confident in his knowledge in this area, so that unsettles him from the very get-go. A good test-taker would roll with it. But, since Joe is already prognosticating he isn’t strong in this area, he is already doubting himself and pushing himself into the binary trap. Number two, while he has identified as an upper airway infection above the neck, he isn’t 100 percent sure about the pathogen, and that leads him to think he has to know everything for a hundred percent certainty to get it right. And that’s just not always true with these questions. A good test-taker uses the parts of what they know and trusts what they think they know. A bad test-taker, especially a bad binary test-taker like Joe, on the other hand, lets the fact that he isn’t a hundred percent sure push him away from the stuff that he knows. This is a weird, but very persistent phenomenon that we see all the time.
So, he says, “I think this requires anaerobic coverage, “since it’s above the neck, “but I’m not sure.” This statement of the “But I’m not sure” variety is the poison pill that leads him to picking the wrong answer. He does this all the time, but he’s unaware of it. This is bad binary test-taking 101. So, he goes through the answer options one by one, as we train him to, and he eliminates A, then B, then C, then he gets to D, and this is where the binary mentality gets him D is ampicillin and sulbactam, and he says, “Yeah, this covers anaerobes, that’s a great choice “if I’m right, “but maybe I should go a step up, coverage wise, “and hit it with something broader.” Okay, look, this is the right answer. This should be a lock for him, given what he knows. But, since he’s not a hundred percent sure and does not 100 percent trust himself, he’s already pushing away from the correct answer.
I would categorize this as what I call the safe answer for him. D is the safe answer since it connects the parts of what he thinks he knows with the question being asked. But bad binary test-takers like Joe innately push away from these scenarios. He’s also likely flooding his working memory with these other thoughts, which serves to secondarily sort of unhook him from the specific scenario and the specific question being asked, which adds to the problem. So, then he looks at the next and final option, option E, ceftazidime, and he says, “Well, this is broad-spectrum, “but wait, does this cover anaerobes? “I’m not sure. “Maybe it does, so I’ll say, yeah, let’s use this,” and then clicks this as the answer and misses it. Oh boy, it’s like watching a slow-motion car crash.
Let me clarify this. If a good test-taker is tied between ampicillin and sulbactam, option D, and ceftazidime, option E in this scenario, they would do something like this. Now, this might be conscious or it might be subconscious, with their reading strategies plugging into it, but this is the way somebody would navigate this question if they had the fragmentary knowledge that Joe’s carrying, along with some of the self-doubt, but the good test-taking would carry them through. They would realize the question is asking, which of the following treatments is most appropriate for this patient’s upper airway abscess above the neck? They would say, “I think I need to cover anaerobes,” and they would not let the idea that they might be wrong influence this decision. They would look at ampicillin and sulbactam and say, “I think this covers anaerobes, so this is a good choice.” Then they would look at ceftazidime and say, “I don’t know if this covers anaerobes,” then they would pick ampicillin and sulbactam because that is the safest choice, using and trusting the parts of what they know, and be done with it.
As you know, Joe is not a good test-taker, and, in fact, he is a tried and true bad test-taker. Instead, he lets his binary mentality push him away from the right answer, even though he knew more than enough to get it right. He looked at ampicillin and sulbactam and said, “This fits what I want, but I might not be right,” and then he looked at ceftazidime and said, “I don’t know if this covers “the thing I needed to cover anaerobes, “but it might, and plus this is a broad spectrum drug, “so let’s do it,” sort of turning it into a wild card and it can fit whatever he needs it to. This is what a bad test-taker does, and I think they get pushed into that sort of thought trap due to the binary mentality, completely pushing him away from what he knows and losing sight of what he’s trying to do with this patient in this question.
So, this is an example of being a binary test-taker. Is Joe born this way or is this a learned behavior? Are test-takers who avoid these binary traps and rely on flexible, partial, are just born that way, or are they shaped into being that way? The answer is I don’t know. I don’t think they’re ever taught that way explicitly. I think they just innately grow into that, but this is getting into reading theory and the way different reading brains evolve. All I can tell you is this binary, need to know it all mentality makes test-taking at the board’s level really, really hard. But bad test-taking is bad behavior, so we can use a control system to swap these bad behaviors with good behaviors. That’s pragmatic, and where the rubber meets the road, so that’s what we focus on.
Let’s identify the behavior, let’s name the behavior, let’s sort of see it in self-reflection and grow self-monitoring skills, so you can regulate yourself moving forward. Absolutely doable. So, we’ll stop here and come back for the second and final part of this exploration about whether being a good medical boards test-taker is innate or something we learn in our next episode. Thanks for listening, and we’ll be back soon.
– [Narrator] Thanks for tuning into the STATMed podcast. Join us next week for part two. If you like the show, we hope you’ll subscribe. You can find more test-taking and studying strategies specifically developed for med students and physicians over at our blog on STATMedlearning.com. Thanks for listening.