Pitfalls to Avoid When Taking Medical Board Exams and Other Tests
Over the past decade, we’ve seen the students and physicians we work with make a lot of mistakes on tests. But we’ve come to realize that the types of mistakes they make are surprisingly consistent. So much so, in fact, that we’ve been able to identify 13 common errors medical students and doctors make that cause them to miss points on board exams and other tests.
In this podcast mini-series, we dig deep into each “miss type,” or type of error we’ve seen. We start with evaluating whether it’s truly a test-taking issue or whether you simply don’t know the material well enough — a miss type we call a “knowledge miss.”
From there, we outline 12 additional ways test-takers can go wrong on exam day.
Test-Taking Mistakes: Part 1
In this episode, Ryan and David discuss how to tell if you’re a bad test taker — or if you just weren’t prepared for the test. Then they dig into three common mistakes med students and doctors make on board exams.
More Test-Taking Misses
We’ll discuss other test-taking pitfalls in future episodes. So, be sure to sign up for our mailing list to stay up to date on our latest podcast episodes, videos, and posts!
Subscribe to the STATMed Podcast!
Episode Transcript
Announcer: Welcome to the STATMed Podcast, where we teach you how to study in med school and how to pass board style tests. Your hosts are Ryan Orwig and David LaSalle, learning specialists who have decades of experience working with med students and physicians. In this episode, Ryan and Dave discuss how to tell if you’re a bad test taker, or if you just weren’t prepared for the test. Then they dig into three common mistakes med students and doctors make on board exams.
Ryan Orwig: A truly bad test taker, they’re just repeating the same three, four, maybe five mistake patterns over, and over, and over on like an infinite loop. And since they can’t see the pattern, there’s no way they can fix it. And that’s why these things like, even if they’ve done hundreds or thousands of questions, they haven’t fixed the behavior.
Announcer: Here are Ryan and Dave.
Ryan: Hey Ryan Orwig and David LaSalle here with STATMed Learning. We talk about studying, timing, and testing in medical education. And today we’re gonna talk about, types of test taking misses bad test takers experience.
David: Yeah, so people call us up because they are struggling with board exams, maybe they’ve failed a board exam, maybe they’ve failed some shelf exams, or maybe they’re taking practice tests, and they’re learning from their practice tests that they are at risk of failing, or really just not where they wanna be in terms of test taking. So, one of the first things that we need to parse out is, is it actually a test taking issue? Because we deal with folks who are struggling to bring content on board to lock it into longterm memory in effective ways so they can get at it when they need it. So that’s really more of a study issue. But then we also deal with plenty of people who are specifically struggling with interfacing with the exams themselves, who are experiencing test taking problems. What we found is that there are patterns that show up again, and again, and again. So we’ve been able to identify some specific miss types that happen as people are working with these questions. And there’s about 15 of these things, but the good news is, that most people don’t experience 15 of them. You know, everybody’s got their own particular constellation of test taking misses, their favorites, if you will. So chances are, if you’re struggling with test taking, that three or four of these might pertain to you, but they’re costing you a lot of points.
Ryan: Yeah, what I see with it is, it’s like a truly bad test taker, they’re just repeating the same three, four, maybe five, mistake patterns, over, and over, and over on like an infinite loop. And since they can’t see the pattern, there’s no way they can fix it. And that’s why these things like even if they’ve done hundreds or thousands of questions, they haven’t fixed the behavior, because they can’t see it. It’s just like, oh, that was a dumb mistake. Oh, how did that happen? Or that was careless, or however they define it, in really fuzzy language, they’re not able to see where the boundary lines, the inbounds and out of bounds are. And therefore they don’t know when they’re skidding out of bounds, right?
David: Yeah, I think an analogy that I think you used to use, I don’t think I made this up, is the idea of like when my car stops functioning and I pop open the hood of the car, I can look at the motor, and I could be like, well, this car is not working. It is not doing what I want it to do, and I don’t know why. I couldn’t even tell you what the different parts of the engine are. So, yeah. So job one is to figure out what are the different parts of the engine? What are the different parts of the problem and how do we attack them? And then we could figure out what exactly is it that’s going on so that we can begin to fix it.
Ryan: The other analogy I use is, imagine you’re a really good basketball player, and you have all the tools technically, tactically, strategically, to be a really good basketball player, except you see the sidelines as about two feet narrower than they really are. And then the baseline’s about three or four feet deeper than they really are. So you’re out there playing with everybody and you’re a good basketball player, but then sometimes people are like inbounding the ball and you slap it out of their hands. Or you’re dribbling like way out of bounds, and it’s clear to everybody else you’re out of bounds like, what is wrong with him? You’re not really a good basketball player in that sense. Nobody wants to play with you, you can’t effectively operate within the game. So if you can, you know, give them the special glasses to see the boundary lines it’s like, oh, whoa! All of a sudden you go from useless to really good. And for something that’s relatively not that important to the game of basketball, or, you know, it’s sort of a fundamental thing that people overlook, like, they just write that basketball player off. So that’s the other one. So you got the old car thing is an old one, with the engine, but then the basketball, seeing the boundary lines. Like, how can you be a good basketball player if you can’t understand those boundary lines, you can’t. So that sort of gets into the test taking stuff, I think, right? You know, there are going to be times for the good test taker that they just don’t know enough and therefore they missed it. Somebody might contact us and say, I need help, I’m failing my boards, I’m failing my shelves, I’m failing my inservices I’m failing my pre-test, whatever. And therefore I’m a bad test taker. Maybe, right. Maybe that’s the problem, or maybe you just don’t know enough. And that’s where the conversation has to start. The conversation has to start with, okay, you’re contacting us because there’s an issue with test performance. I’m not performing on tests the way that I would like. Well, why do you think you should be doing better? Well, I think I should be doing better because, well, maybe student A will say, I study all the time. What’s that mean to me, you know? And so again, part of this is like, we gotta help them suss it out. We gotta help them pull that. I mean, that’s a fair thing to call us and say, it’s just that then we have to ask the right questions to get to where we wanna go and that’s a fine starting point. So then with student A, you say, well, okay, you talk to them about their study, you talk to them about what that looks like. But then I think we have to go into, I mean, the easiest template would be, okay, let’s imagine you do a bunch of UWorld or whatever bank you’re using. If it’s an emergency medicine doc, Rosh or whatever you use.
David: Rosh, yeah.
Ryan: Yeah, whatever you’re using, right? Let’s imagine you do 10 questions and you miss 10 questions. No, I’m sorry, You do X amount of questions, whatever, and you miss 10. You have 10 missed questions, you read the answer explanations, how many of those do you feel like would be just, I was going to miss that no matter what, either I studied it and couldn’t remember it, which is still a knowledge miss. Some people wanna say that’s a test taking miss, like, oh, I should’ve known it. No, I don’t care if you’ve studied it, I don’t care if you’ve heard it, if you did not remember it, it is a knowledge miss. You know, never saw it, or I had it encoded wrong, learned wrong. All those are all knowledge misses, right? But the other side is a test taking miss. How many of them do you realize, oh, I looked at this one clue, I ignored the other clues. The thing I chose wasn’t really answering the question being asked. I narrowed down to two, talked myself out of the right answer, I went to the wrong answer. I realize now that’s a bad thing. I added a clue that wasn’t there, I lost a clue that was there. All that stuff’s test taking misses, right? And so I think that’s where you have to start when you’re talking about this knowledge versus test taking split. Is that fair or where else might we start with that?
David: For sure, for sure, yeah. And there’s remedies to both. I mean, you know, if you’re having a knowledge deficit there’s remedies to that, they’re just on the study end of things.
Ryan: Yes, it’s like, you gotta go through multiple doors to get to, I can pass my test. I mean, it could just be as simple as door number one is knowledge, like in encoding, retrieval, random access, all that in the study side. And then the other is the door two is the test taking, showing what I know consistently on the test. I mean, but you’ve gotta be able to do the first one versus the second. I don’t know if you know this anecdote. I don’t know if I’ve told you this one. So think about it with the knowledge versus the test taking. Let’s say you wanna build an airplane in you’re in your garage. Have you heard the story? I’ll tell it anyway. Like my grandfather-
David: I don’t think we have.
Ryan: My grandfather built a World War I era, like, biplane, like a two seater biplane. He was a pilot. Things I remember about him is he had a clock in his house and the clock was like the propeller of a plane, and had the clock, fixed it, and then the picture beside it was like a plane, like Cessna on its nose, in a field. So that propeller came from the, you know, the plane he wrecked and he was like, hey any landing you can walk away from, so. I never flew with grandpa because-
David: Right, yeah.
Ryan: My mom wasn’t letting that happen, wisely. So he then decides to build this biplane in his garage. And this is scary, like genetic legacy stuff, I guess. But he built this thing for years. And at some point when it’s almost done, he makes this horrific revelation. He realizes like he can’t get it out of the garage. It’s not going to fit So there’s probably this moment where he’s like, he’s got two choices. He can take the plane apart to get it out, or you can deconstruct the garage.
David: Yeah. Heartbreaking.
Ryan: And so what he did, he took the garage apart. For months. So yeah, and then he flew that thing, and he survived, I mean, he never crashed it. But I never went up with it. So how does that relate? You know, so if it’s a test taking issue, you can build the airplane, right? You can have this fully functional airplane, that’s your knowledge in this convoluted metaphor, right? Like, you can have the airplane that totally works, it’s gonna hold up, up, down, land, all that jazz, right? But if you can’t get it out of the garage, it’s no good. So the garage, being the test taking, right? So it gets there. So right, but that’s where I think it has to start. It has to start with, is this a knowledge miss or is it a test taking miss? And I think there we’re getting the conversation where we start, it’s okay, so you’ve got this knowledge thing, and again, it’s not an exact science, trying to figure this out, but if the overwhelming majority of the misses are test taking in nature, I mean, again, it’s gotta be fixed, you can’t get the plane out of the garage. It’s gotta be fixed because, otherwise, I mean, well, let’s… Okay, if you’re a bad test taker, what’s your take on, how do you compensate? How do you get by? How do people who are bad test takers survive in this medical field with classroom exams, with steps and levels, with specialty boards, how do you think they survive?
David: I think, well, gosh. I think there’s a lot of energy output, and I think there’s a lot of struggle. Like a heartbreaking amount of struggle. I think that there is a lot of grinding on question banks and trying to know everything. I think that’s oftentimes people’s remedy, is they want to know everything. In order to pass the exam.
Ryan: That’s my short take on it.
David: Yeah.
Ryan: Is if you are a systematically bad, consistently bad test taker, that’s not going to go anywhere. It’s just there, it’s part of the system. The way they compensate is just knowing so much more than everybody else. It’s like being overly muscular instead of like fluid and fluent. I mean, I’m not a big swimmer, but you know, I grew up swimming, and like lifeguard, and pool, and all that stuff, but I’m not like somebody who knows a ton about swimming but the idea would be somebody who just isn’t like swimming and just churning the raw muscle as opposed to somebody who really understands how to like glide to the water with proper strokes. It’s dramatically different, you know?
David: So we’re saying that bad test taking is behavioral. That there are things that you are doing or not doing that are causing you to miss questions. The way I think of it, it’s not like we are the plumbers where you come to us and you say, “Okay, my sink isn’t working, please fix my sink.” And then we go and we do whatever we do and all of a sudden your sink works. That’d be nice if that were the case. It’s more like we are the physical therapists where you come in and you say, “I’m having this issues with my back or my shoulder.” And we’ve got to figure out exactly what it is that’s going on. We’ve gotta be able to take it apart, diagnose it. And then what we’re gonna do, is we’re gonna help you understand where the breakdown is happening. We’re gonna help you identify what it is that you need to be doing consistently, differently, in order to remedy the situation. And even then you’re still not done. You still got to go run the exercise a whole bunch of times in order to strengthen those muscles and get that limberness before the problem is gonna be resolved. The good news is there’s not an infinite number of errors types that people make. There’s a finite number of these. I don’t know exactly what that number is. I feel like every once in a while, I stumbled across a new one, I have one right now percolating that I think is interesting but there’s, you know, 14, 15, 16 that we have categorized as specific test taking misses that we see people do again, and again, and again. Specific discrete behaviors that people are engaging in during working a question, that are costing them those questions. So the first one that we can talk about, and when we have these sort of broadly categorized. We break down the working of a test question into three phases. Our first phase, I think of it as the reading phase, that’s where we’re really interacting with the prompt, the specific question being asked, and the vignette. And then we’ve got our second phase, where we’re dealing with the answer set. And then our third phase doesn’t always play out, but if we have a tie, a couple of answers that are equally strong, then we’ve got to have some tie-breaking methodology down there as well. So that’s our third phase that kicks in sometimes. So in our first phase, in our reading phase the very first thing that we tell our clients to do, every single time that they engage with a question is to go and read that prompt first. That last sentence in the vignette, the one that tells you exactly what question is being asked here. So right out of the gate, there are people who are attacking the prompt in ways that are biting them, in ways that are costing them in the question.
Ryan: Yeah, and that has become a very… That’s like a pretty well known strategy now, start with the last sentence, as opposed to say 15 years ago, that was much more of a novel approach. So a lot of people, like, that if you’re looking for test taking advice, that’s probably one of the ones that people have heard of, start with the last sentence first. And unfortunately just starting with the last sentence first, doesn’t fix all that ails you. Even with this problem alone, that’s the most, like, that’s like one of the craziest parts about it, is people say like, “Oh, I know I’m supposed to do it. And I only do it sometimes.” Well, if you’re not doing it all the time you’re putting yourself at risk. Because it’s such an essential anchor point. If your reading is getting overwhelmed, we know from reading theory, if we know what we’re looking for, it’s going to augment the engagement and retention, but if you choose not to always do it, it’s almost like saying… And then I’ll ask people, well, when will you read the last sentence? Well, when I remember it, or when I think it’s important. How do you know when it’s gonna be important? And again, look, if you’re a good test taker or if your friend’s a good test taker and that’s what they do, more power to them. With the prompt, if you selectively read it when you need it, that’s like saying I’m only gonna put my kid in their car seat when I feel like we might be in a crash. Otherwise I’m just gonna let them roll around in the back like it’s 1980 and they can have a good time. But to me reading that prompt first should be something you do all the time, every time. And then there’s other problems with the prompt as well, right?
David: Yeah. This is a little bit of a tangent, I guess but just to piggyback on what you were saying, I also am a big believer in stripping extraneous decisions out of testing. I mean, what an exam is, what a board exam is, or a shelf exam, it’s really just 1,000 tiny decisions that you have to make one after another, after another, after another. There’s a pretty well known story about Barack Obama, that when he was the president, he wore more or less the same suit, all the time in just a few different shades. And he was talking about it, I think, with Forbes magazine. And they were talking about that, asking him about that. And he said, “Yeah, that’s because my job is to make tons, and tons, and tons of very weighty decisions all day long. So that’s one fewer decision that I have to make every day.” Or we can think of Steve jobs and his turtleneck or whatever it might be. So I think if we build in a decision point right away where it’s like, well, sometimes I’m gonna do it this way, and sometimes I’m gonna do it that way, we’re gonna reach decision fatigue sooner.
Ryan: That’s right.
David: And this decision fatigue is real,
Ryan: It’s real.
David: And it’s going to hit during your exam. So let’s see if we can push that off as long as we can. Let’s strip out all those extraneous decisions about do I want to take this path this time or this path this time. You gotta get rid of extra junk in there.
Ryan: A burden, that they’re putting on.
David: Yeah.
Ryan: So yeah, so a big part of our philosophy is we want you to approach every question, the same way, all the time, every time. Now, when we say all the time, every time there always like very few exceptions. There are exceptions that we teach once you get into the system, but you can’t build a system built around the exceptions. We have to build a blueprint, a graphical overlay. that you as a test taker… You know, a test taker who wants to change their process can use all the time, every time.
David: So that’s one obvious issue with the prompt, is not tackling it first. And oftentimes what that leads to is a really general reading of a prompt. So we make that prompt generic. And, you know, I say again, and again, in testing, like, the general is our enemy, when it comes to testing. The specific is our friend. You know, it’s funny to hear you say, like people think about the prompt, like why should I bother? But it is the most important sentence in the world, at that moment. It is the only question in the world that matters. And I can’t tell you how many times I see people do a nice job of working their question and land on the wrong answer because they’re answering the wrong question.
Ryan: And maybe what that means is they did read the prompt deliberately, at first.
David: Yeah.
Ryan: And it could be that they don’t, from the outset. Okay, got it. Your brain is not gonna allow you to operate with like a giant, like, no question is being asked, like, your brain is gonna autofill.
David: Yeah.
Ryan: So you’re gonna make it generic or you’re thinking too clinically. So you’re thinking in the sense of… I mean, as a physician, I was talking to this interventional cardiologist, that I’m working with. I was talking to him the other day and this guy, I mean, like, he’s operating on people’s hearts every day. This is, like, his concert. This guy’s been doing this for longer than I’ve been alive. And his problem is, he thinks too clinically. So what does that mean? Well, as a physician, he comes in, he’s like I’m I… He’s building up the timeline. So he’s thinking about all the things the patient’s done and what they’ve been through, and what their chief complaint is, and why they’re coming in. And then they’re trying to think like, this is where we are. And then like, this is where we might go, depending on what we find out and what happens. Always forking, potential timelines, potential continuums. These tests are built to emulate that, and look like that, but they’re illusions of it. They’re just asking for a single fixed point on that timeline. And so, if you’re operating too clinically, and again, I think people say don’t think… Sometimes they’ll say think clinically, sometimes they say don’t think clinically whatever, that’s all just a bunch of noise. That’s kind of what’s happening, at least for people like him. And so he’s got to learn to really see that timeline and see, this is where we are, this is what I would do next or whatever the question’s asking. But yeah, people lose sight of the question being asked. Maybe from the onset, maybe they have it as they’re navigating through the passage, through the clinical vignette and then they lose it once they get to the answer options. Or maybe they keep a hold of it all the way through, and they’re working the answer options, in that second phase, and then they lose it, or they get to a tie break and then they lose it. And at the end they end up choosing something that is not the specific question being asked. And again, you can see it anywhere, from they lose it in the first five to 10 seconds, or later through, at the 70 seconds, or 80 seconds, or whatever it might be. Is that fair, is that sort of what you’re saying?
David: Yeah, and it’s such unnecessary point bleed. It’s so unnecessary, because the good news about the prompt is, it’s really easy to find. It’s always in the same place and you can check back in with it at any point along the way
Ryan: I guess let’s pivot to another test taking miss. So when we talk about other types of tests taking misses, the next one might be what we call the triangulation fail. Failure of what we call the act of triangulation. So maybe you can say what triangulation means, where that happens, and how it works.
David: Sure, yeah. So after you’ve read the prompt, after you’ve read the question, the last sentence in the passage that tells you the specific question that you’re trying to answer, then you’re gonna go ahead and actually read your vignette. So, as you’re reading that clinical vignette, we wanna read that clinical vignette neutrally. And we’ll talk in more detail about that in another video. But after we have seen everything in the passage, after we have read one time through the passage, what we are going to ask ourselves is, what’s in there that’s valuable? What are the most valuable clues in there to me as I go to assess my answers down below. What are the specifics that are involved in this passage? And we are going to grab three.
Ryan: Go back like three clues. And again, like maybe as you get better at this, maybe this always becomes just kind of a discreet step, maybe you end up learning how to, sort of, sort and cipher as you go. But what you wanna to do is go back and say like, you know, what are the three main clues? Now, but main initial complaints against this are gonna be, there’s more than three key clues.
David: Of course. I need everything, yeah
Ryan: We know. We know, we know, it’s all valuable but we want you to go back and select three key clues. Now what are some just big picture arguments for doing a triangulation? Like why is that a valuable thing. Even if there’s more than three clues what would you say are the main arguments for the strategy?
David: So, I mean, I think the number one argument is that oftentimes, what people are trying to do, is they’re trying to hoover up everything. They’re trying to grab every detail, and cram it in, and hang onto it. And then they’re gonna hang on to all of that as they go to work answers. What’s gonna happen there, is that we’re gonna overload. Your working memory system, your sort of mental chalkboard where you put your little notes while you’re working on something complex, that system gets overloaded, and then what happens is, you end up dropping clues without even realizing that it’s happening and then you don’t know what you have, and what you don’t have. So it’s not like we’re saying pick three things because we like the number three or because we are trying to instill some weird discipline. We’re saying pick three, because we have to be selective. It’s not like we wanna be selective, we have to be selective.
Ryan: Yeah.
David: You know, when I go on vacation, I would love to bring all of my clothes with me, who knows what party I might get invited to where I need a tuxedo, you know? But my luggage only holds so much. So I have to be selective about what I’m going to pack and bring with me. Same thing here. My working memory can only hold so many items at a time, effectively.
Ryan: Yeah.
David: So I’ve got to be choosy
Ryan: Right, yeah, you have to be… So and the act of being selective, is much more constructive and interactive. One thing that can happen is, if you’re not, you know, forced to have some kind of construct like a triangulation, you either maybe really remember nothing specific. So you get down there, and, I mean, you remember stuff, but it’s so random. Maybe you’re only remembering clues that are like just super general, making any random options fit. And what happens, you remember a bunch of stuff. and one key clue.
David: Yep.
Ryan: Then who knows what kind of damage you can do with, what we call, a single point of contact, down below. So triangulation offsets that, and it builds a framework. It builds a micro framework that you can then build off of and other things can stick to, and you can chunk, and link other stuff to it. So it’s an important act, again, to prime your brain for and to train with. Other thoughts on people who… So again, these are miss types. So someone might say, well, I had a triangulation fail. So what does that mean, just succinctly, you think?
David: I mean, you know, one version of it is I really just didn’t triangulate. I really just didn’t do it. I saw one exciting clue and I had that aha moment, and lightning struck, and I knew exactly what was going on. And that contaminated all my reading from that moment forward.
Ryan: Yeah.
David: And that’s just broken process. I don’t think that we have to be perfectionistic in our triangulation. I think people get very hung up sometimes about like, I gotta find exactly the right triangulation as though the question writer had like buried this secret treasure map somewhere in the question. And they had to pick out exactly the right things. I don’t buy that. Yeah, I don’t buy that. I mean, the answer is preset and everything in that passage has to support that answer. So as long as you do a decent, thoughtful job of triangulation, we’re going to be okay. We don’t want to be casual or careless about it but we don’t have to be perfectionistic or too, too, too terribly precious about it either. That’s the good news.
Ryan: 100%. I like using it to say, “Okay, you missed this question. How did you do up in the passage with your triangulation?” Well, here are the main clues that I really grabbed on to. And then we then make a decision, well, were those clues effective enough to get you to where you needed to be? Yeah, they were. Okay good. But let’s say, you know, actually they weren’t because I basically filled up on all these, like, really generic, low-hanging, like, non differential type clues, and then I really ignored this really weird clue over here. And this really like contradictory clue, down here. I just, kind of, didn’t know what to do with it, so I left it off. And then we say, okay, so this would have been your triangulation when you missed it.
David: Yeah.
Ryan: And it’s better to write it out, make it external and explicit, people don’t wanna do it but sorry, we’re just the messengers. And that’s how you change your behavior. And then we say, now write out what your triangulation should have been. And again, it depends on severity of issue. And again, I don’t think that that’s certainly not the majority of this stuff but we do see people that really struggle with it. So by reflecting on what they did wrong, these are the things that I chose and all the time it’s like, I didn’t really choose anything or I chose these random things. Why did you choose these random things? And then you get into rationale there. And then that tends to shine a light on the behavior, because it’s like a microcosm that then plugs into a bigger picture. And then you can find ways to hack it, with like the geography of the question or maybe like what we call the pointy edge or what have you. And then you can figure a way around it.
David: Yeah, and you can get a little formulaic with it. If somebody is really struggling, I mean, there are classes of information that we probably want to grab onto. There are some categories of information that are oftentimes useful in triangulation. So we can point towards those. I like your mention of somebody, you know, choosing to leave out contradictory clues or clues that they don’t know how they factor in.
Ryan: Yeah.
David: I think you see that a lot, and I think so much bad test taking is emotionally motivated and it is driven by this need to feel comfortable and secure while we’re taking these tests. And I think the sooner we get okay with the idea that we are not gonna be comfortable during this exam, that there’s gonna be a lot of times we’re not gonna know exactly what’s going on, we’re not gonna have a firm diagnosis, we’re not going to know the answer for sure, the better off we are.
Ryan: Well, and that ties into the idea that we are supposed to know the answer when we click on it. Like, I think, in a lot of people’s hearts and certainly, a lot of bad test takers hearts, the idea is like, when I click C, I wanna know like, yeah. And I think that’s the expectation. That is a toxic, toxic, like, expectation. Like do not expect that at all. I was working with this anesthesiologist years ago and she was like well, the good news is, and she failed her boards say twice, and it was back when you could take it once a year, and she’s like, “Well, the good news is, when I take it this time, I’m gonna feel good.” It was like right at the end of the workshop, I was like, “Whoa, whoa, whoa! No, you’re not!” And she was like, “What, I’m not going to feel good?” I was like, “No!” It was like it led to like an extra 30 minutes of the meeting and we were like signing off. I was like, whoa! And it was a talk, and then I think I had to talk to her again, like a month later, cause her test was like, say three or four months out. And it was really just about like, where are you mentally with this? It is not about knowing, like, this is the answer. You’re not gonna know, I mean, like, I ask, like, my emergency medicine docs, like, what percentage of the time when you click the answer you’re like a 100%? And if they tell me 50%. I’m like, you’re lying. And they’re like, you’re right, you’re right.
David: Yeah.
Ryan: That’s what I wanted. Like, they say, “I thought I was low balling it,” but it’s like, no, you know? I mean, even scaling it back to like prediction. Like, how often can you predict the answer?
David: Yeah. That’s like a deep-seated myth in this testing. That’s like one of those things that we just cling to. And I think it influences triangulation where, you know, I’m looking at a passage and I say, well, if I pick these three clues, this looks a lot like scabies, but if I include this clue, I don’t know what it is then. And so let me steer towards the comfortable answer and leave out that clue. Let me push that clue, so let me sweep it under the rug. So I don’t have to deal with it but really what my thinking should be at that point is well, it looks a lot like scabies, but I know it isn’t because I know this clue doesn’t fit scabies. So instead of then… Yeah, instead of my thinking, then being like well, let me go into my answers, looking for scabies, now I’m like, well, I think I’ve ruled out scabies because I know this clue doesn’t fit it. And even if I don’t know what’s going on, at least now I’ve got an accurate, honest picture of where I am. Instead of hiding clues from myself, pretending they’re not there, so that I get to feel more comfortable during the test, I would rather prioritize passing the test than feeling comfortable as I am choosing my triangulation points.
Ryan: Well, yeah. So then, with a failed triangulation makes you feel better in the test because it lets you pick scabies with confidence.
David: Right.
Ryan: Even though, in the final analysis you would realize, well, if anything, I should not pick scabies because I know that this clue I’m disregarding, doesn’t fit with scabies.
David: Yeah.
Ryan: So it’s interesting. I try to talk about these individual miss types you know, if it filters into, and tangles into other stuff. So yeah, triangulation fail is to me, whenever we can go back and say, like, I grabbed onto this clues or I didn’t really grab on any clues. And then we say well, if I had grabbed onto these three clues or these three, I mean, it can be more than one often.
David: Yeah.
Ryan: Then I’m better off. And that’s sort of how the triangulation there works. You have to make the mistake. You have to understand the name of the mistake and how the mistake works. Then you do a bunch of questions, you find examples where you did mess up, like, oh right here, look, I grabbed this. And if I had grabbed this, then I would have gotten it right. This is how I missed it, this how I would have gotten it right. That’s how you realize it. And it’s through that self reflection that you grow self-monitoring skills. So that you can regulate yourself in the run of play. Again, this is common practice in sports and other aspects of behavioral change. We just need to, I mean, it totally applies in test taking, you just need a construct to practice that. I think that wraps up triangulation failure, right?
David: I think so.
Ryan: Good. So another miss type would be what we call the neutral read failure. So we’re still up in the vignette, in the main paragraph of the question and what we want, part of our philosophy is that you need to read that passage neutrally. You’re really collecting the information and saying, what do I have here? So a neutral read failure means you didn’t do that, you contaminated it. Is that a fair summary of a neutral read failure?
David: Yeah, 100%, yeah. And I think that contamination can happen a couple of different ways, right? One way that contamination can happen is that as I start reading the question, I read my prompt, I start reading my passage, and then my emotions run away with me, one way or another. It might be like, oh good, I love this topic, I’m totally gonna crush this one. Or it might be, oh no, I hate this topic. I was really hoping they weren’t gonna ask this question. And either way, when that stuff pops up, and it’s fine that it pops up, right? You’re not gonna to turn yourself into a robot, it’s going to happen.
Ryan: Yeah. How does it influence you, right? How does it manifest?
David: Right, and ideally, should that change what you’re doing at all. You’ve got a process to run, you’ve got a job to do, right? So it doesn’t matter if I feel overjoyed at this question or desolate at this question, it doesn’t change what I have to do, it’s irrelevant.
Ryan: It’s fine to feel any and all of those emotions, throughout the course of the test, but you can’t let that up into your boat and flip you over. And it’s like, here I am, okay. And then you can’t let it flip you over, but you also can’t let it to throw you off course. And again, it could be like, you see a one clue at the beginning, and then it makes you yeah… Then you just, blah, blah blah, wa, wa, wa, wa, over the rest of the clues, you never factor them in. It could be that you skim over the answer choices, I don’t like that, but you skim over the answer choices and you see neurofibromas down there and then you shape the reading of the passage to fit neurofibromas, that’s not neutral. Like you said, it might be like, oh no, this is a terrible topic for me. And then you’re, you know, you’ve got like an anxiety haze glazing over. Again, if I have a system, I can lean on the system and sort of cut through that fog. And so then, you know, part of this read failure has to be recognized after the fact, you know, like, you know what? Like, I didn’t read neutrally because of X, Y, Z. You have to be able to name it. This is what I did that made me not read it neutrally. This is what biased me. And again, that’s not gonna fix it up front, but by continuing to use the language and the categorization, then you start to get your head wrapped around, and you start to get your emotions under control. You start to regulate, but that might take several iterations. But you’re not gonna get better unless you can name it. You gotta name the enemy before you can contain that enemy, that behavior. Does that cover all the neutral read stuff? I mean, again, this is one of those ones that you sort of explain upfront and then you really start to troubleshoot it once you start seeing if it’s happening in the run of play with feedback, I think.
David: Yeah. I guess I just want to mention, you had, sort of, mentioned spring boarding off of one clue. I think the neutral read is hard because it runs counter to what we do in our everyday reading. What we do in our everyday reading-
Ryan: Anticipation, and all that stuff.
David: Yeah. I mean our brains, are hypothesis machines. They’re gonna pick up hypothesis and that’s fine, right?
Ryan: Yeah, that’s how reading works. Reading works by making predictions and then revising those predictions as we go. And we’re also not saying not to do that. That’s where it gets, like, you’ve got to really just see this organically in the run of play. This is definitely one of the more like, it’s vague but very customized to the individual. So we’re not saying like, don’t think. Like, as you read, you should be thinking and predicting but it’s a line, there’s a line there, And I think so many of our people get so far over that line with predicting, with anticipating. Like, the interaction is so over the top that by pulling it back to the neutral read, maybe it helps calm things down, I mean.
David: Yeah. I think the error lies in when your brain kicks up a hypothesis, do you fall in love with it? Do you grab onto it with both hands with a death grip? Because what happens is, and it’s so easy for that to happen, where we get one or two clues on board and we’re like, oh, I think I know what’s going on. And it feels good, and I can relax a little bit more. But then what happens is as I continue to read, my brain has set up a filter. If I think that the diagnosis is hypertrophic cardiomyopathy, as I continue to read, the only clues that are making their way through my filter are the ones that fit hypertrophic cardiomyopathy.
Ryan: Cognitive dissonance. It’s a form of cognitive dissonance, where at that point I’m only going to, you know, recognize the clues that validate the hypothesis instead of trying to shoot holes in it.
David: And the it’s amazing because when you do that work of looking back after the fact, that’s when people are like, oh my gosh, I didn’t even know that was in there. That clue that was like three quarters of the way down the passage, it was invisible. And it’s like it aims the magic trick our brains did.
Ryan: Yeah, they’re not gonna say, like, I didn’t read it. They’re like my eyes went over it. I’ll have people to, like, I used to like sit and have people… The way I build all this, is like having people read out loud and you would think like reading out loud, they would catch it. Like, I heard you say those words. Like, yeah, I don’t know. But yeah, it’s because they set up that filter, and that is absolutely a failure of the neutral read. Very learnable, very learnable.
David: Yeah.
Ryan: But again, and this is more of an insidious, like underlying test taking issue. And it does tie into other aspects of the system. This is why we don’t go in and try to fix, like, we don’t go in and say, we’re going to try to fix neutral read. That’s that’s too in. Like, I wanna tear down, we wanna tear down the entire system and install our system, and you learn our system, hope, in the end all the way through. And then that helps us fix it. Does that that’s probably a fair saying, right?
David: Yeah, I mean, everything is connected. It’s all connected.
Ryan: It’s all connected.
Announcer: Thanks for tuning in to this episode of the STATMed podcast. In future episodes, Ryan and Dave will discuss additional ways test takers can go wrong on the boards. If you liked the show, be sure to rate it on iTunes, Stitcher, Spotify, or wherever you listen to your podcast. And be sure to subscribe so you don’t miss future episodes. You can find more test taking and studying strategies, specifically designed for med students and physicians over at our blog on statmedlearning.com. Thanks for listening.