Don’t Let Common Test-Taking Pitfalls — Like Predicting, Ruling In, and Code Failures — Sabotage Your Performance
Over the years, we’ve worked with many students who struggled to take medical board exams and other tests. And we’ve noticed specific patterns in the types of errors students and doctors make. In this podcast mini-series, we’re breaking down 13 of the most common test-taking mistakes we see on the medical board exams and other tests.
This is episode three of the three-part series.
Common Test-Taking Mistakes: Part 3
In this episode, Ryan and David discuss more common test-taking mistakes students and doctors make on medical board exams, including predicting, ruling in, partial true versus partial false, and code-failure misses.
More Test-Taking Misses
Interested in more test-taking pitfalls to avoid? Be sure to stay up to date on our latest podcast episodes, videos, and posts by joining our mailing list!
Subscribe to the STATMed Podcast!
Announcer: Welcome to the “STATMed” Podcast, where we teach you how to study in med school and how to pass board-style exams. Your hosts are Ryan Orwig and David LaSalle, learning specialists who have decades of experience working with med students and physicians. This is the third episode in a three-part series where Ryan and Dave discuss common mistakes, med students and doctors make on board exams. If you haven’t listened to parts one and two yet, we’d encourage you to check those out.
David LaSalle: Because what it means is that you don’t have to know the answer to the question in order to get the question right. It’s empowering. It means that you can take the bits and pieces you do know, and there are things you can do and steps you can take that will get you to the right answer even if you don’t know what the right answer is.
Announcer: Here are Ryan and Dave.
Ryan Orwig: Hey, Ryan Orwig and David LaSalle here with STATMed Learning. We talk about studying timing and testing in medical education. And today we’re gonna talk about types of test-taking misses, bad test takers’ experience. So the next mistype that we talk about a lot is the prediction trap. What do you think about that? What can you tell us about the prediction trap?
David: The production trap comes with a lot of baggage, right? Because I think that somewhere along the line, pretty much everybody that we work with has been told that they should be able to predict the answer to the question before they even look at their answer options, that that is like a good testing strategy. I think that that bites people over and over and over again. I think that even while they are still trying to read and process the clues on the page, people start synthesizing that and locking in, they’re like calcifying their predictions, which is super dangerous. What that leads to is that, well, two things, number one, as I’m reading, anything that doesn’t match my prediction, I’m gonna sweep it under the rug ’cause it feels so uncomfortable. I don’t know I’m doing it, but I’ll do it. And number two, it leads to being unable to assess each answer on its own merits. I am now taking those answers, and instead of checking them against the clues from the passage, the actual data points that I’m given by the test, instead I’m testing them against my prediction as though it were established fact, which it isn’t. So that causes me to knock out anything that doesn’t match exactly what was in my head, and I never get a chance to objectively weigh whether it was in fact a viable answer for that question. So those are a few of my initial thoughts on prediction trap.
Ryan: Yeah, I think that this is like for other question constructs, predicting what you think the answer is a good choice. And so this is something that you got to review board review programs, probably a lot of MCAT classes, and they talk about this important aspect, the importance of predicting, and I just think it’s a trap. It’s a trap, and then it’s like it pushes them into this binary, the binary mentality of do I know it or do I not? And they’re trying to predict. And then when they get to the end and they can’t predict, it’s like game over, man. Yes, I don’t know. And that’s not really how consistently working these questions at a board’s level should be. You have to work through the question and collect the clues, figure out what’s going on, generate some sort of hypothesis, no matter how flawed or broken, eliminate the wrong options, things that are partially wrong or wrong, and then choose from the best of what’s left. I think if your just one of your key tenants is like this idea that I’m supposed to be able to predict the answer, you’re cutting yourself off at the knees and you’re not gonna be able to do that very process. Some people like to look at the answer options first, just to see what’s up, to get a lay of the land and to see what neighborhood we’re in. I get the idea of it, but what might happen is you might see option D is something you know a lot about or you’re familiar with. And then the way you shape your reading because of your, you’re essentially predicting that answer option, you then shape the clues, you gravitate to this clue that does connect with that, you ignore these clues and then you twist this clue. All of a sudden bing, bing, bing, D is it, and then you click it, you’re like, nailed it. And that’s not how these questions are gonna work either, it doesn’t allow for that cognitive dissonance stuff where you signal stuff that doesn’t go with a prediction. There’s so many things that happen here. What else are you thinking?
David: Well, it just, it doesn’t serve. I think there’s so much anxiety around these exams anyway, that I think that that places such an undue unrealistic burden on the tester, because the only way that predicting works is if you know the answer to every question that comes up. Yeah, and I feel like we’re a little bit like, don’t predict. This sounds like a negative thing. I think at the end of the day, I think this is a real positive. I think the fact that predicting doesn’t work is a real positive because what it means is that you don’t have to know the answer to the question in order to get the question right. It it’s empowering.
Ryan: It is.
David: Because it means that you can take the bits and pieces you do know, and there are things you can do and steps you can take that will get you to the right answer even if you don’t know what the right answer is just cold. So I think that’s good, it’s gotta be good news.
Ryan: Just gotta, yeah, well, you can get, you can eliminate things that are partially false. A little bit of false is all false. A little something’s a little bit dead. Sorry, yeah. But then you’re choosing from the safest of what’s left that connects with the clues and what you know, and that is a dynamic paradigm shift in the way that we’re reading and thinking and processing questions. But just to sort of wrap up talking about the prediction trap, so many people come in with this preset notion that prediction is a key part of good test taking. And we would say that is not right
Ryan: And that’s a big part of our philosophy. All right, well, the next mistype that we talk about is, and I sort of alluded to this just now, is the partial true miss. This is when we miss a question because we hung onto the answer option or we validated or we chose it because it is indeed partially true.
Ryan: And the opposite of this guy is the partial false. So this is partial true versus partial false. Wrong answers are designed to be partially true. So when you look at A, B, C, D, E, just for the sake of our argument, A, B, C, D, E, say A’s right, B, C, D, E are wrong, they’re not gonna just be outlandish. They’re gonna have some, or for the most part, they’re gonna have some part that connects to part of the clue, the clinical scenario, the association and what have you. So what happens with, and this ties into prediction, it ties into expectation, it’s all like an interconnected organism, but when somebody comes on this thing and they say, well, I like B because it connects here, I’m gonna pick it. Well, maybe, right? But I guess let me hand this off to you for a minute. Like what do you see with the whole the partial true which is bad versus partial false, which is a better filter? What can you say about that?
David: Yeah, I think partial true is like a desperate testing strategy. I think that people enter into test questions feeling like they’re really in a position of weakness and anxiety sometimes, and that partial true is really what they pin everything on. It’s when we go into testing, hoping that there is just gonna be something that connects, hoping there’s just gonna be something we recognize that links one of these answers to one of clues. And when we see it, ah, huge sigh of relief. Oh, thank goodness. I can pick C because I know it would have a very high fever that would come on suddenly. And there it is, if there is a high fever and now I can just, ah, at least I’ve got something to go on. Oh yeah.
Ryan: This is somebody who’s really doing a lot of partial true rationalizing. It wrecks havoc, obviously, but if you can then flip it, then really the bar really goes up with this.
David: Yeah, and it’s just in a simple simplest formulation, you sort of said it, if an answer is partially false, it’s all false.
Ryan: It’s all false.
David: So that’s so beautifully cut and dry. We make a big deal out of the fact that we don’t do testing tips because as you said, right? It’s like a whole organism and everything is interconnected. And it’s really true. What we’re talking about is a systemic overhaul for a bad test taker to become a good test taker. But if I had five minutes to teach somebody to be a better test taker, this is one of the things I would tell them is if you see an answer and you can point to any piece of that answer that doesn’t match, it’s gotta go. No ifs, ands, or buts, treat it like it’s the law. Just knock that out of there.
Ryan: There’s only one answer can be right for the question to be valid. We see people all say, well, I really narrowed down to two and I really can’t parse it. There’s usually a pattern underneath that. Partial truth can happen with the way you validated a clue, can be the way that you read the prompt, it can be the way you’re weighing an option that can get you at the tiebreak. And you just have to sort of figure out where that’s happening. But if you can, like you said, as a tip, this is a pretty strong one, if this is the kind of thing you’re doing. Get away from this idea of like being so excitable and falling for the partial trues, which is fine. Of course you acknowledge it, but really you really wanna look for that partial false. You see something that’s partially false, it’s gotta be out.
David: Feels scary. Like you said, it feels scary to slash out answers, and I think that’s oftentimes why people don’t do this.
David: But paradoxically, it feels scary. It’s actually making a safer. Every answer I can knock out for cause is increasing the odds that I am going to choose the right answer so I can get over the fact that it feels a little scary, I’m in a better position.
Ryan: Yeah, you’re gonna win out over the course of the test, because it’s never about the individual question, it’s about the entire scope of the test where you have so many questions. So if you’re always behaving in as positive manner, all the way through you eliminate this whole, like for in this sense since the partial true trap, it’s the rising tides gonna lift all shifts throughout the test. So I think the partial true is definitely a huge one. So that brings us around to what we call code failure misses. And this is like an umbrella of a variety of very specific types of misses. If we talk about our first phase up the way that you navigate your clues up in the passage, working through the prompt of the question being asked, the clues in the clinical scenario, lab values and the imagery, however you might sort of pull that all together, but then you come down and you have to navigate through the answer options. We like to take each option one by one in sequence, weighing it and making a decision in the moment about how viable it is. Now, just upfront, I’m not saying that’s the only way to do it, but that is a key part of our process is this idea that you take each one, one by one, put it on the scale, make a determination about how viable it is, and we use this coding process, and I think it’s very teachable, it’s very learnable, it’s just different. And I’m not saying it’s the only way to do it, but this is the way we do it.
David: It’s a skill.
David: And it runs counter to habits that you’ve been employing for years and years. So you’ve gotta practice the skill. You’ve gotta identify it as valuable and practice this skill. But it is absolutely doable to break each answer out and deal with it on its own merits, taking it, comparing it back against the clues from the passage against the prompt instead of to the other answers down in the answer step below, ’cause that is a fast path to getting all tangled up and chasing your tail down in that answer.
Ryan: If you are a bad test taker, there’s all kinds of entanglements that happen there that we can tell you. I’ve had people refer to it as building silos or partitions or learning how to partition. And it’s almost like you said, it’s to build these barriers in your head as you read, and it’s so much more doable than people think. But again, it’s a reading skill. People think this is a tangent about reading, but people think like literacy is literacy, deep literacy is like, can I read yes or no? Am I fast or slow? And how much do I remember?
Ryan: I guess like how people think about literacy and it’s so much more nuance of that, is so many sub-skills, subsets, and these questions are of a very unique construction, and based on that construction, you have to use a specific set of reading skills to unlock it. And I feel like people are using very ineffective skills, especially the bad test takers, and they’re using the wrong reading strategies to plug into these things. And I think the way that they’re navigating these questions, we can’t say like, read answer option carefully.
David: Yeah, what does that even mean? That means squint a lot or file your brow while you’re reading? What does that even mean?
Ryan: Keeping my head through the screen. I hate that kind of advice. And so we wanna be more deliberate. So to me, that’s, and our clients aren’t gonna know what that means for them until they roll their sleeves up, and they start applying the methodology to the questions, which is of course what we do in the workshop with them. And so we’ve created a list of ways people might botch or mess up working a question while they’re doing this weighing process. So the first one is what we call soft coding. So soft coding, ’cause again, and again, you don’t have to worry too much about our process, and we were sort of talking about the abstract when we teach it as extremely explicit and linear, but we make them put an option on the scale and make a decision as to its viability, and you have to give it a code of one of our five credits, like is it strong? Yeah, I get yes, this is right. And you kind of answering is this viable? Is this right? A strong maybe, a weak maybe, and I’m sure you’ve had people try to just make some maybes and put them together, and it’s so important to learn how to put those to, it will be easier for us if it was just four, but making you split that right there in a vacuum by itself is important, of the question mark, I don’t know this, I don’t recall this, and we’ll talk about ways you can provide that or slash, I know it’s wrong or right to pick it. So a soft code, what would you say a soft code is? What what’d you say the definition or an example of how soft coding might work?
David: So a soft code, I think one of the ways that I talk about siloing these answers is to take each answer option and almost assess it on its own as its own independent question separate from all the other answer options, and weigh it on a scale, sort of a sliding scale of true to false.
David: Like yeah, this could very well be true, or I really don’t think this is true and I can knock it out. And then we’ve got several sort of way stations in between. I think a soft code is something like, yeah, I think this is actually a really good answer. I think this is probably true, but you know what? Maybe I’m wrong. Maybe I’m not sure about, maybe I don’t know something about this that would make it wrong. And so instead of giving it that yes that it deserves, I’m gonna slide it down into the middle. Or at the other end of the scale would be like, no, I don’t think this is it, but it could be it? Maybe sometimes in fringe cases, this is a kid maybe that’s different, and now we’re sliding back again into the squishy middle. So soft coding is taking what we know. I think it happens because we’re testing-scared. I think that’s what it is. I think a lot of times it is because we’re not trusting what we know. We’re not willing to lean on what we know. So we’re trying to account for what we know and anything else that might be true that we might not know or might’ve forgotten or might be wrong about. You just can’t test that way.
Ryan: And that kind of perverts, and that perverts our interpretation of the readout of the rock on the scale of the code, and then it’s unreliable. So, and would you talk about is the importance of trusting what you know?
Ryan: That is a key tenant to all test taking, it’s gotta-
David: Which is terrifying, right? For a lot of people it’s like, but I’m wrong, I don’t know things. That’s like a revolutionary idea in some, to a lot of bad testers.
Ryan: Yeah, to put it in such bold terms, all, look, all you have is what you know. That’s all you have. Are you gonna miss some questions ’cause you don’t know stuff? Absolutely. Can test taking have you wiggle your way to 100%? No. You’re going to miss questions. It’s not about that, it’s about consistently plugging into the questions, showing what you know, choosing the safest option and moving forward. You’re gonna miss questions, but if you do that consistently, you’re gonna be okay on the other side of it if you know enough. But again, it’s also readjusting what we think about what it means to know stuff. You don’t have to know everything about every question to get a substantial number of questions right. Sometimes, absolutely. You might not know that one little factoid and that undoes the whole thing and you’re not gonna get it right. There’s nothing to be done about that. You’re gonna miss that question, and we call that a fair miss. Now, but if you’re coming in you and you find that one little clue you don’t know and you’re like, game over, it undoes, then you have no chance. You’re cutting yourself off at the knees, you have no chance. Or you see an answer option that you don’t know anything about and you flip out about it, or you lock onto a prediction and the prediction is not even down there in the thing you’re trying to predict, and then it’s actually the answer A, but it’s like phrased differently or worded differently. You don’t even fairly ever put it on the scale. You’re just like, that’s not the thing I’ve predicted, so therefore it’s out, and then you’re eyeballing it. But yeah, but it all does come back to trusting what you know. And again, that is scary, but I think if you can unlock, again, I wouldn’t ever just end with that, right? Like, hey guys, better test taking, trust what you know, good luck, thanks for coming to our TED Talk, right? It’s like that is just one of the underlying principles. And you cannot code accurately if you don’t trust the parts of what you know. And so oftentimes, soft code, we can see these codes and we can teach you to see that the codes are soft. And then we gotta understand, why? And often it is like that, not trusting myself, you’re adding those twisty disclaimers of what if or but, maybe, saying like, I think this is right, but I could be wrong. What is all the studying and the clinical experience worth if you’re going to just chop it off with those little disclaimers at the end, and then all your codes end up going to the middle, you’re keeping free radicals in play that should have been eliminated, you’re pushing way too many things to tiebreaks? Some questions should go to tiebreak, but if you’re artificially pushing way too many things to tiebreak, which is overburdening an already really overburdened system’s bad idea. So again, this is a, it’s a symptom that is often, I think that we’re often dealing with soft codes. You’re not gonna usually come out of the first homework assignment with really good hard code, which is what you want. So it’s a pretty normal place to start, right?
David: Yeah. Yeah. And I think in some ways it comes from a fundamental misunderstanding of how you give codes in the first place, which people don’t know because this is our system that we’ve created so you don’t come in pre-built with that. But it is important I think to recognize that these codes aren’t based on like, how do I feel about this answer? Right. I think a lot of people are coding based on my gut feeling about this kinda like this one, I’d rather not choose this one.
Ryan: It’s just so much about the, people wanna talk about the gut. You know what? Whatever the gut is, it’s such a big vague, amorphous thing that we talked about culturally. If you’re really good at your gut and test-taking at this level, you’re probably not calling us. Right. So we wanna get rid of that. And again, I don’t wanna be like, go with your gut. That’s not the kind of advice we want, but right you’re there saying like this gut thing, right? And we’re taking that, the ambiguity out of that and turning it into something that’s more reliable with more concrete feedback that we can analyze and then change that behavior to get to this harder codes.
David: Yeah, the way I think of it is that when I give a code, I wanna always be able to explain why I gave that code. I wanna be able to say what my rationale was. Why do I think that this deserves a strong maybe? And the kind of rationale that I should be able to give is the kind of argument that I would make in a court of law, not the kind of argument that I would make at a backyard barbecue with my friends, right? At a backyard barbecue with my friends, I’m gonna be like, oh, come on, you just know it’s true, it feels true. It’s obviously true, it’s gotta be true. Right?
David: And in a court of law, imagine me saying that to the judge, like, come on your honor, it just feels right, doesn’t know. I gotta present evidence. I gotta be able to point to concrete evidence that supports that answer. Where do I get that evidence? Well, it’s right up there in the passage for me, and it is in all of the knowledge that I’ve acquired through all of my hard study. So I put those things together, and I should be able to say, I am giving this answer a strong maybe because it matches this clue, it matches this clue, this connects to what I know, but there’s still some pieces I’m not sure about, there’s some unknown out there, so it’s not quite a yes, but I’ve got enough evidence pulling me into strong maybe territory.
Ryan: And one of the things that I find that we have to really, like a certain fire we have to light in our clients is the use of because or why, these rationale pieces. I gave a slash because of this, but I should have given it a plus because of this, because of these clues. And what a lot of people wanna do is not go into that. They don’t wanna necessarily go to the why that because when we make them, which is like, sorry, you gotta give me that. And then once you ignite that fire, that sort of illuminates the path, illuminates the track, and that’s how they grow it, I think, or at least in part. And again, when we talk about this test taking stuff, but we have a very specific construct and scope, but it just depends on how an individual slots into it, right? To then figure out the path. When we were building the board’s workshop, there was an idea of like, hey, can we just make these like a four-hour video series or an eight-hour video series and just let people buy it for cheap, and then everybody has it and everybody wins? But you can put some stuff out there, but this, the back half of it, you really need that customized. You need somebody like you or me really walking through it and troubleshooting and giving that feedback and helping them build the construct, build the track and for how it plugs into them. What do you say?
David: Yeah, oh, for sure. Yeah.
David: Yeah. And it’s exciting work, I think for clients. And I don’t think people expect that, that they come in and you really start to see, oh my gosh, this is what I’ve been doing and I had no idea that this is what I was doing, and there is a path that I can follow that can get me to where I wanna be. I think it does it lights a fire that way too. I think people get excited about it.
Ryan: Well, I think, yeah, even up front, when they watch like the first few units, they’re like, wow! That was creepy. He was like, you were talking about me. It’s like you were in my head. They say that, right?
Ryan: So another, the second of the code fail misses that we can talk about is the abuse of the question mark. So one of our codes, and this is sort of an off-to-the-side, ’cause on a pretty tight continuum of yes, strong, maybe weak, maybe out, but then down there sort of near the bottom is also this question mark where you can code something and say, I don’t know what this is. I’ve never heard of this drug. Oh, I know I’ve seen this enzyme, I’ve heard of it, but I know nothing about it. I don’t know how just say like, I have no idea how this idea where this thing relates question mark. So how might somebody abuse the question mark?
David: Yeah, I think there are sort of two different causes of abusing the question mark. I think the first one is it’s just a subtype of soft coding where we see something and we say, well, I know some things about it, but I don’t know everything about it. I don’t know enough to feel confident, enough to get rid of it or enough to feel confident enough to put it in the maybe category, so I’m just gonna put a question mark on there ’cause I can’t get down in the weeds like that. So I think it’s pretty informal.
Ryan: They’re just counting on it. They’re like, I don’t wanna make an actual decision, so I’m just gonna give it a question mark, right?
David: Yeah, right. And sometimes that is because it is a considered answer, they look at it and they’re like, well, I have some information about it, but I don’t know if I have the right information. I don’t know if I have enough.
Ryan: Or enough, I don’t have enough.
David: Yeah. And then sometimes it is just a quick-jump thing that people do where they see something on the page and they don’t recognize it, and right away, like, don’t recognize it question mark, move on. And that happens sometimes because I’ve already got an answer that I kinda like.
David: And so I’m just gonna-
Ryan: The predicted, it’s not the thing I predicted, therefore it’s out. I was talking to somebody yesterday and she was talking about this question she missed between multiple sources and ALS. And so she had predicted whatever it was like ALS. And so she gets down to like say the right answers D, and it’s ALS by some other, just make some other description or some other label or something to do with it. And since it didn’t literally say that, she just like moved right on. And so she strikes it, but you could also be like, well, I don’t know what that is ’cause it’s not the thing that I’m thinking about. So you could just give it a bad slash, slashing up the right answer, or I think she was just like, well, I don’t know. And she didn’t really put it on the scale to see what she thought. I think it’s really easy to listen to us talk about this and be like, how on earth could you ever do this stuff under the rigid time confines? Right?
Ryan: We are gonna always sound like we’re not aware of time because we’re talking about doing all this stuff. Our ultimate concern is time. If you can’t do this up on time, then forget it. So we are very concerned about time, but it’s ridiculous for us for people of our expertise and our process to worry about time upfront. We have to get all the other ducks in a row first. Efficiency will come later.
David: Yeah, and I think we’ve covered it. It’s it just serves as a placeholder, right? There’s a kind of a kind of testing I think about, I think some of our clients fall into, I think of it as two-basket testing where the first thing I’m gonna do is go through my answers and sort them into a no basket and a maybe basket.
Ryan: Yeah, it’s like they wouldn’t do a quick knockout skim or something like that, right?
David: Right, yeah. And so oftentimes the question mark is just something, it’s just like a placeholder, like it lets me keep it around. Like I don’t wanna get rid of it, it lets me keep it around, and it’s a superficial processing. At some point, I’ve gotta dig in and see what I know and see what I don’t know if I wanna have a chance of getting to the right answer. But it’s unpleasant to deal with an answer that I don’t recognize right out of the gate or it’s unpleasant to deal with an answer that I don’t know that much about, it forces me to take a good hard look at what I know and what I don’t know, which is a little bit uncomfortable.
Ryan: So another mistype is what we call comparative coding. This is one of those things it’s like, it’s real bad, I think I know why but I like to know why in a lot of situations, but this is just like, look, I’m telling you, this is bad, don’t do it. So can you define comparative coding for us?
David: Yeah. We already talked about sort of siloing our answers, taking them one by one by one and dealing with them in isolation. Comparative coding is just any time when I start treating my answer options set globally and I start bouncing answers off of other answers and saying, well, I like this answer, but not as much as I like that other answer.
Ryan: Boom! Right there. Right there. And that is the nutshell of it, right?
David: Yeah. Yeah, and it’s hard to avoid. I fully grant that getting out of that habit is really hard because for most people, I think most people start taking multiple-choice tests in third grade, and oftentimes the first testing advice we ever get in our lives is look at all the answers, cross off the bad ones, compare what’s left, which I think is great testing advice if you’re taking a dinosaurs quiz in third grade. I don’t think it’s good testing advice anymore for something that is this complicated.
Ryan: Yeah, I don’t know when that advice does become obsolete.
David: I can tell you, well, I could tell you why I think it’s so dangerous. I think it’s dangerous because if I take my first answer, answer A, and I look at it and I’m trying to decide how strong that answer is, right? Is this a safe choice for me? What am I gonna look at? I’m gonna look at my prompt, I’m gonna look at my clues from the passage, and those are what’s gonna inform the code that I ended up giving answer A. So I’m testing it against things I know to be true because they’re given to me right there on the page. But if I then take answer B, and instead of taking it back to those clues on the page, I just bounce it off of answer A, now I’m comparing an unknown to an unknown. I don’t have anything solid to push off against. And if I was wrong at all in my assessment of A, well, that’s gonna cause me to miss code B and then C and then D and right on down the line. So I think comparative coding is among other things are really great way to take one error and turn it into a whole cascade of errors.
Ryan: Wow, of course multiplier against itself. And I think the other thing is if we think that working memory is at the heart of a lot of bad test taking, then you start on comparing A, B to C, all of a sudden your working memory is starting to overwrite a key clue here or there.
Ryan: And all of a sudden, who knows? You might have just lost a really the thing that may be less good, and then all of a sudden you just overwrote that, and then all of a sudden B becomes better because the clue that was weakest against it is now lost. I think that’s also part of it. And so there’s this idea of comparative coding is just, it’s consistently an issue, and then it’s just by, and it is consistently an issue, and then we see when people clean up their process, it just sort of goes away and it improves outcomes, right?
David: Yeah, for sure. And it’s 100% behavioral, which means that it can be learned and it can be changed.
Ryan: Yeah, and all of this is. It’s amazing that the behavioral aspects of all this stuff. Then the next one is what we call hugging the familiar.
Ryan: Which makes so much sense. But can you define what hugging the familiar means?
David: Yes. So hugging the familiar is when you’re working a question and you’re feeling maybe some anxiety around the question because you feel anxiety around testing or because this is not your favorite topic or whatever it might be, and then we see an answer and, ah, beautiful, that answer I know something about. That answer, I just looked at a couple of answers and I was like, I don’t know if these are right or not, but that one, I know I’ve studied that answer, and I know maybe even one piece of it connects, maybe not, I don’t know, but it jumps out at us and we grab onto it. We grab onto it like a life ring in the ocean.
Ryan: It’s humorous, it obscures everything. It’s like a blast radius around this thing.
Ryan: And it makes things, it makes the concrete stuff buzzier, it warps our reading of the clues, it might promote twisting, it might promote rounding down clues because it might play into some cognitive dissonance where we only pay attention to the stuff that validates it, or what is turning the question to a referendum on which of these answer options do I know more about?
Ryan:And then you end up picking the thing that you know more about in general without vetting it against the concrete clues in the question, and then you end up using your knowledge to damage yourself instead of, using your knowledge to validate a wrong answer, instead of using that same exact set of knowledge to knock out that same answer option, the thing that you would know enough to say, I at least know this one’s wrong, then if it’s a five-answer options, then it’s at least you’re picking one out of four instead of one out of five. And, it’s a really painful thing, but you can see how, I guess a lot of these coding issues are from having fuzzy process, instead of having a really crystallized process. Process for working through each question all the time, every time, which is at the heart of our principle. So you’re not making all these decisions that are superfluous and unnecessary and burdensome, but, and then also clarity of thought, like I’m not saying you’re always gonna have clarity of thought, it’s super stressful. You are worried about your career, your wife, your financial outcome, your dreams, your residency outcomes, your work-life balance, it’s all that stuff. And that stuff’s not gonna go away. But if you have a really rigid system that you train with, that gives you feedback that you can get better at, then you can lean on the system on test day, and that’s gonna help. And this idea of hugging the familiar is just, it’s a real trap, and I think it’s something that you can really cut out. What’s that?
David: Yeah, if you track the logic of hugging the familiar out, it’s easy to see how absurd it is, right? Because hugging the familiar is built on this logic that the test is written specifically and only for you so that the correct answer is always the one that you know the most about, that you are most familiar with, which once you say it out loud, you’re like, that’s ridiculous, of course that’s not true. But that’s sort of what’s underlying that decision to grab onto an answer just because I know more about it. I have like a dumb analogy for hugging the familiar of course. The way I think of hugging the familiar is like, imagine that you have been traveling abroad, you’ve been away from home for a couple of weeks. Sounds terrific but in this case, you haven’t spoken English to anybody in a couple of weeks and you’re feeling a little lonely and you’re in a crowded city and you see somebody you know from back home, and the next thing you know, you run across the street and you throw your arms around them and give them a great big hug and a squeeze before stopping to think, wait a second, is this somebody that I like from back home or is this somebody that I can’t stand from back home? And I think that’s the remedy to hugging the familiar ’cause familiarity is good. That means you know something about it, but you’ve gotta ask yourself the second question. The second question is okay, now, given what I know about it, does that pull me toward this answer or push me away from it so it lets me make an informed decision? But I’ve gotta get to that second question. If I stop at is this familiar or not? Then I got my arms wrapped around somebody who’s gonna drive me crazy on my European vacation for the next week and a half.
Ryan: It’s a nightmare.
David: It’s a nightmare.
Ryan: That’d be bad. All right, very good, I like that. Then that leads into gaming the test, which I was gonna start talking about just now before I realized that was our next one. And gaming the test, ugh! I don’t know. It’s like just the, so again, it’s sort of thinking about that there are people in a shadowy room sitting around a dark table plotting your demise, and you’re trying to get into their heads and counter their moves against you. it’s sort of how I think about it? How would you think of like gaming the test?
David: Yeah, that completely tallies with what my thinking is. My red flag for gaming the test is anytime I hear a client use the pronoun they.
David: Immediately I’m like, oh, hang on, freeze. In the context of like, well, when they want you to choose this, they always give this kinda clue, or they are trying to trick me by putting this in here. This they, all of a sudden, not only am I trying to assess what’s on the page and remember what I know about this content and put all that together, now I am also playing mental chess with somebody who I’ve never met and will never meet. I’ve got this, this figure out there who I’ve got to fence with as I work the question, is too much, and it’s the wrong game. I’m playing the wrong game at that point. Nobody that I work with is an expert in test design. They’re experts, our clients are experts in medicine. That’s your expertise is you’re like-
Ryan: That’s how they’re definitely spending their, that’s what they’re spending their 10,000 hours on. That’s their goal in life, right?
David: Exactly. Exactly. So that is the turf. That’s the turf that you should be, that’s where you should be focusing your game. That’s where you wanna be playing the game is are your medical expertise not out of your half-understood attempt to hack-test writer’s psychology?
Ryan: Well, and it’s also that they’re grandfathering in these old deductive reasoning strategies. They took SAT classes, ACT classes, MPAC classes, ways to rule things out. People have to understand there are psychometricians that statistically norm these questions. There’s very specific criteria being used, right? These questions And some bad test- taking habits might just be ingrained. Some might come from an SAT prep class, that doesn’t really translate to these constructs, or they were even more abused in their classroom learning years of med school by interacting with professor-generated question which are often not built to the same standard, not statistically evaluated nor by psychometricians. And so you come in and we’re gonna untangle this stuff, and it’s like, you’ve gotta get rid of all that. And again, it’s not just like saying, hey, stop doing that, that’s terrible, that’s very frustrating advice, but it’s like, it’s the system. And you gotta find out where the friction point is. So it’s like we really wanna make sure that we’re not distorting strategy to get these weird outliers. Instead it’s like, stay right down the center with it, and don’t get into this idea of gaming the test. And said something to me a few months ago about something that’s related about overly thinking about test design, about the distractors.
Ryan: The role of distractors. Everybody that comes, so many people, they wanna talk about foils and distractors and… How often do we talk about foils and distractors and stuff like that, do you think?
David: Only when clients bring it up.
Ryan: It’s interesting. I think it’s like a really like, nobody’s like, I really wish he does talk more about distractors at the end.
Ryan: And why do you think that is?
David: I think we’ve got enough work cut out for us looking at what clues are in the passage that we can grab and put together and synthesize and put together with our knowledge without also, ah, let me back up. I am a big believer in doing one thing at a time, right? And especially in testing, like just the cognitive burden is such that I cannot afford to be running parallel processes at the same time, the system’s gonna crash if I do that. So I can choose either I can focus on what is in this passage that is of use to me and that I can put together with what I know to try and make sense out of these answers down below? Or I can focus on looking for distractors that may or may not be in there, and trying to assess which clues are distractors in which clues are not, and try and sniff out anything that might be trying to lead me off the path. I have to choose between one or the other of those. I know which one I’m gonna choose, ’cause I can’t do both. I can’t do both. It’s gonna just be a big tangled up mess. And I know which one makes no sense.
Ryan: And this is definitely true for the bad test taker. I don’t know about the good test taker. I don’t know, it’s a closed system, I can’t see it. I can’t see the gears. A bad test taker, we can see the gears, and it’s like to try to listen to music and watch a TV show at the same time. You’re not gonna get enjoyment out of either of those things. And I think, yeah, I just feel like it’s too macro. They’re trying to look at all this stuff and it just ties into the gaming the test, is just something you don’t need to be trying to figure out.
David: And I think we have already built in a tool that takes care of distractors, which is that partial false.
David: Distractors are there, if we even buy the concept of distractors. And extraneous information. Extraneous information is there to maybe be attractive and make an answer partially true, but id we don’t-
Ryan: I think they do, they’re there, the wrong answers partially connect to clues. They’re not just randomly sitting down there.
Ryan: So, yeah. So it’s built in to our system to debug those things, to pluck them out. And so we don’t need to think about it. Again, I feel like people get so caught up in it that they’re thinking of this the big chess board type thing, and it’s just, we don’t need it, we don’t need to go there. And I think that ties into gaming tests. We don’t need to try to game the test, we just need to focus step by step, one foot in front of the other, working through the process, eliminate the wrong answer choices, choose from the safe so what’s left. So sometimes you get through your second-phase work through the answer options and you have one high code, and you just pick it and you’re done, no second-guessing for us, but sometimes you’re gonna be tired. You might get down to you’ve four slashes, two question marks and two minuses, one of those two minuses are all of the stuff that with maybes. So sometimes you’re gonna go to a tiebreak. By having a tighter system working your way through, you don’t wanna push everything to ties. And one of the first things we wanna find out is how good are you at tiebreaks? And we have very specific strategies we teach for various tiebreak moves is it can’t be a prescriptive thing. I think you need a grab bag of a handful of concrete things you can do when you find yourself in a tie and then you figure out how to use them. Is that fair? Is that sorta how you would look at the ties in general, big picture?
David: Yeah. Yeah, I think that’s absolutely true. I think tiebreakers, for whatever reason, is one place where a lot of my clients tend to default back into old habits. Yeah, good solid process and methodology, first phase, the reading phase, second phase, the coding phase, get down and the tiebreaker and just be like, I don’t know this one feels better, I’m just picking this and I gotta keep going. And sometimes I think that’s because they’re worried about time. I think that’s because sometimes things are getting real nitty gritty down there and they don’t wanna really dig in and do that very granular work in order to break out which of those two options is really my safer choice? So what I think is so important in our process is that we give a mechanism, an option. An option, a menu of mechanisms.
David: But you pick one. You pick one and you say, okay, this is what I’m gonna do for this tiebreaker, and I’m going to take the information, then I’m gonna plug it into this little machine and whatever comes out the other end as my safer choice, that’s what I’m gonna pick and I’m gonna move on.
Ryan: The one of the most common things people will say to us when they come to us and say I always narrow it down to two and I always pick the wrong one. Well, they don’t mean that 100% of the time, but they’re saying, so more often than not. So if that is the case, that means you’re doing something. You’re putting your finger on the scale inadvertently to tip you to going to the wrong thing. And there are a few patterns for those, and it’s gonna be dependent on the individual. You’ve gotta learn how to navigate with that tie-break process. I guess the other one is people end up defaulting to a partial true, or trying to rule an answer option in as opposed to a partial false rule and answer option out. It’s easy to rule a wrong answer in and very dangerous. And it’s how some people are. That’s like the default setting for the bad test taker, and that might be unrelated to working memory, or it could be tied into it. It could be we’re reading and thinking intersect. It doesn’t really matter. Once you see it, it’s like, okay, well, we need to now learn better ways to work around it. But if you don’t know the name of the phenomenon and to be able to see it after the fact, you can’t fix it moving forward. So that’s kind of a big picture type thing. We can talk about tiebreaks forever. Do you have any thoughts specific to tire brakes right now?
David: I have one thing that you and I have not talked about, so I’m curious just to get your read on it. This is something new I’m batting around with tiebreakers, is the idea that when we go into tiebreaker, we have to start back at square one with both answers in the tiebreaker, meaning I think that oftentimes we go into break process and we already have a favorite answer, and then a lot of times what we end up doing is stacking evidence for that answer or stacking evidence against the other answer without putting that finger on the scale. So I think in order to combat that, one of the things we can do is like a hard reset at the tiebreaker, like now I’m working in brand-new question, here I am with A and B, and I’ve got a process in place. I’m just gonna run it through that process and that’s gonna tell me, but I’ve gotta sort of shake loose whatever my preferences are coming in. Does that sound right to you?
Ryan: Oh, 100%. So that’s why I like us to, in our training tool, I liked them to do a dot or the check mark, like I’m going into tiebreak. This idea of not just sliding from second phase initial way into a tiebreak, I like there to be like this hard click.
David: Yeah, and a break point, yup.
Ryan: Yeah, and a scramble, reshuffle, pivot back to it. I had this guy years ago, one of my Seminole students for the board’s workshop. And he calls me like a week before the testing, like man, I think my tiebreak’s really bad.
Ryan: I was like, you think your tiebreak? And I like when people call me and ask me questions, I don’t think I’m just, but I knew him really well so I could proudly give him a little bit. I was like, you think your tiebreak is really bad? He’s like, yeah, yeah. Why don’t you know? And this is before, now we make people tell you the date is really easy to see it in large part because of the single student. That’s how we build everything is listening to our clients and building stuff moving forward. But I was like, well, go back and add them all up. Add up how many tally, now I know this is automatic, but I was like, go back and tally how many questions out of your last three practice block she did, how many went to tiebreak, and then what your success rate was on them? He’s like, I’m afraid. I’m afraid. And as any caring person would say, I was like, I don’t care, go.
Ryan: Go add them up and call me back. But clicking it, it might’ve been harsher than that, it might’ve been nicer, but who know? Who knows? So he calls me back, he’s like, hey! Like, how it go? He was like, everything’s great. He’s like, I added them up and I’m like 68% of the time I narrowed down to two, I picked the right one. I was like, wow, that’s great. He’s like, that’s great. So do you feel better? He was like, I feel so much better. Said, well, good luck. He’s like, yeah, I’m good. Can we probably talked about soccer or something or “Star Wars” or something? And then he got off the phone. And he took his test and he passed, he’s fine. But he was so freaked out about thinking he was bad because he wasn’t tallying it. And now of course that conversation led to that being built into this thing. But you have to know what the score is. You have to see what you’re doing, have to then be able to categorize, sub-categorize, find those sub-patterns and then work your way through it, which again is, I don’t know how somebody’s supposed to build this on their own. This is what you and I have been doing for 15 years, building this stuff out, listening to those stories. Like when that guy called me, I was like, that’s important. That’s important. That’s something that’s gotta be built into the system. And now it is, but I would hate to have to probably time-traveled and went back in time and lost my, I didn’t have my computer with me and lost my methodology somehow, I’d hate to have to build it again, because it was so lucky listening to these people and having a single doctor, the single med student, single veterinarian where it’s like that experience informed us to sort of build these things out ’cause the tiebreak stuff is in the whole system is just so dependent on various steps that have evolved over the years.
David: Yeah, that was happening.
David: That work is still ongoing, yeah.
Ryan: It’s ongoing and I think so many years ago, it’s like, don’t you get bored doing the same thing over and over. I’m like, it’s not the same thing over and over though. It’s the system, but it’s navigating that individual through it. That is pretty fascinating, and seeing smart, motivated, hungry people makes it really worthwhile. So the last test-taking thing, and we’re not gonna get too down in the nitty gritty on it, is timing issues. If you run out of time, that is a test-taking issue, I guess big picture, if there’s like a concrete reading disability, there’s a real issue with reading, our background is working with people with reading disorders, reading disabilities, you’re not really gonna get in there and really change the way that the brain reads that information. I think like the speed reading and stuff like that. I think what you’re gonna do is just get the whole system, it’s just gotta be super tight. The whole system has gotta be so much more calibrated and get rid of all the inefficiency and really become robustly efficient with the steps. I took a guy who had ADHD and dyslexia, and he had failed step two because he wasn’t getting to 10 questions per block. And we installed this system and his score took such a jump. I think he might’ve gotten investigated, ’cause it was so statistically improbable to make that jump because we’re busting the curve because we were changing the whole system. So I know it’s a learnable thing, but that’s like on one side over here. The same system works, it just gotta become hyper-robust. Is that fair to say on the reading issue side?
David: Yeah. Yeah, I think for sure. There are things that people do to remediate the specific learning differences and reading issues. That’s not what we’re gonna do if you come to us three months prior to taking your exam and say, I need
Ryan: Right. But there are ways to do that with the system. Now, the majority is more just people coming in like I’m always running out of time, I’m rushing through the last five, I’m not getting to the last three or five, it could be more, it could be 10, it depends on how just utterly inefficient and ineffective you are just churning and hemorrhaging time all over the way you’re tracking the way you’re training. When we’re thinking about that, like people who say like they’re really worried about time.
David: Yeah. Which is everybody, right?
Ryan: Sure. Because time is like gravity in relation to this test. It’s an inescapable force that’s gonna exert itself on everybody. Well, I’ll have people tell me like, I don’t have time issues, but I get done with 15 minutes left over, or 10 minutes left over. You probably do have time issues. You’re probably flying through this thing, and putting it on a scale, putting on a scale, putting on a scale, but not actually doing any of the measurements or auto-filling. So timing I think can go anywhere. You gotta find that happy middle ground which the workshop will take you through. But what are some thoughts you have, just big picture on the timing stuff?
David: Yeah, there’s a great expression. I think it comes from the Navy Seals where they say slow is smooth and smooth is fast. And I think that’s sort of what we’re all about, which is that first of all, we’ve gotta start slowly enough to build you a system that can run smoothly and efficiently. And once you’ve got that system that can run smoothly and efficiently, well, then that’s gonna bring itself up to speed.
David: So we can’t be frantic, we can’t be chaotic, we can’t be sprinting, we can’t be dragging our heels, what we’ve gotta put in place is the cleanest, most streamlined system that we possibly can, build that from the get-go, and then so many times, just installing that system, and then running it again and again and again and again, practicing, putting in the miles. Time often takes care of itself.
Ryan: Yeah, and that’s exactly right. It often does. And then what we never wanna do, and we’re always really keeping an eye on this for each individual, is we don’t wanna let somebody cut corners early on in foundations, because we have a progression as we work through the workshop where we add time. But you don’t wanna add time when somebody has really obviously a poor foundations, ’cause then you’re trying to work in two different directions at once. We’re trying to move forward with time, and we’re trying to backtrack and backfill and correct these holes. I’m not saying that you have to be perfect with the system before you add time, what I say is you have to be reasonably good with the system. And then it’s not necessarily like an elegant takeoff. It’s oftentimes a little rocky in there, but that’s necessary. It’s just the cost of business. It’s the cost of that skill acquisition. And yeah, it often does take care of itself. And then if it doesn’t, we have protocols where we go through and help find where those gaps are, ’cause it’s usually inefficiencies or just concrete nuts and bolts with the system or something about the philosophy of the mentality, that’s then causing a problem with the system.
David: Yeah. Yeah, I think of it, yeah, when I’m troubleshooting time issues, I think it tends to fall on one end of a spectrum or another where people are either so obsessed with time that they are sprinting through and losing track of accuracy altogether and their processes is falling apart, or they’re so obsessed with perfection and trying to make every move perfectly that they’ve let go of time altogether and so it’s taken them forever, and are looking for a sweet spot.
Ryan: Really precious with each But yeah, I think a lot of people do come in with timing concerns and timing issues, but that has to be addressed through working through everything else to get to that. And it’s gotta be cleared. It’s gotta be cleared, it’s obviously a concern. So yes, so that sort of wraps up our talk on test-taking mistypes.
David: Okay. So as you mentioned previously, when people call us, it’s not like they are doing all of these things wrong. Generally speaking, people are doing three, four or five of these things wrong, which is good news, because that means you’ve got three, four or five behaviors you can change and get you a ton of points, but you gotta be able to take the system apart, see what’s working, see what’s not, install a system that works in a clean, efficient, effective way for you. Yes, if you’re watching this and you heard anything in there that sounds like you, if you just wanna learn more, check out our website, give us a call, we’re happy to talk with you and see if we can help.
Ryan: Very well said. Thank you very much. And we hope to hear from you guys soon. Thanks.
David: Take care, bye bye.
Announcer: Thanks for tuning in to the “STATMed” Podcast. If you liked the show, be sure to rate it on Apple Podcast, Stitcher, Spotify or wherever you listen to your podcasts. And be sure to subscribe so you don’t miss future episodes. You can find more test taking and studying strategies specifically developed for med students and physicians over at our blog on statmedlearning.com. Thanks for listening.
Ryan: Hey there, Ryan Orwig from STATMed Learning. Thanks for watching. If you liked this video, we’d appreciate it if you’d hit the thumbs-up button below, and we’d love to hear from you, so leave us a comment with your thoughts. To make sure you don’t miss future studying and test-taking tips for the medical field, be sure to subscribe to our channel. And when you do, turn on alerts by clicking the bell icon. Thanks again.