Strategies bad test-takers can use to get ready for medical boards
Just like you wouldn’t practice basketball drills to improve your soccer skills, it’s important to approach prep for medical boards so you can “train the way you want to play.” In this episode, host Ryan Orwig is back with Dr. Jim Culhane, Assistant Dean for Student Academic Success Programs and Professor of Pharmaceutical Sciences at Notre Dame of Maryland University School of Pharmacy. They dig into more strategies students can use to prepare for medical boards, including using practice questions and the partial true/partial false strategy to prep for medical boards.
“But if you’re hurrying, picking the first answer that looks right if you’re missing key words, and you know that you’re that kind of test taker because all through pharmacy or medical school or veterinary school, you’re constantly complaining to your professors, ‘I picked the first answer that looked right’, or’ I missed this important phrase in the question’, you’re the type of person that better slow down a little bit and utilize your time resource a little bit more effectively.” – Dr. Culhane
Are you struggling to prep for medical boards? We can help! Are you struggling to prep for medical boards? We can help. Learn more about the STATMed Boards Test-Taking Workshop to get the most out of your study sessions.
Subscribe to the STATMed Podcast!
Transcript
– Welcome to the STATMed Podcast, where we teach you how to study in med school and how to pass Board style exams. Your host is Ryan Orwig, a learning specialist with more than a decade of experience working with med students and physicians. In this episode, Ryan and Dr. Jim Culhane, Assistant Dean for Student Academic Success Programs and Professor of Pharmaceutical Sciences at Notre Dame of Maryland University School of Pharmacy share their insights on preparing for the Medical Board. In part three, they continue their discussion on the dual-track method for preparing for medical boards, including the important role, practice questions can play.
– Right within there. And that’s what the good test taker learns how to do. Good test-taking can absolutely be innate and it can also be developed, but more importantly, you got, anyone can grow better test-taking skills, bad test-taking is bad behavior. So if you let these bad behaviors continue to get oxygen, they’ll continue to grow.
– Here, a podcast just isn’t really gonna work with that, but that kind of technique can help you to achieve that quickly and efficiently, correct?
– Yeah, yeah, yeah, yeah. I mean it’s better, it’s better. And again, it’s not always about being like best here, you know, this is about like, what can we do to cut out some of the worst, lowest yield things, and add some better approaches on the upfront. I mean, it could be as simple as, make yourself at every subheading, try to write down what you know about the topic and then read it, you know? Now this is getting a little more down into the details, this is not as macro, this is like boots on the ground connective, but people don’t wanna do that? I get it. I would want to do it, but it’s better. It’s the right kind of struggle. It’s the right kind of interface between you and your knowledge and this information you’re dealing with. And again, from the learning science side, we know that’s how it has to work, but getting students at this level to engage like this is very, very difficult. I’m always talking to my daughter and her friend, I’m sure they love this. Like when we’re driving to soccer practice, these high achieving ninth-grade freshmen in high school, but I’m just fascinated at how, you know, these same myths perpetuate, you know, there’s the same stuff. Now they are using more like Quizlet, Anky type stuff, they are using technology a little bit better to quiz themselves, but still not so often time, it is mostly dependent upon what the teacher gives them it’s not as much agency.
– Let’s switch gears and go to the practice question side now a little bit. And so again, you know, most of these Board prep materials, as you put as you mentioned before, have copious question banks and, you know, test banks that you can, you can practice board style questions on, what are some suggestions you have for the listeners about how to interface effectively with that component of it, because I’m sure there’s an efficient, effective way to do that, and then there’s a way that may not be so efficient or effective. So as you’re practicing these questions, what would you, what advice would you give the listeners?
– Well, I think number one, you’ve gotta think, I think at the top you have to identify, are you using practice questions, primarily as a learning tool? Or you pass that into actual iterating, and practicing recall and application at that level? So at some point you’re gonna be using practice questions possibly as a primary learning tool. So you’re gonna take much more of a kicking that way, right? You’re gonna miss a lot more questions.
– Yes. Which is not necessarily a bad thing, I always tell my student, right?
– Not at all.
– Even if you miss a question or you can’t answer a question, the active attempting recall on that, will help to encode the correct answer and correct information in your brain, so it’s, you know, if students should not be frustrated if they’re missing questions.
– No, no, that’s why I think it’s so important.
– It should happen in the study phase, absolutely.
– So important that you recognize like this is where you’re starting. Like, “Hey, I’m still early on in this.” Of course, cause they’re, anybody’s gonna be upset if they do 10 questions and they miss, you know, seven or eight or whatever, right? Or five, five or six, it’s like, they’re gonna be like, “Whoa.” But if you’re early on that’s I mean, that’s to be expected. I think that again, you gotta think about what your learning needs are.
– Okay.
– And if you are someone who can learn bottom up, I mean, there are probably a lot of people that can listen to this and be bottom up learners.
– Yeah.
– And okay, so then it’s about, I don’t know, I mean, I could talk about test taking all day, I think a lot, so here’s one of the, if there’s a single mentality adjustment, somebody can make going into these kinds of questions, going into Boards, just because it’s a multiple choice test with A, B, C, D, E doesn’t mean these things are built like other multiple choice tests you’ve seen elsewhere in your life.
– Yes.
– Okay. These multiple choice, clinical vignette scenarios, are second tertiary questions, that you don’t have to know everything to get everything, to get a question right. So I see a lot of struggling test takers. As soon as they stumble upon one thing in the passage, they don’t know, “Oh no, like I don’t know what that means.” I know I’m supposed to know that, I don’t remember that. I can’t figure out what the specific diagnosis is. It doesn’t mean you have to miss the question, but I think a lot of people immediately find that one thing they don’t know and it’s like game over. Game over, I’m gonna miss this question and like in a first order question, I mean, maybe that’s more true, I don’t know, but that’s not how these things work. You should be like, “Okay, well, what are the parts that I do have,” “let me use the parts of what I know.” “Let me work my way through the answer options” “one by one by one,” “looking for anything that’s partially false” “about a given question and learn how to strike out” “anything that is a little bit false.” Now a little bit false is different than a little bit I don’t know. Like if you’re like, I know that this one aspect is false, it has to be out, I’ll see people find something in option A that’s a little bit false, but says, yeah, because there’s these three other pieces that are true about it. So it’s like three is greater than one, therefore I will keep this as a variable in play. No.
– It’s a possibility as an answer is what you’re saying?
– In their minds?
– Yeah, in their mind, they’ll consider it a possibility, even if there’s there’s one false piece and three correct pieces in the answer.
– Yeah 100 percent because three is greater than one, right? No, it’s 100 percent out. And the more you leave those in, it’s like leaving these free radicals in play that is going to put you at risk. It’s either gonna just drag you down and take more churn and grind and burn to get to the right answer. Or you’re leaving a bad answer option in play, and you could actually end up choosing it because as you work a question, your working memory gets overloaded, it gets flooded. You lose sight of the thing that made it false. If you lose the variable, that one thing that made it false, and you’re only looking at the three that are right. 30 seconds later, you could be like, “Bing, bing, bing, I just nailed that question,” because you literally lost the thing that made it false because you left it in play. The longer you keep things in play, the, you know, the more risk you’re putting yourself at. So I would say, as you are starting out or wherever you are on board, your Board prep continuum, your Board training continuum, really train like you wanna play.
– Yeah.
– Really train in the sense of working questions, and if, I mean, I could talk about 100 things about test taking, but I would say like really lean into partial false is entirely false. The game should be to strike out the right answer. Now there are certain kinds of test takers that are what I call bad slashers. These are the ones that are slashing the right answer frequently.
– Okay.
– And that’s usually like, that means you’re impulsive, you’re probably going too fast, you’re maybe grabbing onto a single clue and ignoring other clues, or you’re forcing a prediction. You know, you come up with a prediction, you jam the square peg in the round hole and you pick it and you’re out. So if you’re often slashing out the right answer, that probably means you’re just being like broadly speaking, impulsive, and instead, so you gotta dial it back, but the majority of people are being way too cautious. Especially if you edge into that bad test taker sphere, you’re being way too cautious with that and too conservative with the slashing power. A little bit false is wrong. Let me say this too, like, I think a lot of people think so, you know, people are trained clinically. I mean, we’re not training people in this field to be test-takers. We’re training them to work clinically. So these test questions create the illusion of the real world, the three-dimensional real world, but these are two-dimensional constructs, so the real world,
– There are artificial constructs that we use to assess student learning, because we don’t have anything better, and that’s right from an educator’s mouth, I think that that’s, I agree with you 100 percent and I always tell my students, once you’re done with this, you never have to take another multiple-choice exam again.
– Yeah, for your pharmacist, right? Absolutely.
– Well, you’re right, for maybe medical students, they certainly do,
– Done, done, done, same thing with our vets, right? but here’s what I’ve been telling people. So like in the real world, you know, you come upon the clinical scenario of the person, you’re collecting all the background that led up to them coming to you right now, and then you’re thinking about possible timelines that would offshoot it, do this, then this then find out this, but if I find out this, then I do this, if I find out this, I do this. That’s how you think clinically, these questions have the illusion of that, but they’re only asking for a fixed point on that timeline. And then if it’s things shoot down, down the timeline, well like most appropriate next step, best appropriate initial treatment, I’ll see people say, “Well, this would not be the initial treatment,” “but you could do it later.” And then they leave it in play. When in fact it should 100 percent be ruled out like slash it out right then and there, and that’s what the good test taker learns how to do. Good test taking can absolutely be innate and it can also be developed, but more and importantly, you guys, anyone can grow better test taking skills, bad test taking is bad behavior. So if you let these bad behaviors continue to get oxygen, they’ll continue to grow and balloon and mutate into who knows what, but likewise, like, you know, we’re talking about this at the begin of the Board prep cycle. Like if you’re really getting into questions now I would really try to plant that flag, especially if you feel like that’s a thing that you do. And again, I’m never comfortable speaking to like the masses or speaking to these students as a monolithic entity, I think that’s a big problem in medical education is treating them all the same. So if you’re listening to this and this resonates to you, then yeah, probably start trying to grow that, it’s not gonna hurt you, it’s not gonna hurt you to really start growing that right now.
– I’ve got two, yeah, that’s great, absolutely. I’ve got two final questions for you. The first one is, can you just talk for a few minutes about, you know, when you’re working with practice questions, the importance of interleaving questions from different topics as a way to interface effectively with your practice question banks.
– Yeah, yeah, so interleaving sounds like we’re mispronouncing interweaving.
– Yeah.
– But interleaving, is this idea of like topic A then topic B, then topic C then back to topic A as opposed to topic A, topic A, topic A, topic A and topic B, topic B, topic B, topic B. Interleaving we know is super important, right? So I will talk to people, I mean, I’ve had this conversation this week where they’re like, yeah, when I do, like, if I do my PIDs, I’m really good with my PIDs questions, but then I go to like OBGYN questions, and then it’s like a harder transition, then I come back to PIDs later, like a month later whenever, and then it’s like, I’m back to where I started, right? Or maybe the better analogy would be, I do really well when I’m just doing PIDs or I’m just doing OBGYN, but when they’re mixed, they’re all mixed, like my timing is off.
– Right, it induces something we call a desirable difficulty into your learning, right? And that’s a term from cognitive psychology, that we, you and I have discussed before on a previous podcast.
– Yeah. So the problem there is they’re training primarily in these closed chunks, they’re not making their brains recognize, like they’re benefiting from entering the question and already knowing the neighborhood it’s in, as opposed to doing blind, mixed questions, which will make them have to identify, what is it, is this a psych question? Is this a PIDs question? Is this an OBGYN question? Like how I think to like, it’s higher up, it’s a level higher up. So by mixing questions you are inherently bringing interleaving by mixing the categories and you’re training your brain to, I almost add that higher up level of hierarchical discernment, and so you can travel those highways faster and all of a sudden be, go from like, oh, I’m in Psych boom, I’m in PIDs, I’m in, I am, and be able to toggle between, does that make sense? Am I answering that in the way?
– Yeah, no actually, I mean, so like for our athletes or musicians out there that can really, I think, tune into this because, you know, for example, and you and I have talked about this kind of analogy before our kids both play soccer, right? So it’s sort of like taking a kid, and teaching them how to dribble, pass the ball and shoot, and then throw them right into a game and expect them to put all of those things together all at once and perform effectively versus interleaving is sort of like, you know, in some ways it’s like scrimmaging, where you’re practicing, putting all those skills together, you’re dribbling at some point you’re passing at some point, you’re shooting at some point, and it’s more difficult in that way, but the benefit that you get from that type of rehearsal is way better than just compartmentalizing those skills by themselves.
– For sure, yeah, absolutely. And you, and I mean, I hear about this problem all the time, and people are a little shocked by it, you know? It’s like, no, no, no. it’s okay. It actually makes sense that you’re having that problem because you have not been adding that mix. Here’s another ripple on that. I will talk to people about timing, the influence of timing on their test taking, and obviously some people say, well, you know, time’s a real problem for me. I mean, I only have 72 seconds or 84 seconds or whichever, whatever your test is per question, that’s a problem for me. So they know, and so they train under time circumstances, hopefully. Okay, good. But what about the person that comes into me? This is somebody who comes to me and they’re struggling on their Boards, and they say, well, this is like maybe somebody who failed their Boards, and one of the first things I say, “Well, what’s the role of timing for you?” “Well, time’s not a problem.” “Oh, do elaborate.” “Like, yeah, time’s not a problem.” “Like, I get plenty, I get to finish” “with plenty of time left over.” “Well, I’d say, well, I would say time’s a problem.” “Well, why?” “Because you are going too fast.” “You need some checks and balances.” “You need to deconstruct it.” “Oh, okay.” “So did you practice using time?” “No, I don’t need to practice with time,” “cause time’s not a problem.” But the time is on the test. It’s a dooms day clock ticking down, up in the corner of this test. You can see it the whole time, it’s like gravity, it’s influencing everything. So time is a variable that has to be practiced no matter your relationship to time, and so is, I mean, that’s not as much interleaving as much as it is variable conditioning.
– That’s a really good point too, because, and what I really loved what you said there too, is most students focus on one end of that spectrum in terms of running out of time, right? But what I love what you say and I have a, you know, there’s always a couple of students in every one of my classes, right? They always finish an exam in 20 minutes, right? Or 30 minutes, and then they’re out the door, and you know, so time is also a resource, right? That you should, you should use, you should maximize that resource, right? So, or no.
– Maybe.
– All right, tell me about that.
– I don’t know, well, just because I, look if you get done and you’ve got 20 minutes left, I don’t necessarily think going back.
– And you get the grade, right?
– Just to clarify on that.
– But if you’re hurrying, picking the first answer that looks right, if you’re missing key words, you know, and you know, that you’re that kind of test taker because all through pharmacy or medical school or veterinary school, you’re constantly complaining to your professors, I picked the first answer that looked right, or I missed this important phrase in the question, you’re the type of person that better slow down a little bit and utilize your time resource a little bit more effectively, would you agree with that?
– Oh yeah, of course, but the only way that’s truly good advice is if they train that way.
– Right, exactly, right, right.
– So like, if you’re like, “Like, I’m so fast, I don’t need to worry about,” “I just need to have better behavior on test day.” You’re talking about like, you’ve got ingrained behaviors and you’re expecting just to tell yourself to behave better on test day? Well, cause I think people don’t think about it as behavioral, I think people think of it as like it’s just part of my personality, like, no, it’s huge. Like this is this, I mean, in sport, we care about this stuff so much, you know, like if the super bowl team plays on grass all year and they’re going to the super bowl on turf, they’re only gonna play on turf those final two weeks. You know, like, because that matters because every little thing’s an edge. I think when you come to, when it comes to test taking, you need to be aware of your strengths and weaknesses, and really install structure that really helps support the weaknesses and work around the weaknesses throughout your training, throughout the use of practice questions under that two siloed system that I was talking about right? And really train rigidly, and don’t be like, “Well, it doesn’t matter cause it’s practice.” Oh my gosh.
– Oh, it does matter, yes.
– I mean like that like when I, when I coached my, like my daughter’s a club soccer player now, but when I coached them when they were little, people were like, “Oh, you’re such a good coach.” I’m like, “Not really.” Like, when they’re like three or when they’re like four. “Yeah but my kids are running around” “with the ball in their hands and yours aren’t.” it’s yeah, because in practice, I never let them touch the ball with their hands.
– Right.
– Like I see you guys, you guys are picking, like the parents are picking the ball up with their hands, the kids are picking up, “Like, oh, it’s cute, it’s funny.” And then they go to the game and it’s like, “This is crazy.” It’s like, no, like, you know, teach them, like show them, like model for the, I mean, it’s super simple, right? But like, you’re thinking it’s easier to think about training and conditioning kids when they’re younger than thinking about highly intelligent, critical thinking adults, but there’s a wild animal in here for the test taker, and if your wild animal is hurting you, you have to really condition that thing. Through rigid, consistent practice, like you wanna play, exactly. So I don’t know that that’s definitely where I come from on this when it comes to like thinking about test taking and Board prep and all that jazz.
– Fantastic, all right. So one final question. I hope I’m not opening up a can of worms here, but I think this is really important, especially for my pharmacy students or other pharmacy students out there who, you know, can schedule the NAPLEX exam post-graduation at some point, it’s not a set date that they have to take it. How do you know when you’re ready? How do you know when you’re ready to go in there and take that exam? Are there any, you know, any indicators, you know, I have some in my mind, but I wanted to hear from you in terms of what you think.
– That’s a good question. How do you know? Well, I mean, you’re taking.
– When you’re ready to schedule that exam and take it right
– Well, I mean, most of these Boards have some sort of measuring tool. We don’t use the word predictive of course, but you’re using some kind of measuring tool. I mean, I think if your scores are super erratic, that should give you pause. And so like, like if you’re sometimes way above sometimes way below, I mean, I definitely know people who take the test, took their Boards and fail and they come back and they’re like, “Yeah I never got above passing” “on these measures these tools.”
– Practice tests, yeah.
– Yeah. I never, I never hit above passing. I was just hoping, I was just hoping,
– Yeah right, yeah right, if you can’t pass the practice test, right?
– It doesn’t make sense, right. So I think like you want to be like one and a half deviations above the passing is like, but that’s getting a little more into the psychometrics of it, a little outside of my scope on it. But yeah, I mean, you can’t, I think you wanna see level more consistent scores across, and I think you want to, for my people coming in, we’re trying to take those test taking mistakes that they’re making, cause we talking about knowledge versus test taking mistakes and really reduce and dramatically contain those test taking mistakes.
– In your Boards, I’m sorry. In your Board’s prep class that you do with students that you guys work with, I mean, you do, there’s again, that kind of two phases you’ve got, you know, the study phase in terms of here’s the study methodology and approach to, you know, studying the material and then you’ve got a whole test taking component to that as well, is that correct?
– Well, the STATMed class has both, but we have I think called the Board’s Workshop, which is just test taking.
– Just test taking, okay.
– Those people are coming in because they say, “I truly think my knowledge is good enough.” “My issue is my test taking.” So that’s usually a little downstream from a lot of this, and that’s where, I mean, we are banking on their self assessment, that their knowledge is indeed good enough, usually it is. Now sometimes people get in there and they find, “Okay, I cleaned up my test taking,” “and this actually shows me where my knowledge gaps are.” They couldn’t find the knowledge gap cause the test taking was so erratic, but yeah, so I think it’s a matter of, I mean, are you seeing signs that based on your test that those indicators tell us you are going to pass. That I mean, I think that’s kind of how I look at it. What are your thoughts on that?
– No, again, you know I’ve not done much work in this area at all, but my go-to would be again, based on my knowledge is really looking, like you said, at the results of yourself testing or retrieval practice, and like you said, whether it’s that, you know, the daily retrieval practice that you’re doing or the practice questions that you’re doing or practice tests that you’re taking, you know, if you start to see very positive, consistent results over time, to me that’s an indicator that, “Hey, you’re probably ready to go.” And like you said, okay, most of these things have, you know, those practice tests that you can take.
– Okay, so right there, now I would hear a mixing of talking about some study methods side,
– Sure, yep.
– And some practice question side, right? So if my self testing, my retrieval practice is going well, that’s all good, but that’s more study side foundational side. So I’m probably looking more toward the actual gateway indicators of like whatever the test is that we’re taking.
– You know, and I think, think that’s, you know, that’s really interesting too, because I think that that’s maybe a different perspective that you and I have about practice questions, right? Because you know, from my perspective, doing practice questions is a form of retrieval practice.
– It is. I draw a distinct separation between the two.
– Yeah, tell me a little bit about, tell me a little bit about that, what’s your separation?
– I look at it like this, so it’s not that practice questions are not retrieval practice. It is an act of retrieval practice, but you’re, let’s say I studied infectious diseases, a certain aspect of this ID and I do 20 practice questions on ID. That’s a pretty broad, I mean like that’s pretty hit or miss as far as like what I’m hitting on the width and breadth of that ID section. So like that’s very different than if I went through and did a framework self test on the content of that, say like the content in the review section is 400 factoids. I might have only hit 20 of them on that question, on the practice question. So I think of it like this, if I’m a soccer coach, which I’m not really, but if I’m a soccer coach, I’m thinking about all my technical development for my player, all the way they use the services of their foot to control the ball, to trap ball with their body, to move the ball, manipulate the ball with their feet, their body, whatever, that’s all like technical. So the technical training is very important. That’s what I consider under the retrieval practice side, the self testing on the knowledge side of the silo, okay? Simulated play and scrimmages, are what I would consider practice questions. So if I’m a coach, I want to mix technical training and simulated scrimmages, and I want to interleave them and swap them back and forth and overlap them and build off of them and progress through them.
– Yeah, I love that analogy.
– So that, and you know, if you’re like a really good soccer coach, like your session would start with some individual technical stuff that would build up and progress through the session, the training session. At the end there’s some kind of simulated play that’s tying in what you were working on technically, in some sort of simulated, not like an old school scrimmage, like we would’ve played when we were kids, like when we were kids, scrimmages we’re like roll the ball out there and go get it, you know?
– Right, right.
– Like, no, this is like, we’re working on defensive transition, but you’re still playing a game with some goals and stuff, you know, like, so again, like, I don’t wanna just do technical training and I don’t just wanna do scrimmages. I want to mix the balance, so to me, I separate. Cause what I’m really worried about is when I’m teaching study skills to somebody who’s at a Board prep level, is that what happens in? So let’s say we give them an array of study tools and test taking tools three months out from their test, and then they’re really into retrieval practice. Really just sewing all the stuff, all the knowledge in really building it in really fortifying, consolidating, building robust retrieval pathways, filling in all these holes for the first month and a half, and then they shift to practice questions more heavily. So, so far so good, but then they stop doing any retrieval practice of the content they studied and they are now only doing practice questions.
– Got it.
– I don’t like that, I don’t like that.
– I like that soccer analogy, I think that really resonates quite a bit.
– So yeah, does that give you a little different perspective? Like what does that shift,
– I see on that you’re coming from, with that? No absolutely, and I think it’s very clear to our listeners.
– Okay, and then, and last piece on that, so what would happen is I would ask somebody, and this is an example of inconsistent vocabulary usage, which always drives me crazy, and so I’d be like, “Oh, so you’re still doing,” So let’s say I’m talking to that same student a month before their test, like we had a three month window, I knew they were doing really good retrieval practice in that first month, cause I saw some of their stuff and they’re like, “Yeah, yeah, yeah.” “I’m doing all kinds of retrieval,” “I’m doing so much retrieval practice.” “Oh, that’s awesome.” “Yeah, like it’s all you world.” And “I’m like, wait, what?” “Yeah, like I’m doing so much retrieval practice” “with practice questions.” I’m like, “Wait, so what you really mean” “is you’re doing a ton of practice questions,” “which is in the other silo.” “Yeah, yeah, yeah.” “That’s all I’m doing.” “But you just said retrieval practice,” but see those are, those are different to me. So make sense?
– Absolutely, yeah, definitely.
– So that’s a distinction.
– Thank you for that distinction.
– Thank you, thank you. Well, I just wanted to thank you so much for letting me hijack your podcast today, it was a ton of fun to be able to talk to you. I learned an awful lot about Board prep and some strategies there, and I think our listeners will too. If you’re interested in some of the work that Ryan has done in this area, I really encourage you to check out his website at www.statmedlearning.com. I can shamelessly plug your business cause I’m hosting the podcast and you’re not, he’s got a lot of great content on YouTube and Instagram. So definitely check that if you’re interested, and if you want more detailed information, give Ryan a shout.
– Well, thank you.
– Thank you so much for joining us today.
– I appreciate it.
– On your own podcast.
– He’s right, I’m sure we’ll be back with more, thank you.
– All right, thanks, bye-bye.
– Thank you for tuning into the STATMed Podcast. If you liked the show, we hope you’ll subscribe. You can find more test-taking and studying strategies, specifically developed for med students and physicians over at our blog at statmedlearning.com. Thanks for listening.