On the “Ask STATMed” series, we dig into your top questions about studying and test-taking in med school
For the Ask STATMed podcast miniseries, we dig into the most common questions we get asked about studying and board-style test-taking in medical school. In Part 2, we look a little deeper into the nature versus nurture discussion and determine how to tell if a missed question on a test or medical board exam is from a test-taking miss or a knowledge miss. In this episode, we explore specific test-taking issues like working memory limitations and the binary test-taking mentality.
“My view is test-taking methods,won’t cover or bridge these knowledge gaps. Test-taking does not hack the test. Test-taking is about cleaning up the user interface so you can clearly show what you know, but you have to know enough. And that’s addressed, through augmented streamlined study methods.” – Ryan Orwig
Subscribe to the STATMed Podcast!
If you’re struggling with test-taking misses on the boards, check out our STATMed Boards Test-Taking Workshop to learn more.
Transcript
Narrator: Welcome to The STATMed Podcast, where we teach you how to study in med school and how to pass board-style exams. Your host is Ryan Orwig, a learning specialist with more than a decade of experience, working with med students and physicians.
In this episode, Ryan focuses on answering top questions about studying and test taking in med school and on medical boards. Today, he’ll dig into how to determine if you’re missed question on tests or medical boards, come from bad test taking strategies or knowledge misses.
Ryan Orwig: Bad test takers, reduce the question or lose sight of it, or have it mutate into something else. Over the course of the 30 to 75 seconds of initially reading it and ultimately picking the answer. Your reading brain will not allow, the question parameter in your memory to remain blank. It will auto fill or auto correct, the question parameter.
Hey everyone, Ryan Orwig back here again, with our ongoing ask STATMed a questions, solo series. While I’m going to continue to talk about last episodes question regarding whether good boards test takers are made or born.
Today, I’m going to start by answering this related question, which is this. When I miss a question, how can I tell if it’s a knowledge miss or a test taking miss? Answering this is a hugely important step, since we need to know if test performance issues are based on bad test taking, meaning, not showing what you know on boards or if the issues are more foundational in the way the learner has studied, organized and coded retrieved and applied the information. Meaning, more of a study side pre-test based issue. Okay?
So one of the first things we needed to determine. When someone is struggling to perform well on medical board-style exams, like the USMLE, COMLEX Shelves, NAVLE Specialty Boards et cetera. Is this, are scores low due to, number one knowledge issues, or are they low due to number two, test-taking issues? So is it a knowledge or test-taking issue?
We determine this by looking for what I call the miss ratio. This is a hypothetical crude measuring stick, that gives us a starting point assessment-wise. So when I talk to a prospective student or doctor, I ask them, “Okay, imagine you’ve just done a bunch of practice questions, maybe 20, 30, whatever, and let’s imagine you missed 10. And let’s imagine each missed question is represented by a coin and we have the stack of 10 coins on the table in front of us.”
We want to sort this pile of 10 missed coins, into two stacks. A knowledge missed stack over here on the left and a test taking missed stack over here on the right. So it’s a knowledge miss if you just flat out, didn’t know enough to get it right, that’s a fair miss.
A knowledge miss include having never seen it before, maybe you learning it wrong, or not being able to recall the details, even though you did study them at some point. It could also include being generally familiar, but your knowledge doesn’t go deep enough. Meaning your knowledge goes three levels deep, but the question requires knowing the fourth or fifth level.
Keep in mind, it’s a knowledge miss. Even if you say, “I studied this and I should know it, but I don’t.” That’s still just a flat out knowledge miss. Everyone will always have knowledge misses. It’s just a matter of how many and the split between knowledge and test taking misses.
On the other side, it’s a test taking miss. If you realize, after reading the answer and explanation that, hey, I should have gotten that right. This includes just flat out realizing, you knew it and were just too hasty or impulsive or reckless in the way you read and selected your answer.
This includes, simple reading and thinking mistakes among other things. It might mean you narrow down to two and picked the wrong one. And you realize at this phase, you actually knew enough to pick correctly. There are some very specific patterns we can expect to see underneath this kind of behavior pattern.
Maybe you latched onto a single clue and ignored other clues, to validate the wrong answer option. Perhaps you picked the answer option you were familiar with, but didn’t actually connect it to the clinical scenario, and or the specific question being asked.
This is a weird common phenomenon where the bad test taker, turns these options into a referendum, on which option they knew more about. Essentially using their knowledge against themselves within the context of the question. This is a particularly, ugly way to miss questions.
Other patterns go like this. Maybe you force a prediction or you pound a square peg into a round hole, or you otherwise twist clues to fit a desired diagnosis or option. Regardless, it’s a test taking miss if you realize, after the fact that you could have gotten it right, using the parts of what you knew, at the time, along with proper test taking methodology.
So then I ask, “Okay, what’s your general miss ratio?” We’re thinking about that hypothetical, representative, 10 missed questions. How would they sort, into those two stacks for you. Is your miss ratio, eight knowledge and two test taking or nine knowledge and one test taking? So eight to two, nine to one. Okay. Then that sounds like a knowledge base issue.
So we need to address the way you’re studying and organizing and encoding and retrieving and applying. We have to fix this, by fixing the way you study. While test taking could be an issue, under the surface we can’t tell yet. And you can’t paper over knowledge issues.
My view is test taking methods, won’t cover or bridge these knowledge gaps. Test taking does not hack the test. Test taking is about cleaning up the user interface so you can clearly show what you know, but you have to know enough. And that’s addressed, through augmented streamlined study methods.
But what if the ratio is more like five to five. Saying 50% of your misses are test taking, or more like a four knowledge, to six test taking. A four to six split, or even more extreme, like three knowledge to seven test taking. I’ve bet those people too, they exist, this is real. But it depends on the individual. Even a six knowledge, to four test taking split. A four to six ratio can be the difference between passing and failing depending on the person.
However, you shake it. These are all test taking based issues. Could they benefit from study side help? Maybe, it depends on their needs and their scores, but most of these people need to address test taking first and foremost, this is urgent, important and necessary.
The test taking, has to be fixed when it’s this impaired from my perspective. So I think a related question would be something like, how have they compensated thus far, if these issues exist? For me, I’d say, it has to be overstudy and over knowledge, just sheer muscle brute force. Which means they’re always at risk, and that this compensatory process might run out of viability at some point.
But with these extreme miss ratios showing around half their misses or test taking misses, we have to address test taking issues first. So that’s how we start with sorting these issues. Looking between knowledge based test performance issues and test taking issues on boards.
So then someone might say, “Okay, but how can you identify bad test taking? Other than saying, I just read too fast or I missed a keyword, like not or except.” This ties back into what I was talking about last episode, regarding if good test takers are made or born.
I talked about two specific bad test taking issues then, working memory limitations and the binary test taking mentality. Let’s talk about some more now, and we can see how these might be obvious or maybe hidden. Test taking issues, that can inform us if a miss, is knowledge based or test taking in nature.
Along the way, we can also consider if these are nature or nurture issues. So another pattern has to do with the way we read and think in a more diversified, sensitive manner. I describe this as we’re reading and thinking, intersect. So in this pattern, we see reading issues manifest and cause problems.
As I’m always saying, reading is much more nuanced and sensitive than many people think. And if you don’t read as well as the average person taking the same board exam. So if you don’t read as well as the average med student or veterinarian or physician, you’re exposing yourself to risk. Exposing yourself to maybe missing boards questions that you shouldn’t, otherwise miss, due to issues where reading and thinking intersect.
Now, I think when most people hear someone say they made a reading mistake on a question, they usually think the test taker missed a negative word, like not or except. I mean, if that happens, that’ll get you. But reading mistakes are often way more subtle than this.
Let me lay out some ways. Reading issues, actually manifest. One of the simplest reading issues we see is when a test taker misses a question by selecting an answer that does not, precisely answered the exact question being asked. This is that all important last sentence.
It has become commonplace, to recommend that test takers read the last sentence first. This is great, that this has permeated the board’s test taking culture over the last 10 plus years. But it can also foster a false sense of security.
At the end of the day, the good test taker reads and maintains and anchors to the specific question being asked. Bad test takers, reduce the question or lose sight of it, or have it mutate into something else, over the course of the 30 to 75 seconds of initially reading it and ultimately picking the answer.
Your reading brain will not allow, the question parameter in your memory to remain blank. It will auto fill or auto correct the question parameter. Good test takers, generate, moderate, self check and self correct this aspect of reading and test taking.
Again, this is where reading and thinking are sort of intersecting. Some bad test takers, simply do not. And this causes all kinds of problems. Are they born this way or made this way? I think, they can be born in a way that this just won’t be a problem or it will be a problem, or they can evolve through their reading and learning development in a good way or a bad way.
At this point, it’s more about identifying if they are good or bad. And if they’re bad, installing methods to overwrite the bad behavior patterns with good behaviors.
So here’s an example of how this might work. Let’s go back to our pediatrician, Joe. Joe Knows his stuff, but he is a terrible test taker. Keep in mind when I meet a bad test taker, they’re not usually doing all the patterns we’ve identified, we’ve named and identified and modeled a little over a dozen patterns.
A bad test taker like Joe, is usually doing three to five of the same missed pattern types again and again and again. He can’t see them or name them, so he can’t control and extinguish them. He just blames himself and endeavors to study more and do more questions.
So anyway, let’s say he’s trying to answer a question, about how you would confirm compliance with dietary therapy for a kid with celiac disease. Joe ends up choosing an option, that has more to do with celiac in general, but not specifically about dietary compliance. So he narrows down to two. Option A, which is normal stools would be the thing you would find. And option B would be normal tTg-IgA levels.
With option A. He already starts to move away from the specific question, that’s being asked by saying to himself, this patient probably presented with diarrhea and constipation. And if we are treating celiac, then her stool should start to normalize. But remember, the question is actually asking this, how would you confirm compliance with dietary therapy for a kid with celiac disease?
So he’s up valuing the viability of this option by adding to the scenario by adding probably and what if. But is unhooking from the specifics of the question. Then with option B, normal tTg-IgA levels he’s pushing away from this, the right answer by saying, this is how you diagnose celiac. If it is high, you have celiac it should be normal, but can be weekly positive in a normal person, but that’s rare.
But then he pushes further away from option B, the right answer. Thinking about outliers and exceptions. I think this, in part floods his working memory and blurs out the actual question being asked, allowing him to start maybe-ing option A and wiggles into option A, by losing sight of the question being asked.
Upon missing this question, because he chose option A, normal stool. We can see he was not answering the specific question being asked when he clicked on, a normal stools.
This highlights another common issue where reading and thinking intersect. He set to a rule options in mentality, which is super dangerous and super risky. Most, if not all wrong answer options are designed to be partially true. So if you are only looking to rule in partial true answer options, you’re risking creating a trap of your own design to get caught in.
Instead, we want to foster and grow a rule options out mentality. That’s sort of central to what we had to do with Joe, and help Joe become a good test taker instead of a bad test taker. You can call this rule in, versus rule out.
Good readers and test takers, are always aggressively trying to rule options out based on any aspect that is partially false. A little false is all false. The easiest way to do this, is to anchor to the question being asked. So we model out, how he should have weighed them and worked them, these answer options.
Anchoring to the question, being asked and leaning into partial false. He realizes he should have weighed and judged the options like this. Option A normal stools, would this confirm compliance with dietary therapy? He says, “Well, I want this to say closer to normal, as opposed to confirm, makes me want to know about her history, but I have to just keep it simple and say, this is a strong maybe, and realize I don’t have to know for a hundred percent.”
It helps, when I realize there is nothing false about it though. So there’s nothing to rule it out. So he has to keep that in play. And then when he goes to option B, normal tTg-IgA levels and says, “Would this confirm compliance with dietary therapy? I mean, no. No, it wouldn’t. You can’t confirm compliance based on this. So this is straight up, out.”
This illustrates, how he missed the question and how he should have worked, the question. This will be the framework we use to self reflect on his issues to grow the need of self-monitoring he needs, to eventually be able to play the game of doing questions our way, the right way, for… And for him that means learning to look, through the filter of partial false, while always anchoring to and refreshing on the specific question being asked. So that sort of illustrates two ways, how reading and thinking issues and tangling create, test taking issues.
Others include, rounding down key clues, which I call genericafication, meaning your brain takes a specific clue and makes it generic, wildcarding it to make it fit anything. This is another dangerous way to let you use your medical knowledge against yourself, within a given question.
For example, let’s say you see a PEDS question about a kid with a purple rash on the kid’s backside, as one of several clues and a most likely diagnosis, question. Then you see an answer option like toxic shock syndrome, and you rule it in, because you genericafy, this very specific rash in the passage and call it generically, a rash. And then you genericafy toxic shocks rash, which is not a purple rash on the buttocks.
And say, this kid has a rash, and toxic shock has a rash, and you match them up, because you have made both rashes generic. So you say, “Hey, that’s a possibility.” This is how a bad test taker, genericafies clues, and uses their knowledge against themselves.
The good test taker on the other hand, keeps the clues specific and uses them to rule out. They can easily rule toxic shock out, based simply on the fact that the rash does not look like that. Doesn’t matter if every other clue fits toxic shock. If it’s partially false, it’s all false and can’t be the right answer. So that’s just one more quick example, of how reading and thinking can intersect to cause test taking issues.
Then the last pattern I’ll talk about here is this. Sometimes bad test takers are just using the wrong strategy based on their needs. This means that the way, they’re working through these questions is not the right fit for them, based on their cognitive wiring or maybe the way, they read and process questions.
Too often, it seems like the powers that be in medical education and boards preparation, treat all students the same. Seeming to rely on a one size fits all approach to medical test taking, which is a misfire in my book.
Maybe this comes from a blindly elitist belief that all med students are created equally. I don’t know, regardless it’s a narrow minded approach at best. And then it’s usually the best test takers trying to help the worst test takers.
The problem is, the best test taker is often giving advice while looking through her clear glass windowpane, telling the bad test taker to look through the bad test takers, muddy warped window. It has to be so frustrating. This is often some version of someone who is born a good test taker, with the robust working memory and innately tune reading and thinking skills, trying to force the bad test taker, who doesn’t have these invisible assets and abilities, to act as if they do.
It’s like the well trained elite athlete, trying to tell the non-athletic person to, just do it like me. This is not helpful, to put it nicely, but that’s what the bad test takers deal with all the time. Related, another wrong strategy for the wrong person, is the old idea that we have to predict the answer.
This can, of course be fine. If it works for someone that’s cool. After all, making predictions is a key part of good and effective reading and thinking. But it can also be wildly damaging to others, when we talk about medical boards test taking.
So some reasons predicting is good, can go like this. It helps access background knowledge. It fires up the retrieval pathways. It’s a natural part of good reading and thinking, and it lets you think about the answer, without being biased by all the other answer options.
This is a classic test taking advice. And if it works for someone, that’s great, that makes sense. Don’t change it. But here’s the thing, it doesn’t work for everyone at the medical board’s level. And for some test takers, this is actually a very toxic and damaging strategy.
Why could predicting be bad? Simply put, these are not first order questions, when we’re talking about medical board exam questions. So a lot of the time, meaning, I don’t know, 50% or more. Many boards test takers will not be able to predict an answer. And if that makes them feel like it’s game over, the sky is falling, that they’re doomed to miss the question simply because they can’t predict the answer. Then that’s a bad strategy for them. Yes. This works in other testing arenas and for other test constructs and yes, it works well for some board-style test takers, but for others, it is actually quite harmful.
Another way prediction can hurt a bad test taker is this. It can create a situation where they predict the answer, then twist the clues to validate that prediction, or you see that option down below and you pounds on it. Never even fairly considering other potentially viable options.
If you’re a good test taker, all this stuff might just seem really innocuous or easy to fix. But for the bad test taker, this can be very damaging if it’s often intertwined with their whole process.
Not how I’m not saying, it’s the same for everyone. None of these issues are a one size fits all situation. In fact, I’m saying it depends on the individual’s needs. We need to treat test takers differently. Shocking, I know. And then this… Other last thought, I guess. Some bad test takers have always been bad test takers from the SAT and ACT to the MCAT, through the classroom and through their boards. All the way through. That’s probably pretty obvious.
But bad test taking can just happen too. It can just emerge or manifest, where before there was never a problem and then all of a sudden at your emergency medicine in services, it manifest or wherever it might be step three, or what have you.
Sometimes people had no problem or were even great with stuff like the SAT, ACT or the MCAT or in the classroom med school exam. It might be, that they did well on certain step exams and then trouble starts later. Can this happen? Yes, indeed. Why? Likely, because first of all, the design of board-style questions is unique, in places unique demands on the test taker, as the test cover more scope and depth.
Perhaps that’s the tipping point, or perhaps once you’re in practice, you get away from the form of studying that you used to compensate, previously on your steps. There are loads of reasons this can happen. And I see it all the time.
So just know that yes, previously good test takers can manifest as bad test takers at any point in their careers. So if you struggle or hit a wall, board’s wise and someone tells you, “You can’t be a bad test taker, look at how far you made it.” I would respectfully tell you, they don’t know what they’re talking about. I’m sure they mean well, but that doesn’t make them qualified to objectively assess your situation.
So is this born or made? Again, this is getting into the way the med student or physician reads at a pretty deep and nuanced level, getting into the transactional manner of the reading process. And with this, I don’t know, and for our purposes, I would say the cause is not as important at this point as the manifestation.
I will say, I don’t think good test takers were ever necessarily sat down and told, “What you need to do is avoid twisting and watch out for rounding down key clues and rule options out instead of ruling them in.” Et cetera, et cetera.
I think most med students and physicians, simply read well enough in a sensitive and accurate enough manner, that they interface with questions in a way that deploys the right skills at the right time, in the right way to avoid these issues. Usually in an invisible subconscious automatic way, like an elite athlete, running a random obstacle course.
But for our bad test takers, this manifestation needs to be addressed in a way that is hyperspecific to these test taking constructs, meaning they should train externally and explicitly with the strategies, that will offset test taking issues to level the playing field for them. So they can reduce test taking misses and more accurately show what they know on exams.
I think that more than wraps up this conversation. Be sure to check back with us for more episodes, talking about studying, time management and board-style test taking for med students, physicians, and those in related fields. Thanks for listening.
Narrator: Thanks for tuning in to The STATMed Podcast. If you like this show, we hope you’ll subscribe. You can find more test taking and studying strategies, specifically developed for med students and physicians over at our blog @statmedlearning.com. Thanks for listening.