Confidence, Metacognition, and Problem Solving

PUBLISHED: Dec 11, 2020

In This Episode.

Confidence when solving problems might sound like a good thing, but Dave and Steve explore the perils of confidence, especially when people lack the metacognitive awareness to understand the difference between rational confidence and irrational arrogance.  A quick reference to the original Wargames demonstrates how to recognize knowledge acquisition vs. knowledge application, and how to exercise them to make more successful decisions.

Episode Archive

Confidence, Metacognition, and Problem Solving

December 11, 2020

Podcast Transcript

Voiceover: They’re here to do two things. Chew bubble gum and make you smarter, and they’re all out of bubble gum smarter with Dave and Steve.

Steve Pearlman: Dave, you know this. You know that DEFCON or defense condition refers to the state of readiness in the United States relative to war.

Dave Carillo: I’ve heard that. Yeah. Mm hmm.

Steve Pearlman: Yeah. Could you guess which my least favorite defense condition or defcon would be?

Dave Carillo: Well, that’s a tough call. Remind me I will say this.

Steve Pearlman: It’s a five point scale. I was just

Dave Carillo: I just wanted to know like, how are we rating these defense conditions? So it’s a five point scale, and I’m guessing that five is the worst or is five the best,

Steve Pearlman: Five is the best,

Dave Carillo: Five is the best. So I think that your least favorite would be DEFCON three because you don’t know if you’re at peace and you don’t know if you’re at war. It’s just a terrible sort of situation to be in, you know, just all that ambiguity. That’s your, well, that’s your least favorite, right?

Steve Pearlman: No, my least favorite DEFCON is one, really, because one global thermonuclear war, that’s a thing on the whole, right? We want to avoid that’s even worse than worldwide pandemic. The reason we’re talking about defense condition or defcon is that today’s essential clip comes from the one and only war games. Nineteen eighty three.

Dave Carillo: Yeah, Steve. Wait. I’m going to stop you there for the purposes of making sure that our listeners are informed, I’m going to have to go ahead and say that they remade war games. What? Yeah. Wait, what? No, they did. 2008 they called it war games. Colon the dead code. And wait, why? Well, I guess because in 2008, I think that the Hollywood community felt that, you know, the Soviet Union was no longer a threat and our constant need for some sort of existential threat needed to be updated. This is from I am db actually war games dead code. The United States Department of Homeland Security is led to believe in American teen hacker playing a terrorist attack simulator game online is a real terrorist out to destroy the United States.

Steve Pearlman: Ok, that’s a stupid plot. Now let me talk about the real war game.

Dave Carillo: The entire United States.

Steve Pearlman: All it was. Let me tell you about the real war games. The nineteen eighty three war games. All right. Basically, we have an artificial intelligence computer that runs the entire defense apparatus for the United States military and is hacked by Matthew Broderick and through a series of events. What happens is that the computer thinks that it’s playing a game with Matthew Broderick, but is actually potentially launching missile strikes or starting a war against the USSR. The clip you’re going to hear is inside your ad, which is the North American Aerospace Defense Command.

Dave Carillo: For the most part, all you see in the background is a huge wall of screens that are monitors for their various computer programs, and people are scrambling around and they’re running to and fro and there’s like lines and rows of desks with computers. It actually looks like a much darker Apollo.

Steve Pearlman: 13 The control room in Houston. Right, right.

Dave Carillo: Exactly. Darker, bigger, if you can picture that. But at any rate, what we have here is a bunch of folks watching the main computer that called that the Whopper go through this war game that they don’t necessarily know is real, right?

Steve Pearlman: Characters in the film are seeing on the screen. These digitized not very advanced by today’s standards, but these digitized lines off of global maps of nukes coming from Russia. And we as the audience know that that it’s fake. But the people in the film don’t know that it’s fake. Make sure you pay attention to what’s being called out over the loudspeaker, and then you’re going to hear the general giving an order about the DEFCON and make sure you pay attention to what the DEFCON is. Spoiler alert it might have to do with my least favorite DEFCON. Ok, so here’s a clip from war games. Nineteen eighty three. We have much affection, we have

War Games: A Soviet launch detection is confirmed, a massive attack. It’s a warning. No malfunction. Confidence is high. I repeat confidence is high. Is this an exercise? This is an exercise. General DBS is tracking 300 inbound Soviet ICBM.

War Games: Tell me this is one of your simulations is not. All right. Flush the bombers. Get the subs in launch mode. We are at

Dave Carillo: Defcon one, the one who’s responsible for calling these things out on the PR and in what capacity are they able to riff off of what’s going on? You know, we heard and I repeat, confidence is high. Is there any sort of leeway to say something like confidence is high? But let’s try to think this through because we would be incinerating, you know, much of the globe if we go ahead with what we’re proposing here

Steve Pearlman: Or even something more mundane, like I repeat, confidence is high, confidence is high. Also, please make sure you validate your parking tickets prior to leaving the building.

Dave Carillo: You know, I would be really upset if I had to deal with validating my parking ticket if I worked at Nerad. But the reason why we’re talking about this clip today is specifically because of what that particular human being announces over the loudspeaker, which is that confidence is high. And we have a really interesting study that speaks specifically to this idea of confidence and how metacognitive confidence judgments are related to and potentially suggest a greater ability, a stronger ability to solve complex problems. And this clip fits nicely because what more complex a problem could there be than whether or not to launch all our missiles and drop all our bombs based on what we’re seeing on a computer screen when we’re not entirely sure that that’s actually happening? Confidence is high, but it’s probably the confidence that the computer is projecting that the human is actually communicating.

Steve Pearlman: Yeah, and getting back to the article, which is metacognitive confidence judgments, and they’re linked to complex problem solving by Rudolph, nipple grief, goldhammer and kronor. What makes this article particularly interesting among the swath of research that’s been done on complex problem solving is that it teases out confidence as a particular variable and controls for reasoning in the process. As the authors note, complex problem solving typically has been broken down into two phases. There’s the knowledge acquisition phase and then the knowledge application phase. So we get the knowledge and then we act on the knowledge. That’s not very complicated, and what they have found is that capacity to reason is one of the main factors that influences our capacity for complex problem solving. But another one is not just the ability to think through a problem, but also the role of metacognition, which is more to be aware of the process we’re using and the thinking process we’re using to reason through the problem. And those are different, right? There’s the ability to think through the problem, and then there’s the ability to know what your thinking process is in thinking through the problem. Those are two separate things. Both have been shown to be important to our capacity for complex problem solving, right?

Dave Carillo: In fact, there’s a lot of research on that, and that’s something that the authors of this article are very careful to include in their introduction, they say quote among the core aspects of self-regulation is metacognitive self monitoring that is the observing and judging of one’s own performance. So they’re using this research that’s already been shown to validate this idea of metacognition in problem solving and critical thinking. And now they’re looking very specifically at this idea of confidence within those abilities.

Steve Pearlman: Yeah, they have two very simple hypotheses, and the first is that confidence is linked to complex problem solving ability. And that second is that even when controlling for reasoning, confidence is still a factor in complex problem solving ability. Now, before we get any further into this muck of this article, which we’re going to keep really simple. One key preface here is that their use of the word confidence isn’t exactly the same as the popular use of the word confidence. When we popularly hear confidence, we think of somebody who’s very self-assured in their decision who might be cocksure in that decision, who might be arrogant about that decision. That person is super confident, pounding the table about the capacity to have the right answer to the problem. And that’s a little bit different than what we’re talking about here. Confidence in this case doesn’t just refer to how ultimately sure we are about the decision we made. It’s about having confidence in the self enough to reflect on and adjust the problem-solving process so that it improves as we go. It’s another it’s as a confidence in ourselves not to have to go to the first answer, not to just think that we’re right, but confidence to be able to even question the process that we’re using. So it’s a humbler kind of confidence, but a more important kind of confidence. So if you see that person at your office or that person, you know who is always arrogant about how right they are, that is not the kind of confidence we’re talking about here now.

Dave Carillo: That’s the kind of confidence that. Leads us to nuclear war, that person in the office who thinks that they’re absolutely right when they launch all our missiles,

Steve Pearlman: And that brings us right back to the clip with the two phases that we talked about, we have the knowledge acquisition phase and we have the knowledge application phase and the knowledge acquisition phase. We hear the people having looked at the computer data saying confidence is high and then we have the general going quick to a decision and saying, Well, then take us to DEFCON one. That’s the knowledge application phase post. Those people are acting with great certainty, but again, that’s different than the kind of confidence we’re talking about here. So let’s look at what the article does, right?

Dave Carillo: So here’s what they did. They had two phases of this study knowledge acquisition and knowledge application. They took a bunch of students and gave them a set of complex tasks regarding the best way to make a handball team into a more effective handball team.

Steve Pearlman: And this is through some complex computer model that’s not entirely realistic, but nevertheless has a number of variables that the students can move around.

Dave Carillo: Right? In this study, students needed to grapple with a fair amount of complex variables, and then they had to create their own model of what variables would be most valuable in relation to making this handball team a better handball team. In the knowledge acquisition phase, they were given as much time as they wanted to create this model. And then before they knew whether their model was right, they are asked to discuss the extent to which they thought that that model was correct, and that’s how they judged confidence in that knowledge acquisition phase. Here’s what you need to produce now before we tell you whether you’re right or how close you were to write or whether you’re totally ridiculous, tell us to what extent are you confident that you were able to produce a model that would work in the knowledge application phase? They were giving a series of tasks, but each task built on the last one and they weren’t asked at the end whether they were confident they were actually asked after they completed each task. To what extent do you think you’re on the right track? How confident are you that you’ve done this task well enough that the next task is going to play out that again create an authentic discussion regarding the level of confidence in solving these problems

Steve Pearlman: And what the study resulted in is informed by our clip from war games. So if you listen to the clip, you hear confidence is high, confidence is high and then you hear the general without really a moment’s pause. And we understand why in that situation, sometimes there can’t be a lot of pause. If you have to revert the nukes that have already been launched against the country, there’s not a lot of time to pause and reflect, but nevertheless it illustrates our point that we don’t see any moment of that from the general. He has this knowledge acquisition moment. Confidence is high, and then he goes right into knowledge application, really without reflection. What they found, however, is that when it comes to complex problem solving, the opposite strategy is needed. We don’t need people who react to the data or just follow a protocol with respect to data. We don’t need people who just react to the report on that data. We need people who are able to do something better, and that really affects the caliber of the decision making that occurs again from a fully realistic military standpoint. This isn’t always possible in the field, but it’s illustrating our point here. And what they found was that quote the decision about whether adaptation is necessary relies on students confidence. So they found that confidence was a factor in the decision making process. And the reason it was a factor is because when students were confident in their abilities and their ability to monitor their ability to regulate, then they were able to adapt their process as they reflected in moving forward.

Steve Pearlman: The students who are less confident, more insecure about it were the ones least likely or least able to make adaptations to their process. And that’s why when we have that arrogant person in our sphere who’s pounding the table for a particular answer, it immediately thinks their worries right. They’re often the person that isn’t. They might seem secure on the outside, but they’re insecure on the inside and really lack the capacity to question themselves and their process and their conclusions. Now what’s more interesting here, in fact, is that they not only found that confidence was a factor, but remember, we said earlier that they wanted to control for reasoning ability and see if beyond reasoning, ability confidence was still a statistically significant factor. And they did quote the results indicated that confidence in complex problem solving was still strongly linked to complex problem solving performance, even after reasoning had been controlled for. And that’s absolutely critical, and it’s really the premise of the podcast. Good thinking doesn’t just have to do with how much raw intelligence somebody has to have some very smart people who lack reasoning capacities in certain arenas or certain ways or could improve upon their reasoning capacity. They have the right tools, and that’s what this podcast is all about. And what we’re seeing here is that having the confidence to reflect on, examine and adapt our process is an important factor beyond just being able to think well and an important factor in complex problem solving overall, right?

Dave Carillo: And not to give too much of the movie away, but the movie builds to a crescendo where they literally have to make the computer pause and reflect on what it’s doing.

Steve Pearlman: The irony right? The computer that ultimately pauses and reflects.

Dave Carillo: I don’t think there was going to be any other way to stop nuclear war in this particular situation, other than to help the computer build the confidence enough to stop playing this game. But I digress.

Steve Pearlman: Dave, I don’t think you’d digress at all. I think that’s an essential point, right? Is that the actual way that they are able to address this issue more broadly when they have this artificial intelligence is teaching the artificial intelligence to have the confidence to reflect on its process and adapt its process and learn instead of just reasoning through it in a more rote fashion.

Dave Carillo: I think your point is a fair one, Steve. I just I know that if I get going on war games, I’ll turn this very brief discussion into, you know, a three hour thesis on how rockin the movie is. And I just don’t think anybody’s really prepared for that or wants that.

Steve Pearlman: Well, nobody wants to hear two hours and forty five minutes about your crush on Ally Sheedy.

Dave Carillo: I agree. And so therefore, I’m going to move us along because Steve, when we saw this article, we immediately knew that. I mean, right off the top of our heads, we saw at least two key takeaways that we wanted to share with our listeners. So let’s get to that.

Steve Pearlman: It’s the first key takeaway is immediately applicable to your life, which is to have the confidence in yourself as you approach decisions that need to be made to pause and reflect on the process that you’re using and consider what other possibilities are and adapt that process as needed. Don’t just do things the way you’ve always done them. Don’t jump to conclusions because instead you want to be able to reflect on that and don’t feel as though that that’s demonstrating a lack of good reasoning if you can’t immediately reason through the problem. In fact, better reasoning a substantial amount of the time is to be able to pause and think about it. So think about what evidence do you have and your knowledge acquisition phase? What are the sources of the evidence? Who’s giving that evidence to you? What are their biases? What are your biases? Are they a stakeholder in certain ways? Do they have a stake in certain outcomes? How are you interpreting things and so on and so forth? And as you move into knowledge acquisition? Think about different possible outcomes. And as things start to move forward, don’t be afraid to adapt that process when there are opportunities to change course. So all of this saying again, don’t have the confidence of cocky shrewdness, but rather have the confidence that if you engage the process reflectively and are willing to adapt that you’ll get to a better outcome.

Dave Carillo: And the second takeaway actually extends the ideas that Steve shared with you in group situations when you’re leading your team at work or at school or somewhere else through a complex problem or situation. Use the concept of confidence as a key moment to engage the people in your group in one last evaluative moment where you go back and think through what led you to this moment of confidence. Use this moment of confidence as a more generative moment.

Steve Pearlman: And if you’re a team leader, this is a good way to teach and probe your team members and ask them, How did you come to this decision? What processes you go through? How confident are you in it? Where do you see room to change it? Use this as a way to recognize those two faces and to push your team gently into being more confident that you value their willingness to reflect on and adapt the process as things move forward.

Dave Carillo: So, look, you might never find yourself in a situation where you’re the one having to go from DEFCON two to DEFCON one based on what a computer who thinks they’re just playing a game, but is projecting the actual oncoming events of World War Three onto the computer screen. But oftentimes the problems that we encounter in the actual real world, not the war games world, are more complex than we would like to think. And rather than being confident such that we just charge into the first solution we think of or the first solution that makes us feel good, let’s start to be a little bit more confident in our ability to rethink and to adapt. And as such, take that time to reflect on what’s brought us to the moment of confidence. And can we come to a better solution?

Steve Pearlman: And whatever you do, please always exercise considerable care when a computer asks you,

Dave Carillo: Shall we play a game?


Submit a Comment