Will Chuck Hagel Be the Next SecDef? A Case Study in How (Not) to Forecast

Yesterday, Foreign Policy defense blogger Tom Ricks tabled a pessimistic forecast on the fate of Sen. Chuck Hagel’s nomination to succeed Leon Panetta as U.S. Secretary of Defense:

Will Hagel withdraw? I’d say 50-50…But declining by the day. Bottom line: Every business day that the Senate Armed Services Committee doesn’t vote to send the nomination to the full Senate, I think the likelihood of Hagel becoming defense secretary declines by about 2 percent.

Ricks’ gloomy outlook was quickly derided by a few of the national-security pros I follow on Twitter. Political scientist Dan Drezner wrote, “I’ll bet anyone who thinks he’ll withdraw,” prompting Slate reporter Dave Weigel to quip, “Dan, Dan, people withdraw when they have the votes necessary to confirm ALL THE TIME.”

I don’t normally blog about (or even pay much attention to, really) the inside baseball of U.S. politics, but I suspect that Drezner and co. are right on this one, and I thought an explication of the reasons why would make a nice case study in how (not) to forecast.

So: let’s put ourselves in Ricks’ shoes and decide we want to assess the odds that Hagel is going to become the next Secretary of Defense. Where do we start? Experienced forecasters often start with the base rate—that is, the observed frequency of the event of interest over many previous trials. If, for example, we want to estimate how likely it is that Lebron James will sink his next free throw, the single most-useful piece of information is usually going to be his career or season free-throw percentage (as it happens, both about 75%).

Okay, so what do we know about the outcomes of the past nominations for Secretary of Defense? Since the establishment of the position soon after World War II, it looks like only one of 24 official nominees has been rejected by the Senate (John Tower), and none has withdrawn. (See here for the list of approved candidates and here for the rejections and withdrawals.) If we think there’s something unique about that position, we have a base rate of about 96 percent. If we think there isn’t anything particularly unique about the approval process for Secretary of Defense, we might decide to draw on the record of nominations to all Cabinet posts. At that broader level, the approval rate is about the same, perhaps even a bit higher. As Matt Dabrowski noted when I asked about this on Twitter, “Officially, only 10 actual nominees for any Cabinet office have failed since Reconstruction,” and that’s out of several hundred.

Given a base rate in the high 90-percent range, the safe money for any Cabinet nomination that reaches the Senate-hearings stage is going to be on approval. Surely Ricks has some sense of this history, so why is he predicting that Hegel’s nomination will probably fail? Apparently, it was Hagel’s performance at his Senate hearing that swayed Ricks.

That hearing last week didn’t reflect well on the U.S. Senate. But [Hagel] didn’t do well in it, either. He didn’t appear that interested in the job.

Now, let’s give Ricks the benefit of the doubt here and allow a) that Hagel didn’t do so hot and b) that a weak performance before the Armed Services Committee would damage his candidacy. Now the question becomes, “How much damage?”

One way to answer this is with Bayes’ theorem. Taking the base approval rate as our prior and the hearing performance as an impetus for updating, Bayes’ theorem requires us to estimate two things: 1) how likely are we to see a poor Senate performance when the nominee is destined to fail and 2) how likely are we to see a poor performance when the nominee is bound for approval?

I don’t have data on the relative frequency of these pathways, but let’s give Ricks the benefit of the doubt and assume the nominee’s performance in his or her Senate hearing is very revealing. For the sake of argument, I’ll assume that only one of every five nominees bound for success does poorly in confirmation hearings, but 19 of 20 bound for failure do. (If you’re wondering why not 20 of 20, it’s easy to imagine some cases arising where some scandal emerges after the confirmation hearings that derails the nominee.) I think that 1-in-5 figure for successful nominees is probably too low, but again, I’m trying to tip the scales in Ricks’ favor here for illustrative purposes.

Okay, so what happens to our predicted probability of approval when we plug those values into Bayes’ theorem? It tumbles from 96 percent to a lowly…83 percent. Even if we accept Ricks’ judgment that Hagel flubbed his hearing and assume that a flubbed hearing tells us a lot about a nominee’s prospects, Bayes’ theorem tells us that the odds still overwhelmingly favor Hagel’s accession to the post. In what I consider to be a more realistic world where “flubbing” is in the eye of the beholder and confirmation hearings are essentially pro forma, Hagel’s mediocre showing would have little impact on a careful estimate of his odds, which would still hover around 90 percent. Either way, Ricks’ “50 – 2(business days)” estimate is looking pretty wild.

So where is Ricks coming from? I have never met the man and haven’t spoken to him about this blog post, so I can only guess, but a comment he made about John Tower suggests a common culprit: availability bias. As wiseGEEK explains, “Availability bias is a human cognitive bias that causes us to overestimate probabilities of events associated with memorable or vivid occurrences.” In explaining why he thinks Hagel will probably withdraw, Ricks writes:

SecDef nominees have blown up on the launch pad before: Remember John Tower (picked by the first President Bush) and Bobby Inman (picked by President Clinton to replace Les Aspin)?

It sure looks like a couple of recent and salient failures have stuck in Ricks’ mind, tempting him to ignore the overwhelming number of successful nominations. (Note: Inman didn’t factor into the base rate calculations above because he withdrew before Senate confirmation hearings were held, as have a number of other Cabinet nominees over the years, and I’m assuming that nominations which progress to that stage are different from ones that don’t.)

Of course, the base rate is often but not always the most useful piece of information. In situations where we have specific and compelling evidence about the case in question—say, word from a trusted source that Hagel actually plans to withdraw—we’ll want to discount the base rate and weight that specific evidence more heavily. The only specifics Ricks offers on Hagel, however, are his personal impressions of Hagel’s performance in front of the Armed Services Committee, which we’ve already addressed above, and a vague sense that “no one much wants him running the Pentagon.” With “specific” evidence as squishy as this, we’re probably better off sticking with the base rate.

UPDATE: On Tuesday, February 26, the Senate voted 58-41 to confirm Hagel’s appointment, and he was sworn into office the next morning.

Posted by Jay Ulfelder on February 9, 2013

https://dartthrowingchimp.wordpress.com/2013/02/09/will-chuck-hagel-win-senate-approval-a-case-study-in-how-not-to-forecast/