Will Chuck Hagel Be the Next SecDef? A Case Study in How (Not) to Forecast

Yesterday, Foreign Policy defense blogger Tom Ricks tabled a pessimistic forecast on the fate of Sen. Chuck Hagel’s nomination to succeed Leon Panetta as U.S. Secretary of Defense:

Will Hagel withdraw? I’d say 50-50…But declining by the day. Bottom line: Every business day that the Senate Armed Services Committee doesn’t vote to send the nomination to the full Senate, I think the likelihood of Hagel becoming defense secretary declines by about 2 percent.

Ricks’ gloomy outlook was quickly derided by a few of the national-security pros I follow on Twitter. Political scientist Dan Drezner wrote, “I’ll bet anyone who thinks he’ll withdraw,” prompting Slate reporter Dave Weigel to quip, “Dan, Dan, people withdraw when they have the votes necessary to confirm ALL THE TIME.”

I don’t normally blog about (or even pay much attention to, really) the inside baseball of U.S. politics, but I suspect that Drezner and co. are right on this one, and I thought an explication of the reasons why would make a nice case study in how (not) to forecast.

So: let’s put ourselves in Ricks’ shoes and decide we want to assess the odds that Hagel is going to become the next Secretary of Defense. Where do we start? Experienced forecasters often start with the base rate—that is, the observed frequency of the event of interest over many previous trials. If, for example, we want to estimate how likely it is that Lebron James will sink his next free throw, the single most-useful piece of information is usually going to be his career or season free-throw percentage (as it happens, both about 75%).

Okay, so what do we know about the outcomes of the past nominations for Secretary of Defense? Since the establishment of the position soon after World War II, it looks like only one of 24 official nominees has been rejected by the Senate (John Tower), and none has withdrawn. (See here for the list of approved candidates and here for the rejections and withdrawals.) If we think there’s something unique about that position, we have a base rate of about 96 percent. If we think there isn’t anything particularly unique about the approval process for Secretary of Defense, we might decide to draw on the record of nominations to all Cabinet posts. At that broader level, the approval rate is about the same, perhaps even a bit higher. As Matt Dabrowski noted when I asked about this on Twitter, “Officially, only 10 actual nominees for any Cabinet office have failed since Reconstruction,” and that’s out of several hundred.

Given a base rate in the high 90-percent range, the safe money for any Cabinet nomination that reaches the Senate-hearings stage is going to be on approval. Surely Ricks has some sense of this history, so why is he predicting that Hegel’s nomination will probably fail? Apparently, it was Hagel’s performance at his Senate hearing that swayed Ricks.

That hearing last week didn’t reflect well on the U.S. Senate. But [Hagel] didn’t do well in it, either. He didn’t appear that interested in the job.

Now, let’s give Ricks the benefit of the doubt here and allow a) that Hagel didn’t do so hot and b) that a weak performance before the Armed Services Committee would damage his candidacy. Now the question becomes, “How much damage?”

One way to answer this is with Bayes’ theorem. Taking the base approval rate as our prior and the hearing performance as an impetus for updating, Bayes’ theorem requires us to estimate two things: 1) how likely are we to see a poor Senate performance when the nominee is destined to fail and 2) how likely are we to see a poor performance when the nominee is bound for approval?

I don’t have data on the relative frequency of these pathways, but let’s give Ricks the benefit of the doubt and assume the nominee’s performance in his or her Senate hearing is very revealing. For the sake of argument, I’ll assume that only one of every five nominees bound for success does poorly in confirmation hearings, but 19 of 20 bound for failure do. (If you’re wondering why not 20 of 20, it’s easy to imagine some cases arising where some scandal emerges after the confirmation hearings that derails the nominee.) I think that 1-in-5 figure for successful nominees is probably too low, but again, I’m trying to tip the scales in Ricks’ favor here for illustrative purposes.

Okay, so what happens to our predicted probability of approval when we plug those values into Bayes’ theorem? It tumbles from 96 percent to a lowly…83 percent. Even if we accept Ricks’ judgment that Hagel flubbed his hearing and assume that a flubbed hearing tells us a lot about a nominee’s prospects, Bayes’ theorem tells us that the odds still overwhelmingly favor Hagel’s accession to the post. In what I consider to be a more realistic world where “flubbing” is in the eye of the beholder and confirmation hearings are essentially pro forma, Hagel’s mediocre showing would have little impact on a careful estimate of his odds, which would still hover around 90 percent. Either way, Ricks’ “50 – 2(business days)” estimate is looking pretty wild.

So where is Ricks coming from? I have never met the man and haven’t spoken to him about this blog post, so I can only guess, but a comment he made about John Tower suggests a common culprit: availability bias. As wiseGEEK explains, “Availability bias is a human cognitive bias that causes us to overestimate probabilities of events associated with memorable or vivid occurrences.” In explaining why he thinks Hagel will probably withdraw, Ricks writes:

SecDef nominees have blown up on the launch pad before: Remember John Tower (picked by the first President Bush) and Bobby Inman (picked by President Clinton to replace Les Aspin)?

It sure looks like a couple of recent and salient failures have stuck in Ricks’ mind, tempting him to ignore the overwhelming number of successful nominations. (Note: Inman didn’t factor into the base rate calculations above because he withdrew before Senate confirmation hearings were held, as have a number of other Cabinet nominees over the years, and I’m assuming that nominations which progress to that stage are different from ones that don’t.)

Of course, the base rate is often but not always the most useful piece of information. In situations where we have specific and compelling evidence about the case in question—say, word from a trusted source that Hagel actually plans to withdraw—we’ll want to discount the base rate and weight that specific evidence more heavily. The only specifics Ricks offers on Hagel, however, are his personal impressions of Hagel’s performance in front of the Armed Services Committee, which we’ve already addressed above, and a vague sense that “no one much wants him running the Pentagon.” With “specific” evidence as squishy as this, we’re probably better off sticking with the base rate.

UPDATE: On Tuesday, February 26, the Senate voted 58-41 to confirm Hagel’s appointment, and he was sworn into office the next morning.

Leave a comment

6 Comments

  1. thalevy@gmail.com

     /  February 10, 2013

    As usual, enjoy your thoughtful posts. Supreme Court is different but similar to Defense Secretary. Before Robert Bork, when to bork became a verb, judicial nominations were not especially politicized. But now they are. How does you know that you’re calculating a base rate for the right thing? Are the conditions right that the nomination process for Secretary of Defense will change?
    Sent from my Verizon Wireless BlackBerry

    Reply
    • Hey, Tammy. That’s an important question to ask, and my short answer is that you don’t know, so you make an informed guess. You can do stuff like look empirically for change points in the base rate, or even calculate different base rates based on different periodizations of the data and then average across those with weights representing your relative confidence in the relevance of the various versions. The strongest evidence that something’s changed would be a noticeable decline in the accuracy of forecasts based on the old assumptions.

      On the specifics of Cabinet and Supreme Court appointments, it does seem like they get a ton of scrutiny nowadays, but I wonder if that’s increased the rate of withdrawals before the nomination is officially made and hearings are held instead. It’s like a big, ugly, public vetting process, but most of it unfolds before the president officially names someone as his choice. The Susan Rice situation comes to mind there, too. If that’s right, then we might expect to see more names floated and withdrawn than in the past without seeing much change in the approval rate for formal nominees.

      Reply
  2. Grant

     /  February 11, 2013

    In Ricks’ defense I don’t think he was being serious about the exact numbers as much as going with a ‘gut feeling’ approach. Not the best writing on his part but also not an attempt to cloak it in probabilities.

    Reply
    • I think you’re probably right, but that’s kind of my point here: just making a “gut call” will often carry you pretty far from the mark, and it’s really not that hard to make a better forecast.

      Reply
      • Grant

         /  February 11, 2013

        I thought that the point was that Ricks’ methodology was bad (which it is).
        Still, Ricks might have a good basic point that Hagel’s chances aren’t perfect and that the longer this takes the worse they might be. I think a good gut feeling is really just an unconscious noticing of a trend. As a child I was able to guess who would be president on basis of whose face I saw on television the most*. That’s hardly methodologically sound but it’s a starting point.

        *Looking back on it with far more knowledge now I suspect it was because I was raised in Pennsylvania in the middle class, moderate Republican suburbs. Prior to 2000 the candidate who had enough ads to get them and the state probably won the election.

  1. Lost in the Fog of Civil War in Syria | Dart-Throwing Chimp

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 7,173 other followers

%d bloggers like this: