Lost in the Fog of Civil War in Syria

On Twitter a couple of days ago, Adam Elkus called out a recent post on Time magazine’s World blog as evidence of the way that many peoples’ expectations about the course of Syria’s civil war have zigged and zagged over the past couple of years. “Last year press was convinced Assad was going to fall,” Adam tweeted. “Now it’s that he’s going to win. Neither perspective useful.” To which the eminent civil-war scholar Stathis Kalyvas replied simply, “Agreed.”

There’s a lesson here for anyone trying to glean hints about the course of a civil war from press accounts of a war’s twists and turns. In this case, it’s a lesson I’m learning through negative feedback.

Since early 2012, I’ve been a participant/subject in the Good Judgment Project (GJP), a U.S. government-funded experiment in “wisdom of crowds” forecasting. Over the past year, GJP participants have been asked to estimate the probability of several events related to the conflict in Syria, including the likelihood that Bashar al-Assad would leave office and the likelihood that opposition forces would seize control of the city of Aleppo.

I wouldn’t describe myself as an expert on civil wars, but during my decade of work for the Political Instability Task Force, I spent a lot of time looking at data on the onset, duration, and end of civil wars around the world. From that work, I have a pretty good sense of the typical dynamics of these conflicts. Most of the civil wars that have occurred in the past half-century have lasted for many years. A very small fraction of those wars flared up and then ended within a year. The ones that didn’t end quickly—in other words, the vast majority of these conflicts—almost always dragged on for several more years at least, sometimes even for decades. (I don’t have my own version handy, but see Figure 1 in this paper by Paul Collier and Anke Hoeffler for a graphical representation of this pattern.)

On the whole, I’ve done well in the Good Judgment Project. In the year-long season that ended last month, I ranked fifth among the 303 forecasters in my experimental group, all while the project was producing fairly accurate forecasts on many topics. One thing that’s helped me do well is my adherence to what you might call the forecaster’s version of the Golden Rule: “Don’t neglect the base rate.” And, as I just noted, I’m also quite familiar with the base rates of civil-war duration.

So what did I do when asked by GJP to think about what would happen in Syria? I chucked all that background knowledge out the window and chased the very narrative that Elkus and Kalyvas rightly decry as misleading.

Here’s a chart showing how I assessed the probability that Assad wouldn’t last as president beyond the end of March 2013, starting in June 2012. The actual question asked us to divide the probability of his exiting office across several time periods, but for simplicity’s sake I’ve focused here on the part indicating that he would stick around past April 1. This isn’t the same thing as the probability that the war would end, of course, but it’s closely related, and I considered the two events as tightly linked. As you can see, until early 2013, I was pretty confident that Assad’s fall was imminent. In fact, I was so confident that at a couple of points in 2012, I gave him zero chance of hanging on past March of this year—something a trained forecaster really never should do.

gjp assad chart

Now here’s another chart showing my estimates of the likelihood that rebels would seize control of Aleppo before May 1, 2013. The numbers are a little different, but the basic pattern is the same. I started out very confident that the rebels would win the war soon and only swung hard in the opposite direction in early 2013, as the boundaries of the conflict seemed to harden.

gjp aleppo chart

It’s impossible to say what the true probabilities were in this or any other uncertain situation. Maybe Assad and Aleppo really were on the brink of falling for a while and then the unlikely-but-still-possible version happened anyway.

That said, there’s no question that forecasts more tightly tied to the base rate would have scored a lot better in this case. Here’s a chart showing what my estimates might have looked like had I followed that rule, using approximations of the hazard rate from the chart in the Collier and Hoeffler paper. If anything, these numbers overstate the likelihood that a civil war will end at a given point in time.

gjp baserate chart

I didn’t keep a log spelling out my reasoning at each step, but I’m pretty confident that my poor performance here is an example of motivated reasoning. I wanted Assad to fall and the pro-democracy protesters who dominated the early stages of the uprising to win, and that desire shaped what I read and then remembered when it came time to forecast. I suspect that many of the pieces I was reading were slanted by similar hopes, creating a sort of analytic cascade similar to the herd behavior thought to drive many financial-market booms and busts. I don’t have the data to prove it, but I’m pretty sure the ups and downs in my forecasts track the evolving narrative in the many newspaper and magazine stories I was reading about the Syrian conflict.

Of course, that kind of herding happens on a lot of topics, and I was usually good at avoiding it. For example, when tensions ratcheted up on the Korean Peninsula earlier this year, I hewed to the base rate and didn’t substantially change my assessment of the risk that real clashes would follow.

What got me in the case of Syria was, I think, a sense of guilt. The Assad government has responded to a legitimate popular challenge with mass atrocities that we routinely read about and sometimes even see. In parts of the country, the resulting conflict is producing scenes of absurd brutality. This isn’t a “problem from hell,” as Samantha Powers’ book title would have it; it is a glimpse of hell. And yet, in the face of that horror, I have publicly advocated against American military intervention. Upon reflection, I wonder if my wildly optimistic forecasting about the imminence of Assad’s fall wasn’t my unconscious attempt to escape the discomfort of feeling complicit in the prolongation of that suffering.

As a forecaster, if I were doing these questions over, I would try to discipline myself to attend to the base rate, but I wouldn’t necessarily stop there. As I’ve pointed out in a previous post, the base rate is a valuable anchoring device, but attending to it doesn’t mean automatically ignoring everything else. My preferred approach, when I remember to have one, is to take that base rate as a starting point and then use Bayes’ theorem to update my forecasts in a more disciplined way. Still, I’ll bring a newly skeptical eye the flurry of stories predicting that Assad’s forces will soon defeat Syria’s rebels and keep their patron in power. Now that we’re a couple years into the conflict, quantified history tells us that the most likely outcome in any modest slice of time (say, months rather than years) is, tragically, more of the same.

And, as a human, I’ll keep hoping the world will surprise us and take a different turn.

Leave a comment

11 Comments

  1. Grant

     /  May 21, 2013

    Considering that Assad is, in all likelihood, going to continue to receive continued financial support and supplies from friendly nations and fighters from Lebanon, Iran (and reportedly now Iraq) isn’t it possible that this is a case where time is more on the side of the Assad regime than on the side of the rebels?

    Reply
    • Well, yes, it’s certainly possible. The hard part about probabilistic forecasting is figuring out what you think phrases like that mean and then quantifying your judgment about the likelihood of the event of interest. I’m guessing you mean that the probability of Assad losing the war declines with time. So, then, if you were asked how likely it is that he will still be president of Syria at the end of 2013, what number would you give? And then, conditional on his surviving that long, end of 2014?

      Reply
      • Grant

         /  May 22, 2013

        I try to avoid specific numbers, especially since I’m still not a full political scientist. Also we can’t overlook the facts that reportedly Qatar and Saudi Arabia have been supplying rebel groups with at least some weapons, the sectarian nature of the war likely means that the rebel groups don’t lack for recruits and the fact that Iran, Hezhbollah and Iraq (or at least someone in Iraq) wouldn’t be sending fighters to help the Assad government if they thought he was going to win this soon. Additionally we can’t predict international events like a general Sunni/Shia war in the Middle East that might change fortunes.

        Still, with the better training of the Syrian military, the support (monetary, military and political) coming in from friends fearful of Assad’s defeat and a Sunni takeover and the fractured nature of the rebels it seems to me that unless a major event occurs in aid of the rebel groups (a politically unified front, the U.S. supplying them with weapons, a NATO intervention etc), time is more on Assad’s side. I will admit though, that I’m actually rather surprised that the rebellion has lasted as long as it has. I could be biased into presuming an Assad victory because of the 1970s Syrian civil war*.

        *It should be fascinating to see if Syria suffers more civil wars in the future from its sectarian splits regardless of the outcome of this one.

  2. Admirably honest Jay. I remember thinking that if even you thought Assad was going to fall, then surely he would.

    There is a chance that you are now (subconsciously) over-compensating for your earlier overreaction by downplaying Assad’s advances, which would be another example of motivated reasoning. Not suggesting you are, but it would be horribly ironic.

    Reply
    • Thanks, Richard. What you say about overcompensating is quite possible and had occurred to me, too. That’s actually what I had in mind when I picked a title for the post.

      Reply
  3. Oral Hazard

     /  May 22, 2013

    Jay, were you invited to be a “super forecaster” next season?

    Reply
    • I’m actually working as a consultant to the project for the next season, helping to identify and write the questions.

      Reply
      • Oral Hazard

         /  May 22, 2013

        Interesting! And congratulations — I imagine it’s a very exciting project to be associated with as a researcher. Supposedly IARPA wants to increase the # of questions next season, so I’m sure they’ll make good use of your skills. Too bad, in a way, because you would have made an awesome teammate sharing personal insights and how they apply to forecasting principals in an interesting and accessible way. It was how I found your site, actually. Best of luck. Now I’ll have someone to blame for the particularly dastardly ones.

  4. Great post, dartthrowingchip. Just to amplify your point about how most civil wars are very long-running, the same is even more true for “subnational conflicts.” The World Development Report from a couple years back estimated that the average duration for subnational conflicts is 41 years. Since Syria could easily degenerate into a subnational conflict, that should be sobering for anyone who thinks the conflict there is likely to end soon. Thinking more in terms of regional history, the civil war in Lebanon, where the ethnic mix is closest to Syria’s, lasted 15 years. Finally, there’s an embedded assumption in several of those charts that if “Aleppo falls” or if “Assad goes” that this means the end of the civil war — which is some kind of a joke, albeit in poor taste.

    In sum, there’s every reason to think the civil war could last a long time.

    Reply
    • Thanks, Nils. You’re right, of course, that the fall of Assad wouldn’t directly end the war. I can’t be sure because I didn’t keep careful notes, but I think that awareness of that point might actually have been part of how I rationalized pegging those probabilities so high in spite of what I knew about the base rate of civil-war duration. That loose connection, in turn, hints at one of the challenges of probabilistic forecasting: the base rate of what? Good topic for a future post, that…

      Reply

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 7,098 other followers

%d bloggers like this: