No, Pope Francis, this is not World War Three

In the homily to a mass given this morning in Italy, at a monument to 100,000 soldiers killed in World War I, Pope Francis said:

War is madness… Even today, after the second failure of another world war, perhaps one can speak of a third war, one fought piecemeal, with crimes, massacres, destruction.

There are a lot of awful things happening around the world, and I appreciate the pope’s advocacy for peace, but this comparison goes too far. Take a look at this chart of battle deaths from armed conflict around the world from 1900 to 2005, from a study by the Peace Research Institute of Oslo:

The chart doesn’t include the past decade, but we don’t need all the numbers in one place to see what a stretch this comparison is. Take Syria’s civil war, which has probably killed more than 150,000 (source) and perhaps as many as 300,000 or more people over the past three years, for an annual death rate of 50,000–100,000. That is a horrifying toll, but it is vastly lower than the annual rates in the several millions that occurred during the World Wars. Put another way, World War II was like 40 to 80 Syrian civil wars at once.

The many other wars of the present do not substantially close this gap. The civil war in Ukraine has killed approximately 3,000 so far (source). More than 2,000 people have died in the fighting associated with Israel’s Operation Protective Edge in Gaza this year (source). The resurgent civil war in Iraq dwarfs them both but still remains well below the intensity of the (interconnected) war next door (source). There are more than 20 other armed conflicts ongoing around the world, but most of them are much less lethal than the ones in Syria and Iraq, and their cumulative toll does not even begin to approach the ones that occurred in the World Wars (source).

I sympathize with the Pope’s intentions, but I don’t think that hyperbole is the best way to realize them. Of course, Pope Francis is not alone; we’ve been hearing a lot of this lately. I wonder if violence on the scale of the World Wars now lies so far outside of our lived experience that we simply cannot fathom it. Beyond some level of disorder, things simply become terrible, and all terrible things are alike. I also worry that the fear this apparent availability cascade is producing will drive other governments to react in ways that only make things worse.

Refugee Flows and Disorder in the Global System

This

The number of people displaced by violent conflict hit the highest level since World War II at the end of 2013, the head of the United Nations refugee agency, António Guterres, said in a report released on Friday…

Moreover, the impact of conflicts raging this year in Central African Republic, South Sudan, Ukraine and now Iraq threatens to push levels of displacement even higher by the end of 2014, he said.

…is, I think, another manifestation of the trends I discussed in a blog post here last September:

If we think on a systemic scale, it’s easier to see that we are now living through a period of global disorder matched in recent history only by the years surrounding the disintegration of the Soviet Union, and possibly exceeding it. Importantly, it’s not just the spate of state collapses through which this disorder becomes evident, but also the wider wave of protest activity and institutional transformation to which some of those collapses are connected.

If that’s true, then Mr. Guterres is probably right when he predicts that this will get even worse this year, because things still seem to be trending toward disorder. A lot of the transnational activity in response to local manifestations is still deliberately inflammatory (e.g., materiel and cash to rebels in Syria and Iraq, Russian support for separatists in Ukraine), and international efforts to quell some of those manifestations (e.g., UN PKOs in CAR and South Sudan) are struggling. Meanwhile, in what’s probably both a cause and an effect of these processes, global economic growth still has not rebounded as far or as fast as many had expected a year or two ago and remains uncertain and uneven.

In other words, the positive feedback still seems to be outrunning the negative feedback. Until that turns, the systemic processes driving (and being driven by) increased refugee flows will likely continue.

Addendum: The quote at the start of this post contains what I think is an error. A lot of the news stories on this report’s release used phrases like “displaced persons highest since World War II,” so I assumed that the U.N. report included the data on which that statement would be based. It turns out, though, that the report only makes a vague (and arguably misleading) reference to “the post-World War II era.” In fact, the U.N. does not have data to make comparisons on numbers of displaced persons prior to 1989. With the data it does have, the most the UNHCR can say is this, from p. 5: “The 2013 levels of forcible displacement were the highest since at least 1989, the first year that comprehensive statistics on global forced displacement existed.”

The picture also looks a little different from the press release if we adjust for increases in global population. Doing some rough math with the number of displaced persons in this UNHCR chart as the numerator and the U.S. Census Bureau’s mid-year estimates of world population as the denominator, here are some annual statistics on displaced persons as a share of the global population:

1989: 0.65%
1992: 0.84%
2010: 0.63%
2014: 0.72%

In no way do I mean to make light of what’s obviously a massive global problem, but as a share of the global population, the latest numbers are not (yet) even the worst since 1989, the first year for which UNHCR has comparable data.

A Brief Response to Anne-Marie Slaughter on Iraq and Syria

Anne-Marie Slaughter has an op-ed in today’s New York Times in which she argues that the U.S. government should launch air strikes now against targets in Iraq and Syria as a way to advance America’s and the world’s strategic and humanitarian interests. Here is the crux of the piece:

President Obama should be asking the same question in Iraq and Syria. What course of action will be best, in the short and the long term, for the Iraqi and Syrian people? What course of action will be most likely to stop the violence and misery they experience on a daily basis? What course of action will give them the best chance of peace, prosperity and a decent government?

The answer to those questions may well involve the use of force on a limited but immediate basis, in both countries. Enough force to remind all parties that we can, from the air, see and retaliate against not only Al Qaeda members, whom our drones track for months, but also any individuals guilty of mass atrocities and crimes against humanity. Enough force to compel governments and rebels alike to the negotiating table. And enough force to create a breathing space in which decent leaders can begin to consolidate power.

For the moment, let’s take for granted her assertions about the strategic interests at stake; the U.S.’s responsibility to protect civilians in other countries, by force if necessary; and the propriety of taking such action without prior approval from the U.N. Security Council.

Conceding all of that ground, it’s easier to see that, as a practical matter, Slaughter’s recommendation depends on strong assumptions about the efficacy of the action she proposes. Specifically, she asserts that the U.S. should conduct air strikes (“use of force on a limited but immediate basis,” “from the air”) against targets in Iraq and Syria because doing so will have three main effects:

  1. Deter atrocities (“to remind all parties that we can…see and retaliate against…any individuals guilty of mass atrocities and crimes against humanity”);
  2. Spur talks among warring parties (“to compel governments and rebels alike to the negotiating table”); and
  3. Enable positive political development (“to create a breathing space in which decent leaders can begin to consolidate power”)

If you believe, as Slaughter apparently does, that limited air strikes a) will almost certainly achieve all of these goals and b) will not produce other harmful strategic or humanitarian consequences that could partially offset or even outweigh those gains, then you should probably endorse this policy.

If, however, you are unsure about the ability of limited air strikes on yet-to-be-named targets in Iraq and Syria to accomplish these ends, or about the unintended strategic and humanitarian consequences those strikes could also have, then you should hesitate to support this policy and think through those other possible futures.

Beware the Confident Counterfactual

Did you anticipate the Syrian uprising that began in 2011? What about the Tunisian, Egyptian, and Libyan uprisings that preceded and arguably shaped it? Did you anticipate that Assad would survive the first three years of civil war there, or that Iraq’s civil war would wax again as intensely as it has in the past few days?

All of these events or outcomes were difficult forecasting problems before they occurred, and many observers have been frank about their own surprise at many of them. At the same time, many of those same observers speak with confidence about the causes of those events. The invasion of Iraq in 2003 surely is or is not the cause of the now-raging civil war in that country. The absence of direct US or NATO military intervention in Syria is or is not to blame for continuation of that country’s civil war and the mass atrocities it has brought—and, by extension, the resurgence of civil war in Iraq.

But here’s the thing: strong causal claims require some confidence about how history would have unfolded in the absence of the cause of interest, and those counterfactual histories are no easier to get right than observed history was to anticipate.

Like all of the most interesting questions, what causality means and how we might demonstrate it will forever be matters for debate—see here on Daniel Little’s blog for an overview of that debate’s recent state—but most conceptions revolve around some idea of necessity. When we say X caused Y, we usually mean that had X not occurred, Y wouldn’t have happened, either. Subtler or less stringent versions might center on salience instead of necessity and insert a “probably” into the final phrase of the previous sentence, but the core idea is the same.

In nonexperimental social science, this logic implicitly obliges us to consider the various ways history might have unfolded in response to X’ rather than X. In a sense, then, both prediction and explanation are forecasting problems. They require us to imagine states of the world we have not seen and to connect them in plausible ways to to ones we have. If anything, the counterfactual predictions required for explanation are more frustrating epistemological problems than the true forecasts, because we will never get to see the outcome(s) against which we could assess the accuracy of our guesses.

As Robert Jervis pointed out in his contribution to a 1996 edited volume on counterfactual thought experiments in world politics, counterfactuals are (or should be) especially hard to construct—and thus causal claims especially hard to make—when the causal processes of interest involve systems. For Jervis,

A system exists when elements or units are interconnected so that the system has emergent properties—i.e., its characteristics and behavior canot be inferred from the characteristics and behavior of the units taken individually—and when changes in one unit or the relationship between any two of them produce ramifying alterations in other units or relationships.

As Jervis notes,

A great deal of thinking about causation…is based on comparing two situations that are the same in all ways except one. Any differences in the outcome, whether actual or expected…can be attributed to the difference in the state of the one element…

Under many circumstances, this method is powerful and appropriate. But it runs into serious problems when we are dealing with systems because other things simply cannot be held constant: as Garret Hardin nicely puts it, in a system, ‘we can never do merely one thing.’

Jervis sketches a few thought experiments to drive this point home. He has a nice one about the effects of external interventions on civil wars that is topical here, but I think his New York traffic example is more resonant:

In everyday thought experiments we ask what would have happened if one element in our world had been different. Living in New York, I often hear people speculate that traffic would be unbearable (as opposed to merely terrible) had Robert Moses not built his highways, bridges, and tunnels. But to try to estimate what things would have been like, we cannot merely subtract these structures from today’s Manhattan landscape. The traffic patterns, the location of businesses and residences, and the number of private automobiles that are now on the streets are in significant measure the product of Moses’s road network. Had it not been built, or had it been built differently, many other things would have been different. Traffic might now be worse, but it is also possible that it would have been better because a more efficient public transportation system would have been developed or because the city would not have grown so large and prosperous without the highways.

Substitute “invade Iraq” or “fail to invade Syria” for Moses’s bridges and tunnels, and I hope you see what I mean.

In the end, it’s much harder to get beyond banal observations about influences to strong claims about causality than our story-telling minds and the popular media that cater to them would like. Of course the invasion of Iraq in 2003 or the absence of Western military intervention in Syria have shaped the histories that followed. But what would have happened in their absence—and, by implication, what would happen now if, for example, the US now re-inserted its armed forces into Iraq or attempted to topple Assad? Those questions are far tougher to answer, and we should beware of anyone who speaks with great confidence about their answers. If you’re a social scientist who isn’t comfortable making and confident in the accuracy of your predictions, you shouldn’t be comfortable making and confident in the validity of your causal claims, either.

There Is No Such Thing as Civil War

In a 2008 conference paper, Jim Fearon and David Laitin used statistics and case narratives to examine how civil wars around the world since 1955 have ended. They found that deadly fights between central governments and domestic challengers usually only end after an abrupt change in the relative fighting power of one side or the other, and that these abrupt changes are usually brought on by the beginning or end of foreign support. This pattern led them to ruminate thus (emphasis in original):

We were struck by the high frequency of militarily significant foreign support for government and rebels. The evidence suggests that more often than not, civil wars either become – or may even begin as –the object of other states’ foreign policies…Civil wars are normally studied as matters of domestic politics. Future research might make progress by shifting the perspective, and thinking about civil war as international politics by other means.

Their study recently came to mind when I was watching various people on Twitter object to the idea that what’s happening in Ukraine right now could be described as civil war, or at least the possible beginnings of one. Even if some of the separatists mobilizing in eastern Ukraine really were Ukrainian nationals, they argued, the agent provocateur was Russia, so this fight is properly understood as a foreign incursion.

As Jim and David’s paper shows, though, strong foreign hands are a common and often decisive feature of the fights we call civil wars.

In Syria, for example, numerous foreign governments and other external agents are funding, training, equipping, and arming various factions in the armed conflict that’s raged for nearly three years now. Some of that support is overt, but the support we see when we read about the war in the press is surely just a fraction of what’s actually happening. Yet we continue to see the conflict described as a civil war.

In the Central African Republic, it’s Chad that’s played “an ambiguous and powerful role” in the conflict that has precipitated state collapse and ethnic cleansing there. As the New York Times described in April,

[Chad] was accused of supporting the overthrow of the nation’s president, and then later helped remove the rebel who ousted him, making way for a new transitional government. In a statement on Thursday, the Chadian government said that its 850 soldiers had been accused of siding with Muslim militias in sectarian clashes with Christian fighters that have swept the Central African Republic for months.

At least a couple of bordering states are apparently involved in the civil war that’s stricken South Sudan since December. In a May 2014 report, the UN Mission to South Sudan asserted that government forces were receiving support from “armed groups from the Republic of Sudan,” and that “the Government has received support from the Uganda People’s Defence Force (UPDF), notably in Juba and Jonglei State.” The report also claimed that “some Darfuri militias have allied with opposition forces in the northern part of Unity State,” which borders Sudan. And, of course, there is a nearly 8,000-strong UN peacekeeping operation that is arguably shaping the scale of the violence there, even if it isn’t stopping it.

Pick a civil war—any civil war—and you’ll find similar evidence of external involvement. This is what led Jim and David to muse about civil wars as “international politics by other means,” and what led me to the deliberately provocative title of this post. As a researcher, I see analytic value in sometimes distinguishing between interstate and intrastate wars, which may have distinct causes and follow different patterns and may therefore be amenable to different forms of prevention or mitigation. At the same time, I think it’s clear that this distinction is nowhere near as crisp in reality as our labels imply, so we should be mindful to avoid confusing the typology with the reality it crudely describes.

A Notable Year of the Wrong Kind

The year that’s about to end has distinguished itself in at least one way we’d prefer never to see again. By my reckoning, 2013 saw more new mass killings than any year since the early 1990s.

When I say “mass killing,” I mean any episode in which the deliberate actions of state agents or other organizations kill at least 1,000 noncombatant civilians from a discrete group. Mass killings are often but certainly not always perpetrated by states, and the groups they target may be identified in various ways, from their politics to their ethnicity, language, or religion. Thanks to my colleague Ben Valentino, we have a fairly reliable tally of episodes of state-led mass killing around the world since the mid-1940s. Unfortunately, there is no comparable reckoning of mass killings carried out by non-state actors—nearly always rebel groups of some kind—so we can’t make statements about counts and trends as confidently as I would like. Still, we do the best we can with the information we have.

With those definitions and caveats in mind, I would say that in 2013 mass killings began:

Of course, even as these new cases have developed, episodes of mass killings have continued in a number of other places:

In a follow-up post I hope to write soon, I’ll offer some ideas on why 2013 was such a bad year for deliberate mass violence against civilians. In the meantime, if you think I’ve misrepresented any of these cases here or overlooked any others, please use the Comments to set me straight.

The Fog of War Is Patchy

Over at Foreign Policy‘s Peace Channel, Sheldon Himmelfarb of USIP has a new post arguing that better communications technologies in the hands of motivated people now give us unprecedented access to information from ongoing armed conflicts.

The crowd, as we saw in the Syrian example, is helping us get data and information from conflict zones. Until recently these regions were dominated by “the fog war,” which blinded journalists and civilians alike; it took the most intrepid reporters to get any information on what was happening on the ground. But in the past few years, technology has turned conflict zones from data vacuums into data troves, making it possible to render parts the conflict in real time.

Sheldon is right, but only to a point. If crowdsourcing is the future of conflict monitoring, then the future is already here, as Sheldon notes; it’s just not very evenly distributed. Unfortunately, large swaths of the world remain effectively off the grid on which the production of crowdsourced conflict data depends. Worse, countries’ degree of disconnectedness is at least loosely correlated with their susceptibility to civil violence, so we still have the hardest time observing some of the world’s worst conflicts.

The fighting in the Central African Republic over the past year is a great and terrible case in point. The insurgency that flared there last December drove the president from the country in March, and state security forces disintegrated with his departure. Since then, CAR has descended into a state of lawlessness in which rival militias maraud throughout the country and much of the population has fled their homes in search of whatever security and sustenance they can find.

We know this process is exacting a terrible toll, but just how terrible is even harder to say than usual because very few people on hand have the motive and means to record and report out what they are seeing. At just 23 subscriptions per 100 people, CAR’s mobile-phone penetration rate remains among the lowest on the planet, not far ahead of Cuba’s and North Korea’s (data here). Some journalists and NGOs like Human Rights Watch and Amnesty International have been covering the situation as best they can, but they will be among the first to tell you that their information is woefully incomplete, in part because roads and other transport remain rudimentary. In a must-read recent dispatch on the conflict, anthropologist Louisa Lombard noted that “the French colonists invested very little in infrastructure, and even less has been invested subsequently.”

A week ago, I used Twitter to ask if anyone had managed yet to produce a reasonably reliable estimate of the number of civilian deaths in CAR since last December. The replies I received from some very reputable people and organizations makes clear what I mean about how hard it is to observe this conflict.

C.A.R. is an extreme case in this regard, but it’s certainly not the only one of its kind. The same could be said of ongoing episodes of civil violence in D.R.C., Sudan (not just Darfur, but also South Kordofan and Blue Nile), South Sudan, and in the Myanmar-China border region, to name a few. In all of these cases, we know fighting is happening, and we believe civilians are often targeted or otherwise suffering as a result, but our real-time information on the ebb and flow of these conflicts and the tolls they are exacting remains woefully incomplete. Mobile phones and the internet notwithstanding, I don’t expect that to change as quickly as we’d hope.

[N.B. I didn't even try to cover the crucial but distinct problem of verifying the information we do get from the kind of crowdsourcing Sheldon describes. For an entry point to that conversation, see this great blog post by Josh Stearns.]

How Long Will Syria’s Civil War Last? It’s Really Hard to Say

Last week, political scientist Barbara Walter wrote a great post for the blog Political Violence @ a Glance called “The Four Things We Know about How Civil Wars End (and What This Tells Us about Syria),” offering a set of base-rate forecasts about how long Syria’s civil war will last (probably a lot longer) and how it’s likely to end (with a military victory and not a peace agreement).

The post is great because it succeeds in condensing a large and complex literature into a small set of findings directly relevant to an important topic of public concern. It’s no coincidence that this post was written by one of the leading scholars on that subject. A “data scientist” could have looked at the same data sets used in the studies on which Walter bases her summary and not known which statistics would be most informative. Even with the right statistics in hand, a “hacker” probably wouldn’t know much about the relative quality of the different data sources, or the comparative-historical evidence on relevant causal mechanisms—two things that could (and should) inform their thinking about how much confidence to attach to the various results. To me, this is a nice illustration of the point that, even in an era of relentless quantification, subject-matter expertise still matters.

The one thing that seems to have gotten lost in the retellings and retweetings of this distilled evidence, though, is the idea of uncertainty. Apparently inspired by Walter’s post, Max Fisher wrote a similar one for the Washington Post‘s Worldviews blog under the headline “Political science says Syria’s civil war will probably last at least another decade.” Fisher’s prose is appropriately less specific than that (erroneous) headline, but if my Twitter feed is any indication, lots of people read Walter’s and Fisher’s post as predictions that the Syrian war will probably last 10 years or more in total.*

If you had to bet now on the war’s eventual duration, you’d be right to expect an over-under around 10, but the smart play would probably be not to bet at all, unless you were offered very favorable odds or you had some solid hedges in place. That’s because the statistics Walter and Fisher cite are based on a relatively small number of instances of a complex phenomenon, the origins and dynamics of which we still poorly understand. Under these circumstances, statistical forecasting is inevitably imprecise, and the imprecision only increases the farther we try to peer into the future.

We can visualize that imprecision, and the uncertainty it represents, with something called a prediction interval. A prediction interval is just an estimate of the range in which we expect future values of our quantity of interest to fall with some probability. Prediction intervals are sometimes included in plots of time-series forecasts, and the results typically look like the bell of a trumpet, as shown in the example below. The farther into the future you try to look, the less confidence you should have in your point prediction. When working with noisy data on a stochastic process, it doesn’t take a lot of time slices to reach the point where your prediction interval practically spans the full range of possible values.

prediction interval

Civil wars are, without question, one of those stochastic processes with noisy data. The averages Walter and Fisher cite are just central tendencies from a pretty heterogenous set of cases observed over a long period of world history. Using data like these, I think we can be very confident that the war will last at least a few more months and somewhat confident that it will last at least another year or more. Beyond that, though, I’d say the bell of our forecasting trumpet widens very quickly, and I wouldn’t want to hazard a guess if I didn’t have to.

* In fact, neither Walter nor Fisher specifically predicted that the war would last x number of years or more. Here’s what Walter actually wrote:

1. Civil wars don’t end quickly. The average length of civil wars since 1945 have been about 10 years. This suggests that the civil war in Syria is in its early stages, and not in the later stages that tend to encourage combatants to negotiate a settlement.

I think that’s a nice verbal summary of the statistical uncertainty I’m trying to underscore. And here’s what Fisher wrote under that misleading headline:

According to studies of intra-state conflicts since 1945, civil wars tend to last an average of about seven to 12 years. That would put the end of the war somewhere between 2018 and 2023.

Worse, those studies have identified several factors that tend to make civil wars last even longer than the average. A number of those factors appear to apply to Syria, suggesting that this war could be an unusually long one. Of course, those are just estimates based on averages; by definition, half of all civil wars are shorter than the median length, and Syria’s could be one of them. But, based on the political science, Syria has the right conditions to last through President Obama’s tenure and perhaps most or all of his successor’s.

President Obama, You’re the Fish in this Morbid Game of Poker

I believe the Obama administration’s planned punitive strikes on Syria are wrong for larger reasons (see here for a 2012 post that’s still relevant today), but I’m also convinced that they’re likely to be ineffective for the narrower goal of deterring the use of chemical weapons in Syria and beyond. I don’t mean to make light of a horrible situation, but I think a gaming analogy can help show why.

Think of the repeated interactions between the Assad regime and the U.S. as a single game of poker with several hands. In 2011, President Obama said Assad had to go, and the U.S. hinted that it would intervene to support the Syrian opposition. That was a raise, Assad called it, and the U.S. effectively folded that hand by not following through on its initial raise.

More recently, President Obama declared the use of chemical weapons a “red line” that Assad’s forces must not cross, and then they crossed that line, apparently more than once before the massive attack near Damascus on 21 August. Again, the administration made a raise in hopes of driving Assad off his hand, but again Assad re-raised.

Now the Obama administration is threatening to strike Assad’s forces to punish him for the CW attack. While making this threat, though, the administration is simultaneously signaling that a) the attack will be limited and b) the administration hopes not to have to do more. These terms are more or less written into the authorization Congress is now considering, and they are being reiterated every time a member of the administration makes a public case for a military response.

In poker terms, this approach is like trying to drive your opponent off a pot with a modest bet when you hold a weak hand. Unless your opponent has really weak cards, that kind of bet is usually more effective at enticing that opponent to stay in the hand, not encouraging him to fold. In the Syrian case, the Assad regime has repeatedly signaled that it will play every hand to the end, so this kind of bet will almost certainly not have the desired effect.

That outcome is even more likely if the opponent has good reason to think your hand is weak. When the Obama administration can’t muster much domestic or international support for its punitive strikes and whatever support it can muster is predicated on those strikes being very limited in their scope and intent, then I’d say that’s easy to read as a weak hand. It’s a bit like waving around a pair of eights and threatening to make a small raise. To drive a committed rival to fold, you need to really change the expected value of the pot, and this approach simply doesn’t do that to a regime that has shown itself to be deeply committed to playing every hand to the end.

Some supporters of punitive strikes seem to think the effect those strikes would have on Assad’s forces is less important than the signal this action would send to potential future violators. The goal is not to hurt Assad as much as it’s to reinforce the norm. Unfortunately, the same problem extends to future hands with other players, too. If I were a ruler considering using chemical weapons at some later date, the lesson I think I’d have learned from Syria so far is that the rest of the world actually isn’t willing to pay a steep cost to reinforce this supposed norm for its own sake. In fact, we’ve developed a tell: if the stakes are high for other reasons, our initial raise will probably be a bluff, and it probably won’t be that costly to stay in the hand and see if that’s right.

I can see two paths out of the current situation. One is to acknowledge that our tepid raise has failed to drive Assad off this pot and go ahead and fold this hand. The outcome is essentially the same, and we don’t incur bigger losses getting there. The other is to change the hand we’re playing by committing to do whatever it takes to prevent Assad’s forces from using chemical weapons again. In other words, we commit to regime-defeating war if necessary and we signal that stronger commitment to Assad’s forces and their backers as clearly as possible.

If this more aggressive approach isn’t both feasible and desirable—and I believe it’s neither—it’s hard for me to see what’s gained by continuing to pretend that’s the hand we’re playing when everyone knows it isn’t and calling yet another of the Assad regime’s horrible raises.

What Should the U.S. Do in Syria: Survey Results and Lessons on Process

A few days ago, I used the All Our Ideas platform to create a pairwise wiki survey asking, “Which action would you rather see the United States take next in Syria?” I did this partly to get a better sense of peoples’ views on the question posed, and partly to learn more about how to use this instrument. Now, I think it’s a good time to take stock on both counts.

First, some background. A pairwise wiki survey involves a single question with many possible answers. Respondents are presented with answers in pairs, one pair at a time, and asked to cast a vote for one or the other item. The overarching question determines what that vote is about, but the choice always entails a comparison (more, better, more likely, etc.). Respondents can also choose not to decide, and they can propose their own answers to add to the mix. Here’s a screenshot from my survey on U.S. policy in Syria that shows how that looks in action:

syria wiki survey respondent interface screenshot

You vote by clicking on one of the big blue boxes or the smaller “I can’t decide” button tucked under them, or you propose your own answer by writing it into in the “Add your own ideas here…” field at the bottom. Once you vote on one pair, you’re presented with another pair, and you can repeat this process as many times as you like. To make each vote as informative as possible, the All Our Ideas platform doesn’t select answers for each pairing at random. Instead, it uses an algorithm that favors answers with fewer completed appearances. This adaptive approach spreads the votes evenly across the field of answers, and it helps newly-added answers quickly catch up with older ones. The resulting pairwise votes are converted into aggregate ratings using a Bayesian hierarchical model that estimates a set of collective preferences that’s most consistent with the observed data.

I’m already experimenting with pairwise wiki surveys as a way to forecast rare events, but this question about how the U.S. should respond to events in Syria is closer to their original purpose of identifying and ranking a set of options that aren’t exhaustive or mutually exclusive. In situations like these, it’s often easy to criticize or tout each option on its own. Comparing them all in a coherent way is usually much harder, and that’s what the pairwise wiki survey helps us do.

So, what did the respondents to my survey think the U.S. should do now about Syria? The screenshot below shows where things stood around 8:45 AM Eastern time on Tuesday, September 3, after more than 1,400 votes had been cast in nearly 100 unique user sessions. (For the latest results, click here.)

syria wiki survey results 20130903 0842

Clearly, the crowd that’s found its way to this survey so far is not keen on the Obama administration’s plan for military strikes in Syria in response to the chemical-weapons attack that took place on 21 August. The two options that come closest to that stated plan—limited strikes on targets associated with Syria’s chemical weapons capability or limited strikes to degrade various aspects of its military—both rank in the bottom half, below “Do nothing” and “Strongly condemn the Syrian regime.” The idea of military strikes targeting Assad and other senior regime officials—the so-called decapitation approach—ranks last, and increased military aid to Syrian rebels, another option the U.S. government is already pursuing, doesn’t rank much higher. What this crowd wants from the U.S. instead are increased humanitarian aid to Syrian civilians, broader and tighter sanctions on the Syrian regime and its “enablers,” and more pushing for formal talks among the warring parties.

Now, I don’t mean to imply that the results of this survey accurately capture the contours of public opinion in the U.S. or anywhere else. Frankly, I don’t really know how representative they are. As implemented here, a pairwise wiki survey is a form of crowdsourcing. The big advantage of crowdsourcing is the ability quickly and cheaply to get feedback from large group, but that efficiency sometimes comes at the cost of not knowing a lot about who is responding. I know that the participants in my Syria survey come from six continents (see the map below). but I don’t collect any information about the respondents as they vote, so I can’t say anything about how how representative my crowd is of any larger population, or how the characteristics of individual respondents relate to the preferences they express. All I can say with confidence is that these results are probably a reliable gauge of the views of the crowd that became aware of the survey through my blog post and Twitter and other social-media shares and were motivated to respond. I think it’s reassuring that the results of my wiki survey generally accord with the results of traditional public-opinion surveys in the U.S. (e.g., here and here) and elsewhere (e.g., Germany), but it would be irresponsible to make any strong claims about public opinion from these data alone.

Syria wiki survey vote map 20130903 0908

I hope to put this instrument to more ambitious uses in the future, so I’ll close with a lesson learned about how to do it better: respondents really need to be given some explanation about how the survey works before they’re asked to start voting. I rushed to get the Syria survey online because I was trying to get out the door for a bike ride and didn’t include anything in my blog post or tweets about how the voting process works. From the things some people wrote in the submit-your-own-idea field, it quickly became clear that many visitors were confused. Some apparently thought the initial pair presented were the only options being considered, so they either complained directly about that (“This survey, I hope, is designed to demonstrate to takers the way questioners of surveys control the outcome with push-polling”) or proposed adding ideas that were already covered (e.g., “Neither” when “Do nothing” was already on the list, or “Aid to refugees and camps” when “Increase humanitarian aid to Syrian civilians” was an option). I also think the “I can’t decide” button and the options it offers (press it and see) are a really important feature that respondents may overlook because it can be hard to see. Next time, I won’t share a direct link to the survey and will instead embed the link at the bottom of a blog post that describes the voting process and calls out the “I can’t decide” feature first.

Follow

Get every new post delivered to your Inbox.

Join 7,156 other followers

%d bloggers like this: