What Should the U.S. Do in Syria: Survey Results and Lessons on Process

A few days ago, I used the All Our Ideas platform to create a pairwise wiki survey asking, “Which action would you rather see the United States take next in Syria?” I did this partly to get a better sense of peoples’ views on the question posed, and partly to learn more about how to use this instrument. Now, I think it’s a good time to take stock on both counts.

First, some background. A pairwise wiki survey involves a single question with many possible answers. Respondents are presented with answers in pairs, one pair at a time, and asked to cast a vote for one or the other item. The overarching question determines what that vote is about, but the choice always entails a comparison (more, better, more likely, etc.). Respondents can also choose not to decide, and they can propose their own answers to add to the mix. Here’s a screenshot from my survey on U.S. policy in Syria that shows how that looks in action:

You vote by clicking on one of the big blue boxes or the smaller “I can’t decide” button tucked under them, or you propose your own answer by writing it into in the “Add your own ideas here…” field at the bottom. Once you vote on one pair, you’re presented with another pair, and you can repeat this process as many times as you like. To make each vote as informative as possible, the All Our Ideas platform doesn’t select answers for each pairing at random. Instead, it uses an algorithm that favors answers with fewer completed appearances. This adaptive approach spreads the votes evenly across the field of answers, and it helps newly-added answers quickly catch up with older ones. The resulting pairwise votes are converted into aggregate ratings using a Bayesian hierarchical model that estimates a set of collective preferences that’s most consistent with the observed data.

I’m already experimenting with pairwise wiki surveys as a way to forecast rare events, but this question about how the U.S. should respond to events in Syria is closer to their original purpose of identifying and ranking a set of options that aren’t exhaustive or mutually exclusive. In situations like these, it’s often easy to criticize or tout each option on its own. Comparing them all in a coherent way is usually much harder, and that’s what the pairwise wiki survey helps us do.

So, what did the respondents to my survey think the U.S. should do now about Syria? The screenshot below shows where things stood around 8:45 AM Eastern time on Tuesday, September 3, after more than 1,400 votes had been cast in nearly 100 unique user sessions. (For the latest results, click here.)

Clearly, the crowd that’s found its way to this survey so far is not keen on the Obama administration’s plan for military strikes in Syria in response to the chemical-weapons attack that took place on 21 August. The two options that come closest to that stated plan—limited strikes on targets associated with Syria’s chemical weapons capability or limited strikes to degrade various aspects of its military—both rank in the bottom half, below “Do nothing” and “Strongly condemn the Syrian regime.” The idea of military strikes targeting Assad and other senior regime officials—the so-called decapitation approach—ranks last, and increased military aid to Syrian rebels, another option the U.S. government is already pursuing, doesn’t rank much higher. What this crowd wants from the U.S. instead are increased humanitarian aid to Syrian civilians, broader and tighter sanctions on the Syrian regime and its “enablers,” and more pushing for formal talks among the warring parties.

Now, I don’t mean to imply that the results of this survey accurately capture the contours of public opinion in the U.S. or anywhere else. Frankly, I don’t really know how representative they are. As implemented here, a pairwise wiki survey is a form of crowdsourcing. The big advantage of crowdsourcing is the ability quickly and cheaply to get feedback from large group, but that efficiency sometimes comes at the cost of not knowing a lot about who is responding. I know that the participants in my Syria survey come from six continents (see the map below). but I don’t collect any information about the respondents as they vote, so I can’t say anything about how how representative my crowd is of any larger population, or how the characteristics of individual respondents relate to the preferences they express. All I can say with confidence is that these results are probably a reliable gauge of the views of the crowd that became aware of the survey through my blog post and Twitter and other social-media shares and were motivated to respond. I think it’s reassuring that the results of my wiki survey generally accord with the results of traditional public-opinion surveys in the U.S. (e.g., here and here) and elsewhere (e.g., Germany), but it would be irresponsible to make any strong claims about public opinion from these data alone.

I hope to put this instrument to more ambitious uses in the future, so I’ll close with a lesson learned about how to do it better: respondents really need to be given some explanation about how the survey works before they’re asked to start voting. I rushed to get the Syria survey online because I was trying to get out the door for a bike ride and didn’t include anything in my blog post or tweets about how the voting process works. From the things some people wrote in the submit-your-own-idea field, it quickly became clear that many visitors were confused. Some apparently thought the initial pair presented were the only options being considered, so they either complained directly about that (“This survey, I hope, is designed to demonstrate to takers the way questioners of surveys control the outcome with push-polling”) or proposed adding ideas that were already covered (e.g., “Neither” when “Do nothing” was already on the list, or “Aid to refugees and camps” when “Increase humanitarian aid to Syrian civilians” was an option). I also think the “I can’t decide” button and the options it offers (press it and see) are a really important feature that respondents may overlook because it can be hard to see. Next time, I won’t share a direct link to the survey and will instead embed the link at the bottom of a blog post that describes the voting process and calls out the “I can’t decide” feature first.

Posted by Jay Ulfelder on September 3, 2013

https://dartthrowingchimp.wordpress.com/2013/09/03/what-should-the-u-s-do-in-syria-survey-results-and-lessons-on-process/