The Ethics of Political Science in Practice

As citizens and as engaged intellectuals, we all have the right—indeed, an obligation—to make moral judgments and act based on those convictions. As political scientists, however, we have a unique set of potential contributions and constraints. Political scientists do not typically have anything of distinctive value to add to a chorus of moral condemnation or declarations of normative solidarity. What we do have, hopefully, is the methodological training, empirical knowledge and comparative insight to offer informed assessments about alternative courses of action on contentious issues. Our primary ethical commitment as political scientists, therefore must be to get the theory and the empirical evidence right, and to clearly communicate those findings to relevant audiences—however unpalatable or inconclusive they might be.

That’s a manifesto of sorts, nested in a great post by Marc Lynch at the Monkey Cage. Marc’s post focuses on analysis of the Middle East, but everything he writes generalizes to the whole discipline.

I’ve written a couple of posts on this theme, too:

  • This Is Not a Drill,” on the challenges of doing what Marc proposes in the midst of fast-moving and politically charged events with weighty consequences; and
  • Advocascience,” on the ways that researchers’ political and moral commitments shape our analyses, sometimes but not always intentionally.

Putting all of those pieces together, I’d say that I wholeheartedly agree with Marc in principle, but I also believe this is extremely difficult to do in practice. We can—and, I think, should—aspire to this posture, but we can never quite achieve it.

That applies to forecasting, too, by the way. Coincidentally, I saw this great bit this morning in the Letter from the Editors for a new special issue of The Appendix, on “futures of the past”:

Prediction is a political act. Imagined futures can be powerful tools for social change, but they can also reproduce the injustices of the present.

Concern about this possibility played a role in my decision to leave my old job, helping to produce forecasts of political instability around the world for private consumption by the U.S. government. It is also part of what attracts me to my current work on a public early-warning system for mass atrocities. By making the same forecasts available to all comers, I hope that we can mitigate that downside risk in an area where the immorality of the acts being considered is unambiguous.

As a social scientist, though, I also understand that we’ll never know for sure what good or ill effects our individual and collective efforts had. We won’t know because we can’t observe the “control” worlds we would need to confidently establish cause and effect, and we won’t know because the world we seek to understand keeps changing, sometimes even in response to our own actions. This is the paradox at the core of applied, empirical social science, and it is inescapable.

Another Chicken Little Post on China

Last fall, I described what I saw as an “accumulating risk of crisis” in China. Recent developments in two parts of the country only reinforce my sense that the Communist Party of China (CPC) is entering a period during which it will find it increasingly hard to sustain its monopoly on state authority.

The first part of the country drawing fresh attention is Hong Kong, where pro-democracy activists have mobilized a new nonviolent challenge to the Party’s authority in spite of the center’s pointed efforts to discourage them. Organizing under the Occupy Central label, these activists recently held an unofficial referendum that drew nearly 800,000 voters who overwhelmingly endorsed proposals that would allow the public to nominate candidates for elections in 2017—an idea that Beijing has repeatedly and unequivocally rejected. Today, on 1 July, tens of thousands of people marched into the city’s center to press those same demands.

1 July 2014 rally in Hong Kong (AP via BBC News)

The 1 July rally looks set to be one of the island’s largest protests in years, and it comes only weeks after Beijing issued a white paper affirming its “comprehensive jurisdiction” over Hong Kong. Although the official line since the 1997 handover has been “one country, two systems,” the expectation has generally been that national leaders would only tolerate differences that didn’t directly challenge their authority, and the new white paper made that implicit policy a bit clearer. Apparently, though, many Hong Kong residents aren’t willing to leave that assertion unchallenged, and the resulting conflict is almost certain to persist into and beyond those 2017 elections, assuming Beijing doesn’t concede the point before then.

The second restive area is Xinjiang Uyghur Autonomous Region, where Uyghurs have agitated for greater autonomy or outright independence since the area’s incorporation into China in 1949. Over the past year or so, the pace of this conflict has intensified again.

The Chinese government describes this conflict as a fight against terrorism, and some of the recent attacks—see here and here, for example—have targeted and killed large numbers of civilians. As Assaf Moghadam argues in a recent blog post, however, the line between terrorism and insurgency is almost always blurry in practice. Terrorism and insurgency—and, for that matter, campaigns of nonviolent resistance—are all tactical variations on the theme of rebellion. In Xinjiang, we see evidence of a wider insurgency in recent attacks on police stations and security checkpoints, symbols of the “occupying power” and certainly not civilian targets. Some Uyghurs have also engaged in nonviolent protests, although when they have, the police have responded harshly.

In any case, the tactical variation and increased pace and intensity of the clashes leads me to believe that this conflict should now be described as a separatist rebellion, and that this rebellion now poses a significant challenge to the Communist Party. Uyghurs certainly aren’t going to storm the capital, and they are highly unlikely to win sovereignty or independence for Xinjiang as long as the CPC still rules. Nevertheless, the expanding rebellion is taxing the center, and it threatens to make Party leaders look less competent than they would like.

Neither of these conflicts is new, and the Party has weathered flare-ups in both regions before. What is new is their concurrence with each other and with a number of other serious political and economic challenges. As the conflicts in Xinjiang and Hong Kong intensify, China’s real-estate market finally appears to be cooling, with potentially significant effects on the country’s economy, and pollution remains a national crisis that continues to stir sporadic unrest among otherwise “ordinary” citizens. And, of course, Party leaders are simultaneously pursuing an anti-corruption campaign that is hitting higher and higher targets. This campaign is ostensibly intended to bolster the economy and to address popular frustration over abuses of power, but like any purge, it also risks generating fresh enemies.

For reasons Barbara Geddes helps to illuminate (here), consolidated single-party authoritarian regimes like China’s tend to be quite resilient. They persist because they usually do a good job suppressing domestic opponents and co-opting would-be rivals within the ruling party. Single-party regimes are better than others at co-opting internal rivals because, under all but exceptional circumstances, regime survival reliably generates better payoffs for all factions than the alternatives.

Eventually, though, even single-party regimes break down, and when they do, it’s usually in the face of an economic crisis that simultaneously stirs popular frustration and weakens incentives for elites to remain loyal (on this point, see Haggard and Kaufman, too). Exactly how these regimes come undone is a matter of local circumstance and historical accident, but generally speaking, the likelihood increases as popular agitation swells and the array of potential elite defectors widens.

China’s slowing growth rate and snowballing financial troubles indicate that the risk of an economic crisis is still increasing. At the same time, the crises in Hong Kong, Xinjiang, and the many cities and towns where citizens are repeatedly protesting against pollution and corruption suggest that insiders who choose to defect would have plenty of potential allies to choose from. As I’ve said before, I don’t believe that the CPC regime is on the brink of collapse, but I would be surprised to see it survive in its current form—with no legal opposition and direct elections in rural villages only—to and through the Party’s next National Congress, due in in 2017.

Scientific Updating as a Social Process

Cognitive theories predict that even experts cope with the complexities and ambiguities of world politics by resorting to theory-driven heuristics that allow them: (a) to make confident counterfactual inferences about what would have happened had history gone down a different path (plausible pasts); (b) to generate predictions about what might yet happen (probable futures); (c) to defend both counterfactual beliefs and conditional forecasts from potentially disconfirming data. An interrelated series of studies test these predictions by assessing correlations between ideological world view and beliefs about counterfactual histories (Studies 1 and 2), experimentally manipulating the results of hypothetical archival discoveries bearing on those counterfactual beliefs (Studies 3-5), and by exploring experts’ reactions to the confirmation or disconfirmation of conditional forecasts (Studies 6-12). The results revealed that experts neutralize dissonant data and preserve confidence in their prior assessments by resorting to a complex battery of belief-system defenses that, epistemologically defensible or not, make learning from history a slow process and defections from theoretical camps a rarity.

That’s the abstract to a 1999 AJPS paper by Phil Tetlock (emphasis added; ungated PDF here). Or, as Phil writes in the body of the paper,

The three sets of studies underscore how easy it is even for sophisticated professionals to slip into borderline tautological patterns of thinking about complex path-dependent systems that unfold once and only once. The risk of circularity is particularly pronounced when we examine reasoning about ideologically charged historical counterfactuals.

As noted in a recent post, ongoing debates over who “lost” Iraq or how direct U.S. military intervention in Syria might or might not have prevented wider war in the Middle East are current cases in point.

This morning, though, I’m intrigued by Phil’s point about the rarity of defections from theoretical camps tied to wider belief systems. If that’s right—and I have no reason to doubt that it is—then we should not put much faith in any one expert’s ability to update his or her scientific understanding in response to new information. In other words, we shouldn’t expect science to happen at the level of the individual. Instead, we should look wherever possible at the distribution of beliefs across a community of experts and hope that social cognition is more powerful than our individual minds are.

This evidence should also affect our thinking about how scientific change occurs. In The Structure of Scientific Revolutions, Thomas Kuhn (p. 19 in the 2nd Edition) described scientific revolutions as a process that happens at both the individual and social levels:

When, in the development of a natural science, an individual or group first produces a synthesis able to attract most of the next generation’s practitioners, the older schools gradually disappear. In part their disappearance is caused by their members’ conversion to the new paradigm. But there are always some men who cling to one or another of the older views, and they are simply read out of the profession, which thereafter ignores their work.

If I’m reading Tetlock’s paper right, though, then this description is only partly correct. In reality, scientists who are personally and professionally (or cognitively and emotionally?) invested in existing theories probably don’t convert to new ones very often. Instead, the recruitment mechanism Kuhn also mentions is probably the more relevant one. If we could reliably measure it, the churn rate associated with specific belief clusters would be a fascinating thing to watch.

Retooling

Over the next year, I plan to learn how to write code to do text mining.

I’m saying this out loud for two reasons. The first is self-centered; I see a public statement about my plans as a commitment device. By saying publicly that I plan to do this thing, I invest some of my credibility in following through, and my credibility is personally and professionally valuable to me.

I’m also saying this out loud, though, because I believe that the thinking behind this decision might interest other people working in my field. There are plenty of things I don’t know how to do that would be useful in my work on understanding and forecasting various forms of political instability. Three others that spring to mind are Bayesian data analysis, network theory, and agent-based modeling.

I’m choosing to focus on text mining instead of something else because I think that the single most significant obstacle to better empirical analysis in the social sciences is the scarcity of data, and I think that text mining is the most promising way out of this desert.

The volume of written and recorded text we produce on topics of interest to social scientists is incomprehensibly vast. Advances in computing technology and the growth of the World Wide Web have finally made it possible to access and analyze those texts—contemporary and historical—on a large scale with efficiency. This situation is still new, however, so most of this potential remains unrealized. There is a lot of unexplored territory on the other side of this frontier, and that territory is still growing faster than our ability to map it.

Lots of other people in political science and sociology are already doing text mining, and many of them are probably doing it better than I ever will.  One option would be to wait for their data sets to arrive and then work with them.

My own restlessness discourages me from following that strategy, but there’s also a principled reason not just to take what’s given: we do better analysis when we deeply understand where our data come from. The data sets you know the best are the ones you make. The data sets you know second-best are the ones someone else made with a process or instruments you’ve also used and understand. Either way, it behooves me to learn what these instruments are and how to apply them.

Instead of learning text mining, I could invest my time in learning other modeling and machine-learning techniques to analyze available data. My modeling repertoire is pretty narrow, and the array of options is only growing, so there’s plenty of room for improvement on that front, too.

In my experience, though, more complex models rarely add much to the inferential or predictive power we get from applying relatively simple models to the right data. This may not be true in every field, but it tends to be true in work on political stability and change, where the phenomena are so complex and still so poorly understood. On these topics, the best we can usually do is to find gross patterns that recur among data representing theoretically coherent processes or concepts.

Relatively simple models usually suffice to discover those gross patterns. What’s harder to come by are the requisite data. I think text mining is the most promising way to make them, so I am now going to learn how to do it.

From Thailand, Evidence of Coups’ Economic Costs

Last year, I used this space to report a statistical analysis I’d done on the impact of successful coups on countries’ economic growth rates. Bottom line: they usually hurt. As summarized in this chart from a post at FiveThirtyEight, my analysis indicated that, on average, coups dent a country’s economy by a couple of percentage points in the year they happen and another point the year after. Those are not trivial effects.

What makes this question so tricky to analyze is the possibility of a two-way relationship between these things. It’s not hard to imagine that coups might damage national economies, but it’s also likely that countries suffering slower growth are generally more susceptible to coups. With country-year data and no experimental controls, we can’t just run a model with growth as the dependent variable and the occurrence of a coup as a predictor and expect to get a reliable estimate of the former on the latter.

In my statistical analysis, I tried to deal with this problem by using coarsened exact matching to focus the comparison on sets of country-years with comparable coup risk in which coups did or did not happen. I believe the results are more informative than what we’d get from a pooled sample of all country-years, but they certainly aren’t the last word. After all, matching does not magically resolve deeper identification problems, even if it can help.

Under these circumstances, a little process-tracing can go a long way. If we look at real-world cases and see processes linking the “treatment” (coups) to the hypothesized effect (economic damage), we bolster our confidence that the effect we saw in our statistical analysis is not ephemeral.

Here, the recent coup in Thailand is serving up some intriguing evidence. In the past week, I have seen two reports  identifying specific ways in which the coup itself, and not the instability that preceded and arguably precipitated it, is damaging Thailand’s economy. First, I saw this Reuters story (emphasis mine):

Thai officials said on Tuesday that the mass departure of Cambodian laborers would dent the economy as thousands more migrant workers, fearing reprisals from the new military government, poured across the border.

Around 170,000 Cambodian workers have headed home in the past week, according to the International Organization for Migration (IOM), although the exodus is now slowing. Many left after hearing rumors that Thailand’s junta was bent on cracking down on illegal migrants.

Then I saw this tidbit in a Credit Suisse analysis shared on Twitter by Andrew Marshall (emphasis mine):

The coup and martial laws have produced stronger negative impact on Thai tourism, worsening the 2014 earnings outlook and could affect the magnitude of recovery anticipated for 2015…

Based on our conversations with [Airports of Thailand Plc], the coup…appears to have had a stronger impact on its international passenger volumes than the political conflicts… While the coup has restored peace in Bangkok and Thailand, and comforted the Thai people [!], we reckon tourists may take this more negatively and have chosen to go to other destinations.

In a country where international tourism contributes nearly 10 percent of the gross domestic product (here), that impact is a serious issue.

What’s important about both of these reports for the question at hand is the explicit connection they make between the occurrence of the coup and the economy-damaging behavior that followed. To me, these look like consequences rather than coincidences. Neither report definitively proves that the occurrence of a coups usually has an independent, negative effect on a country’s economic growth rate, of course. But they do make me more confident that the effect I saw in my statistical analysis is not just an artifact of some deeper forces I failed to consider.

Refugee Flows and Disorder in the Global System

This

The number of people displaced by violent conflict hit the highest level since World War II at the end of 2013, the head of the United Nations refugee agency, António Guterres, said in a report released on Friday…

Moreover, the impact of conflicts raging this year in Central African Republic, South Sudan, Ukraine and now Iraq threatens to push levels of displacement even higher by the end of 2014, he said.

…is, I think, another manifestation of the trends I discussed in a blog post here last September:

If we think on a systemic scale, it’s easier to see that we are now living through a period of global disorder matched in recent history only by the years surrounding the disintegration of the Soviet Union, and possibly exceeding it. Importantly, it’s not just the spate of state collapses through which this disorder becomes evident, but also the wider wave of protest activity and institutional transformation to which some of those collapses are connected.

If that’s true, then Mr. Guterres is probably right when he predicts that this will get even worse this year, because things still seem to be trending toward disorder. A lot of the transnational activity in response to local manifestations is still deliberately inflammatory (e.g., materiel and cash to rebels in Syria and Iraq, Russian support for separatists in Ukraine), and international efforts to quell some of those manifestations (e.g., UN PKOs in CAR and South Sudan) are struggling. Meanwhile, in what’s probably both a cause and an effect of these processes, global economic growth still has not rebounded as far or as fast as many had expected a year or two ago and remains uncertain and uneven.

In other words, the positive feedback still seems to be outrunning the negative feedback. Until that turns, the systemic processes driving (and being driven by) increased refugee flows will likely continue.

Addendum: The quote at the start of this post contains what I think is an error. A lot of the news stories on this report’s release used phrases like “displaced persons highest since World War II,” so I assumed that the U.N. report included the data on which that statement would be based. It turns out, though, that the report only makes a vague (and arguably misleading) reference to “the post-World War II era.” In fact, the U.N. does not have data to make comparisons on numbers of displaced persons prior to 1989. With the data it does have, the most the UNHCR can say is this, from p. 5: “The 2013 levels of forcible displacement were the highest since at least 1989, the first year that comprehensive statistics on global forced displacement existed.”

The picture also looks a little different from the press release if we adjust for increases in global population. Doing some rough math with the number of displaced persons in this UNHCR chart as the numerator and the U.S. Census Bureau’s mid-year estimates of world population as the denominator, here are some annual statistics on displaced persons as a share of the global population:

1989: 0.65%
1992: 0.84%
2010: 0.63%
2014: 0.72%

In no way do I mean to make light of what’s obviously a massive global problem, but as a share of the global population, the latest numbers are not (yet) even the worst since 1989, the first year for which UNHCR has comparable data.

A Brief Response to Anne-Marie Slaughter on Iraq and Syria

Anne-Marie Slaughter has an op-ed in today’s New York Times in which she argues that the U.S. government should launch air strikes now against targets in Iraq and Syria as a way to advance America’s and the world’s strategic and humanitarian interests. Here is the crux of the piece:

President Obama should be asking the same question in Iraq and Syria. What course of action will be best, in the short and the long term, for the Iraqi and Syrian people? What course of action will be most likely to stop the violence and misery they experience on a daily basis? What course of action will give them the best chance of peace, prosperity and a decent government?

The answer to those questions may well involve the use of force on a limited but immediate basis, in both countries. Enough force to remind all parties that we can, from the air, see and retaliate against not only Al Qaeda members, whom our drones track for months, but also any individuals guilty of mass atrocities and crimes against humanity. Enough force to compel governments and rebels alike to the negotiating table. And enough force to create a breathing space in which decent leaders can begin to consolidate power.

For the moment, let’s take for granted her assertions about the strategic interests at stake; the U.S.’s responsibility to protect civilians in other countries, by force if necessary; and the propriety of taking such action without prior approval from the U.N. Security Council.

Conceding all of that ground, it’s easier to see that, as a practical matter, Slaughter’s recommendation depends on strong assumptions about the efficacy of the action she proposes. Specifically, she asserts that the U.S. should conduct air strikes (“use of force on a limited but immediate basis,” “from the air”) against targets in Iraq and Syria because doing so will have three main effects:

  1. Deter atrocities (“to remind all parties that we can…see and retaliate against…any individuals guilty of mass atrocities and crimes against humanity”);
  2. Spur talks among warring parties (“to compel governments and rebels alike to the negotiating table”); and
  3. Enable positive political development (“to create a breathing space in which decent leaders can begin to consolidate power”)

If you believe, as Slaughter apparently does, that limited air strikes a) will almost certainly achieve all of these goals and b) will not produce other harmful strategic or humanitarian consequences that could partially offset or even outweigh those gains, then you should probably endorse this policy.

If, however, you are unsure about the ability of limited air strikes on yet-to-be-named targets in Iraq and Syria to accomplish these ends, or about the unintended strategic and humanitarian consequences those strikes could also have, then you should hesitate to support this policy and think through those other possible futures.

Beware the Confident Counterfactual

Did you anticipate the Syrian uprising that began in 2011? What about the Tunisian, Egyptian, and Libyan uprisings that preceded and arguably shaped it? Did you anticipate that Assad would survive the first three years of civil war there, or that Iraq’s civil war would wax again as intensely as it has in the past few days?

All of these events or outcomes were difficult forecasting problems before they occurred, and many observers have been frank about their own surprise at many of them. At the same time, many of those same observers speak with confidence about the causes of those events. The invasion of Iraq in 2003 surely is or is not the cause of the now-raging civil war in that country. The absence of direct US or NATO military intervention in Syria is or is not to blame for continuation of that country’s civil war and the mass atrocities it has brought—and, by extension, the resurgence of civil war in Iraq.

But here’s the thing: strong causal claims require some confidence about how history would have unfolded in the absence of the cause of interest, and those counterfactual histories are no easier to get right than observed history was to anticipate.

Like all of the most interesting questions, what causality means and how we might demonstrate it will forever be matters for debate—see here on Daniel Little’s blog for an overview of that debate’s recent state—but most conceptions revolve around some idea of necessity. When we say X caused Y, we usually mean that had X not occurred, Y wouldn’t have happened, either. Subtler or less stringent versions might center on salience instead of necessity and insert a “probably” into the final phrase of the previous sentence, but the core idea is the same.

In nonexperimental social science, this logic implicitly obliges us to consider the various ways history might have unfolded in response to X’ rather than X. In a sense, then, both prediction and explanation are forecasting problems. They require us to imagine states of the world we have not seen and to connect them in plausible ways to to ones we have. If anything, the counterfactual predictions required for explanation are more frustrating epistemological problems than the true forecasts, because we will never get to see the outcome(s) against which we could assess the accuracy of our guesses.

As Robert Jervis pointed out in his contribution to a 1996 edited volume on counterfactual thought experiments in world politics, counterfactuals are (or should be) especially hard to construct—and thus causal claims especially hard to make—when the causal processes of interest involve systems. For Jervis,

A system exists when elements or units are interconnected so that the system has emergent properties—i.e., its characteristics and behavior canot be inferred from the characteristics and behavior of the units taken individually—and when changes in one unit or the relationship between any two of them produce ramifying alterations in other units or relationships.

As Jervis notes,

A great deal of thinking about causation…is based on comparing two situations that are the same in all ways except one. Any differences in the outcome, whether actual or expected…can be attributed to the difference in the state of the one element…

Under many circumstances, this method is powerful and appropriate. But it runs into serious problems when we are dealing with systems because other things simply cannot be held constant: as Garret Hardin nicely puts it, in a system, ‘we can never do merely one thing.’

Jervis sketches a few thought experiments to drive this point home. He has a nice one about the effects of external interventions on civil wars that is topical here, but I think his New York traffic example is more resonant:

In everyday thought experiments we ask what would have happened if one element in our world had been different. Living in New York, I often hear people speculate that traffic would be unbearable (as opposed to merely terrible) had Robert Moses not built his highways, bridges, and tunnels. But to try to estimate what things would have been like, we cannot merely subtract these structures from today’s Manhattan landscape. The traffic patterns, the location of businesses and residences, and the number of private automobiles that are now on the streets are in significant measure the product of Moses’s road network. Had it not been built, or had it been built differently, many other things would have been different. Traffic might now be worse, but it is also possible that it would have been better because a more efficient public transportation system would have been developed or because the city would not have grown so large and prosperous without the highways.

Substitute “invade Iraq” or “fail to invade Syria” for Moses’s bridges and tunnels, and I hope you see what I mean.

In the end, it’s much harder to get beyond banal observations about influences to strong claims about causality than our story-telling minds and the popular media that cater to them would like. Of course the invasion of Iraq in 2003 or the absence of Western military intervention in Syria have shaped the histories that followed. But what would have happened in their absence—and, by implication, what would happen now if, for example, the US now re-inserted its armed forces into Iraq or attempted to topple Assad? Those questions are far tougher to answer, and we should beware of anyone who speaks with great confidence about their answers. If you’re a social scientist who isn’t comfortable making and confident in the accuracy of your predictions, you shouldn’t be comfortable making and confident in the validity of your causal claims, either.

There Is No Such Thing as Civil War

In a 2008 conference paper, Jim Fearon and David Laitin used statistics and case narratives to examine how civil wars around the world since 1955 have ended. They found that deadly fights between central governments and domestic challengers usually only end after an abrupt change in the relative fighting power of one side or the other, and that these abrupt changes are usually brought on by the beginning or end of foreign support. This pattern led them to ruminate thus (emphasis in original):

We were struck by the high frequency of militarily significant foreign support for government and rebels. The evidence suggests that more often than not, civil wars either become – or may even begin as –the object of other states’ foreign policies…Civil wars are normally studied as matters of domestic politics. Future research might make progress by shifting the perspective, and thinking about civil war as international politics by other means.

Their study recently came to mind when I was watching various people on Twitter object to the idea that what’s happening in Ukraine right now could be described as civil war, or at least the possible beginnings of one. Even if some of the separatists mobilizing in eastern Ukraine really were Ukrainian nationals, they argued, the agent provocateur was Russia, so this fight is properly understood as a foreign incursion.

As Jim and David’s paper shows, though, strong foreign hands are a common and often decisive feature of the fights we call civil wars.

In Syria, for example, numerous foreign governments and other external agents are funding, training, equipping, and arming various factions in the armed conflict that’s raged for nearly three years now. Some of that support is overt, but the support we see when we read about the war in the press is surely just a fraction of what’s actually happening. Yet we continue to see the conflict described as a civil war.

In the Central African Republic, it’s Chad that’s played “an ambiguous and powerful role” in the conflict that has precipitated state collapse and ethnic cleansing there. As the New York Times described in April,

[Chad] was accused of supporting the overthrow of the nation’s president, and then later helped remove the rebel who ousted him, making way for a new transitional government. In a statement on Thursday, the Chadian government said that its 850 soldiers had been accused of siding with Muslim militias in sectarian clashes with Christian fighters that have swept the Central African Republic for months.

At least a couple of bordering states are apparently involved in the civil war that’s stricken South Sudan since December. In a May 2014 report, the UN Mission to South Sudan asserted that government forces were receiving support from “armed groups from the Republic of Sudan,” and that “the Government has received support from the Uganda People’s Defence Force (UPDF), notably in Juba and Jonglei State.” The report also claimed that “some Darfuri militias have allied with opposition forces in the northern part of Unity State,” which borders Sudan. And, of course, there is a nearly 8,000-strong UN peacekeeping operation that is arguably shaping the scale of the violence there, even if it isn’t stopping it.

Pick a civil war—any civil war—and you’ll find similar evidence of external involvement. This is what led Jim and David to muse about civil wars as “international politics by other means,” and what led me to the deliberately provocative title of this post. As a researcher, I see analytic value in sometimes distinguishing between interstate and intrastate wars, which may have distinct causes and follow different patterns and may therefore be amenable to different forms of prevention or mitigation. At the same time, I think it’s clear that this distinction is nowhere near as crisp in reality as our labels imply, so we should be mindful to avoid confusing the typology with the reality it crudely describes.

A Useful Data Set on Political Violence that Almost No One Is Using

For the past 10 years, the CIA has overtly funded the production of a publicly available data set on certain atrocities around the world that now covers the period from January 1995 to early 2014 and is still updated on a regular basis. If you work in a relevant field but didn’t know that, you’re not alone.

The data set in question is the Political Instability Task Force’s Worldwide Atrocities Dataset, which records information from several international press sources about situations in which five or more civilians are deliberately killed in the context of some wider political conflict. Each record includes information about who did what to whom, where, and when, along with a brief text description of the event, a citation for the source article(s), and, where relevant, comments from the coder. The data are updated monthly, although those updates are posted on a four-month lag (e.g., data from January become available in May).

The decision to limit collection to events involving at least five fatalities was a pragmatic one. As the data set’s codebook notes,

We attempted at one point to lower this threshold to one and the data collection demands proved completely overwhelming, as this involved assessing every murder and ambiguous accidental death reported anywhere in the world in the international media. “Five” has no underlying theoretical justification; it merely provides a threshold above which we can confidently code all of the reported events given our available resources.

For the past three years, the data set has also fudged this rule to include targeted killings that appear to have a political motive, even when only a single victim is killed. So, for example, killings of lawyers, teachers, religious leaders, election workers, and medical personnel are nearly always recorded, and these events are distinguished from ones involving five or more victims by a “Yes” in a field identifying “Targeted Assassinations” under a “Related Tactics” header.

The data set is compiled from stories appearing in a handful of international press sources that are accessed through Factiva. It is a computer-assisted process. A Boolean keyword search is used to locate potentially relevant articles, and then human coders read those stories and make data from the ones that turn out actually to be relevant. From the beginning, the PITF data set has pulled from Reuters, Agence France Press, Associated Press, and the New York Times. Early in the process, BBC World Monitor and CNN were added to the roster, and All Africa was also added a few years ago to improve coverage of that region.

The decision to restrict collection to a relatively small number of sources was also a pragmatic one. Unlike GDELT, for example—the routine production of which is fully automated—the Atrocities Data Set is hand-coded by people reading news stories identified through a keyword search. With people doing the coding, the cost of broadening the search to local and web-based sources is prohibitive. The hope is eventually to automate the process, either as a standalone project or as part of a wider automated event data collection effort. As GDELT shows, though, that’s hard to do well, and that day hasn’t arrived yet.

Computer-assisted coding is far more labor intensive than fully automated coding, but it also carries some advantages. Human coders can still discern better than the best automated coding programs when numerous reports are all referring to the same event, so the PITF data set does a very good job eliminating duplicate records. Also, the “where” part of each record in the PITF data set includes geocoordinates, and its human coders can accurately resolve the location of nearly every event to at least the local administrative area, a task over which fully automated processes sometimes still stumble.

Of course, press reports only capture a fraction of all the atrocities that occur in most conflicts, and journalists writing about hard-to-cover conflicts often describe these situations with stories that summarize episodes of violence (e.g., “Since January, dozens of villagers have been killed…”). The PITF data set tries to accommodate this pattern by recording two distinct kinds of events: 1) incidents, which occur in a single place in short period of time, usually a single day; and 2) campaigns, which involve the same perpetrator and target group but may occur in multiple places over a longer period of time—usually days but sometimes weeks or months.

The inclusion of these campaigns alongside discrete events allows the data set to capture more information, but it also requires careful attention when using the results. Most statistical applications of data sets like this one involve cross-tabulations of events or deaths at a particular level during some period of time—say, countries and months. That’s relatively easy to do with data on discrete events located in specific places and days. Here, though, researchers have to decide ahead of time if and how they are going to blend information about the two event types. There are two basic options: 1) ignore the campaigns and focus exclusively on the incidents, treating that subset of the data set like a more traditional one and ignoring the additional information; or 2) make a convenient assumption about the distribution of the incidents of which campaigns are implicitly composed and apportion them accordingly.

For example, if we are trying to count monthly deaths from atrocities at the country level, we could assume that deaths from campaigns are distributed evenly over time and assign equal fractions of those deaths to all months over which they extend. So, a campaign in which 30 people were reportedly killed in Somalia between January and March would add 10 deaths to the monthly totals for that country in each of those three months. Alternatively, we could include all of the deaths from a campaign in the month or year in which it began. Either approach takes advantage of the additional information contained in those campaign records, but there is also a risk of double counting, as some of the events recorded as incidents might be part of the violence summarized in the campaign report.

It is also important to note that this data set does not record information about atrocities in which the United States is either the alleged perpetrator or the target (e.g., 9/11) of an atrocity because of legal restrictions on the activities of the CIA, which funds the data set’s production. This constraint presumably has a bigger impact on some cases, such as Iraq and Afghanistan, than others.

To provide a sense of what the data set contains and to make it easier for other researchers to use it, I wrote an R script that ingests and cross-tabulates the latest iteration of the data in country-month and country-year bins and then plots some of the results. That script is now posted on Github (here).

One way to see how well the data set is capturing the trends we hope it will capture is to compare the figures it produces with ones from data sets in which we already have some confidence. While I was writing this post, Colombian “data enthusiast” Miguel Olaya tweeted a pair of graphs summarizing data on massacres in that country’s long-running civil war. The data behind his graphs come from the Rutas de Conflicto project, an intensive and well-reputed effort to document as many as possible of the massacres that have occurred in Colombia since 1980. Here is a screenshot of Olaya’s graph of the annual death counts from massacres in the Rutas data set since 1995, when the PITF data pick up the story:

Annual Deaths from Massacres in Colombia by Perpetrator (Source: Rutas de Conflicta)

Annual Deaths from Massacres in Colombia by Perpetrator (Source: Rutas de Conflicta)

Now here is a graph of deaths from the incidents in the PITF data set:

deaths.yearly.colombia

Just eyeballing the two charts, the correlation looks pretty good. Both show a sharp increase in the tempo of killing in the mid-1990s; a sustained peak around 2000; a steady decline over the next several years; and a relatively low level of lethality since the mid-2000s. The annual counts from the Rutas data are two or three times larger than the ones from the PITF data during the high-intensity years, but that makes sense when we consider how much deeper of a search that project has conducted. There’s also a dip in the PITF totals in 1999 and 2000 that doesn’t appear in the Rutas data, but the comparisons over the larger span hold up. All things considered, this comparison makes the PITF data look quite good, I think.

Of course, the insurgency in Colombia has garnered better coverage from the international press than conflicts in parts of the world that are even harder to reach or less safe for correspondents than the Colombian highlands. On a couple of recent crises in exceptionally under-covered areas, the PITF data also seems to do a decent job capturing surges in violence, but only when we include campaigns as well as incidents in the counting.

The plots below show monthly death totals from a) incidents only and b) incidents and campaigns combined in the Central African Republic since 1995 and South Sudan since its independence in mid-2011. Here, deaths from campaigns have been assigned to the month in which the campaign reportedly began. In CAR, the data set identifies the upward trend in atrocities through 2013 and into 2014, but the real surge in violence that apparently began in late 2013 is only captured when we include campaigns in the cross-tabulation (the dotted line).

deaths.monthly.car

The same holds in South Sudan. There, the incident-level data available so far miss the explosion of civilian killings that began in December 2013 and reportedly continue, but the combination of campaign and incident data appears to capture a larger fraction of it, along with a notable spike in July 2013 related to clashes in Jonglei State.

deaths.monthly.southsudan

These examples suggest that the PITF Worldwide Atrocities Dataset is doing a good job at capturing trends over time in lethal violence against civilians, even in some of the hardest-to-cover cases. To my knowledge, though, this data set has not been widely used by researchers interested in atrocities or political violence more broadly. Probably its most prominent use to date was in the Model component of the Tech Challenge for Atrocities Prevention, a 2013 crowdsourced competition funded by USAID and Humanity United. That challenge produced some promising results, but it remains one of the few applications of this data set on a subject for which reliable data are scarce. Here’s hoping this post helps to rectify that.

Disclosure: I was employed by SAIC as research director of PITF from 2001 until 2011. During that time, I helped to develop the initial version of this data set and was involved in decisions to fund its continued production. Since 2011, however, I have not been involved in either the production of the data or decisions about its continued funding. I am part of a group that is trying to secure funding for a follow-on project to the Model part of the Tech Challenge for Atrocities Prevention, but that effort would not necessarily depend on this data set.

Follow

Get every new post delivered to your Inbox.

Join 6,640 other followers

%d bloggers like this: