A Useful Data Set on Political Violence that Almost No One Is Using

For the past 10 years, the CIA has overtly funded the production of a publicly available data set on certain atrocities around the world that now covers the period from January 1995 to early 2014 and is still updated on a regular basis. If you work in a relevant field but didn’t know that, you’re not alone.

The data set in question is the Political Instability Task Force’s Worldwide Atrocities Dataset, which records information from several international press sources about situations in which five or more civilians are deliberately killed in the context of some wider political conflict. Each record includes information about who did what to whom, where, and when, along with a brief text description of the event, a citation for the source article(s), and, where relevant, comments from the coder. The data are updated monthly, although those updates are posted on a four-month lag (e.g., data from January become available in May).

The decision to limit collection to events involving at least five fatalities was a pragmatic one. As the data set’s codebook notes,

We attempted at one point to lower this threshold to one and the data collection demands proved completely overwhelming, as this involved assessing every murder and ambiguous accidental death reported anywhere in the world in the international media. “Five” has no underlying theoretical justification; it merely provides a threshold above which we can confidently code all of the reported events given our available resources.

For the past three years, the data set has also fudged this rule to include targeted killings that appear to have a political motive, even when only a single victim is killed. So, for example, killings of lawyers, teachers, religious leaders, election workers, and medical personnel are nearly always recorded, and these events are distinguished from ones involving five or more victims by a “Yes” in a field identifying “Targeted Assassinations” under a “Related Tactics” header.

The data set is compiled from stories appearing in a handful of international press sources that are accessed through Factiva. It is a computer-assisted process. A Boolean keyword search is used to locate potentially relevant articles, and then human coders read those stories and make data from the ones that turn out actually to be relevant. From the beginning, the PITF data set has pulled from Reuters, Agence France Press, Associated Press, and the New York Times. Early in the process, BBC World Monitor and CNN were added to the roster, and All Africa was also added a few years ago to improve coverage of that region.

The decision to restrict collection to a relatively small number of sources was also a pragmatic one. Unlike GDELT, for example—the routine production of which is fully automated—the Atrocities Data Set is hand-coded by people reading news stories identified through a keyword search. With people doing the coding, the cost of broadening the search to local and web-based sources is prohibitive. The hope is eventually to automate the process, either as a standalone project or as part of a wider automated event data collection effort. As GDELT shows, though, that’s hard to do well, and that day hasn’t arrived yet.

Computer-assisted coding is far more labor intensive than fully automated coding, but it also carries some advantages. Human coders can still discern better than the best automated coding programs when numerous reports are all referring to the same event, so the PITF data set does a very good job eliminating duplicate records. Also, the “where” part of each record in the PITF data set includes geocoordinates, and its human coders can accurately resolve the location of nearly every event to at least the local administrative area, a task over which fully automated processes sometimes still stumble.

Of course, press reports only capture a fraction of all the atrocities that occur in most conflicts, and journalists writing about hard-to-cover conflicts often describe these situations with stories that summarize episodes of violence (e.g., “Since January, dozens of villagers have been killed…”). The PITF data set tries to accommodate this pattern by recording two distinct kinds of events: 1) incidents, which occur in a single place in short period of time, usually a single day; and 2) campaigns, which involve the same perpetrator and target group but may occur in multiple places over a longer period of time—usually days but sometimes weeks or months.

The inclusion of these campaigns alongside discrete events allows the data set to capture more information, but it also requires careful attention when using the results. Most statistical applications of data sets like this one involve cross-tabulations of events or deaths at a particular level during some period of time—say, countries and months. That’s relatively easy to do with data on discrete events located in specific places and days. Here, though, researchers have to decide ahead of time if and how they are going to blend information about the two event types. There are two basic options: 1) ignore the campaigns and focus exclusively on the incidents, treating that subset of the data set like a more traditional one and ignoring the additional information; or 2) make a convenient assumption about the distribution of the incidents of which campaigns are implicitly composed and apportion them accordingly.

For example, if we are trying to count monthly deaths from atrocities at the country level, we could assume that deaths from campaigns are distributed evenly over time and assign equal fractions of those deaths to all months over which they extend. So, a campaign in which 30 people were reportedly killed in Somalia between January and March would add 10 deaths to the monthly totals for that country in each of those three months. Alternatively, we could include all of the deaths from a campaign in the month or year in which it began. Either approach takes advantage of the additional information contained in those campaign records, but there is also a risk of double counting, as some of the events recorded as incidents might be part of the violence summarized in the campaign report.

It is also important to note that this data set does not record information about atrocities in which the United States is either the alleged perpetrator or the target (e.g., 9/11) of an atrocity because of legal restrictions on the activities of the CIA, which funds the data set’s production. This constraint presumably has a bigger impact on some cases, such as Iraq and Afghanistan, than others.

To provide a sense of what the data set contains and to make it easier for other researchers to use it, I wrote an R script that ingests and cross-tabulates the latest iteration of the data in country-month and country-year bins and then plots some of the results. That script is now posted on Github (here).

One way to see how well the data set is capturing the trends we hope it will capture is to compare the figures it produces with ones from data sets in which we already have some confidence. While I was writing this post, Colombian “data enthusiast” Miguel Olaya tweeted a pair of graphs summarizing data on massacres in that country’s long-running civil war. The data behind his graphs come from the Rutas de Conflicto project, an intensive and well-reputed effort to document as many as possible of the massacres that have occurred in Colombia since 1980. Here is a screenshot of Olaya’s graph of the annual death counts from massacres in the Rutas data set since 1995, when the PITF data pick up the story:

Annual Deaths from Massacres in Colombia by Perpetrator (Source: Rutas de Conflicta)

Annual Deaths from Massacres in Colombia by Perpetrator (Source: Rutas de Conflicta)

Now here is a graph of deaths from the incidents in the PITF data set:

deaths.yearly.colombia

Just eyeballing the two charts, the correlation looks pretty good. Both show a sharp increase in the tempo of killing in the mid-1990s; a sustained peak around 2000; a steady decline over the next several years; and a relatively low level of lethality since the mid-2000s. The annual counts from the Rutas data are two or three times larger than the ones from the PITF data during the high-intensity years, but that makes sense when we consider how much deeper of a search that project has conducted. There’s also a dip in the PITF totals in 1999 and 2000 that doesn’t appear in the Rutas data, but the comparisons over the larger span hold up. All things considered, this comparison makes the PITF data look quite good, I think.

Of course, the insurgency in Colombia has garnered better coverage from the international press than conflicts in parts of the world that are even harder to reach or less safe for correspondents than the Colombian highlands. On a couple of recent crises in exceptionally under-covered areas, the PITF data also seems to do a decent job capturing surges in violence, but only when we include campaigns as well as incidents in the counting.

The plots below show monthly death totals from a) incidents only and b) incidents and campaigns combined in the Central African Republic since 1995 and South Sudan since its independence in mid-2011. Here, deaths from campaigns have been assigned to the month in which the campaign reportedly began. In CAR, the data set identifies the upward trend in atrocities through 2013 and into 2014, but the real surge in violence that apparently began in late 2013 is only captured when we include campaigns in the cross-tabulation (the dotted line).

deaths.monthly.car

The same holds in South Sudan. There, the incident-level data available so far miss the explosion of civilian killings that began in December 2013 and reportedly continue, but the combination of campaign and incident data appears to capture a larger fraction of it, along with a notable spike in July 2013 related to clashes in Jonglei State.

deaths.monthly.southsudan

These examples suggest that the PITF Worldwide Atrocities Dataset is doing a good job at capturing trends over time in lethal violence against civilians, even in some of the hardest-to-cover cases. To my knowledge, though, this data set has not been widely used by researchers interested in atrocities or political violence more broadly. Probably its most prominent use to date was in the Model component of the Tech Challenge for Atrocities Prevention, a 2013 crowdsourced competition funded by USAID and Humanity United. That challenge produced some promising results, but it remains one of the few applications of this data set on a subject for which reliable data are scarce. Here’s hoping this post helps to rectify that.

Disclosure: I was employed by SAIC as research director of PITF from 2001 until 2011. During that time, I helped to develop the initial version of this data set and was involved in decisions to fund its continued production. Since 2011, however, I have not been involved in either the production of the data or decisions about its continued funding. I am part of a group that is trying to secure funding for a follow-on project to the Model part of the Tech Challenge for Atrocities Prevention, but that effort would not necessarily depend on this data set.

Introducing A New Venue for Atrocities Early Warning

Starting today, the bits of this blog on forecasting and monitoring mass atrocities are moving to their proper home, or at least the initial makings of it. Say hi to the (interim) blog of the Early Warning Project.

Since 2012, I have been working as a consultant to the U.S. Holocaust Memorial Museum’s Center for the Prevention of Genocide (CPG) to help build a new global early-warning system for mass atrocities. As usual, that process is taking longer than we had expected. We now have working versions of the project’s two main forecasting streams—statistical risk assessments and a “wisdom of (expert) crowds” system called an opinion pool—and CPG has hired a full-time staffer (hi, Ali) to manage their day-to-day workings. Unfortunately, though, the web site that will present, discuss, and invite discussion of those forecasts is still under construction. Thanks to Dartmouth’s DALI Lab, we’ve got a great prototype, but there’s finishing work to be done, and doing it takes a while.

Well, delays, be damned. We think the content we’re producing is useful now, so we’re not waiting for that site to get finished to start sharing it. Instead, we’re launching this interim blog to go ahead and start doing things like:

When the project’s full-blown web site finally goes up, it will feature a blog, too, and all of the content from this interim venue will migrate there. Until then, if you’re interested in atrocities early warning and prevention—or applied forecasting more generally—please come see what we’re doing, share what you find interesting, and help us think about how to do it even better.

Meanwhile, Dart-Throwing Chimp will keep plugging along on its core themes of democratization, political instability, and forecasting. If you’ve got the interest and the bandwidth, I hope you’ll find time to watch and engage with both channels.

Early Results from a New Atrocities Early Warning System

For the past couple of years, I have been working as a consultant to the U.S. Holocaust Memorial Museum’s Center for the Prevention of Genocide to help build a new early-warning system for mass atrocities around the world. Six months ago, we started running the second of our two major forecasting streams, a “wisdom of (expert) crowds” platform that aggregates probabilistic forecasts from a pool of topical and area experts on potential events of concern. (See this conference paper for more detail.)

The chart below summarizes the output from that platform on most of the questions we’ve asked so far about potential new episodes of mass killing before 2015. For our early-warning system, we define a mass killing as an episode of sustained violence in which at least 1,000 noncombatant civilians from a discrete group are intentionally killed, usually in a period of a year or less. Each line in the chart shows change over time in the daily average of the inputs from all of the participants who choose to make a forecast on that question. In other words, the line is a mathematical summary of the wisdom of our assembled crowd—now numbering nearly 100—on the risk of a mass killing beginning in each case before the end of 2014. Also:

  • Some of the lines (e.g., South Sudan, Iraq, Pakistan) start further to the right than others because we did not ask about those cases when the system launched but instead added them later, as we continue to do.
  • Two lines—Central African Republic and South Sudan—end early because we saw onsets of mass-killing episodes in those countries. The asterisks indicate the dates on which we made those declarations and therefore closed the relevant questions.
  • Most but not all of these questions ask specifically about state-led mass killings, and some focus on specific target groups (e.g., the Rohingya in Burma) or geographic regions (the North Caucasus in Russia) as indicated.
Crowd-Estimated Probabilities of Mass-Killing Onset Before 1 January 2015

Crowd-Estimated Probabilities of Mass-Killing Onset Before 1 January 2015

I look at that chart and conclude that this process is working reasonably well so far. In the six months since we started running this system, the two countries that have seen onsets of mass killing are both ones that our forecasters promptly and consistently put on the high side of 50 percent. Nearly all of the other cases, where mass killings haven’t yet occurred this year, have stuck on the low end of the scale.

I’m also gratified to see that the system is already generating the kind of dynamic output we’d hoped it would, even with fewer than 100 forecasters in the pool. In the past several weeks, the forecasts for both Burma and Iraq have risen sharply, apparently in response to shifts in relevant policies in the former and the escalation of the civil war in the latter. Meanwhile, the forecast for Uighurs in China has risen steadily over the year as a separatist rebellion in Xinjiang Province has escalated and, with it, concerns about a harsh government response. These inflection points and trends can help identify changes in risk that warrant attention from organizations and individuals concerned about preventing or mitigating these potential atrocities.

Finally, I’m also intrigued to see that our opinion pool seems to be sorting cases into a few clusters that could be construed as distinct tiers of concern. Here’s what I have in mind:

  • Above the 50-percent threshold are the high risk cases, where forecasters assess that mass killing is likely to occur during the specified time frame.  These cases won’t necessarily be surprising. Some observers had been warning on the risk of mass atrocities in CAR and South Sudan for months before those episodes began, and the plight of the Rohingya in Burma has been a focal point for many advocacy groups in the past year. Even in supposedly “obvious” cases, however, this system can help by providing a sharper estimate of that risk and giving a sense of how it is trending over time. In the case of Burma, for example, it is the separation that has happened in the last several weeks that tells the story of a switch from possible to likely and thus adds a degree of urgency to that warning.
  • A little farther down the y-axis are the moderate risk cases—ones that probably won’t suffer mass killing during the period in question but could more readily tip in that direction. In the chart above, Iraq, Sudan, Pakistan, Bangladesh, and Burundi all land in this tier, although Iraq now appears to be sliding into the high risk group.
  • Clustered toward the bottom are the low risk cases where the forecasters seem fairly confident that mass killing will not occur in the near future. In the chart above, Russia, Afghanistan, and Ethiopia are the cases that land firmly in this set. China (Uighurs) remains closer to them than the moderate risk tier, but it appears to be creeping toward the moderate-risk group. We are also running a question about the risk of state-led mass killing in Rwanda before 2015, and it currently lands in this tier, with a forecast of 14 percent.

The system that generates the data behind this chart is password protected, but the point of our project is to make these kinds of forecasts freely available to the global public. We are currently building the web site that will display the forecasts from this opinion pool in real time to all comers and hope to have it ready this fall.

In the meantime, if you think you have relevant knowledge or expertise—maybe you study or work on this topic, or maybe you live or work in parts of the world where risks tend to be higher—and are interested in volunteering as a forecaster, please send an email to us at ewp@ushmm.org.

Alarmed By Iraq

Iraq’s long-running civil war has spread and intensified again over the past year, and the government’s fight against a swelling Sunni insurgency now threatens to devolve into the sort of indiscriminate reprisals that could produce a new episode of state-led mass killing there.

The idea that Iraq could suffer a new wave of mass atrocities at the hands of state security forces or sectarian militias collaborating with them is not far fetched. According to statistical risk assessments produced for our atrocities early-warning project (here), Iraq is one of the 10 countries worldwide most susceptible to an onset of state-led mass killing, bracketed by places like Syria, Sudan, and the Central African Republic where large-scale atrocities and even genocide are already underway.

Of course, Iraq is already suffering mass atrocities of its own at the hands of insurgent groups who routinely kill large numbers of civilians in indiscriminate attacks, every one of which would stun American or European publics if it happened there. According to the widely respected Iraq Body Count project, the pace of civilian killings in Iraq accelerated sharply in July 2013 after a several-year lull of sorts in which “only” a few hundred civilians were dying from violence each month. Since the middle of last year, the civilian toll has averaged more than 1,000 fatalities per month. That’s well off the pace of 2006-2007, the peak period of civilian casualties under Coalition occupation, but it’s still an astonishing level of violence.

Monthly Counts of Civilian Deaths from Violence in Iraq (Source: Iraq Body Count)

Monthly Counts of Civilian Deaths from Violence in Iraq (Source: Iraq Body Count)

What seems to be increasing now is the risk of additional atrocities perpetrated by the very government that is supposed to be securing civilians against those kinds of attacks. A Sunni insurgency is gaining steam, and the government, in turn, is ratcheting up its efforts to quash the growing threat to its power in worrisome ways. A recent Reuters story summarized the current situation:

In Buhriz and other villages and towns encircling the capital, a pitched battle is underway between the emboldened Islamic State of Iraq and the Levant, the extremist Sunni group that has led a brutal insurgency around Baghdad for more than a year, and Iraqi security forces, who in recent months have employed Shi’ite militias as shock troops.

And this anecdote from the same Reuters story shows how that battle is sometimes playing out:

The Sunni militants who seized the riverside town of Buhriz late last month stayed for several hours. The next morning, after the Sunnis had left, Iraqi security forces and dozens of Shi’ite militia fighters arrived and marched from home to home in search of insurgents and sympathizers in this rural community, dotted by date palms and orange groves.

According to accounts by Shi’ite tribal leaders, two eyewitnesses and politicians, what happened next was brutal.

“There were men in civilian clothes on motorcycles shouting ‘Ali is on your side’,” one man said, referring to a key figure in Shi’ite tradition. “People started fleeing their homes, leaving behind the elders and young men and those who refused to leave. The militias then stormed the houses. They pulled out the young men and summarily executed them.”

Sadly, this escalatory spiral of indiscriminate violence is not uncommon in civil wars. Ben Valentino, a collaborator of mine in the development of this atrocities early-warning project, has written extensively on this topic (see especially here , here, and here). As Ben explained to me via email,

The relationship between counter-insurgency and mass violence against civilians is one of the most well-established findings in the social science literature on political violence. Not all counter-insurgency campaigns lead to mass killing, but when insurgent groups become large and effective enough to seriously threaten the government’s hold on power and when the rebels draw predominantly on local civilians for support, the risks of mass killing are very high. Usually, large-scale violence against civilians is neither the first nor the only tactic that governments use to defeat insurgencies. They may try to focus operations primarily against armed insurgents, or even offer positive incentives to civilians who collaborate with the government. But when less violent methods fail, the temptation to target civilians in the effort to defeat the rebels increases.

Right now, it’s hard to see what’s going to halt or reverse this trend in Iraq. “Things can get much worse from where we are, and more than likely they will,” Daniel Serwer told IRIN News for a story on Iraq’s escalating conflict (here). Other observers quoted in the same story seemed to think that conflict fatigue would keep the conflict from ballooning further, but that hope is hard to square with the escalation of violence that has already occurred over the past year and the fact that Iraq’s civil war never really ended.

In theory, elections are supposed to be a brake on this process, giving rival factions opportunities to compete for power and influence state policy in nonviolent ways. In practice, this often isn’t the case. Instead, Iraq appears to be following the more conventional path in which election winners focus on consolidating their own power instead of governing well, and excluded factions seek other means to advance their interests. Here’s part of how the New York Times set the scene for this week’s elections, which incumbent prime minister Nouri al-Maliki’s coalition is apparently struggling to win:

American intelligence assessments have found that Mr. Maliki’s re-election could increase sectarian tensions and even raise the odds of a civil war, citing his accumulation of power, his failure to compromise with other Iraqi factions—Sunni or Kurd—and his military failures against Islamic extremists. On his watch, Iraq’s American-trained military has been accused by rights groups of serious abuses as it cracks down on militants and opponents of Mr. Maliki’s government, including torture, indiscriminate roundups of Sunnis and demands of bribes to release detainees.

Because Iraq ranked so high in our last statistical risk assessments, we posted a question about it a few months ago on our “wisdom of (expert) crowds” forecasting system. Our pool of forecasters is still relatively small—89 as I write this—but the ones who have weighed in on this topic so far have put it in what I see as a middle tier of concern, where the risk is seen as substantial but not imminent or inevitable. Since January, the pool’s estimated probability of an onset of state-led mass killing in Iraq in 2014 has hovered around 20 percent, alongside countries like Pakistan (23 percent), Bangladesh (20 percent), and Burundi (19 percent) but well behind South Sudan (above 80 percent since December) and Myanmar (43 percent for the risk of a mass killing targeting the Rohingya in particular).

Notably, though, the estimate for Iraq has ticked up a few notches in the past few days to 27 percent as forecasters (including me) have read and discussed some of the pre-election reports mentioned here. I think we are on to something that deserves more scrutiny than it appears to be getting.

The Rwanda Enigma

For analysts and advocates trying to assess risks of future mass atrocities in hopes of preventing them, Rwanda presents an unusual puzzle. Most of the time, specialists in this field readily agree on which countries are especially susceptible to genocide or mass killing, either because those countries are either already experiencing large-scale civil conflict or because they are widely considered susceptible to it. Meanwhile, countries that sustain long episodes of peace and steadily grow their economies are generally presumed to have reduced their risk and eventually to have escaped this trap for good.

Contemporary Rwanda is puzzling because it provokes a polarized reaction. Many observers laud Rwanda as one of Africa’s greatest developmental successes, but others warn that it remains dangerously prone to mass atrocities. In a recent essay for African Arguments on how the Rwandan genocide changed the world, Omar McDoom nicely encapsulates this unusual duality:

What has changed inside Rwanda itself since the genocide? The country has enjoyed a remarkable period of social stability. There has not been a serious incident of ethnic violence in Rwanda for nearly two decades. Donors have praised the country’s astonishing development.  Economic growth has averaged over 6% per year, poverty and inequality have declined, child and maternal mortality have improved, and primary education is now universal and free. Rwanda has shown, in defiance of expectations, that an African state can deliver security, public services, and rising prosperity.

Yet, politically, there is some troubling continuity with pre-genocide Rwanda. Power remains concentrated in the hands of a small, powerful ethnic elite led by a charismatic individual with authoritarian tendencies. In form, current president Paul Kagame and his ruling party, the RPF, the heroes who ended the genocide, appear to exercise power in a manner similar to former president Juvenal Habyarimana and his ruling MRND party, the actors closely-tied to those who planned the slaughter. The genocide is testament to what unconstrained power over Rwanda’s unusually efficient state machinery can enable.

That duality also emerges from a comparison of two recent quantitative rankings. On the one hand, The World Bank now ranks Rwanda 32nd on the latest edition of its “ease of doing business” index—not 32nd in Africa, but 32nd of 189 countries worldwide. On the other hand, statistical assessments of the risk of an onset of state-led mass killing identify Rwanda as one of the 25 countries worldwide currently most vulnerable to this kind of catastrophe.

How can both of these things be true? To answer that question, we need to have a clearer sense of where that statistical risk assessment comes from. The number that ranks Rwanda among the 25 countries most susceptible to state-led mass killing is actually an average of forecasts from three models representing a few different ideas about the origins of mass atrocities, all applied to publicly available data from widely used sources.

  • Drawing on work by Barbara Harff and the Political Instability Task Force, the first model emphasizes features of countries’ national politics that hint at a predilection to commit genocide or “politicide,” especially in the context of political instability. Key risk factors in Harff’s model include authoritarian rule, the political salience of elite ethnicity, evidence of an exclusionary elite ideology, and international isolation as measured by trade openness.
  • The second model takes a more instrumental view of mass killing. It uses statistical forecasts of future coup attempts and new civil wars as proxy measures of things that could either spur incumbent rulers to lash out against threats to their power or usher in an insecure new regime that might do the same.
  • The third model is really not a model but a machine-learning process called Random Forests applied to the risk factors identified by the other two. The resulting algorithm is an amalgamation of theory and induction that takes experts’ beliefs about the origins of mass killing as its jumping-off point but also leaves more room for inductive discovery of contingent effects.

All of these models are estimated from historical data that compares cases where state-led mass killings occurred to ones where they didn’t. In essence, we look to the past to identify patterns that will help us spot cases at high risk of mass killing now and in the future. To get our single-best risk assessment—the number that puts Rwanda in the top (or bottom) 25 worldwide—we simply average the forecasts from these three models. We prefer the average to a single model’s output because we know from work in many fields—including meteorology and elections forecasting—that this “ensemble” approach generally produces more accurate assessments than we could expect to get from any one model alone. By combining forecasts, we learn from all three perspectives and hedge against the biases of any one of them.

Rwanda lands in the top 25 worldwide because all three models identify it as a relatively high-risk case. It ranks 15th on the PITF/Harff model, 28th on the “elite threat” model, and 30th on the Random Forest. The PITF/Harff model sees a relatively low risk in Rwanda of the kinds of political instability that typically trigger onsets of genocide or politicide, but it also pegs Rwanda as the kind of regime most likely to resort to mass atrocities if instability were to occur—namely, an autocracy in which elites’ ethnicity is politically salient in a country with a recent history of genocide. Rwanda also scores fairly high on the “elite threat” model because, according to our models of these things, it is at relatively high risk of a new insurgency and moderate risk of a coup attempt. Finally, the Random Forest sees a very low probability of mass killing onset in Rwanda but still pegs it as a riskier case than most.

Our identification of Rwanda as a relatively high-risk case is echoed by some, but not all, of the other occasional global assessments of countries’ susceptibility to mass atrocities. In her own applications of her genocide/politicide model for the task of early warning, Barbara Harff pegged Rwanda as one of the world’s riskiest cases in 2011 but not in 2013. Similarly, the last update of Genocide Watch’s Countries at Risk Report, in 2012, lists Rwanda as one of more than a dozen countries at stage five of seven on the path to genocide, putting it among the 35 countries worldwide at greatest risk. By contrast, the Global Centre for the Responsibility to Protect has not identified Rwanda as a situation of concern in any of its R2P Monitor reports to date, and the Sentinel Project for Genocide Prevention does not list Rwanda among its situations of concern, either. Meanwhile, recent reporting on Rwanda from Human Rights Watch has focused mostly on the pursuit of justice for the 1994 genocide and other kinds of human-rights violations in contemporary Rwanda.

To see what our own pool of experts makes of our statistical risk assessment and to track changes in their views over time, we plan to add a question to our “wisdom of (expert) crowds” forecasting system asking about the prospect of a new state-led mass killing in Rwanda before 2015. If one does not happen, as we hope and expect will be the case, we plan to re-launch the question at the start of next year and will continue to do so as long as our statistical models keep identifying it as a case of concern.

In the meantime, I thought it would be useful to ask a few country experts what they make of this assessment and how a return to mass killing in Rwanda might come about. Some were reluctant to speak on the record, and understandably so. The present government of Rwanda has a history of intimidating individuals it perceives as its critics. As Michaela Wrong describes in a recent piece for Foreign Policy,

A U.S. State Department spokesperson said in mid-January, “We are troubled by the succession of what appear to be politically motivated murders of prominent Rwandan exiles. President Kagame’s recent statements about, quote, ‘consequences’ for those who betray Rwanda are of deep concern to us.”

It is a pattern that suggests the Rwandan government may have come to see the violent silencing of critics—irrespective of geographical location and host country—as a beleaguered country’s prerogative.

Despite these constraints, the impression I get from talking to some experts and reading the work of others is that our risk assessment strikes nearly all of them as plausible. None said that he or she expects an episode of state-led mass killing to begin soon in Rwanda. Consistent with the thinking behind our statistical models, though, many seem to believe that another mass killing could occur in Rwanda, and if one did, it would almost certainly come in reaction to some other rupture in that country’s political stability.

Filip Reyntjens, a professor at the University of Antwerpen who wrote a book on Rwandan politics since the 1994 genocide, was both the most forthright and the most pessimistic in his assessment. Via email, he described Rwanda as

A volcano waiting to erupt. Nearly all field research during the last 15 years points at pervasive structural violence that may, as we know, become physical, acute violence following a trigger. I don’t know what that trigger will be, but I think a palace revolution or a coup d’etat is the most likely scenario. That may create a situation difficult to control.

In a recent essay for Juncture that was adapted for the Huffington Post (here), Phil Clark sounds more optimistic than Reyntjens, but he is not entirely sanguine, either. Clark sees the structure and culture of the country’s ruling party, the Rwandan Patriotic Front (RPF), as the seminal feature of Rwandan politics since the genocide and describes it as a double-edged sword. On the one hand, the RPF’s cohesiveness and dedication to purpose has enabled it, with help from an international community with a guilty conscience, to make “enormous” developmental gains. On the other hand,

The RPF’s desire for internal cohesion has made it suspicious of critical voices within and outside of the party—a feature compounded by Rwanda’s fraught experience of multi-party democracy in the early 1990s, which saw the rise of ethnically driven extremist parties and helped to create an environment conducive to genocide. The RPF’s singular focus on rebuilding the nation and facilitating the return of refugees means it has often viewed dissent as an unaffordable distraction. The disastrous dalliance with multipartyism before the genocide has only added to the deep suspicion of policy based on the open contestation of ideas.

Looking ahead, Clark wonders what happens when that intolerance for dissent bumps up against popular frustrations, as it probably will at some point:

For the moment, there are few signs of large-scale popular discontent with the closed political space. However, any substantial decline in socio-economic conditions in the countryside will challenge this. The RPF’s gamble appears to be that the population will tolerate a lack of national political contestation provided domestic stability and basic living standards are maintained. For now, the RPF seems to have rightly judged the popular mood but that situation may not hold.

Journalist Kris Berwouts portrays similarly ambiguous terrain in a recent piece for the Dutch magazine Mo that also appeared on the blog African Arguments (here). Berwouts quotes David Himbara, a former Rwandan regime insider who left the country in 2010 and has vocally criticized the Kagame government ever since, as telling him that “all society has vanished from Rwanda, mistrust is complete. It has turned Rwanda into a time bomb.” But Berwouts juxtaposes that dire assessment with the cautiously optimistic view of Belgian journalist Marc Hoogsteyns, who has worked in the region for years and has family ties by marriage to its Tutsi community. According to Hoogsteyns,

Rwanda is a beautiful country with many strengths and opportunities, but at the same time it is some kind of African version of Brave New World. People are afraid to talk. But they live more comfortably and safely than ever before, they enjoy high quality education and health care. They are very happy with that. The Tutsi community stands almost entirely behind Kagame and also most Hutu can live with it. They obviously don’t like the fact that they do not count on the political scene, but they can do what they want in all other spheres of live. They can study and do business etcetera. They can deal with the level of repression, because they know that countries such as Burundi, Congo or Kenya are not the slightest bit more democratic. Honestly, if we would have known twenty years ago, just after the genocide, that Rwanda would achieve this in two decades, we would have signed for it immediately.

As people of a certain age in places like Sarajevo or Bamako might testify, though, stability is a funny thing. It’s there until it isn’t, and when it goes, it sometimes goes quickly. In this sense, the political crises that sometimes produce mass killings are more like earthquakes than elections. We can spot the vulnerable structures fairly accurately, but we’re still not very good at anticipating the timing and dynamics of ruptures in them.

In the spirit of that last point, it’s important to acknowledge that the statistical assessment of Rwanda’s risk to mass killing is a blunt piece of information. Although it does specifically indicate a susceptibility to atrocities perpetrated by state security forces or groups acting at their behest, it does not necessarily implicate the RPF as the likely perpetrators. The qualitative assessments discussed above suggest that some experts find that scenario plausible, but it isn’t the only one consistent with our statistical finding. A new regime brought to power by coup or revolution could also become the agent of a new wave of mass atrocities in Rwanda, and the statistical forecast would be just as accurate.

Egypt’s recent past offers a case in point. Our statistical assessments of susceptibility to state-led mass killing in early 2013 identified Egypt as a relatively high-risk case, like Rwanda now. At the time, Mohammed Morsi was president, and one plausible interpretation of that risk assessment might have centered on the threat the Muslim Brotherhood’s supporters posed to Egypt’s Coptic Christians. Fast forward to July 2013, and the mass killing we ended up seeing in Egypt came at the hands of an army and police who snatched power away from Morsi and the Brotherhood and then proceeded to kill hundreds of their unarmed sympathizers. That outcome doesn’t imply that Coptic Christians weren’t at grave risk before the coup, but it should remind us to consider a variety of ways these systemic risks might become manifest.

Still, after conversations with a convenience sample of regional experts, I am left with the impression that the risk our statistical models identify of a new state-led mass killing in Rwanda is real, and that it is possible to imagine the ruling RPF as the agents of such violence.

No one seems to expect the regime to engage in mass violence without provocation, but the possibility of a new Hutu insurgency, and the state’s likely reaction to it, emerged from those conversations as perhaps the most likely scenario. According to some of the experts with whom I spoke, many Rwandan Hutus are growing increasingly frustrated with the RPF regime, and some radical elements of the Hutu diaspora appear to be looking for ways to take up that mantle. The presence of an insurgency is the single most-powerful predictor of state-led mass killing, and it does not seem far fetched to imagine the RPF regime using “scorched earth” tactics in response to the threat or occurrence of attacks on its soldiers and Tutsi citizens. After all, this is the same regime whose soldiers pursued Hutu refugees into Zaire in the mid-1990s and, according to a 2010 U.N. report, participated in the killings of tens of thousands of civilians in war crimes that were arguably genocidal.

Last but not least, we can observe that Rwanda has suffered episodes of mass killing roughly once per generation since independence—in the early 1960s, in 1974, and again in the early 1990s, culminating in the genocide of 1994 and the reprisal killings that followed. History certainly isn’t destiny, but our statistical models confirm that in the case of mass atrocities, it often rhymes.

It saddens me to write this piece about a country that just marked the twentieth anniversary of one of the most lethal genocides since the Holocaust, but the point of our statistical modeling is to see what the data say that our mental models and emotional assessments might overlook. A reprisal of mass killing in Rwanda would be horribly tragic. As Free Africa Foundation president George Ayittey wrote in a recent letter of the Wall Street Journal, however, “The real tragedy of Rwanda is that Mr. Kagame is so consumed by the 1994 genocide that, in his attempt to prevent another one, he is creating the very conditions that led to it.”

Whither Organized Violence?

The Human Security Research Group has just published the latest in its series of now-annual reports on “trends in organized violence around the world,” and it’s essential reading for anyone deeply interested in armed conflict and other forms of political violence. You can find the PDF here.

The 2013 edition takes Steven Pinker’s Better Angels as its muse and largely concurs with Pinker’s conclusions. I’ll sheepishly admit that I haven’t read Pinker’s book (yet), so I’m not going to engage directly in that debate. Instead, I’ll call attention to what the report’s authors infer from their research about future trends in political violence. Here’s how that bit starts, on p. 18:

The most encouraging data from the modern era come from the post–World War II years. This period includes the dramatic decline in the number and deadliness of international wars since the end of World War II and the reversal of the decades-long increase in civil war numbers that followed the end of the Cold War in the early 1990s.

What are the chances that these positive changes will be sustained? No one really knows. There are too many future unknowns to make predictions with any degree of confidence.

On that point, political scientist Bear Braumoeller would agree. In an interview last year for Popular Science (here), Kelsey Atherton asked Braumoeller about Braumoeller’s assertion in a recent paper (here) that it will take 150 years to know if the downward trend in warfare that Pinker and others have identified is holding. Braumoeller replied:

Some of this literature points to “the long peace” of post-World War II. Obviously we haven’t stopped fighting wars entirely, so what they’re referring to is the absence of really really big wars like World War I and World War II. Those wars would have to be absent for like 70 to 75 more years for us to have confidence that there’s been a change in the baseline rate of really really big wars.

That’s sort of a separate question from how we know whether there are trends in warfare in general. We need to understand that war and peace are both stochastic processes. We need a big enough sample to rule out the historical average, which is about one or two big wars per century. We just haven’t had enough time since World War I and World War II to rule out the possibility that nothing’s changed.

I suspect that the authors of the Human Security Report would not dispute that claim, but after carefully reviewing Pinker’s and their own evidence, they do see causes for cautious optimism. Here I’ll quote at length, because I think it’s important to see the full array of forces taken into consideration to increase our confidence in the validity of the authors’ cautious speculations.

The case for pessimism about the global security future is well rehearsed and has considerable support within the research community. Major sources of concern include the possibility of outbreaks of nuclear terrorism, a massive transnational upsurge of lethal Islamist radicalism, or wars triggered by mass droughts and population movements driven by climate change.

Pinker notes reasons for concern about each of these potential future threats but also skepticism about the more extreme claims of the conflict pessimists. Other possible drivers of global violence include the political crises that could follow the collapse of the international financial system and destabilizing shifts in the global balance of economic and military power—the latter being a major concern of realist scholars worried about the economic and military rise of China.

But focusing exclusively on factors and processes that may increase the risks of large-scale violence around the world, while ignoring those that decrease it, also almost certainly leads to unduly pessimistic conclusions.

In the current era, factors and processes that reduce the risks of violence not only include the enduring impact of the long-term trends identified in Better Angels but also the disappearance of two major drivers of warfare in the post–World War II period—colonialism and the Cold War. Other post–World War II changes that have reduced the risks of war include the entrenchment of the global norm against interstate warfare except in self-defence or with the authority of the UN Security Council; the intensification of economic and financial interdependence that increases the costs and decreases the benefits of cross-border warfare; the spread of stable democracies; and the caution-inducing impact of nuclear weapons on relations between the major powers.

With respect to civil wars, the emergent and still-growing system of global security governance discussed in Chapter 1 has clearly helped reduce the number of intrastate conflicts since the end of the Cold War. And, at what might be called the “structural” level, we have witnessed steady increases in national incomes across the developing world. This is important because one of the strongest findings from econometric research on the causes of war is that the risk of civil wars declines as national incomes—and hence governance and other capacities—increase. Chapter 1 reports on a remarkable recent statistical study by the Peace Research Institute, Oslo (PRIO) that found that if current trends in key structural variables are sustained, the proportion of the world’s countries afflicted by civil wars will halve by 2050.

Such an outcome is far from certain, of course, and for reasons that have yet to be imagined, as well as those canvassed by the conflict pessimists. But, thanks in substantial part to Steven Pinker’s extraordinary research, there are now compelling reasons for believing that the historical decline in violence is both real and remarkably large—and also that the future may well be less violent than the past.

After reading the new Human Security Report, I remain a short-term pessimist and long-term optimist. As I’ve said in a few recent posts (see especially this one), I think we’re currently in the thick of period of systemic instability that will continue to produce mass protests, state collapse, mass killing, and other forms of political instability at higher rates than we’ve seen since the early 1990s for at least the next year or two.

At the same time, I don’t think this local upswing marks a deeper reversal of the long-term trend that Pinker identifies, and that the Human Security Report confirms. Instead, I believe that the global political economy is continuing to evolve in a direction that makes political violence less common and less lethal. This system creep is evident not only in the aforementioned trends in armed violence, but also in concurrent and presumably interconnected trends in democratization, socio-economic development, and global governance. Until we see significant and sustained reversals in most or all of these trends, I will remain optimistic about the directionality of the underlying processes of which these data can give us only glimpses.

States Aren’t the Only Mass Killers

We tend to think of mass killing as something that states do, but states do not have a monopoly on this use of force. Many groups employ violence in an attempt to further their political and economic agendas; civilians often suffer the consequences of that violence, and sometimes that suffering reaches breathtaking scale.

This point occurred to me again as I thought about the stunning acts of mass violence that Boko Haram has carried out in northern Nigeria in the past few weeks. The chart below comes from the Council on Foreign Relations’ Nigeria Security Tracker, an online interface for a data set that counts deaths from “violent incidents directed at government property, places of worship, and suicide bombings.” The sharp upward bend at the far right of that red line represents the sudden and brutal end of several hundred lives in the past two months in various towns and villages in a part of the world that surely isn’t as alien to Americans as many of us assume. In Nigeria, too, parents wake up and set about the business of providing for themselves and their families, and many kids toddle off to school to learn and fidget and chatter with friends. Over the past few years, Boko Haram has repeatedly interrupted those daily routines with scores of attacks resulting in thousands of murders.

boko.haram.killings.chart.20140307

I suspect the tendency to see mass killing as the purview of states is driven by the extraordinary salience of two archetypal cases—the Holocaust, of course, but also the Rwandan genocide. From those examples, we infer that violence on this scale requires resources, organization, and opportunity on a scale that in “modern” times only states are supposed to possess. The Holocaust took this bureaucratic logic to unique extremes, but many accounts of the Rwandan genocide also emphasize state planning and propaganda as necessary conditions for that episode of mass murder in extremis.

It’s true that resources, organization, and opportunity facilitate mass violence, and that states are much more likely to have them. In some contexts, though, rebel groups and other non-state actors can accumulate enough resources and become well enough organized to kill on a comparable scale. This is especially likely in the same contexts in which states usually perpetrate mass killing, namely, in civil wars. In some wars, rebels manage to establish governance systems of their own, and the apparent logic of the atrocities committed by these quasi-states looks very similar to the logic behind the atrocities perpetrated by their foes: destroy your rival’s base of support, and scare civilians into compliance or complicity.

Rebels don’t need to govern to carry out mass killings, though, a point driven home by groups like the RUF in Sierra Leone, the Seleka and anti-balaka militias in the Central African Republic, and, of course, Boko Haram. Sometimes the states we now expect to protect civilians against such violence are so weak or absent or uncaring that those non-state groups don’t need deep pockets and sprawling organizations to accomplish mass murder. On Boko Haram, CFR’s John Campbell observes that, “Several of the most recent incidents involve government security forces unaccountably not at their posts, allowing Boko Haram freedom of movement. The governor of Borno state publicly said that Boko Haram fighters outgun government forces.” Campbell also notes that those security forces might be shirking their duty because they are poorly paid and equipped, and because they simply fear a group that “has a long tradition of killing any person in the security services that it can.” With a state like that, the resources and organization required to accomplish mass murder are, unfortunately, not so vast. What is required is a degree of ruthlessness that most of us find hard to understand, but that incomprehensibility should not be confused with impossibility.

Acts we conventionally describe as “terrorism” nowadays are also atrocities by another name, and so-called terrorist groups occasionally succeed in their lethal business on an extraordinary scale. Al Qaeda’s attacks on September 11, 2001, certainly qualify as a mass killing as we conventionally define it. Nearly 3,000 noncombatant civilians from a discrete group (Americans) were deliberately killed as part of a wider political conflict, and all in a single day. The torrent of car bombings and other indiscriminate attacks in Iraq in recent months has surely crossed that arbitrary 1,000-death threshold by now, too.

For analytical purposes, it would be useful to have a catalog of episodes in which non-state organizations committed atrocities on such a large scale. That catalog would allow us to try to glean patterns and develop predictive models from their comparison to each other and, more important, to situations in which those episodes did not occur. Even more useful would be a reliable assemblage of data on the incidents comprising those episodes, so we could carefully study how and where they arise and accumulate over time, perhaps with some hope of halting or at least mitigating future episodes as they develop.

Unfortunately, the data we want usually aren’t the data we have, and that’s true here, too. The Uppsala Conflict Data Program (UCDP) has compiled a data set on “one-sided violence,” defined as “intentional attacks on civilians by governments and formally organized armed groups,” that includes low, high, and best estimates of deaths attributed to each perpetrator group in cases where that annual estimate is 25 deaths or more (here). These data are an excellent start, but they only cover years since 1989, so the number of episodes involving non-state groups as perpetrators is still very small. The Armed Conflict Location & Event Data Project (ACLED) compiles detailed data (here) on attacks by non-state groups, among others, but it only covers Africa since 1997. New developments in the automated production of political event data hint at the possibility of analyzing deliberate violence against civilians around the world at a much higher resolution in the not-too-distant future. As I’ve discovered in an ongoing efforts to adapt one of these data sets to this purpose, however, we’re not quite there yet (see here).

In the meantime, we’ll keep seeing accounts of murderous sprees by groups like Boko Haram (here and here, to pick just two) and CAR’s Seleka (here) and anti-balaka (here) alongside the thrum of reporting on atrocities from places like Syria and Sudan. And as we read, we would do well to remember that people, not states, are the the common denominator.

PS. In the discussion of relevant data sets, I somehow forgot to mention that the Political Instability Task Force also funds the continuing collection of data on “atrocities” around the world involving five or more civilian fatalities (here). These data, which run all the way back to January 1995, are carefully compiled under the direction of a master of the craft, but they also suffer from the inevitable problems of reporting bias that plague all such efforts and so must be handled with care (see Will Moore here and here on this subject).

Watch Experts’ Beliefs Evolve Over Time

On 15 December 2013, “something” happened in South Sudan that quickly began to spiral into a wider conflict. Prior research tells us that mass killings often occur on the heels of coup attempts and during civil wars, and at the time South Sudan ranked among the world’s countries at greatest risk of state-led mass killing.

Motivated by these two facts, I promptly added a question about South Sudan to the opinion pool we’re running as part of a new atrocities early-warning system for the U.S. Holocaust Memorial Museum’s Center for the Prevention of Genocide (see this recent post for more on that). As it happened, we already had one question running about the possibility of a state-led mass killing in South Sudan targeting the Murle, but the spiraling conflict clearly implied a host of other risks. Posted on 18 December 2013, the new question asked, “Before 1 January 2015, will an episode of mass killing occur in South Sudan?”

The criteria we gave our forecasters to understand what we mean by “mass killing” and how we would decide if one has happened appear under the Background Information header at the bottom of this post. Now, shown below is an animated sequence of kernel density plots of each day’s forecasts from all participants who’d chosen to answer this question. A kernel density plot is like a histogram, but with some nonparametric estimation thrown in to try to get at the distribution of a variable’s “true” values from the sample of observations we’ve got. If that sound like gibberish to you, just think of the peaks in the plots as clumps of experts who share similar beliefs about the likelihood of mass killing in South Sudan. The taller the peak, the bigger the clump. The farther right the peak, the more likely that clump thinks a mass killing is.

kplot.ssd.20140205

I see a couple of interesting patterns in those plots. The first is the rapid rightward shift in the distribution’s center of gravity. As the fighting escalated and reports of atrocities began to trickle in (see here for one much-discussed article from the time), many of our forecasters quickly became convinced that a mass killing would occur in South Sudan in the coming year, if one wasn’t occurring already. On 23 December—the date that aforementioned article appeared—the average forecast jumped to approximately 80 percent, and it hasn’t fallen below that level since.

The second pattern that catches my eye is the appearance in January of a long, thin tail in the distribution that reaches into the lower ranges. That shift in the shape of the distribution coincides with stepped-up efforts by U.N. peacekeepers to stem the fighting and the start of direct talks between the warring parties. I can’t say for sure what motivated that shift, but it looks like our forecasters split in their response to those developments. While most remained convinced that a mass killing would occur or had already, a few forecasters were apparently more optimistic about the ability of those peacekeepers or talks or both to avert a full-blown mass killing. A few weeks later, it’s still not clear which view is correct, although a forthcoming report from the U.N. Mission in South Sudan may soon shed more light on this question.

I think this set of plots is interesting on its face for what it tells us about the urgent risk of mass atrocities in South Sudan. At the same time, I also hope this exercise demonstrates the potential to extract useful information from an opinion pool beyond a point-estimate forecast. We know from prior and ongoing research that those point estimates can be quite informative in their own right. Still, by looking at the distribution of participant’s forecasts on a particular question, we can glean something about the degree of uncertainty around an event of interest or concern. By looking for changes in that distribution over time, we can also get a more complete picture of how the group’s beliefs evolve in response to new information than a simple line plot of the average forecast could ever tell us. Look for more of this work as our early-warning system comes online, hopefully in the next few months.

UPDATE (7 Feb): At the urging of Trey Causey, I tried making another version of this animation in which the area under the density plot is filled in. I also decided to add a vertical line to show each day’s average forecast, which is what we currently report as the single-best forecast at any given time. Here’s what that looks like, using data from a question on the risk of a mass killing occurring in the Central African Republic before 2015. We closed this question on 19 December 2013, when it became clear through reporting by Human Rights Watch and others that an episode of mass killing has occurred.

kplot2.car.20140207

Background Information

We will consider a mass killing to have occurred when the deliberate actions of state security forces or other armed groups result in the deaths of at least 1,000 noncombatant civilians over a period of one year or less.

  • A noncombatant civilian is any person who is not a current member of a formal or irregular military organization and who does not apparently pose an immediate threat to the life, physical safety, or property of other people.
  • The reference to deliberate actions distinguishes mass killing from deaths caused by natural disasters, infectious diseases, the accidental killing of civilians during war, or the unanticipated consequences of other government policies. Fatalities should be considered intentional if they result from actions designed to compel or coerce civilian populations to change their behavior against their will, as long as the perpetrators could have reasonably expected that these actions would result in widespread death among the affected populations. Note that this definition also covers deaths caused by other state actions, if, in our judgment, perpetrators enacted policies/actions designed to coerce civilian population and could have expected that these policies/actions would lead to large numbers of civilian fatalities. Examples of such actions include, but are not limited to: mass starvation or disease-related deaths resulting from the intentional confiscation, destruction, or medicines or other healthcare supplies; and deaths occurring during forced relocation or forced labor.
  • To distinguish mass killing from large numbers of unrelated civilian fatalities, the victims of mass killing must appear to be perceived by the perpetrators as belonging to a discrete group. That group may be defined communally (e.g., ethnic or religious), politically (e.g., partisan or ideological), socio-economically (e.g., class or professional), or geographically (e.g., residents of specific villages or regions). In this way, apparently unrelated executions by police or other state agents would not qualify as mass killing, but capital punishment directed against members of a specific political or communal group would.

The determination of whether or not a mass killing has occurred will be made by the administrators of this system using publicly available secondary sources and in consultation with subject-matter experts. Relevant evidence will be summarized in a blog post published when the determination is announced, and any dissenting views will be discussed as well.

Will Unarmed Civilians Soon Get Massacred in Ukraine?

According to one pool of forecasters, most probably not.

As part of a public atrocities early-warning system I am currently helping to build for the U.S. Holocaust Memorial Museum’s Center for the Prevention of Genocide (see here), we are running a kind of always-on forecasting survey called an opinion pool. An opinion pool is similar in spirit to a prediction market, but instead of having participants trade shares tied the occurrence of some future event, we simply ask participants to estimate the probability of each event’s occurrence. In contrast to a traditional survey, every question remains open until the event occurs or the forecasting window closes. This way, participants can update their forecasts as often as they like, as they see or hear relevant information or just change their minds.

With generous support from Inkling, we started up our opinion pool in October, aiming to test and refine it before our larger early-warning system makes its public debut this spring (we hope). So far, we have only recruited opportunistically among colleagues and professional acquaintances, but we already have more than 70 registered participants. In the first four months of operation, we have used the system to ask more than two dozen questions, two of which have since closed because the relevant events occurred (mass killing in CAR and the Geneva II talks on Syria).

Over the next few years, we aim to recruit a large and diverse pool of volunteer forecasters from around the world with some claim to topical expertise or relevant local knowledge. The larger and more diverse our pool, the more accurate we expect our forecasts to be, and the wider the array of questions we can ask. (If you are interested in participating, please drop me a line at ulfelder <at> gmail <dot> com.)

A few days ago and prompted by a couple of our more active members, I posted a question to our pool asking, “Before 1 March 2014, will any massacres occur in Ukraine?” As of this morning, our pool had made a total of 13 forecasts, and the unweighted average of the latest of those estimates from each participating forecaster was just 15 percent. Under the criteria we specified (see Background Information below), this forecast does not address the risk of large-scale violence against or among armed civilians, nor does it exclude the possibility of a series of small but violent encounters that cumulatively produce a comparable or larger death toll. Still, for those of us concerned that security forces or militias will soon kill nonviolent protesters in Ukraine on a large scale, our initial forecast implies that those fears are probably unwarranted.

Crowd-Estimated Probability of Any Massacres in Ukraine Before 1 March 2014

Crowd-Estimated Probability of Any Massacres in Ukraine Before 1 March 2014

Obviously, we don’t have a crystal ball, and this is just an aggregation of subjective estimates from a small pool of people, none of whom (I think) is on the scene in Ukraine or has inside knowledge of the decision-making of relevant groups. Still, a growing body of evidence shows that aggregations of subjective forecasts like this one can often be usefully accurate (see here), even with a small number of contributing forecasters (see here). On this particular question, I very much hope our crowd is right. Whatever happens in Ukraine over the next few weeks, though, principle and evidence suggest that the method is sound, and we soon expect to be using this system to help assess risks of mass atrocities all over the world in real time.

Background Information

We define a “massacre” as an event that has the following features:

  • At least 10 noncombatant civilians are killed in one location (e.g., neighborhood, town, or village) in less than 48 hours. A noncombatant civilian is any person who is not a current member of a formal or irregular military organization and who does not apparently pose an immediate threat to the life, physical safety, or property of other people.
  • The victims appear to have been the primary target of the violence that killed them.
  • The victims do not appear to have been engaged in violent action or criminal activity when they were killed, unless that violent action was apparently in self-defense.
  • The relevant killings were carried out by individuals affiliated with a social group or organization engaged in a wider political conflict and appear to be connected to each other and to that wider conflict.

Those features will not always be self-evident or uncontroversial, so we use the following series of ad hoc rules to make more consistent judgments about ambiguous events.

  • Police, soldiers, prison guards, and other agents of state security are never considered noncombatant civilians, even if they are killed while off duty or out of uniform.
  • State officials and bureaucrats are not considered civilians when they are apparently targeted because of their professional status (e.g., assassinated).
  • Civilian deaths that occur in the context of operations by uniformed military-service members against enemy combatants are considered collateral damage, not atrocities, and should be excluded unless there is strong evidence that the civilians were targeted deliberately. We will err on the side of assuming that they were not.
  • Deaths from state repression of civilians engaged in nonviolent forms of protest are considered atrocities. Deaths resulting from state repression targeting civilians who were clearly engaged in rioting, looting, attacks on property, or other forms of collective aggression or violence are not.
  • Non-state militant or paramilitary groups, such as militias, gangs, vigilante groups, or raiding parties, are considered combatants, not civilians.

We will use contextual knowledge to determine whether or not a discrete event is linked to a wider conflict or campaign of violence, and we will err on the side of assuming that it is.

Determinations of whether or not a massacre has occurred will be made by the administrator of this system using publicly available secondary sources. Relevant evidence will be summarized in a blog post published when the determination is announced, and any dissenting views will be discussed as well.

Disclosure

I have argued on this blog that scholars have an obligation to disclose potential conflicts of interest when discussing their research, so let me do that again here: For the past two years, I have been paid as a contractor by the U.S. Holocaust Memorial Museum for my work on the atrocities early-warning system discussed in this post. Since the spring of 2013, I have also been paid to write questions for the Good Judgment Project, in which I participated as a forecaster the year before. To the best of my knowledge, I have no financial interests in, and have never received any payments from, any companies that commercially operate prediction markets or opinion pools.

What the U.S. Intelligence Community Says About Mass Atrocities in 2014

Here’s what Director of National Intelligence James Clapper said about the risk of mass atrocities this year in the Worldwide Threat Assessment he delivered today to the Senate Select Committee on Intelligence:

The overall risk of mass atrocities worldwide will probably increase in 2014 and beyond. Trends driving this increase include more social mobilization, violent conflict, including communal violence, and other forms of instability that spill over borders and exacerbate ethnic and religious tensions; diminished or stagnant quality of governance; and widespread impunity for past abuses. Many countries at risk of mass atrocities will likely be open to influence to prevent or mitigate them. This is because they are dependent on Western assistance or multilateral missions in their countries, have the political will to prevent mass atrocities, or would be responsive to international scrutiny. Overall international will and capability to prevent or mitigate mass atrocities will likely diminish in 2014 and beyond, although support for human rights norms to prevent atrocities will almost certainly deepen among some non-government organizations. Much of the world will almost certainly turn to the United States for leadership to prevent and respond to mass atrocities.

That’s a ton of analysis crammed into a single paragraph, and I suspect a lot of person-hours went into the construction of those six sentences.

However many hours it was, I think the results are largely correct. After two decades of relative quiescence, we’ve seen a troubling rebound in the occurrence of mass atrocities in the past few years, and the systemic forces that seem to be driving that rebound don’t yet show signs of abating.

One point on which I disagree with the IC’s analysis, though, is the claim that “widespread impunity for past abuses” is helping to fuel the upward trend in mass atrocities. I don’t think this assertion is flat-out false; I just think it’s overblown and over-confident. As Mark Kersten argued last week in a blog post on the debate over whether or not the situation in Syria should be referred to the International Criminal Court (ICC),

Any suggestion that international criminal justice should be pursued in the context of ongoing hostilities in Syria leads us to the familiar “peace versus justice” debate. Within this debate, there are broadly two camps: one which views international criminal justice as a necessary and useful tool which can deter crimes, marginalize perpetrators and even be conducive to peace negotiations; and a second camp which sees judicial interventions as deleterious to peace talks and claims that it creates disincentives for warring parties to negotiate and leads to increased levels of violence.

So who’s right? I think Kersten is when he says this:

It remains too rarely conceded that the Courts effects are mixed and, even more rarely, that they might be negligible.This points to the ongoing need to reimagine how we study and assess the effects of the ICC on ongoing and active conflicts. There is little doubt that the Court can have negative and positive effects on the ability of warring parties and interested actors to transform conflicts and establish peace. But this shouldn’t lead to a belief that the ICC must have these effects across cases. In some instances, the Court may actually have minimal or even inconsequential effects. As importantly, in many if not most cases, the ICC won’t be the be-all and end-all of peace processes. Even when the Court has palpable effects, peace processes aren’t likely to flourish or perish on the hill of international criminal justice.

Finally, I’m not sure what the Threat Assessment‘s drafters had in mind when they wrote that “overall international will and capability to prevent or mitigate mass atrocities will likely diminish in 2014 and beyond.” I suspect that statement is a nod in the direction of declinists who worry that a recalcitrant Russia and rising China spell trouble for the supposed Pax Americana, but that’s just a guess.

In any case, I think the assertion is wrong. Syria is the horror that seems to lurk behind this point, and there’s no question that the escalation and spread of that war represents one of the greatest failures of global governance in modern times. Even as the war in Syria continues, though, international forces have mobilized to stem fighting in the Central African Republic and South Sudan, two conflicts that are already terrible but could also get much, much worse. Although the long-term effects of those mobilizations remain unclear, the very fact of their occurrence undercuts the claim that international will and capability to respond to mass atrocities are flagging.

Follow

Get every new post delivered to your Inbox.

Join 6,445 other followers

%d bloggers like this: