Early Results from a New Atrocities Early Warning System

For the past couple of years, I have been working as a consultant to the U.S. Holocaust Memorial Museum’s Center for the Prevention of Genocide to help build a new early-warning system for mass atrocities around the world. Six months ago, we started running the second of our two major forecasting streams, a “wisdom of (expert) crowds” platform that aggregates probabilistic forecasts from a pool of topical and area experts on potential events of concern. (See this conference paper for more detail.)

The chart below summarizes the output from that platform on most of the questions we’ve asked so far about potential new episodes of mass killing before 2015. For our early-warning system, we define a mass killing as an episode of sustained violence in which at least 1,000 noncombatant civilians from a discrete group are intentionally killed, usually in a period of a year or less. Each line in the chart shows change over time in the daily average of the inputs from all of the participants who choose to make a forecast on that question. In other words, the line is a mathematical summary of the wisdom of our assembled crowd—now numbering nearly 100—on the risk of a mass killing beginning in each case before the end of 2014. Also:

  • Some of the lines (e.g., South Sudan, Iraq, Pakistan) start further to the right than others because we did not ask about those cases when the system launched but instead added them later, as we continue to do.
  • Two lines—Central African Republic and South Sudan—end early because we saw onsets of mass-killing episodes in those countries. The asterisks indicate the dates on which we made those declarations and therefore closed the relevant questions.
  • Most but not all of these questions ask specifically about state-led mass killings, and some focus on specific target groups (e.g., the Rohingya in Burma) or geographic regions (the North Caucasus in Russia) as indicated.
Crowd-Estimated Probabilities of Mass-Killing Onset Before 1 January 2015

Crowd-Estimated Probabilities of Mass-Killing Onset Before 1 January 2015

I look at that chart and conclude that this process is working reasonably well so far. In the six months since we started running this system, the two countries that have seen onsets of mass killing are both ones that our forecasters promptly and consistently put on the high side of 50 percent. Nearly all of the other cases, where mass killings haven’t yet occurred this year, have stuck on the low end of the scale.

I’m also gratified to see that the system is already generating the kind of dynamic output we’d hoped it would, even with fewer than 100 forecasters in the pool. In the past several weeks, the forecasts for both Burma and Iraq have risen sharply, apparently in response to shifts in relevant policies in the former and the escalation of the civil war in the latter. Meanwhile, the forecast for Uighurs in China has risen steadily over the year as a separatist rebellion in Xinjiang Province has escalated and, with it, concerns about a harsh government response. These inflection points and trends can help identify changes in risk that warrant attention from organizations and individuals concerned about preventing or mitigating these potential atrocities.

Finally, I’m also intrigued to see that our opinion pool seems to be sorting cases into a few clusters that could be construed as distinct tiers of concern. Here’s what I have in mind:

  • Above the 50-percent threshold are the high risk cases, where forecasters assess that mass killing is likely to occur during the specified time frame.  These cases won’t necessarily be surprising. Some observers had been warning on the risk of mass atrocities in CAR and South Sudan for months before those episodes began, and the plight of the Rohingya in Burma has been a focal point for many advocacy groups in the past year. Even in supposedly “obvious” cases, however, this system can help by providing a sharper estimate of that risk and giving a sense of how it is trending over time. In the case of Burma, for example, it is the separation that has happened in the last several weeks that tells the story of a switch from possible to likely and thus adds a degree of urgency to that warning.
  • A little farther down the y-axis are the moderate risk cases—ones that probably won’t suffer mass killing during the period in question but could more readily tip in that direction. In the chart above, Iraq, Sudan, Pakistan, Bangladesh, and Burundi all land in this tier, although Iraq now appears to be sliding into the high risk group.
  • Clustered toward the bottom are the low risk cases where the forecasters seem fairly confident that mass killing will not occur in the near future. In the chart above, Russia, Afghanistan, and Ethiopia are the cases that land firmly in this set. China (Uighurs) remains closer to them than the moderate risk tier, but it appears to be creeping toward the moderate-risk group. We are also running a question about the risk of state-led mass killing in Rwanda before 2015, and it currently lands in this tier, with a forecast of 14 percent.

The system that generates the data behind this chart is password protected, but the point of our project is to make these kinds of forecasts freely available to the global public. We are currently building the web site that will display the forecasts from this opinion pool in real time to all comers and hope to have it ready this fall.

In the meantime, if you think you have relevant knowledge or expertise—maybe you study or work on this topic, or maybe you live or work in parts of the world where risks tend to be higher—and are interested in volunteering as a forecaster, please send an email to us at ewp@ushmm.org.

New Leaders and Political Change in Authoritarian Regimes (with an Eye on Ethiopia)

Does leadership change in authoritarian regimes open the door to democratic reform?

A post yesterday by Alula Alex Iyasu at the Royal African Society’s African Arguments blog implies that it does, or at least that it might in the case of present-day Ethiopia, where longtime leader Meles Zenawi is rumored to be gravely ill, dead already, or “recuperating,” depending on whom you ask. Whatever the precise condition of Zenawi’s health, Iyasu sees the leadership crisis spawned by this uncertainty as an opportunity for political and economic reform:

The next Prime Minister of Ethiopia should take this potential and impending leadership crisis and turn it into an opportunity – to reform and improve areas hampered by overreaching government policy and an absence of democratic institutions.  There is a golden opportunity to view the private sector as a true partner in national economic growth and not an entity to be feared and stymied. An opportunity to encourage public-private partnership as a means to raise capital for the kinds of ambitious development goals Ethiopia has outlined but lacks the funds. An opportunity to create democratic institutions with truly independent bodies that facilitate, arbitrate and encourage entrepreneurship.

Iyasu’s post is more prescriptive than diagnostic, but I think it also reflects a widespread view that leadership change in authoritarian regimes opens doors to more fundamental institutional changes. There are at least a few reasons this might be true. It may be that leadership transitions helps cause democratization by stimulating struggles among elites, thereby presenting would-be reformers with new room to maneuver. It may be that leadership change often coincides with democratization because both occur in response to increases in deeper pressures for political reform. A correlation between leadership change and democratization could also arise for sociological reasons; perhaps it’s the leader’s values that matter, and leaders who are personally committed to reform usually launch those changes soon after arriving in office, but you can’t get that effect without changing leaders first. And, of course, all of those statements could be true. Whatever the pathway(s), the point is that we might expect the likelihood of a democratic transition to be higher during the several years after a new leader takes the helm of an authoritarian regime than it is during the rest of their tenures.

So, is it? Talking on Twitter yesterday about Iyasu’s post, I suggested that leadership change wouldn’t make much difference in Ethiopia, but this morning I figured I ought to check that claim.

Details of the statistical analysis I used to do that are at the bottom of this post, but the bottom line is that I was probably wrong: in fact, authoritarian regimes are much more likely to transition to democracy during the several years following a leadership change, other things being equal. According to my estimates, a democratic transition is more than three times as likely to occur during the first three years of a new leader’s tenure as it is after a ruler becomes more established. That estimate comes from a statistical model that also accounts for the age of the authoritarian regime, the civil liberties it allows, and the occurrence of economic recession and nonviolent popular uprisings in the previous year, among other things. In the context of this kind of modeling, it’s a pretty big “effect.”

After seeing that relationship, I wondered if leadership change might also indirectly improve prospects for democratic transition by increasing the likelihood that a nonviolent popular uprising would take shape. To test that conjecture, I added the same indicator of new leadership to a model that tries to predict the starts of civil-resistance campaigns in authoritarian regimes. To my surprise, the association actually seems to run in the opposite direction; other things being equal, popular uprisings are only about half as likely to flare up during the first few years of an authoritarian ruler’s tenure as they are during later years.

For autocracies in general, this pair of results suggests that leadership changes do open the door to democratization, at least temporarily, and that the linkage between those two events does not run through popular uprisings. The association we see probably has more to do with elite infighting, the new leader’s values, or deeper forces that impel change in both leaders and institutions.

For Ethiopia in particular, those results imply that Iyasu has reason to be optimistic about widening opportunities of political reform in that country, whenever and for whatever reason Zenawi leaves office. My statistical forecast of Ethiopia’s prospects for democratic transition put it close to the bottom of the pile of authoritarian regimes this year, but a change in leadership would bump it up toward the middle of the pack, other things being equal. If the new leadership loosened restrictions on civil liberties as Iyasu recommends, those odds would get even better. Even with those changes, history tells us that the authoritarian regime would be more likely to survive than not, but I’ll take a forecast that calls for a few breaks in the clouds over portents of unending rain any day.


My indicators of authoritarian rule and transitions to democracy come from a Political Instability Task Force data set, while my indicator of civil-resistance campaign onset comes from Erica Chenoweth and Maria Stephan’s NAVCO data set. I used R’s generalized linear model (glm) command to estimate logistic regression models with the following covariates.

  • p(democratic transition | authoritarian rule) = any prior democracy (yes/no) + log(duration of authoritarian rule) + [any prior democracy * log(duration of authoritarian rule)] + post-Cold War period (yes/no) + civil liberties index (1-year lag) + any ongoing civil-resistance campaign (yes/no, 1-year lag) + negative annual GDP growth (yes/no, 1-year lag) + natural-resource wealth (categorical by tercile) + new leader (yes/no, 1-year lag)
  • p(onset of civil-resistance campaign | authoritarian rule) = log(infant mortality rate relative to annual global median,1-year lag) + log(total population relative to annual global median, 1-year lag) + post-Cold War period (yes/no) + any national elections (yes/no) + (post-Cold War period * any national elections) + civil liberties index (1-year lag) + negative annual GDP growth (yes/no, 1-year lag)  + new leader (yes/no, 1-year lag)

The estimated odd ratios for periods of new leadership (with lower and upper bounds of a 95% confidence interval) were as follows:

  • Democratic transition: 3.5 (2.2, 5.8)
  • Onset of civil-resistance campaign: 0.5 (0.2, 0.9)

To check the robustness of these results, I re-estimated the models with random intercepts for countries using the ‘glmmML’ package, and the parameters were basically unchanged. I also reran the models with an indicator that extended the “new leader” period to five years from three and got very similar results. For democratic transitions, they were essentially unchanged; for popular uprisings, the association was a bit weaker, suggesting that the period of depressed odds is short. Finally, I reran the models with continuous measures of leader’s time in office (logged) and got the expected patterns: other things being equal, as leaders’ time in office passes, the odds of democratic transition decline while the odds of a civil-resistance campaign forming go up.

If you’re interested in seeing the data and code I used, hit me up at ulfelder <at> gmail.

Statistics Is Not Alchemy

Are aid and investment from China driving crackdowns on the press in some parts of Africa?

I don’t know.

That’s unsatisfying and maybe even a little annoying, but I’m writing a post about it anyway because why I don’t know says a lot about how hard it is to do good quantitative social science, even in the age of Big Data. Here’s the story:

A few Mondays ago, the New York Times ran an op-ed entitled “Africa’s Free Press Problem” in which the author, Mohamed Keita of the advocacy group Committee to Protect Journalists, asserted that press freedom is eroding in Africa, and foreign forces are partially at fault. According to Keita, “Independent African journalists covering the continent’s development are now frequently persecuted for critical reporting on the misuse of public finances, corruption and the activities of foreign investors.” He lays part of the blame for this alleged trend at the feet of Western governments more interested in promoting economic development and stability than democracy, but he sees other forces at work, too:

Then there’s the influence of China, which surpassed the West as Africa’s largest trading partner in 2009. Ever since, China has been deepening technical and media ties with African governments to counter the kind of critical press coverage that both parties demonize as neocolonialist.

In January, Beijing issued a white paper calling for accelerated expansion of China’s news media abroad and the deployment of a press corps of 100,000 around the world, particularly in priority regions like Africa. In the last few months alone, China established its first TV news hub in Kenya and a print publication in South Africa. The state-run Xinhua news agency already operates more than 20 bureaus in Africa. More than 200 African government press officers received Chinese training between 2004 and 2011 in order to produce what the Communist Party propaganda chief, Li Changchun, called “truthful” coverage of development fueled by China’s activities.

When I finished Keita’s piece, I was sympathetic to his concerns, but I was skeptical of his claim that the ebb and flow of press freedom in Africa was being shaped so decisively by China’s recent investments on the continent. From my own reading of politics, I see the kinds of constraints on the press that Keita describes in Ethiopia and Rwanda as normal features of authoritarian rule. By my reckoning, both Ethiopia and Rwanda have been repressing independent journalism for quite a while, so I couldn’t see how China’s recent overtures would have much to do with why that repression is happening. Cause has to precede effect and all that.

Being an empiricist and a blogger, I figured I’d pursue my hunch by taking a look at the data and writing a post. In a day or two, I could run a statistical analysis that would check Keita’s implied claim that Chinese engagement was reducing press freedom in Africa. I knew that both Freedom House and Reporters Without Borders produce annual, country-level measures of press freedom covering at least the past decade, so I was confident that I could observe recent trends on that side of the equation. All I needed was comparable data on aid and foreign direct investment from China, and I could run some simple fixed-effects models to see if changes over time in those money flows really were associated with decreases in press freedom, as Keita’s essay seemed to suggest.

And that’s where I hit a wall. First, I Googled “china foreign investment data” and “china foreign aid data” and came up with next to the nothing. The best I could do was an incomplete, project-level data set of Chinese foreign aid projects in Africa from 1990 through 2005. Next, I posted queries on Twitter and the listserv of the Society for Political Methodology. The latter led me to the University of the Pacific’s Daniel O’Neill, who confirmed my growing suspicion that the data I wanted simply don’t exist. We can see annual outflows of FDI from China, but we can’t see where that money’s gone, and bilateral data on development assistance from China are not available. (Even if they were, I’m not sure I would have trusted the numbers, but that’s beside the point for now.)

So, here we are in 2012, and it’s impossible to answer a seemingly simple question because the data we need to answer that question are nowhere to be found.

In fact, there are a lot of really interesting and important social-science questions where this is true. Income inequality is one of them, as I discussed on this blog a few weeks ago. Unemployment is another. If I had a dollar for every time I heard someone suggested adding unemployment to a global statistical model of political instability, I’d be a lot richer. It turns out, though, that many countries don’t report unemployment rates, and many of the ones that do only started to do so recently. A quick look at the World Bank’s World Development Indicators shows the problem clearly; lots of countries have no observations, and those gaps are correlated with other things that contribute to the risks of political instability–poverty most especially, but also authoritarian rule and recent or ongoing civil violence.

The list of known unknowns is a lot longer, but I think that’s enough to make the problem clear. From popular discussions, you’d think we’re living in an era when anything and everything is routinely quantified and the only problem left is finding the signal in all that noise. For some questions in some (rich) countries, that’s a fair description. For many of the big questions in comparative politics and international relations, though, we’re only just starting to exit the Dark Ages, and the past–and often even the present–are essentially lost to statistical analysis.

Raising the Human-Rights Bar for Development Assistance…But Will It Make a Difference?

The U.S.’s Millennium Challenge Corporation (MCC) has raised the bar for countries seeking its development-assistance grants in 2012 by adopting stricter standards for civil liberties and political rights. The intentions behind this change are clear and laudable, but larger weaknesses in the MCC program and the increased availability of unconditional aid from other sources lead me to expect that this change’s impact on political development in the targeted countries will be negligible.

For readers who aren’t aid wonks, some background is in order. The MCC is a U.S. government-funded but independently managed aid agency that aims to help its recipients reduce poverty by funding programs that are meant to boost economic growth. The MCC was established by President Bush in 2004, but it was the brain child of Stanford international-relations professor Stephen Krasner, who went on to serve as director of the State Department’s Policy Planning Staff for part of Bush’s second term.

The big idea behind the MCC was to give poor countries stronger incentive to improve their economic and political governance by making a big, new pot of aid funding available, but making access to that pot conditional on countries’ performance on a basket of governance indicators. In theory, it’s like setting up a smoothie bar  in a high-school cafeteria and then telling the hungry students they’ll get free smoothies, but only if they’ve done well enough on their report cards. If they’re hungry enough (and like smoothies enough), anticipation of that reward should encourage them to improve their schoolwork, and everyone ends up better off for it.

To be eligible for MCC grants, countries a) have to be relatively poor (“low income” or “low middle income” in World Bank parlance, meaning they have an annual gross national income per capita less than $3,975); and b) have to satisfy a battery of selection criteria across three thematic groups: “economic freedom,” “investing in people,” and “ruling justly.” The MCC spells out its criteria in painstaking detail in an annual report, identifies candidate countries based on income, issues “report cards” on those countries’ governance practices, and then, finally, announces which countries have qualified for its assistance.

The big change announced by the MCC in 2011 for fiscal-year 2012 comes in the way it handles the “ruling justly” category. In the past, countries could qualify by scoring above the median on “controlling corruption” and any two of the five other indicators in that bin: political rights, civil liberties, voice and accountability, government effectiveness, and rule of law. Starting in fiscal-year 2012, however, countries have to score above a threshold on two of those six “ruling justly” indicators: still “controlling corruption,” but now either “political rights” and “civil liberties” as well.

This change is potentially significant. Of the six “ruling justly” indicators, only three are directly indicative of democratization: political rights, civil liberties, and voice and accountability. This meant that, under the old rules, highly undemocratic countries could qualify for MCC grants, as long as they were well administered relative to their low-income peers. Under the new rules, however, countries have to be at least moderately liberalized or democratized to get through the door. (For those of you who are familiar with the Freedom House political rights and civil liberties indices used to measure these dimensions, the minima for 2012 are 4s on both scales.)

To see what this rule change might mean in the real world, I poked around the MCC’s data in search of countries that would have cleared the “ruling justly” hurdle under the old system but fall short under the new one. Instead of trying to determine overall eligibility, which is pretty complicated and sometimes involves additional considerations, I just looked at the “ruling justly” category. This unofficial and possibly error-prone exercise identified the following four countries as ones that would have made the old cut but fail to make the new one:

  • Djibouti
  • Ethiopia
  • Rwanda
  • Vietnam

That list nicely reflects the intentions behind the 2012 rule change. I know little about Djibouti, but Rwanda and Vietnam readily spring to mind as countries that often get lauded for their technocratic performance in spite of their clear failings on human rights and democracy. I would have guessed Ethiopia was more of a mixed bag, but it just barely tops the peer-group thresholds for “control of corruption” and “rule of law” while easily clearing the bar on “government effectiveness.”

Of course, the big question is whether or not MCC’s scoring change will actually help motivate the governments of those four countries to initiate political reforms they otherwise would not have taken. On that count, I’m hopeful but pessimistic. Seven years after its creation, the MCC isn’t having the transformative effects its designers intended, and that pattern isn’t likely to change any time soon.

The basic problem is that the MCC’s pot of money is too small to have the kind of “transformative” effect on the vast political economy of aid that its creators intended.  In part, that’s a function of supply. As originally envisioned, the MCC’s Millennium Challenge Account was supposed to have an annual budget of $5 billion. In fact, the budget has hovered closer to $1 billion per year, thanks to smaller requests from the presidents and smaller allocations from Congress. Given the current state of the federal government’s finances and the domestic politics of foreign aid, it’s hard to imagine that budget growing much larger in the next several years.

Budget woes aside, any transformative effect the MCC might have is also impeded by limited demand. Poor countries seeking development assistance have other options, and most of those other options don’t come with political strings attached. Faced with the choice between adopting political reforms that might threaten their grip on power in order to pursue a modest-sized grant from the MCC or seeking assistance elsewhere, it’s hard to imagine many authoritarian rulers opting for the former. In the 1990s and early 2000s, when the U.S. and Europe were pretty much the only game in town for development assistance, the MCC’s conditional offers might have been more tempting. In recent years, though, rapid growth in foreign assistance from China in particular has expanded the pool of available funds, thereby diluting the power of the MCC’s medicine.

In sum, while I applaud the MCC for making this change, I doubt it will make much difference. They’re trying to do the right thing, but it’s hard to move the world with a short lever and a shaky fulcrum.

  • Author

  • Follow me on Twitter

  • Follow Dart-Throwing Chimp on WordPress.com
  • Enter your email address to follow this blog and receive notifications of new posts by email.

    Join 13,611 other followers

  • Archives

%d bloggers like this: