Measurement Is Hard, Especially of Politics, and Everything Is Political

If occasional readers of this blog remember only one thing from their time here, I’d like it to be this: we may be getting better at measuring political things around the world, but huge gaps remain, sometimes on matters that seem basic or easy to see, and we will never close those gaps completely.

Two items this week reminded me of this point. The first came from the World Bank, which blogged that only about half of the countries they studied for a recent paper had “adequate” data on poverty. As a chart from an earlier World Bank blog post showed, the number of countries suffering from “data deprivation” on this topic has declined since the early 1990s, but it’s still quite large. Also notice that the period covered by the 2015 study ends in 2011. So, in addition to “everywhere”, we’ve still got serious problems with the “all the time” part of the Big Data promise, too.

The other thing that reminded me of data gaps was a post on the Lowy Institute’s Interpreter blog about Myanmar’s military, the Tatmadaw. According to Andrew Selth,

Despite its dominance of Burma’s national affairs for decades, the Tatmadaw remains in many respects a closed book. Even the most basic data is beyond the reach of analysts and other observers. For example, the Tatmadaw’s current size is a mystery, although most estimates range between 300,000 and 350,000. Official statistics put Burma’s defence expenditure this year at 3.7 % of GDP, but the actual level is unknown.

This kind of situation may be especially pernicious. It looks like we have data—350,000 troops, 3.7 percent of GDP—but the subject-matter expert knows that those data are not reliable. For those of us trying to do cross-national analysis of things like conflict dynamics or coup risk, the temptation to plow ahead with the numbers we have is strong, but we shouldn’t trust the inferences we draw from them.

The size and capability of a country’s military are obviously political matters. It’s not hard to imagine why governments might want to mislead others about the true values of those statistics.

Measuring poverty might seem less political and thus more amenable to technical fixes or workarounds, but that really isn’t true. At each step in the measurement process, the people being observed or doing the observing may have reasons to obscure or mislead. Survey respondents might not trust their observers; they may fear the personal or social consequences of answering or not answering certain ways, or just not like the intrusion. When the collection is automated, they may develop ways to fool the routines. Local officials who sometimes oversee the collection of those data may be tempted to fudge numbers that affect their prospects for promotion or permanent exile. National governments might seek to mislead other governments as a way to make their countries look stronger or weaker than they really are—stronger to deter domestic and international adversaries or get a leg up in ideological competitions, or weaker to attract aid or other help.

As social scientists, we dream of data sets that reliably track all sorts of human behavior. Our training should also make us sensitive to the many reasons why that dream is impossible and, in many cases, undesirable. Measurement begets knowledge; knowledge begets power; and struggles over power will never end.

Early Results from a New Atrocities Early Warning System

For the past couple of years, I have been working as a consultant to the U.S. Holocaust Memorial Museum’s Center for the Prevention of Genocide to help build a new early-warning system for mass atrocities around the world. Six months ago, we started running the second of our two major forecasting streams, a “wisdom of (expert) crowds” platform that aggregates probabilistic forecasts from a pool of topical and area experts on potential events of concern. (See this conference paper for more detail.)

The chart below summarizes the output from that platform on most of the questions we’ve asked so far about potential new episodes of mass killing before 2015. For our early-warning system, we define a mass killing as an episode of sustained violence in which at least 1,000 noncombatant civilians from a discrete group are intentionally killed, usually in a period of a year or less. Each line in the chart shows change over time in the daily average of the inputs from all of the participants who choose to make a forecast on that question. In other words, the line is a mathematical summary of the wisdom of our assembled crowd—now numbering nearly 100—on the risk of a mass killing beginning in each case before the end of 2014. Also:

  • Some of the lines (e.g., South Sudan, Iraq, Pakistan) start further to the right than others because we did not ask about those cases when the system launched but instead added them later, as we continue to do.
  • Two lines—Central African Republic and South Sudan—end early because we saw onsets of mass-killing episodes in those countries. The asterisks indicate the dates on which we made those declarations and therefore closed the relevant questions.
  • Most but not all of these questions ask specifically about state-led mass killings, and some focus on specific target groups (e.g., the Rohingya in Burma) or geographic regions (the North Caucasus in Russia) as indicated.
Crowd-Estimated Probabilities of Mass-Killing Onset Before 1 January 2015

Crowd-Estimated Probabilities of Mass-Killing Onset Before 1 January 2015

I look at that chart and conclude that this process is working reasonably well so far. In the six months since we started running this system, the two countries that have seen onsets of mass killing are both ones that our forecasters promptly and consistently put on the high side of 50 percent. Nearly all of the other cases, where mass killings haven’t yet occurred this year, have stuck on the low end of the scale.

I’m also gratified to see that the system is already generating the kind of dynamic output we’d hoped it would, even with fewer than 100 forecasters in the pool. In the past several weeks, the forecasts for both Burma and Iraq have risen sharply, apparently in response to shifts in relevant policies in the former and the escalation of the civil war in the latter. Meanwhile, the forecast for Uighurs in China has risen steadily over the year as a separatist rebellion in Xinjiang Province has escalated and, with it, concerns about a harsh government response. These inflection points and trends can help identify changes in risk that warrant attention from organizations and individuals concerned about preventing or mitigating these potential atrocities.

Finally, I’m also intrigued to see that our opinion pool seems to be sorting cases into a few clusters that could be construed as distinct tiers of concern. Here’s what I have in mind:

  • Above the 50-percent threshold are the high risk cases, where forecasters assess that mass killing is likely to occur during the specified time frame.  These cases won’t necessarily be surprising. Some observers had been warning on the risk of mass atrocities in CAR and South Sudan for months before those episodes began, and the plight of the Rohingya in Burma has been a focal point for many advocacy groups in the past year. Even in supposedly “obvious” cases, however, this system can help by providing a sharper estimate of that risk and giving a sense of how it is trending over time. In the case of Burma, for example, it is the separation that has happened in the last several weeks that tells the story of a switch from possible to likely and thus adds a degree of urgency to that warning.
  • A little farther down the y-axis are the moderate risk cases—ones that probably won’t suffer mass killing during the period in question but could more readily tip in that direction. In the chart above, Iraq, Sudan, Pakistan, Bangladesh, and Burundi all land in this tier, although Iraq now appears to be sliding into the high risk group.
  • Clustered toward the bottom are the low risk cases where the forecasters seem fairly confident that mass killing will not occur in the near future. In the chart above, Russia, Afghanistan, and Ethiopia are the cases that land firmly in this set. China (Uighurs) remains closer to them than the moderate risk tier, but it appears to be creeping toward the moderate-risk group. We are also running a question about the risk of state-led mass killing in Rwanda before 2015, and it currently lands in this tier, with a forecast of 14 percent.

The system that generates the data behind this chart is password protected, but the point of our project is to make these kinds of forecasts freely available to the global public. We are currently building the web site that will display the forecasts from this opinion pool in real time to all comers and hope to have it ready this fall.

In the meantime, if you think you have relevant knowledge or expertise—maybe you study or work on this topic, or maybe you live or work in parts of the world where risks tend to be higher—and are interested in volunteering as a forecaster, please send an email to us at

Reform in Burma Isn’t Unraveling (Yet), But Our Narrative About It Sure Is

If a couple of recent pieces on Foreign Policy‘s website are to be believed, the democratization process that sputtered to life in Burma two and a half years ago has stalled and is now rolling back downhill. In “Hillary’s Burma Problem,” Catherine Traywick and John Hudson argue that “the promise of a free and democratic Myanmar is rapidly receding as sectarian violence escalates and the government backslides on a number of past reforms.” Meanwhile, Democracy Lab blogger Min Zin tells us that, for the past few months, he’s been “unable to escape an ominous sense that the political situation in Burma is on the wrong track,” and he points to a leadership crisis and a growing risk of social unrest as the chief sources of his anxiety.

I won’t dispute any of the facts in those pieces, and I’ve been an avid reader of Min Zin’s excellent Democracy Lab posts as long as he’s been writing them. As I argued on this blog a couple of years ago, though, I think it’s more accurate to think of what’s happened in Burma so far not as a transition to democracy, but as a case of liberalization from above that may or may not produce a try at democratic government in the next few years.

Is that a distinction without a difference? I don’t think so. As O’Donnell and Schmitter propose in their Little Green Book, liberalization involves the expansion of freedoms from arbitrary acts of the state and others, while democratization entails the expansion of popular consultation and accountability. The two processes often coincide, but they are usefully construed as distinct streams of political change. Crucially, while democratic government is impossible without civil liberties—especially freedoms of speech, association, and assembly—liberalization can and sometimes does occur without any democratization.

Understood on those terms, I think the liberalization process in Burma has progressed incrementally but significantly in the past two years and has not yet regressed in any substantial way, with the partial but significant exception of the plight of the Rohingya. What Burma’s liberalization has done is create space for new political and economic activity, and as is often the case, not all of what people are doing with that space is progressive or good. On the positive side of the ledger, freedoms of speech and the press remain incomplete but are much improved. Political prisoners have been released and not restocked. Apparently, there’s even a budding startup scene in Yangon. On the negative side of the ledger, the prospect of new fortunes is spurring land grabs by elites, and attempts to protest those displacements and the pollution that sometimes follow have largely been ignored or harshly repressed. And, of course, some Burmans have responded to the opening by mobilizing around an aggressive chauvinism that has already produced what amounts to a slow-rolling episode of ethnic cleansing and still threatens to slide into genocide.

As is sometimes but not always the case, this partial liberalization has also been accompanied by some significant but still limited elements of democratization, too. Parliamentary by-elections were held in 2012, opposition parties won nearly all the seats at stake, and no one shut the process down. More recently, word came that the National League for Democracy, the party of ostensible opposition leader Aung Saan Suu Kyi, would field a candidate for president in balloting scheduled for next year, even if Suu Kyi herself is not permitted to run.

What we still haven’t seen, though, is any clear sign that deeply entrenched elites plan to allow that process to threaten their station. Rather, what’s emerged so far is more like the arrangements that hold in monarchies like Morocco or Jordan. There, loyal opposition parties are allowed to contest seats in the legislature, and a certain amount of free discourse and even protest is tolerated, but formal and informal rules ensure that incumbent insiders retain control over the political agenda and veto power over all major decisions.

For that to change in Burma, the country’s constitution would have to change. When military elites rewrote that document a few years ago, however, they cleverly ensured that constitutional reform couldn’t happen without their approval. So far, we have seen no signs that they plan to relinquish that arrangement any time soon. Until we do, I think it’s premature to speak of a transition to democracy in Burma. Democratization, yes, but not enough yet to say that the country is between political orders. What we have now, I think, is a partially liberalized authoritarian regime that’s still led by a military elite with uncertain intentions.

To make sure this view wasn’t crazy, I queried Brian Joseph, senior director for Asia and Global Programs at the National Endowment for Democracy and a longtime Burma watcher who also happens to be the father of one of my son’s classmates. In particular, I asked Brian by email if he agreed with Traywick and Hudson’s thesis that the “transition” in Burma was “unraveling.” He pointed me toward Min Zin’s piece as “a more informative analysis” and said he agreed with Min Zin that “the transition’s trajectory is no longer clear” and then added parenthetically: “Not that I ever thought it was in the first place but that was clearly the message of the [international] community.”

Brian’s reference to “the message of the international community” in that aside is crucial to understanding how what I described here can be true and we can still see analyses claiming that Burma’s “transition” is “unraveling.” Best I can tell, what’s coming undone right now isn’t Burma’s reform process, although as Min Zin discusses, that certainly could happen, and there are plenty of reasons to fear that it might.

No, what I think we’re really seeing in articles like the one by Traywick and Hudson is an overdue deflation of the hype balloon Burma’s reforms have pumped up. With some help from various outsiders—some eager to see deeper political transformations occur, others looking to capitalize on the money-making opportunities this new market presents—we let our hopes for Burma’s future drive our narrative about what was happening in the present. The Arab Spring spurred a similar dynamic in American analysis of that part of the world. Let’s hope the whiplash over Burma isn’t as severe.

A Notable Year of the Wrong Kind

The year that’s about to end has distinguished itself in at least one way we’d prefer never to see again. By my reckoning, 2013 saw more new mass killings than any year since the early 1990s.

When I say “mass killing,” I mean any episode in which the deliberate actions of state agents or other organizations kill at least 1,000 noncombatant civilians from a discrete group. Mass killings are often but certainly not always perpetrated by states, and the groups they target may be identified in various ways, from their politics to their ethnicity, language, or religion. Thanks to my colleague Ben Valentino, we have a fairly reliable tally of episodes of state-led mass killing around the world since the mid-1940s. Unfortunately, there is no comparable reckoning of mass killings carried out by non-state actors—nearly always rebel groups of some kind—so we can’t make statements about counts and trends as confidently as I would like. Still, we do the best we can with the information we have.

With those definitions and caveats in mind, I would say that in 2013 mass killings began:

Of course, even as these new cases have developed, episodes of mass killings have continued in a number of other places:

In a follow-up post I hope to write soon, I’ll offer some ideas on why 2013 was such a bad year for deliberate mass violence against civilians. In the meantime, if you think I’ve misrepresented any of these cases here or overlooked any others, please use the Comments to set me straight.

The Fog of War Is Patchy

Over at Foreign Policy‘s Peace Channel, Sheldon Himmelfarb of USIP has a new post arguing that better communications technologies in the hands of motivated people now give us unprecedented access to information from ongoing armed conflicts.

The crowd, as we saw in the Syrian example, is helping us get data and information from conflict zones. Until recently these regions were dominated by “the fog war,” which blinded journalists and civilians alike; it took the most intrepid reporters to get any information on what was happening on the ground. But in the past few years, technology has turned conflict zones from data vacuums into data troves, making it possible to render parts the conflict in real time.

Sheldon is right, but only to a point. If crowdsourcing is the future of conflict monitoring, then the future is already here, as Sheldon notes; it’s just not very evenly distributed. Unfortunately, large swaths of the world remain effectively off the grid on which the production of crowdsourced conflict data depends. Worse, countries’ degree of disconnectedness is at least loosely correlated with their susceptibility to civil violence, so we still have the hardest time observing some of the world’s worst conflicts.

The fighting in the Central African Republic over the past year is a great and terrible case in point. The insurgency that flared there last December drove the president from the country in March, and state security forces disintegrated with his departure. Since then, CAR has descended into a state of lawlessness in which rival militias maraud throughout the country and much of the population has fled their homes in search of whatever security and sustenance they can find.

We know this process is exacting a terrible toll, but just how terrible is even harder to say than usual because very few people on hand have the motive and means to record and report out what they are seeing. At just 23 subscriptions per 100 people, CAR’s mobile-phone penetration rate remains among the lowest on the planet, not far ahead of Cuba’s and North Korea’s (data here). Some journalists and NGOs like Human Rights Watch and Amnesty International have been covering the situation as best they can, but they will be among the first to tell you that their information is woefully incomplete, in part because roads and other transport remain rudimentary. In a must-read recent dispatch on the conflict, anthropologist Louisa Lombard noted that “the French colonists invested very little in infrastructure, and even less has been invested subsequently.”

A week ago, I used Twitter to ask if anyone had managed yet to produce a reasonably reliable estimate of the number of civilian deaths in CAR since last December. The replies I received from some very reputable people and organizations makes clear what I mean about how hard it is to observe this conflict.

C.A.R. is an extreme case in this regard, but it’s certainly not the only one of its kind. The same could be said of ongoing episodes of civil violence in D.R.C., Sudan (not just Darfur, but also South Kordofan and Blue Nile), South Sudan, and in the Myanmar-China border region, to name a few. In all of these cases, we know fighting is happening, and we believe civilians are often targeted or otherwise suffering as a result, but our real-time information on the ebb and flow of these conflicts and the tolls they are exacting remains woefully incomplete. Mobile phones and the internet notwithstanding, I don’t expect that to change as quickly as we’d hope.

[N.B. I didn’t even try to cover the crucial but distinct problem of verifying the information we do get from the kind of crowdsourcing Sheldon describes. For an entry point to that conversation, see this great blog post by Josh Stearns.]

A Faith-Based Initiative of Another Kind

According to AP, the U.S. government is considering deepening its ties with Myanmar’s military again, to include a re-up of the human-rights training programs American soldiers and lawyers do with scores of other countries and have done in Myanmar before.

With the backing of opposition leader Aung San Suu Kyi, U.S. defense legal experts last week made their second trip to Myanmar in two months, scoping out what help they can provide on teaching about human rights and the rule of law…

With a quasi-civilian government in place and national elections due in 2015, the Obama administration argues that talking “soldier-to-soldier” with Myanmar on issues like military justice and military-civilian relations can encourage reform and help the U.S. build ties with a military it knows little about…

Priscilla Clapp, a former U.S. charge d’affaires in Yangon, says standing on the sidelines doesn’t serve U.S. interests. “We need to reach into the organization of the military and help educate people and expose them to new ideas,” she said.

The idea that these training programs deepen the recipient military’s commitment to democracy and human rights is essentially a matter of faith. As a GAO report referenced in the AP story makes clear, we have no idea how effective these programs are because we haven’t really tried to measure their impact.

Since 1976, the International Military Education and Training (IMET) program has provided education and training to foreign military personnel. The program’s objectives include professionalizing military forces and increasing respect for democratic values and human rights…

State and DOD’s ability to assess IMET’s effectiveness is limited by several weaknesses in program monitoring and evaluation. First, State and DOD have not established a performance plan for IMET that explains how the program is expected to achieve its goals and how progress can be assessed through performance measures and targets. Second, State and DOD have limited information on most IMET graduates, due to weaknesses in efforts to monitor these graduates’ careers after training…Third, the agencies’ current evaluation efforts include few of the evaluation elements commonly accepted as appropriate for measuring progress of training programs, and do not objectively measure how IMET contributes to long-term, desired program outcomes.

Even in the absence of rigorous monitoring and evaluation, a cursory review of relevant cases makes it hard to accept the premise that these programs are having the presumed effect. Egypt’s military has been the beneficiary of these programs (and much, much more) from the U.S. for many years, and they’ve just perpetrated a coup and a mass killing in the span of a single summer. As the Washington Post reported last year, the leader of Mali’s 2012 coup, then-Capt. Amadou Sanogo, “received military training in the U.S. on ‘several occasions’,” as did many of his compatriots. A high-profile murder trial underway right now in Indonesia involves a dozen troops from a special-forces unit that received training and assistance from the U.S. for many years, even as they were committing gross human-rights violations. So far, I haven’t even mentioned the School of the Americas. The list goes on and on and on.

And, of course, there’s the profound irony that the U.S. did exactly this kind of training in Myanmar before, for eight years. As that AP story notes,

The U.S. financed $4.7 million in military sales delivered to Myanmar between 1980 and 1988, and trained 167 officers at American military schools under [IMET].

Why did those sales and training suddenly stop in 1988? Oh, yeah

Near the start of this post, I claimed that American officials’ and officers’ belief in these programs’ effectiveness is a matter of faith. A cynic might point out that effectiveness depends on the goal. If the goal is to discourage military partners from intervening in their home countries’ politics and committing gross human-rights violations, the litany of historical counter-examples makes it hard for a civilian social scientist like me to understand how that faith is sustained. If, however, the goal is to provide a fig leaf for partnerships our government pursues for other reasons, then IMET seems to be working just fine.

Mass Killing in Egypt

Let’s define a state-led mass killing as an episode in which state security forces or groups acting at their behest deliberately kill at least 1,000 noncombatant civilians from a discrete group in a relatively short period of time—weeks, months, or maybe even several years. This is a paraphrased version of the definition my colleague Ben Valentino developed for a U.S. government-funded research project, so using it allows us to identify and compare many episodes over time, as I did in another recent post.

Since World War II, nearly all of the state-led mass killings that have occurred around the world have followed one of three basic scenarios, all of them involving apparent threats to rulers’ power.

First and most common, state security forces fighting an insurgency or locked in a civil war kill large numbers of civilians whom they accuse of supporting their rivals, or sometimes just kill indiscriminately. The genocide in Guatemala is an archetypal example of this scenario. In some cases, like Rwanda, the state also enlists militias or even civilians to assist in that killing.

Second, rulers confronting budding threats to their power—usually a nonviolent popular uprising or coup plot—violently repress and attack their challengers in an attempt to quash the apparent threat. The anti-communist massacres in Indonesia in 1965-1966 fit this pattern. In rare cases, like North Korea today, just the possibility of such a threat suffices to draw the state into killing large numbers of civilians. More often, state repression of nonviolent uprisings succeeds in quashing the challenge with fewer than 1,000 civilian deaths, as happened in China in 1989, Uzbekistan in 2005, and Burma in 2007.

Third, rulers who have recently seized power by coup or revolution sometimes kill large numbers of civilian supporters of the faction they have just replaced as part of their efforts to consolidate their power. The mass killings carried out by the Khmer Rouge in Cambodia in the late 1970s are probably the most extreme example of this scenario, but Argentina’s “dirty war” and the long-running political purges that began in several East European countries after World War II also fit the pattern.

What happened in Egypt yesterday looks like a slide into the third scenario. Weeks after a military coup toppled Mohamed Morsi, state security forces violently assaulted crowds using nonviolent action to protest the coup and demand Morsi’s restoration to the presidency. The death toll from yesterday’s ruthless repression has already surpassed 500 and seems likely to rise further as more of the wounded die and security forces continue to repress further attempts at resistance and defiance. What’s more, the atrocities of the past 24 hours come on top of the killings of scores if not hundreds of Brotherhood supporters around the country over the past several weeks (see this spreadsheet maintained by The Guardian for details).

One of the many rationalizations offered for the July 3 coup was the argument that the Muslim Brotherhood had used violence to suppress its political rivals during and after mass protests against Morsi last December. People were right to challenge the Muslim Brotherhood over that thuggery, which was arguably a nascent version of the second scenario described above. In calling on the military to deliver them from that threat, however, some of those challengers seem to have struck a Faustian bargain that is now producing killings on a much grander scale.

%d bloggers like this: