Asking the Right Questions

This is a cross-post from the Good Judgment Project’s blog.

I came to the Good Judgment Project (GJP) two years ago, in Season 2, as a forecaster, excited about contributing to an important research project and curious to learn more about my skill at prediction. I did pretty well at the latter, and GJP did very well at the former. I’m also a political scientist who happened to have more time on my hands than many of my colleagues, because I work as an independent consultant and didn’t have a full plate at that point. So, in Season 3, the project hired me to work as one of its lead question writers.

Going into that role, I had anticipated that one of the main challenges would be negotiating what Phil Tetlock calls the “rigor-relevance trade-off”—finding questions that are relevant to the project’s U.S. government sponsors and can be answered as unambiguously as possible. That forecast was correct, but even armed with that information, I failed to anticipate just how hard it often is to strike this balance.

The rigor-relevance trade-off exists because most of the big questions about global politics concern latent variables. Sometimes we care about specific political events because of their direct consequences, but more often we care about those events because of what they reveal to us about deeper forces shaping the world. For example, we can’t just ask if China will become more cooperative or more belligerent, because cooperation and belligerence are abstractions that we can’t directly observe. Instead, we have to find events or processes that (a) we can see and (b) that are diagnostic of that latent quality. For example, we can tell when China issues another statement reiterating its claim to the Senkaku Islands, but that happens a lot, so it doesn’t give us much new information about China’s posture. If China were to fire on Japanese aircraft or vessels in the vicinity of the islands—or, for that matter, to renounce its claim to them—now that would be interesting.

It’s tempting to forego some rigor to ask directly about the latent stuff, but it’s also problematic. For the forecast’s consumers, we need to be able to explain clearly what a forecast does and does not cover, so they can use the information appropriately. As forecasters, we need to understand what we’re being asked to anticipate so we can think clearly about the forces and pathways that might or might not produce the relevant outcome. And then there’s the matter of scoring the results. If we can’t agree on what eventually happened, we won’t agree on the accuracy of the predictions. Then the consumers don’t know how reliable those forecasts are, the producers don’t get the feedback they need, and everyone gets frustrated and demotivated.

It’s harder to formulate rigorous questions than many people realize until they try to do it, even on things that seem like they should be easy to spot. Take coups. It’s not surprising that the U.S. government might be keen on anticipating coups in various countries for various reasons. It is, however, surprisingly hard to define a “coup” in such a way that virtually everyone would agree on whether or not one had occurred.

In the past few years, Egypt has served up a couple of relevant examples. Was the departure of Hosni Mubarak in 2011 a coup? On that question, two prominent scholarly projects that use similar definitions to track coups and coup attempts couldn’t agree. Where one source saw an “overt attempt by the military or other elites within the state apparatus to unseat the sitting head of state using unconstitutional means,” the other saw the voluntary resignation of a chief executive due to a loss of his authority and a prompt return to civilian-led government. And what about the ouster of Mohammed Morsi in July 2013? On that, those academic sources could readily agree, but many Egyptians who applauded Morsi’s removal—and, notably, the U.S. government—could not.

We see something similar on Russian military intervention in Ukraine. Not long after Russia annexed Crimea, GJP posted a question asking whether or not Russian armed forces would invade the eastern Ukrainian cities of Kharkiv or Donetsk before 1 May 2014. The arrival of Russian forces in Ukrainian cities would obviously be relevant to U.S. policy audiences, and with Ukraine under such close international scrutiny, it seemed like that turn of events would be relatively easy to observe as well.

Unfortunately, that hasn’t been the case. As Mark Galeotti explained in a mid-April blog post,

When the so-called “little green men” deployed in Crimea, they were very obviously Russian forces, simply without their insignia. They wore Russian uniforms, followed Russian tactics and carried the latest, standard Russian weapons.

However, the situation in eastern Ukraine is much less clear. U.S. Secretary of State John Kerry has asserted that it was “clear that Russian special forces and agents have been the catalyst behind the chaos of the last 24 hours.” However, it is hard to find categorical evidence of this.

Even evidence that seemed incontrovertible when it emerged, like video of a self-proclaimed Russian lieutenant colonel in the Ukrainian city of Horlivka, has often been debunked.

This doesn’t mean we were wrong to ask about Russian intervention in eastern Ukraine. If anything, the intensity of the debate over whether or not that’s happened simply confirms how relevant this topic was. Instead, it implies that we chose the wrong markers for it. We correctly anticipated that further Russian intervention was possible if not probable, but we—like many others—failed to anticipate the unconventional forms that intervention would take.

Both of these examples show how hard it can be to formulate rigorous questions for forecasting tournaments, even on topics that are of keen interest to everyone involved and seem like naturals for the task. In an ideal world, we could focus exclusively on relevance and ask directly about all the deeper forces we want to understand and anticipate. As usual, though, that ideal world isn’t the one we inhabit. Instead, we struggle to find events and processes whose outcomes we can discern that will also reveal something telling about those deeper forces at play.

 

Bangladesh as Archetype of Contemporary Political Development

If you want to get a feel for the political muddles that trap most countries for decades on a sine wave of democratization and de-democratization, and why durable exits from those oscillations are so hard to come by, you might want to take a look at Bangladesh.

Bangladesh won its independence from Pakistan in 1971 after a genocidal struggle that left hundreds of thousands dead and displaced tens of millions. Since then, the country has roughly split its time between democratic and authoritarian rule. As happened in many newly independent states in the twentieth century, the champions of national independence came to power through elections and then refused to leave. Also typically, the one-party regime born of that refusal soon fell to a restive military. Seventeen years passed before another fairly-elected civilian government came to power, starting the longest spell of more or less democratic government in the country’s still-short history.

Over the ensuing two decades, the core feature of politics in Bangladesh has been acute polarization. Whenever elections approach, the rival Awami League (AL) and Bangladesh Nationalist Party (BNP) engage in bitter public showdowns that bring tens of thousands of supporters into the streets and often produce low-level violence on the margins. Unsurprisingly, the two parties carry that same animosity into government. “Once a party is in power in Bangladesh,” the Economist recently noted, “it is the unalterable tradition to declare nearly everything decreed by your opponents to be null and void.”

Meanwhile, the military has continued to play a more active role in politics than democratic theory would allow. In 2007, as elections approached and the cyclical clash between the AL and BNP cranked up, Bangladesh’s military leaders apparently saw intervention as the lesser of a few evils and tossed their civilian rulers. Two years passed under a caretaker government of the military’s choosing. Civilian supremacy returned at the end of 2008, when the AL won elections widely regarded as the fairest in Bangladesh’s history, but according to the International Crisis Group, Bangladesh’s military remains “visibly restive”:

On 19 January [2012, the military] announced it had foiled a coup by mid-level and retired officers who sought to install an Islamist government. This followed an assassination attempt on an AL member of parliament in October 2009 by mid-level officers seething over the deaths of 57 officers in a mutiny by their subordinate paramilitary border guards the previous February. Large-scale dismissals, forced retirements, deepening politicisation and a heavy-handed approach to curb dissent and root out militants have created an unstable and undisciplined force.

The systemic result of this struggle between two political rivals and the military is the familiar “truel,” or Mexican standoff, that characterizes politics in many countries stuck between stable dictatorship and durable democracy. The defining feature of this standoff is each player’s uncertainty about its rivals’ intentions; no one trusts that the others won’t make a grab for power and then shut out or destroy the others. That uncertainty, in turn, sharply increases the odds of undemocratic behavior, because even players fully committed to democracy in principle might feel pressed to cement or usurp power in order to block their distrusted rivals from doing the same to them first.

Now, in late 2013, elections are due again, and Bangladesh seems to be spiraling toward another local climax of this cyclical confrontation. As Reuters reports, the AL and BNP have called competing rallies in the capital this Friday, and at least one party leader has told followers to come “prepared with arms.” Already this year, state security forces have killed scores of protesters  in unrest spawned by the workings of a war-crimes tribunal that many BNP sympathizers see as a political bludgeon directed against them. According to my statistical forecasts, Bangladesh ranks among the 20 countries in the world most susceptible to coup attempts this year, a result that confirms many observers’ concerns that the military might respond to wider disorder as it did in 2007.

So how does a country get off of this roller coaster? Attempts to induce democratic consolidation often focus on institutional design, but Bangladesh shows how this prescription is more easily written than filled.

One of the focal points in the current confrontation is the AL government’s recent decision to dispense with an arrangement whereby a caretaker body would replace the elected government in the run-up to elections. The BNP has cast that decision as an attempt by the ruling AL to tilt the upcoming election in its own favor. Ironically, though, the caretaker arrangement has often been the focal point of mutual recriminations in past elections, as the two parties would fight over whether or not the caretakers were sufficiently unbiased.

In other words, the system that was meant to dampen that mutual distrust only seemed to end up stoking it, but when one party finally made a change, that act is seen through the same lens. The fundamental problem with expecting rule changes to induce democratic consolidation is that the process of institutional design and change is itself political, so it is subject to the same pathologies and touches off the same worries.

Outsiders can also exhort party leaders to negotiate in good faith, but parties aren’t unitary actors. Those leaders sit atop a massive pyramid of principal-agent problems, and internal rivals often respond opportunistically to attempts at compromise by stoking fears of capitulation and offering themselves as the bulwark against it. Aware of this risk, those leaders rarely take the first step.

The histories of Europe and Latin America imply that Bangladesh will eventually find a way out of these oscillations onto a new equilibrium that includes durable democracy. Unfortunately, the history of countries born in the past half-century—never mind a cursory look at the politics on the streets of Dhaka right now—suggests this election cycle probably isn’t the moment that’s going to happen.

Two Forecasting Lessons from a Crazy Football Season

My younger son is a huge fan of the Baltimore Ravens, and his enthusiasm over the past several years has converted me, so we had a lot of fun (and gut-busting anxiety) watching the Super Bowl on Sunday.

As a dad and fan, my favorite part of the night was the Baltimore win. As a forecaster, though, my favorite discovery of the night was a web site called Advanced NFL Stats, one of a budding set of quant projects applied to the game of football. Among other things, Advanced NFL Stats produces charts of the probability that either team will win every pro game in progress, including the Super Bowl. These charts are apparently based on a massive compilation of stats from games past, and they are updated in real time. As we watched the game, I could periodically refresh the page on my mobile phone and give us a fairly reliable, up-to-the-minute forecast of the game’s outcome. Since the Super Bowl confetti has settled, I’ve spent some time poking through archived charts of the Ravens’ playoff run, and that exercise got me thinking about two lessons for forecasters.

1. Improbable doesn’t mean impossible.

To get to the Super Bowl, the Ravens had to beat the Denver Broncos in the divisional round of the playoffs. Trailing by seven with 3:12 left in that game, the Ravens turned the ball over to Denver on downs at the Broncos’ 31-yard line. To win from there, the Ravens would need a turnover or quick stop; then a touchdown; then either a successful two-point conversion or a first score in overtime.

As the chart below shows, the odds of all of those things coming together were awfully slim. At that point—just before “Regulation” on the chart’s bottom axis—Advanced NFL Stats’ live win-probability graph gave the Ravens roughly a 1% chance of winning. Put another way, if the game could be run 100 times from that position, we would only expect to see Baltimore win once.

ravens_broncos_realtime_forecast

Well, guess what happened? The one-in-a-hundred event, that’s what. Baltimore got the quick stop they needed, Denver punted, Joe Flacco launched a 70-yard bomb down the right sideline to Jacoby Jones for a touchdown, the Ravens pushed the game into overtime, and two minutes into the second extra period at Mile High Stadium, Justin Tucker booted a 47-yard field goal to carry Baltimore back to the AFC Championship.

For Ravens’ fans, that outcome was a %@$# miracle. For forecasters, it was a great reminder that even highly unlikely events happen sometimes. When Nate Silver’s model indicates on the eve of the 2012 election that President Obama has a 91% chance of winning, it isn’t saying that Obama is going to win. It’s saying he’s probably going to win, and the Ravens-Broncos game reminds us that there’s an important difference. Conversely, when a statistical model of rare events like coups or mass killings identifies certain countries as more susceptible than others, it isn’t necessarily suggesting that those highest-risk cases are definitely going to suffer those calamities. When dealing with events as rare as those, even the most vulnerable cases will escape most years without a crisis.

The larger point here is one that’s been made many times but still deserves repeating: no single probabilistic forecast is plainly right and wrong. A sound forecasting process will reliably distinguish the more likely from the less likely, but it won’t attempt to tell us exactly what’s going to happen in every case. Instead, the more accurate the forecasts, the more closely the frequency of real-world outcomes or events will track the predicted probabilities assigned to them. If a meteorologist’s model is really good, we should end up getting wet roughly half of the times she tells us there’s a 50% chance of rain. And almost every time the live win-probability graph gives a football team a 99% chance of winning, they will go on to win that game—but, as my son will happily point out, not every time.

2. The “obvious” indicators aren’t always the most powerful predictors.

Take a look at the Advanced NFL Stats chart below, from Sunday’s Super Bowl. See that sharp dip on the right, close to the end? Something really interesting happened there: late in the game, Baltimore led on score (34-29) but trailed San Francisco in its estimated probability of winning (about 45%).

ravens_49ers_realtime_forecast

How could that be? Consideration of the likely outcomes of the next two possessions makes it clearer. At the time, San Francisco had a first-and-goal situation from Baltimore’s seven yard line. Teams with four shots at the end zone from seven yards out usually score touchdowns, and teams that get the ball deep in their own territory with a two- or three-point deficit and less than two minutes to play usually lose. In that moment, the live forecast confirmed the dread that Ravens fans were feeling in our guts: even though San Francisco was still trailing, the game had probably slipped away from Baltimore.

I think there’s a useful lesson for forecasters in that peculiar situation: the most direct indicators don’t tell the whole story. In football, the team with a late-game lead is usually going to win, but Advanced NFL Stats’ data set and algorithm have uncovered at least one situation where that’s not the case.

This lesson also applies to efforts to forecasts political processes, like violent conflict and regime collapse. With the former, we tend to think of low-level violence as the best predictor of future civil wars, but that’s not always true. It’s surely a valuable piece of information, but there are other sources of positive and negative feedback that might rein in incipient violence in some cases and produce sudden eruptions in others. Ditto for dramatic changes in political regimes. Eritrea, for example, recently had some sort of mutiny and North Korea did not, but that doesn’t necessarily mean the former is closer to breaking down than the latter. There may be features of the Eritrean regime that will allow it to weather those challenges and aspects of the North Korean regime that predispose it to more abrupt collapse.

In short, we shouldn’t ignore the seemingly obvious signals, but we should be careful to put them in their proper context, and the results will sometimes be counter-intuitive.

Oh, and…THIS:

flacco_confetti

Has Africa Gone Coup-Crazy in 2012?

Guinea-Bissau’s armed forces violently seized control of the country’s capital yesterday in an apparent coup d’etat. This is the second successful coup in West Africa in the past month–the other happened in Mali in mid-March–and, if my Twitter feed is any indication, this pair of events has a lot of people wondering if 2012 is going to be an unusually “hot” year for coups in that part of the world.

Statistically speaking, the answer seems to be “no”–or “not yet,” anyway, and it still has a ways to go to get there.

To see if 2012 is shaping up to be a weird year for coup activity in Africa, I used the ‘bcp’ package in R to apply a technique called Bayesian change point detection (BCP) to annual counts of successful and failed coup attempts in the region from 1946 through 2012 (so far). BCP treats time-series data as a collection of independent and identically distributed partitions and looks for points in that series where the data’s generative parameters appear to change. My data on coup events come from the Center for Systemic Peace.

The results are shown below. The top half of the chart plots the observed annual counts (the dots) and the posterior means for those annual counts (the line). The real action, though, is in the bottom half, which plots the posterior probabilities of a change point. The higher that number, the more confident we are that a particular year marks a sudden change. In this series, we see evidence of three change points: one in the mid-1960s, a few years after the start of decolonization; another in the early 1990s, after the end of the Cold War; and a third in the late 1990s, when the rate of coups in the region takes a sharp dip. Meanwhile, the pair of events observed so far in 2012 looks perfectly normal, just about average for the past decade and still well below the recent peak of six events in 2008.

If two coup bids in 2012 does not an aberration make, how many would we need to see this year to call it a significant change? I reran the BCP analysis several times using ever-larger counts for 2012, and it took a big jump to start moving the posterior probability of a change point in any appreciable way. At five events, the posterior probability still hadn’t moved much. At six, it finally moved appreciably, but only to around 0.2. In the end, it took eight events to push the posterior probability over 0.5.

In other words, it would take a lot more than two coup bids in 2012 to mark a significant change from the recent past, and what we’ve seen this year so far looks like normal variation in a stochastic process. Event counts are often noisy, but our pattern-seeking brains still try to find meaning in those small variations. It’s also harder to remember less recent events, and our brains tend to confuse that difficulty with infrequency. It helps to remember those biases whenever a new event starts you thinking about a trend.

NOTE: This version of the plot and the scenario analysis corrects an error in the data used in the original post. For the first run, I forgot that my analysis file ended in 2010, so the 0 events shown for 2011 was a mistake. There were actually two failed coups in Africa last year, one in the DRC in February and another in Guinea in July. With those two events added to the data set, the first third of 2012 looks even more typical than it did before.

Why We Shouldn’t Be Quite So Surprised by the Coup in Mali

Soldiers toppled the government of Mali in a coup d’etat yesterday. As Stanford Ph.D. candidate Ken Opalo notes on his blog, this turn of events has caught many people by surprise, because Mali has long been regarded as a democratic standout in Africa.

Since (re)democratization in the early 1990s Mali has routinely been cited as a case of democratic consolidation despite seemingly insurmountable odds (poor HDI scores, etc.). The current developments, however, raise serious questions with regard to whether the Malian political and military elite have wholly bought into the idea of settling their battles for power and influence at the ballot.

As it happens, the risk of a coup attempt in Mali in 2012 was more apparent in a statistical forecasting exercise I did at the start of the year. According to that analysis, Mali was the 10th riskiest country in the world, ranking behind nine other African countries–most of which, unlike Mali, have suffered coup attempts in the past few years–and Bangladesh.

The statistical modeling isn’t as complicated as it sounds. That analysis pushed Mali toward the top of the list because Mali’s structural conditions in 2011 look a lot like conditions in other countries that have suffered coup attempts in recent decades.

I wonder, though, if a coup in Mali also seems surprising because we’ve been overstating how democratic that country really was. Since 1992, when Mali began holding competitive multiparty elections, many observers have called out Mali as an African success story, an inspiring example of how democratization can progress under challenging conditions.

That’s not what I saw, however, when I took an admittedly cursory look at politics in Mali several years ago, while making data for a research project on transitions to and from democracy. At the time, I saw legislative elections in early 1997 that had been plagued by serious flaws, and many of the accounts I read implicated members of the leading Alliance for Democracy in Mali (ADEMA) party in the discovered instances of electoral misconduct. Under an agreement between the incumbent president Alfred Konare and several opposition leaders, the Constitutional Court annulled the results later that month, but the court refused to reschedule the presidential contest, and Konare cruised to re-election with nearly 96 percent of the vote when the opposition boycotted. Meanwhile, Amnesty International reported that dozens of members and supporters of the opposition had been arrested ahead of the elections, and some were allegedly tortured. When legislative elections were re-run later in 1997, Konare’s allies won a large majority of seats, effectively consolidating the ruling party’s grip on power by questionable means.

When I mentioned my take on Malian democracy on Twitter this morning, I heard some affirmations, but I also got some pushback. Senam Beheton, for example, argued that, “Regardless of Western plaudits, Mali stood out because the process was driven by Malians based on Mali’s interests.”

I concede that Mali is an ambiguous case, whatever your precise definition of democracy. Still, the surprise many people are expressing about the coup makes me wonder about the consequences of our lowered expectations for democratization in poor countries, and perhaps for Africa in particular. For logistical reasons alone, it’s really hard to hold fair elections. And, as anyone who’s spent any time watching politics can tell you, the logistical challenges are only part of the problem; people everywhere will also do all sorts of things in search of an edge. When we see this stuff in rich countries, we call it a crime. When we see it in poor countries, though, we’re more likely to excuse it as growing pains or technical difficulties.

In school, we’d call that grading on a curve, and it’s not necessarily a bad thing. As I noted in a previous post, there’s often an instrumental quality to Western narratives about democratization in places like Mali. Looking for exemplars that might inspire other societies, we sometimes choose to ignore or downplay procedural flaws that would raise howls in other contexts. For purposes of democracy promotion, that might even be a sound idea.

Still, in the wake of Mali’s coup, I can’t help wondering if all that cheerleading isn’t part of why we’re so surprised and confused today. I see similar problems in our thinking about Senegal, another supposed exemplar of democracy in Africa where an elected president has tightened his grip on power. Ditto for Ukraine, which went from Orange Revolution darling to creeping authoritarianism in about the same amount of time it took Mali to make its slide in the 1990s. When we keep telling ourselves that things are going great, we often stop refreshing our view and miss the signs of decline and change. “Surely it can’t happen here” turns out to be a pretty dangerous idea.

Why Egyptians Should Care about the Maldives Coup

I knew nothing about the Maldives until it popped into the news this week, but what I’m seeing there now looks very familiar, as it should to anyone who studies how new democratic regimes so often sputter and fail.

The Republic of Maldives is a tiny archipelago state off the southern tip of India with a population of only about 314,000. Fish are its leading export, but its economy depends most heavily on beach tourism. The Maldives gained independence from the British in 1965 and was ruled for most of the ensuing 45 years by one man, Mamoun Abdul Gayoom. Years of pro-democracy activism finally spurred the government to open the door to multiparty politics in 2003, and the state became a democracy in 2008 when its first free and fair elections delivered the presidency to longtime activist leader Mohamed Nasheed.

The 2008 elections terminated a long period of authoritarian rule, but they did not instantly transform the fundamentals of the political economy that developed under al-Gayoom’s government. That transformation would require deeper change, and President Nasheed’s efforts to bring about those reforms seem to be what recently got him into trouble. In 2010, the New York Times reported:

The government of the Maldives wants its money back — $400 million to be precise. That is the amount that it estimates was looted by its former president, Maumoon Abdul Gayoom, and his associates. Mr. Gayoom dominated politics in the Maldives, a tiny Indian Ocean nation, for 30 years. After winning six successive single-party elections, he finally bowed to popular pressure and allowed open elections in 2008. He lost. He is one of a number of politically connected figures — some alive, others dead — who are the targets of increasingly coordinated efforts to repatriate misappropriated funds. Results to date have been encouraging, but much more can be done, officials and development experts say. A report from the Maldives’ national auditor released in 2009 reads like a guidebook on self-enrichment. The president’s spending was “out of control,” it said, as Mr. Gayoom used his power to live a lavish lifestyle and extend largesse to those around him.

As President Nasheed’s administration struggled “to get its money back,” it found its efforts impeded. When the president tried to overcome one small piece of that resistance by ordering the arrest of an uncooperative criminal court judge, the judge refused to go, and the president’s opponents took to the streets to protest. The Times described the final spiral this way:

Recently, Mr. Nasheed’s popularity has suffered as the economy of the Maldives has struggled. Then, last month, Mr. Nasheed ordered the military to arrest the criminal court judge, Abdulla Mohamed, accusing the judge of acting on behalf of Mr. Gayoom and compromising the fairness of the country’s courts. The arrest, which was widely condemned, prompted the nightly protests in Male that peaked on Monday. “The real catalyst, last night, was that the police decided that they wouldn’t disperse the protesters,” said Mohamed Hussain Shareef, the spokesman for Mr. Gayoom’s party, the Progressive Party of Maldives. Mr. Shareef contended that soldiers had balked as well as the police. “We were told that the army was also asked to disperse the protesters using live rounds,” he said. The Associated Press reported that troops had initially fired rubber bullets. S. Ahmed Shiyam, a police subinspector in Male, said there were clashes between police officers and soldiers on Monday evening and early Tuesday morning, with some of the protesters joining on the police side. Then some soldiers switched sides as well, he said. An official close to Mr. Nasheed denied that the president had ordered soldiers to fire on the protesters. Rather, he said, the president chose to resign specifically to avoid such violence. “He faced the choice of seeing a lot of blood by asking the military to crack down,” said the official, who asked not to be identified, given the political volatility of the moment. “But he wasn’t prepared to do that.”

What seems apparent from the bits of information I’ve been able to find is that political polarization had amped up long before the recent showdown over judge Mohamed. Some of that polarization seems to have resulted from an economic slowdown that hit the Maldives in 2011, some from a disagreement over the proper role of Islam in politics, but surely some also resulted from the new government’s attempts to discover and dismantle the networks of patronage and corruption left over from the ancien regime. Presidential elections were due in 2012, and President Nasheed’s move against judge Mohamed apparently strengthened their belief that he was willing to do whatever it would take to cement his continuation in office and continue his fight against their interests.

These are the familiar and formidable challenges of democratic consolidation. New democracies are not drawn on blank slates. The development of democratic institutions that persist usually requires a transformation of deeper arrangements in which powerful groups are heavily invested. Wealthy individuals and powerful bureaucrats must be convinced to subject their sinecures to the rule of laws adopted by representatives they do not choose. Men with guns must be convinced that they will be better off refraining from picking sides in partisan fights or seizing direct control of government when they don’t like how much money it spends on them or what it tells them to do.

For democracy to survive under these conditions, political and military leaders have to gain confidence that every political confrontation is not a gladiatorial death match, and that their rivals can’t or won’t try to win those confrontations by simply usurping power and demolishing the arena. This trust is impossible to manufacture. It seems instead to rise and decline fitfully, and the confrontations whose successful resolution might deepen that trust more often lead instead to resumptions of authoritarian rule. We can recognize and even vaguely understand all of this and still not know how to make it happen differently.

So what usually happens instead is what happened this week in the Maldives. Motivated by mixtures of ambition and fear, partisan rivals get stuck in a downward spiral of distrust that leads eventually to an undemocratic resolution. These resolutions often take the form of a military coup, where the guys with guns decide to take matters into their own hands or, as in the Maldives, side with the rebellious opposition. In many other cases, the confrontation is resolved by executive coup, where incumbent officials ensure their continuation in power by tightening the screws on their political rivals or just rigging or scrapping elections. Opposition parties sometimes rebel, but those rebellions very rarely succeed; instead, they are usually either quashed or hijacked by a collaboration of opportunistic political insiders and state security forces.

Anyone who cares about the fate of new democracies in Egypt, Tunisia, or really anywhere should care about this pattern, because it foretells the likely futures of those regimes. I don’t mean this to be a declaration of helplessness in the face of these patterns as much as a frank assessment of the depth of the challenges involved. As I’ve said before, I’m a short-term pessimist but a long-term optimist. In their brilliant overview of political development throughout recorded history, Douglass North, John Wallis, and Barry Weingast drly note (p. 27) that “historic transitions [of the sort described here] occurred within relatively brief periods, typically about fifty years.” Replace that last comma with a pause for comic timing, and you get a better sense of what I have in mind.

PS. For detailed reporting on the coup and the spiral of events leading to it, see this story by Bryson Hull for Reuters.

Assessing Coup Risk in 2012

Which countries around the world are most likely to see coup activity in 2012?

This question popped back into my mind this morning when I read a new post on Daniel Solomon’s Securing Rights blog about widening schisms in Sudan’s armed forces that could lead to a coup attempt. There’s also been a lot of talk in early 2012 about the likelihood of a coup in Syria, where the financial and social costs of repression, sanctions, and now civil war continue to mount. Meanwhile, Pakistan seems to have dodged a coup bullet early this year after a tense showdown between its elected civilian government and military leaders. I even saw one story–unsubstantiated, but from a reputable source–about a possible foiled coup plot in China around New Year’s Day. These are all countries where a coup d’etat would shake up regional politics, and coups in some of those countries could substantially alter the direction of armed conflicts in which government forces are committing mass atrocities, to name just two of the possible repercussions.

To give a statistical answer to the question of coup risk in 2012, I’ve decided to dust off a couple of coup-forecasting algorithms I developed in early 2011 and gin up some numbers. Both of these algorithms…

  1. Take the values of numerous indicators identified by statistical modeling as useful predictors of coup activity (see the end of this post for details);
  2. Apply weights derived from that modeling to those indicators; and then
  3. Sum and transform the results to spit out a score we can interpret as an estimate of the probability that a coup event will occur some time in 2011.

Both algorithms are products of Bayesian model averaging (BMA) applied to logistic regression models of annual coup activity (any vs. none) in countries worldwide over the past few decades. One of the modeling exercises, done for a private-sector client, looked only at successful coups using data compiled by the Center for Systemic Peace. The other modeling exercise was done for a workshop at the Council on Foreign Relations on forecasting political instability; this one looked at all coup attempts, successful or failed, using data compiled by Jonathan Powell and Clayton Thyne. For the 2012 coup risk assessments, I’ve simply averaged the output from the two.

The dot plot below shows the estimated coup risk in 2012 for the 40 countries with the highest values (i.e., greatest risk). The horizontal axis is scaled for probabilities ranging from zero to 1; if you’re more comfortable thinking in percentages, just multiply the number by 100. As usual with all statistical forecasts of rare events, the estimates are mostly close to zero. (On average, only a handful of coup attempts occur worldwide each year, and they’ve become even rarer since the end of the Cold War; see this earlier post for details). For a variety of reasons, the estimates are also less precise than those dots might make them seem, so small differences should be taken with a grain of salt. Even so, these results of this exercise should offer plausible estimates of the chances that we’ll see coup activity in these countries some time in 2012.

Here are a few of things that stand out for me in those results.

  • My forecast supports Daniel’s analysis that the risk of a coup attempt in Sudan in 2012 is relatively high. It ranks 11th on the global list, making it one of the most likely candidates for coup activity this year.
  • Surprising to me, Pakistan barely cracks into the top 40, landing at 38th in the company of Iraq, Cambodia, and Senegal. Those countries all rank higher than 120 others, but the distance between their estimated risk and the risk in most other countries is within the realm of statistical noise. Off the top of my head, I would have identified Pakistan and Iraq as relatively vulnerable countries, and I would not have thought of Cambodia or Senegal as particularly coup-prone cases.
  • Unsurprising to me, China doesn’t even make the top 40. Perhaps there has been some erosion in civilian control in recent years, as Gordon Chang discusses, but it still doesn’t much resemble the countries that have seen full-blown coup attempts in the past few decades.
  • Interestingly, Syria doesn’t show up in the top 40, either. To make sense of this forecast, it’s important to note that assigning a low probability to the occurrence of a coup attempt in Syria in 2012 isn’t the same thing as a prediction that President Bashar al-Assad or his regime will survive the year. It might seem like semantic hair-splitting, but the definitions of coups used to construct the data on which these forecasts are based do not include cases where national leaders resign under pressure or are toppled by rebel groups. So the Syria forecast suggests only that Assad is unlikely to be overthrown by his own security forces. As it happens, my analysis of countries most likely to see democratic transitions in 2012 put Syria in the top 10 on that list.
  • Two of the countries near the top of that list–Guinea and Democratic Republic of Congo–are the ones where the Center for Systemic Peace’s Monty Marshall tells me he saw coup activity meeting his definition in 2011. Those recent coup attempts are influencing the 2012 forecasts, but both countries were also near the top of the 2011 risk list. This boosts my confidence in the reliability of these assessments.

I hope there’s a lot more on (or off) that list that interests readers, and I’d be happy to hear your thoughts on the results in the Comments section. For now, though, I’m going to wrap up this post by providing more information on what those forecasts take into account. The algorithm for successful coups uses just four risk factors, one of which is really just an adjustment to the intercept.

  • Infant mortality rate (relative to annual global median, logged): higher risk in countries with higher rates.
  • Degree of democracy (Polity score, quadratic): higher risk for countries in the mid-range of the 21-point scale.
  • Recent coup activity (yes or no): higher risk if any activity in the past five years.
  • Post-Cold War period: lower risk since 1989.

The algorithm for any coup attempts, successful or failed, uses the following ten risk factors, including all four of the ones used to forecast successful coups.

  • Infant mortality rate (relative to annual global median, logged): higher risk in countries with higher rates.
  • Recent coup activity (count of past five years with any, plus one and logged): higher risk with more activity.
  • Post-Cold War period: lower risk since 1989.
  • Popular uprisings in region (count of countries with any, plus one and logged): higher risk with more of them.
  • Insurgencies in region (count of countries with any, plus one and logged): higher risk with more of them.
  • Economic growth (year-to-year change in GDP per capita): higher risk with slower growth.
  • Regime durability (time since last abrupt change in Polity score, plus one and logged): lower risk with longer time.
  • Ongoing insurgency (yes or no): higher risk if yes.
  • Ongoing civil resistance campaign (yes or no): higher risk if yes.
  • Signatory to 1st Optional Protocol of the UN’s International Covenant on Civil and Political Rights (yes or no): lower risk if yes.

How Risky and Costly Are Coups, Really?

Coups d’etat don’t happen as often as they used to, but they do still happen. In the past few year, coups have toppled leaders in Honduras, Niger, and arguably Egypt, while coup bids have fizzled in Guinea-Bissau (twice), Democratic Republic of CongoMadagascar, and, less than a year after the aforementioned success, Niger. Meanwhile, just this morning, we’re hearing fresh rumblings of a possible coup in Pakistan, which last saw the military openly seize power in 1999. (Whether it’s ever really taken its hands off the levers of power at any point in Pakistan’s history is another matter.)

The persistence of coups is a bit of a puzzle, because coup attempts are typically costly to their perpetrators in at least two ways.

1. Most coup attempts fail. From 1955 to 2008, half of all coup bids worldwide failed (158 of 316). As the chart below shows, the failure rate has been much higher in the past two decades than it was in earlier years. And these are just the coup bids that make it all the way to an overt attempt. If our tally also included all of the plots that were uncovered and foiled before they could be put in motion, the failure rate would be much higher. If coup attempts usually fail, and the punishment for a failed coup is often imprisonment or death, then coup bids would seem to be a pretty risky gamble for their plotters.

 2. Coups take a toll on the economy. The figure below plots the estimated impact of a successful coup on a country’s GDP growth rate over the several years following the coup event (data geeks: see the technical notes at the end of this post for details on the method used to derive these estimates). The impact is substantial: on average, a couple of percentage points knocked off the growth rate in the year of the coup and another point or so the following year. Expected growth rates resume a few years after the coup, and there even seems to be something of a rebound effect several years down the road, but three or four years is a long wait in political time. This knock on the GDP growth rate matters for coup plotters because if they succeed, they’re liable be blamed for the damage done (Pakistani generals, take note!).

In light of these facts, it’s hard to understand why coup plotters keep trying, even if they are trying less often. To attempt a coup, you’ve either got to be ignorant of these facts or to consider them irrelevant to your particular situation. You might consider them irrelevant because you’re exceedingly optimistic about your own coup’s chances for success, as behavioral psychology suggests many military and political leaders will be. Alternatively, you might expect your attempt to fail but still think it’s worth a try because you believe that success will produce large private benefits (like the opportunity to loot the state treasury) or non-economic public benefits that will reflect well on you and your co-conspirators (like liberation from an awful tyrant, or defense against public disorder).

My hunch is that all of these forces–ignorance, optimism, greed, and benevolence–factor into the decision-making behind many coup attempts. We’ll never really know for sure, because even first-hand accounts of plotters’ motivations are highly unreliable. What we can say with some confidence, I think, is that people are going to keep trying anyway.

Technical Notes

The figures and charts in this post are based on coup event data compiled by Monty Marshall at the Center for Systemic Peace. You can find them here, under the Polity IV heading.

To estimate the impact of successful coups on GDP growth, I first used “coarsened exact matching” to pre-process a time-series cross-sectional data set of all countries worldwide for the period 1955-2008. Country-years were matched on several risk factors for successful coups, including infant mortality rates, degree of democracy (Polity scores, quadratic), recent coup activity, and a marker for the post-Cold War period. After matching, I used Zelig to estimate the average “treatment effect” for successful coups. The values used in the plot above are the medians from six iterations of this drill, one for each year away from the coup event. The GDP growth data are from the World Bank’s World Development Indicators.

Fact Check: Are Military Coups Back in Vogue?

In a recent piece for The National, Joshua Kurlantzick claims to spot a resurgence of military coups. He writes:

In Latin America, Africa and Asia, coups, which had been a frequent means of changing governments during the Cold War, had become nearly extinct by the dawn of the new century. But over the past decade, the military has made power grabs in at least 12 states, from Guinea to Honduras, from Thailand to Madagascar.

What does that 12 tell us, though? Have military coups really become more frequent in the past several years?

The answer is a flat “no.” The chart below plots annual counts of successful coups from 1946 through the first half of 2011, using data compiled by the Center for Systemic Peace. As the chart clearly shows, the incidence of coups has fallen substantially in the post-Cold War period and remains historically low. (NB: Those counts don’t adjust for the large increase in the number of countries worldwide in the post-Cold War era. Against that growing denominator, the rate of successful coups has declined even further than the raw counts suggest.)

Maybe coup attempts have become more common, but fewer of them are succeeding? Again, no. The following chart looks at the incidence of failed coup attempts over the same period. Again, there has been a clear decline in the past couple of decades, and that pattern has not changed noticeably in the past several years.

Kurlantzick’s story of a trend that isn’t reminds me of this anecdote from a post by Sarah Slobin at Mix Online called “The 7 1/2 Steps to Successful Infographics,” which I found through Andrew Gelman’s blog.

When I was at the NYT, there was this reporter who drove a thousand miles across country chasing this thesis that population growth was sparked near off-ramps on the interstate. It was a lovely road-trip story; he gathered amazing anecdotes and the editors loved it. Except that when we mapped the census data it didn’t support the thesis. Imagine how much gas he could have saved had he started by looking at the data.

Forget infographics for a moment; the moral of Slobin’s story applies to anyone looking for patterns or trends in the real world. Sure, some questions can’t be answered with numbers; the relevant data may not exist (yet), or the research question might involve aspects of process that are difficult to quantify. The rest of the time, however–and that’s going to be a lot of the time–it’s a good idea to look at available data before getting deeply invested in a particular answer.

How Is Liberia Staying Stable?

Earlier this year, in preparation for a workshop at the Council on Foreign Relations, I developed a set of statistical models to assess the risk of onset of a few forms of political instability–violent rebellion, nonviolent rebellion (link), and coup attempts–and then used those models to generate global forecasts for 2011. Liberia scored in the top five on two of those lists: violent rebellion (a.k.a. civil war) and coup attempts. The models pegged it as having roughly a 15% chance of civil-war onset (3rd highest in the world) and more than a 60% chance of a coup attempt (4th highest) before 2012.

If Liberia is so susceptible to these kinds of political crises, why aren’t they happening now? The country’s horrible 14-year civil war ended in 2003 and has not flared again since then. Elections held in 2005 handed the presidency to Ellen Johnson Sirleaf, and no one has yet tried to depose her by force. For a country supposedly at the leading edge of what Robert Kaplan in 1994 called “the coming anarchy” of state collapse and civil strife (link), that’s a terrific run of political stability.

Of course, Liberia is not out of the woods yet. The year is only half over, and the country is scheduled to hold legislative and executive elections this October. Electoral competition or frustration over its results could trigger civil violence or coup attempts. Fears of exactly that scenario have already prompted more than 20 political parties to sign a Memorandum of Understanding with the Liberian police pledging to conduct their election campaigns with civility (link), but that paper promise is hardly a guarantee against crisis.

Still, let’s take the optimistic view and assume that Liberia crests this hump in its political risk without sliding back into large-scale civil violence or suffering a coup. What explains its ability for nearly a decade now to avoid the “conflict trap” that has plagued so many of the world’s poorest countries after the apparent ends of their civil wars?

Liberia doesn’t get a lot of attention in the U.S. press, but the bits I have seen and heard in the past several years have focused almost exclusively on the figure of President Ellen Johnson Sirleaf. As a plain-talking, Harvard-educated black female economist presiding over a country brutalized by a succession of male warlords, Sirleaf cuts a rare and appealing figure, and foreign governments and international aid organizations seem unusually committed to her government’s success. “We see her as one of us,” U.S. ambassador Linda Thompson-Greenfield told the New York Times. “We don’t want to see her fail.”

I don’t know enough about Liberia to assert anything with confidence, but as a general observer of political instability, my hunch is that international peacekeeping has played a larger role in preserving Liberia’s tenuous stability than the hagiographies of President Sirleaf imply. Since 2003, the United Nations has maintained a large peacekeeping operation (PKO) in Liberia to prevent a return to civil war, support humanitarian work, and train that country’s soldiers and police (link). As of May 2011, that PKO included more than 9,200 uniformed personnel, 1,300 police, nearly 500 international civilian personnel, and roughly 1,000 local staffers and was funded with an annual budget of more than $540 million. That’s a tremendous commitment in a country with a population of about 4 million and a gross domestic product (GDP) of less than $1 billion.

The scale and strength of the peacekeeping efforts in Liberia remind me of the U.N. mission in neighboring Sierra Leone, a neighboring west African country that was also brutalized by civil war and so far has avoided both a resumption of violence and a breakdown of its post-conflict democratic regime. Running from 1999 until 2005, the PKO in Sierra Leone involved as many as 17,000 military personnel at its peak and cost $2.8 billion in total in a country with roughly 5 million residents and a GDP of less than $2 billion (link). At this scale and duration, international peacekeeping operations should stand a better chance of helping domestic rivals overcome the security dilemmas that often drive recurrent conflicts, and Sierra Leone and Liberia’s experiences offer a couple of anecdotes in support of that view. [For excellent academic treatments of civil-war recurrence and settlement, see Barbara Walter and Jack Snyder's 1999 edited volume and Monica Toft's 2010 book.]

In short, I believe Liberia’s comparative stability since 2003 is, above all, a testament to the possibility of effective international peacekeeping, especially in smaller countries where intervening forces can more readily achieve a scale that’s virtually impossible to reach in larger countries such as Iraq and Afghanistan. I didn’t have the data required to consider the effects of PKOs in my statistical analysis of violent rebellion and coup attempts and am now eager to try adding them in the future. The 2011 forecasts based on my already-completed analysis could still turn out to be prescient, but I’m certainly hoping they don’t. In the mean time, I would be very interested in hearing from people with expertise on Liberia about their ideas on how that country is beating the odds to stay stable.

UPDATE: Not long after posting these ruminations, I saw a tweet from African Elections (@Africanelection) with a link to an IRIN story, via AlertNet, about Liberia’s current conditions and upcoming election season. The story didn’t change my views about the country’s conflict and coup risks, but it looks like an excellent backgrounder. You can find it here.

Follow

Get every new post delivered to your Inbox.

Join 7,697 other followers

%d bloggers like this: