Why My Coup Risk Models Don’t Include Any Measures of National Militaries

For the past several years (herehere, here, and here), I’ve used statistical models estimated from country-year data to produce assessments of coup risk in countries worldwide. I rejigger the models a bit each time, but none of the models I’ve used so far has included specific features of countries’ militaries.

That omission strikes a lot of people as a curious one. When I shared this year’s assessments with the Conflict Research Group on Facebook, one group member posted this comment:

Why do none of the covariates feature any data on militaries? Seeing as militaries are the ones who stage the coups, any sort of predictive model that doesn’t account for the militaries themselves would seem incomplete.

I agree in principle. It’s the practical side that gets in the way. I don’t include features of national militaries in the models because I don’t have reliable measures of them with the coverage I need for this task.

To train and then apply these predictive models, I need fairly complete time series for all or nearly all countries of the world that extend back to at least the 1980s and have been updated recently enough to give me a useful input for the current assessment (see here for more on why that’s true). I looked again early this month and still can’t find anything like that on even the big stuff, like military budgets, size, and force structures. There are some series on this topic in the World Bank’s World Development Indicators (WDI) data set, but those series have a lot of gaps, and the presence of those gaps is correlated with other features of the models (e.g., regime type). Ditto for SIPRI. And, of course, those aren’t even the most interesting features for coup risk, like whether or not military promotions favor certain groups over others, or if there is a capable and purportedly loyal presidential guard.

But don’t take my word for it. Here’s what the Correlates of War Project says in the documentation for Version 4.0 of its widely-used data set (PDF) about its measure of military expenditures, one of two features of national militaries it tries to cover (the other is total personnel):

It was often difficult to identify and exclude civil expenditures from reported budgets of less developed nations. For many countries, including some major powers, published military budgets are a catch-all category for a variety of developmental and administrative expenses—public works, colonial administration, development of the merchant marine, construction, and improvement of harbor and navigational facilities, transportation of civilian personnel, and the delivery of mail—of dubious military relevance. Except when we were able to obtain finance ministry reports, it is impossible to make detailed breakdowns. Even when such reports were available, it proved difficult to delineate “purely” military outlays. For example, consider the case in which the military builds a road that facilitates troops movements, but which is used primarily by civilians. A related problem concerns those instances in which the reported military budget does not reflect all of the resources devoted to that sector. This usually happens when a nation tries to hide such expenditures from scrutiny; for instance, most Western scholars and military experts agree that officially reported post-1945 Soviet-bloc totals are unrealistically low, although they disagree on the appropriate adjustments.

And that’s just the part of the “Problems and Possible Errors” section about observing the numerator in a calculation that also requires a complicated denominator. And that’s for what is—in principle, at least—one of the most observable features of a country’s civil-military relations.

Okay, now let’s assume that problem magically disappears, and COW’s has nearly-complete and reliable data on military expenditures. Now we want to use models trained on those data to estimate coup risk for 2015. Whoops: COW only runs through 2010! The World Bank and SIPRI get closer to the current year—observations through 2013 are available now—but there are missing values for lots of countries, and that missingness is caused by other predictors of coup risk, such as national wealth, armed conflict, and political regime type. For example, WDI has no data on military expenditures for Eritrea and North Korea ever, and the series for Central African Republic is patchy throughout and ends in 2010. If I wanted to include military expenditures in my predictive models, I could use multiple imputation to deal with these gaps in the training phase, but then how would I generate current forecasts for these important cases? I could make guesses, but how accurate could those guesses be for a case like Eritrea or North Korea, and then am I adding signal or noise to the resulting forecasts?

Of course, one of the luxuries of applied forecasting is that the models we use can lack important features and still “work.” I don’t need the model to be complete and its parameters to be true for the forecasts to be accurate enough to be useful. Still, I’ll admit that, as a social scientist by training, I find it frustrating to have to set aside so many intriguing ideas because we simply don’t have the data to try them.

Statistical Assessments of Coup Risk for 2015

Which countries around the world are more likely to see coup attempts in 2015?

For the fourth year in a row, I’ve used statistical models to generate one answer to that question, where a coup is defined more or less as a forceful seizure of national political authority by military or political insiders. (I say “more or less” because I’m blending data from two sources with slightly different definitions; see below for details.) A coup doesn’t need to succeed to count as an attempt, but it does need to involve public action; alleged plots and rumors of plots don’t qualify. Neither do insurgencies or foreign invasions, which by definition involve military or political outsiders. The heat map below shows variation in estimated coup risk for 2015, with countries colored by quintiles (fifths).

forecast.heatmap.2015

The dot plot below shows the estimates and their 90-percent confidence intervals (CIs) for the 40 countries with the highest estimated risk. The estimates are the unweighted average of forecasts from two logistic regression models; more on those in a sec. To get CIs for estimates from those two models, I took a cue from a forthcoming article by Lyon, Wintle, and Burgman (fourth publication listed here; the version I downloaded last year has apparently been taken down, and I can’t find another) and just averaged the CIs from the two models.

forecast.dotplot.2015

I’ve consistently used simple two– or three-model ensembles to generate these coup forecasts, usually pairing a logistic regression model with an implementation of Random Forests on the same or similar data. This year, I decided to use only a pair of logistic regression models representing somewhat different ideas about coup risk. Consistent with findings from other work in which I’ve been involved (here), k-fold cross-validation told me that Random Forests wasn’t really boosting forecast accuracy, and sticking to logistic regression makes it possible to get and average those CIs. The first model matches one I used last year, and it includes the following covariates:

  • Infant mortality rate. Deaths of children under age 1 per 1,000 live births, relative to the annual global median, logged. This measure that primarily reflects national wealth but is also sensitive to variations in quality of life produced by things like corruption and inequality. (Source: U.S. Census Bureau)
  • Recent coup activity. A yes/no indicator of whether or not there have been any coup attempts in that country in the past five years. I’ve tried logged event counts and longer windows, but this simple version contains as much predictive signal as any other. (Sources: Center for Systemic Peace and Powell and Thyne)
  • Political regime type. Following Fearon and Laitin (here), a categorical measure differentiating between autocracies, anocracies, democracies, and other forms. (Source: Center for Systemic Peace, with hard-coded updates for 2014)
  • Regime durability. The “number of years since the last substantive change in authority characteristics (defined as a 3-point change in the POLITY score).” (Source: Center for Systemic Peace, with hard-coded updates for 2014)
  • Election year. A yes/no indicator for whether or not any national elections (executive, legislative, or general) are scheduled to take place during the forecast year. (Source: NELDA, with hard-coded updates for 2011–2015)
  • Economic growth. The previous year’s annual GDP growth rate. To dampen the effects of extreme values on the model estimates, I take the square root of the absolute value and then multiply that by -1 for cases where the raw value less than 0. (Source: IMF)
  • Political salience of elite ethnicity. A yes/no indicator for whether or not the ethnic identity of national leaders is politically salient. (Source: PITF, with hard-coded updates for 2014)
  • Violent civil conflict. A yes/no indicator for whether or not any major armed civil or ethnic conflict is occurring in the country. (Source: Center for Systemic Peace, with hard-coded updates for 2014)
  • Country age. Years since country creation or independence, logged. (Source: me)
  • Coup-tagion. Two variables representing (logged) counts of coup attempts during the previous year in other countries around the world and in the same geographic region. (Source: me)
  • Post–Cold War period. A binary variable marking years after the disintegration of the USSR in 1991.
  • Colonial heritage. Three separate binary indicators identifying countries that were last colonized by Great Britain, France, or Spain. (Source: me)

The second model takes advantage of new data from Geddes, Wright, and Frantz on autocratic regime types (here) to consider how qualitative differences in political authority structures and leadership might shape coup risk—both directly, and indirectly by mitigating or amplifying the effects of other things. Here’s the full list of covariates in this one:

  • Infant mortality rate. Deaths of children under age 1 per 1,000 live births, relative to the annual global median, logged. This measure that primarily reflects national wealth but is also sensitive to variations in quality of life produced by things like corruption and inequality. (Source: U.S. Census Bureau)
  • Recent coup activity. A yes/no indicator of whether or not there have been any coup attempts in that country in the past five years. I’ve tried logged event counts and longer windows, but this simple version contains as much predictive signal as any other. (Sources: Center for Systemic Peace and Powell and Thyne)
  • Regime type. Using the binary indicators included in the aforementioned data from Geddes, Wright, and Frantz with hard-coded updates for the period 2011–2014, a series of variables differentiating between the following:
    • Democracies
    • Military autocracies
    • One-party autocracies
    • Personalist autocracies
    • Monarchies
  • Regime duration. Number of years since the last change in political regime type, logged. (Source: Geddes, Wright, and Frantz, with hard-coded updates for the period 2011–2014)
  • Regime type * regime duration. Interactions to condition the effect of regime duration on regime type.
  • Leader’s tenure. Number of years the current chief executive has held that office, logged. (Source: PITF, with hard-coded updates for 2014)
  • Regime type * leader’s tenure. Interactions to condition the effect of leader’s tenure on regime type.
  • Election year. A yes/no indicator for whether or not any national elections (executive, legislative, or general) are scheduled to take place during the forecast year. (Source: NELDA, with hard-coded updates for 2011–2015)
  • Regime type * election year. Interactions to condition the effect of election years on regime type.
  • Economic growth. The previous year’s annual GDP growth rate. To dampen the effects of extreme values on the model estimates, I take the square root of the absolute value and then multiply that by -1 for cases where the raw value less than 0. (Source: IMF)
  • Regime type * economic growth. Interactions to condition the effect of economic growth on regime type.
  • Post–Cold War period. A binary variable marking years after the disintegration of the USSR in 1991.

As I’ve done for the past couple of years, I used event lists from two sources—the Center for Systemic Peace (about halfway down the page here) and Jonathan Powell and Clayton Thyne (Dataset 3 here)—to generate the historical data on which those models were trained. Country-years are the unit of observation in this analysis, so a country-year is scored 1 if either CSP or P&T saw any coup attempts there during those 12 months and 0 otherwise. The plot below shows annual counts of successful and failed coup attempts in countries worldwide from 1946 through 2014 according to the two data sources. There is a fair amount of variance in the annual counts and the specific events that comprise them, but the basic trend over time is the same. The incidence of coup attempts rose in the 1950s; spiked in the early 1960s; remained relatively high throughout the rest of the Cold War; declined in the 1990s, after the Cold War ended; and has remained relatively low throughout the 2000s and 2010s.

Annual counts of coup events worldwide from two data sources, 1946-2014

Annual counts of coup events worldwide from two data sources, 1946-2014

I’ve been posting annual statistical assessments of coup risk on this blog since early 2012; see here, here, and here for the previous three iterations. I have rejiggered the modeling a bit each year, but the basic process (and the person designing and implementing it) has remained the same. So, how accurate have these forecasts been?

The table below reports areas under the ROC curve (AUC) and Brier scores (the 0–1 version) for the forecasts from each of those years and their averages, using the the two coup event data sources alone and together as different versions of the observed ground truth. Focusing on the “either” columns, because that’s what I’m usually using when estimating the models, we can see the the average accuracy—AUC in the low 0.80s and Brier score of about 0.03—is comparable to what we see in many other country-year forecasts of rare political events using a variety of modeling techniques (see here). With the AUC, we can also see a downward trend over time. With so few events involved, though, three years is too few to confidently deduce a trend, and those averages are consistent with what I typically see in k-fold cross-validation. So, at this point, I suspect those swings are just normal variation.

AUC and Brier scores for coup forecasts posted on Dart-Throwing Chimp, 2012-2014, by coup event data source

AUC and Brier scores for coup forecasts posted on Dart-Throwing Chimp, 2012-2014, by coup event data source

The separation plot designed by Greenhill, Ward, and Sacks (here) offers a nice way to visualize the accuracy of these forecasts. The ones below show the three annual slices using the “either” version of the outcome, and they reinforce the story told in the table: the forecasts have correctly identified most of the countries that saw coup attempts in the past three years as relatively high-risk cases, but the accuracy has declined over time. Let’s define a surprise as a case that fell outside the top 30 of the ordered forecasts but still saw a coup attempt. In 2012, just one of four countries that saw coup attempts was a surprise: Papua New Guinea, ranked 48. In 2013, that number increased to two of five (Eritrea at 51 and Egypt at 58), and in 2014 it rose to three of five (Burkina Faso at 42, Ukraine at 57, and the Gambia at 68). Again, though, the average accuracy across the three years is consistent with what I typically see in k-fold cross-validation of these kinds of models in the historical data, so I don’t think we should make too much of that apparent time trend just yet.

cou.scoring.sepplot.2012 cou.scoring.sepplot.2013 cou.scoring.sepplot.2014

This year, for the first time, I am also running an experiment in crowdsourcing coup risk assessments by way of a pairwise wiki survey (survey here, blog post explaining it here, and preliminary results discussed here). My long-term goal is to repeat this process numerous times on this topic and some others (for example, onsets of state-led mass killing episodes) to see how the accuracy of the two approaches compares and how their output might be combined. Statistical forecasts are usually much more accurate than human judgment, but that advantage may be reduced or eliminated when we aggregate judgments from large and diverse crowds, or when we don’t have data on important features to use in those statistical models. Models that use annual data also suffer in comparison to crowdsourcing processes that can update continuously, as that wiki survey does (albeit with a lot of inertia).

We can’t incorporate the output from that wiki survey into the statistical ensemble, because the survey doesn’t generate predicted probabilities; it only assesses relative risk. We can, however, compare the rank orderings the two methods produce. The plot below juxtaposes the rankings produced by the statistical models (left) with the ones from the wiki survey (right). About 500 votes have been cast since I wrote up the preliminary results, but I’m going to keep things simple for now and use the preliminary survey results I already wrote up. The colored arrows identify cases ranked at least 10 spots higher (red) or lower (blue) by the crowd than the statistical models. As the plot shows, there are many differences between the two, even toward the top of the rankings where the differences in statistical estimates are bigger and therefore more meaningful. For example, the crowd sees Nigeria, Libya, and Venezuela as top 10 risks while the statistical models do not; of those three, only Nigeria ranks in the top 30 on the statistical forecasts. Meanwhile, the crowd pushes Niger and Guinea-Bissau out of the top 10 down to the 20s, and it sees Madagascar, Afghanistan, Egypt, and Ivory Coast as much lower risks than the models do. Come 2016, it will be interesting to see which version was more accurate.

coup.forecast.comparison.2015

If you are interested in getting hold of the data or R scripts used to produce these forecasts and figures, please send me an email at ulfelder at gmail dot com.

A Crowd’s-Eye View of Coup Risk in 2015

A couple of weeks ago (here), I used the blog to launch an experiment in crowdsourcing assessments of coup risk for 2015 by way of a pairwise wiki survey. The survey is still open and will stay that way until the end of the year, but with nearly 2,700 pairwise votes already cast, I thought it was good time to take stock of the results so far.

Before discussing those results, though, let me say thank you to all the people who voted in the survey or shared the link. These data don’t materialize from thin air. They only exist because busy people contributed their knowledge and time, and I really appreciate all of those contributions.

Okay, so, what does that self-assembled crowd think about relative risks of coup attempts in 2015? The figure below maps the country scores produced from the votes cast so far. Darker grey indicates higher risk. PLEASE NOTE: Those scores fall on a 0–100 scale, but they are not estimated probabilities of a coup attempt. Instead, they are only measures of relative risk, because that’s all we can get from a pairwise wiki survey. Coup attempts are rare events—in most recent years, we’ve seen fewer than a handful of them worldwide—so the safe bet for nearly every country every year is that there won’t be any coup attempts this year.

wikisurvey.couprisk.2015.map

 

Smaller countries can be hard to find on that map, and small differences in scores can be hard to discern, so I also like to have a list of the results to peruse. Here’s a dot plot with countries in descending order by model score. (It’d be nice to make this table sortable so you could also look for countries alphabetically, but my Internet fu is not up to that task.)

wikisurvey.couprisk.2015.dotplot

This survey is open to the public, and participants may cast as many votes as they like in as many sessions as they like. The scores summarized above come from nearly 2,700 votes cast between the morning of January 3, when I published the blog post about the survey, and the morning of January 14, when I downloaded a report on the current results. At present, this blog has a few thousand followers on Wordpress and a few hundred email subscribers. I also publicized the survey twice on Twitter, where I have approximately 6,000 followers: once when I published the initial blog post, and again on January 13. As the plot below shows, participation spiked around both of those pushes and was low otherwise.

votesovertime.20150114

The survey instrument does not collect identifying information about participants, so it is impossible to describe the make-up of the crowd. What we do know is that those votes came from about 100 unique user sessions. Some people probably participated more than once—I know that I cast a dozen or so votes on a few occasions—so 100 unique sessions probably works out to something like 80 or 90 individuals. But that’s a guess.

usersessions.20150114

We also know that those votes came from lots of different parts of the world. As the map below shows, most of the votes came from the U.S., Europe, and Australia, but there were also pockets of activity in the Middle East (especially Israel), Latin America (Brazil and Argentina), Africa (Cote d’Ivoire and Rwanda), and Asia (Thailand and Bangladesh).

votemap.20150114

I’ll talk a little more about the substance of these results when I publish my statistical assessments of coup risk for 2015, hopefully in the next week or so. Meanwhile, number-crunchers can get a .csv with the data used to generate the map and table in this post from my Google Drive (here) and the R script from GitHub (here). If you’re interested in seeing the raw vote-level data from which those scores were generated, drop me a line.

Schrodinger’s Coup

You’ve heard of Schrödinger’s cat, right? This is the famous thought experiment proposed by Nobel Prize–winning physicist Erwin Schrödinger to underscore what he saw as the absurdity of quantum superposition—the idea that “an object in a physical system can simultaneously exist in all possible configurations, but observing the system forces the system to collapse and forces the object into just one of those possible states.”

Schrodinger designed his thought experiment to refute the idea that a physical object could simultaneously occupy multiple physical states. At the level of whole cats, anyway, I’m convinced.

When it comes to coups, though, I’m not so sure. Arguments over whether or not certain events were or were not coups or coup attempts usually involve reasonable disagreements over definitions, but fundamental uncertainty about the actions and intentions involved often plays a role, too—especially in failed attempts. Certain sets of events exist in a perpetual state of ambiguity, simultaneously coup and not-coup with no possibility of our ever observing the definitive empirical facts that would force the cases to collapse into a single, clear condition.

Two recent examples help show what I mean. The first is last week’s coup/not coup attempt in the Gambia. From initial reports, it seemed pretty clear that some disgruntled soldiers had tried and failed to seize power while the president was traveling. That’s a classic coup scenario, and all the elements present in most coup definitions were there: military or political insiders seeking to overthrow the government through the use or threat of force.

This week, though, we hear that the gunmen in question were diasporans who hatched the plot abroad without any help on the inside. As the New York Times reported,

According to the [Justice Department’s] complaint, filed in federal court in Minnesota, the plot to topple Mr. Jammeh was hatched in October. Roughly a dozen Gambians in the United States, Germany, Britain and Senegal were involved in the plot, the complaint said. The plotters apparently thought, mistakenly, that members of the Gambian armed forces would join their cause…

The plot went awry when State House guards overwhelmed the attackers with heavy fire, leaving many dead or wounded. Mr. Faal and Mr. Njie escaped and returned to the United States, where they were arrested, the complaint said.

So, were the putschists really just a cabal of outsiders, in which case this event would not qualify as a coup attempt under most academic definitions? Or did they have collaborators on the inside who were discovered or finked out at the crucial moment, making a coup attempt look like a botched raid? The Justice Department’s complaint implies the former, but we’ll never know for sure.

Lesotho offers a second recent example of coup-tum superposition. In late August, that country’s prime minister, Thomas Thabane, fled to neighboring South Africa and cried “Coup!” after soldiers shut down radio stations and surrounded his residence and police headquarters in the capital. But, as Kristen van Schie reported for Al Jazeera,

Not so, said the military. It claimed the police were planning on arming UTTA—the government-aligned youth movement accused of planning to disrupt Monday’s march [against Thabane]. It was not so much a coup as a preventative anti-terrorism operation, it said.

The prime minister and the South African government continued to describe the event as a coup attempt despite that denial, but other observers disagreed. As analyst Dimpho Motsamai told van Schie, “Can one call it a coup when the military haven’t declared they’ve taken over government?” Maybe this really was just a misunderstanding accelerated by the country’s persistent factional crisis.

This uncertainty is generic to the study of political behavior, where the determination of a case’s status depends, in part, on the actors’ intentions, which can never be firmly established. Did certain members of the military in the Gambia mean to cooperate with the diasporans who shot their way toward the presidential last week, only to quit or get stopped before the decisive moment arrived? Was the commander of Lesotho’s armed forces planning to oust the prime minister when he ordered soldiers out of the barracks in late August, only to change his mind and tune after Thabane escaped the country?

To determine with certainty whether or not these events were coup attempts, we need clear answers to those questions, but we can’t get them. Instead, we can only see the related actions, and even those are incompletely and unreliably reported in most cases. Sometimes we get post hoc descriptions and explanations of those actions from the participants or close observers, but humans are notoriously unreliable reporters of their own intentions, especially in high-visibility, high-stakes situations like these.

Because this problem is fundamental to the study of political behavior, the best we can do is acknowledge it and adjust our estimations and inferences accordingly. When assembling data on coup attempts for comparative analysis, instead of just picking one source, we might use Bayesian measurement models to try to quantify this collective uncertainty (see here for a related example). Then, before reporting new findings on the causes or correlates of coup attempts, we might ask: which cases are more ambiguous than the others, and how would their removal from or addition to the sample alter our conclusions?

An Experiment in Crowdsourced Coup Forecasting

Later this month, I hope to have the data I need to generate and post new statistical assessments of coup risk for 2015. Meanwhile, I thought it would be interesting and useful to experiment with applying a crowdsourcing tool to this task. So, if you think you know something about coup risk and want to help with this experiment, please cast as many votes as you like here:

2015 Coup Risk Wiki Survey

For this exercise, let’s use Naunihal Singh’s (2014, p. 51) definition of a coup attempt: “An explicit action, involving some portion of the state military, police, or security forces, undertaken with intent to overthrow the government.” As Naunihal notes,

This definition retains most of the aspects commonly found in definitions of coup attempts [Ed.: including the ones I use in my statistical modeling] while excluding a wide range of similar activities, such as conspiracies, mercenary attacks, popular protests, revolutions, civil wars, actions by lone assassins, and mutinies whose goals explicitly excluded taking power (e.g., over unpaid wages). Unlike a civil war, there is no minimum casualty threshold necessary for an event to be considered a coup, and many coups take place bloodlessly.

By this definition, last week’s putsch in the Gambia and November’s power grab by a lieutenant colonel in Burkina Faso would qualify, but last February’s change of government by parliamentary action in Ukraine after President Yanukovich’s flight in the face of popular unrest would not. Nor would state collapses in Libya and Central African Republic, which occurred under pressure from rebels rather than state security forces. And, of course, Gen. Sisi’s seizure of power in Egypt in July 2013 clearly would qualify as a successful coup on these terms.

In a guest post here yesterday, Maggie Dwyer identified one factor—divisions and tensions within the military—that probably increases coup risk in some cases, but that we can’t fold into global statistical modeling because, as often happens, we don’t have the time-series cross-sectional data we would need to do that. Surely there are other such factors and forces. My hope is that this crowdsourcing approach will help spotlight some cases overlooked by the statistical forecasts because their fragility is being driven by things those models can’t consider.

Wiki surveys weren’t designed specifically for forecasting, but I have adapted them to this purpose on two other topics, and in both cases the results have been pretty good. As part of my work for the Early Warning Project, we have run wiki surveys on risks of state-led mass killing onset for 2014 and now 2015. That project’s data-makers didn’t see any such onsets in 2014, but the two countries that came closest—Iraq and Myanmar—ranked fifth and twelfth, respectively, in the wiki survey we ran in December 2013. On pro football, I’ve run surveys ahead of the 2013 and 2014 seasons. The results haven’t been clairvoyant, but they haven’t been shabby, either (see here and here for details).

I will summarize the results of this survey on coup risk in a blog post in mid-January and will make the country– and vote-level data freely available to other researchers when I do.

I don’t necessarily plan to close the survey at that point, though. In fact, I’m really hoping to get a chance to tinker with using it more dynamically. Ideally, we would leave the survey running throughout the year so that participants could factor new information—credible rumors of an impending coup, for example, or a successful post-election transfer of power without military intervention—into their voting decisions, and the survey results would update quickly in response to those more recent votes.

Doing that would require modifying the modeling process that converts the pairwise votes into scores, however, and I’m not sure that I’m up to the task. As developed, the wiki survey effectively weights all votes the same, regardless of when they were cast. To make the survey more sensitive to fresher information, we would need to tweak that process so that recent votes are weighted more heavily—maybe with a time-decaying weighting function, or just a sliding window that closes on older votes after some point. If we wanted to get really fancy, we might find a way to use the statistical forecasts as priors in this process, too, letting the time-sensitive survey results pull cases up or push them down as the year passes.

I can imagine these modifications, but I don’t think I can code them. If you’re reading this and you might like to collaborate on that work (or fund it!) or just have thoughts on how to do it, please drop me a line at ulfelder at gmail dot com.

A Failed Coup Attempt (and Forecast) in the Gambia

This is a guest post by Maggie Dwyer (@MagDwyer). She teaches courses related to security and politics in Africa at University of Saint Andrews and University of Edinburgh.

Things are often not the way they seem in the Gambia, and this may help explain why this week’s coup attempt in Banjul was not anticipated by Jay’s statistical forecasting.

Nicknamed “the smiling coast,” the Gambia has long been known for its beach resorts, which are particularly popular with European tourists. The country has never experienced a civil war and is generally considered peaceful. Its president, Yahya Jammeh, came to power in a coup in 1994 and has won four elections since.

A deeper look at the Gambia, however, shows that the appearance of stability comes at a high cost for the population. The repressive style of the Jammeh government has led to a growing list of human rights abuses. Critics of the regime are often met with harassment, arrest, detention, and disappearance. The country is shrouded in secrecy due to a lack of press freedoms. With no presidential term limits and no significant opposition, many see no end in sight for Jammeh’s regime.

The military is often viewed as the strong arm of the Jammeh regime and is responsible for many of the abuses. Yet, the military is also kept on edge. An endless series of promotions, demotions, firings, and re-hirings leave military personnel in a constant state of uncertainty. The sense of fear within the military is exacerbated by severe punishment for those deemed disloyal. Several military members were officially executed for alleged involvement in coup plots in 2013, and there are many more suspected executions, disappearances, and torture of military personnel on which the government has never commented.

There have also been reports of growing military discontent within the military over preference for Jammeh’s minority ethnic group, the Jola. There are claims that the Jola have been given a disproportionate number of promotions, top positions and opportunities (e.g., training and participation in peacekeeping), and that this favoritism has created divisions and spurred resentment in the military.

Despite the fate of past coup plotters in the Gambia, military personnel have continued to try to oust Jammeh. He has endured at least eight alleged coup attempts during his 20 years in office. Many of the accused plotters had served at the highest military positions, including Army Chief of Staff and Director of the National Intelligence Agency, suggesting divisions at the most senior levels. It should be noted that there is speculation as to whether some of the attempts were real or simply ways to purge members of the military. The ambiguity of these events is another cause of uncertainty and fear within the military.

These tensions, divisions, and dissatisfaction within the Gambian military probably contributed to the most recent and past coup attempts against Jammeh. Unfortunately, internal military tensions are difficult to observe and quantify, especially in repressive states like the Gambia. Because these tensions are hard to quantify, they rarely factor into larger statistical forecasts, even though we have good reason to believe they contribute significantly to coup risks.

The recent coup attempt in the Gambia will switch the “domestic coup activity” variable used in Jay’s models from ‘no’ to ‘yes’ and will thereby increase its ranking in the next iteration of those assessments. The climate of fear in the Gambia will also intensify following the crackdown from this week’s coup attempt, however, and may deter copycats in the near future.

From Thailand, Evidence of Coups’ Economic Costs

Last year, I used this space to report a statistical analysis I’d done on the impact of successful coups on countries’ economic growth rates. Bottom line: they usually hurt. As summarized in this chart from a post at FiveThirtyEight, my analysis indicated that, on average, coups dent a country’s economy by a couple of percentage points in the year they happen and another point the year after. Those are not trivial effects.

What makes this question so tricky to analyze is the possibility of a two-way relationship between these things. It’s not hard to imagine that coups might damage national economies, but it’s also likely that countries suffering slower growth are generally more susceptible to coups. With country-year data and no experimental controls, we can’t just run a model with growth as the dependent variable and the occurrence of a coup as a predictor and expect to get a reliable estimate of the former on the latter.

In my statistical analysis, I tried to deal with this problem by using coarsened exact matching to focus the comparison on sets of country-years with comparable coup risk in which coups did or did not happen. I believe the results are more informative than what we’d get from a pooled sample of all country-years, but they certainly aren’t the last word. After all, matching does not magically resolve deeper identification problems, even if it can help.

Under these circumstances, a little process-tracing can go a long way. If we look at real-world cases and see processes linking the “treatment” (coups) to the hypothesized effect (economic damage), we bolster our confidence that the effect we saw in our statistical analysis is not ephemeral.

Here, the recent coup in Thailand is serving up some intriguing evidence. In the past week, I have seen two reports  identifying specific ways in which the coup itself, and not the instability that preceded and arguably precipitated it, is damaging Thailand’s economy. First, I saw this Reuters story (emphasis mine):

Thai officials said on Tuesday that the mass departure of Cambodian laborers would dent the economy as thousands more migrant workers, fearing reprisals from the new military government, poured across the border.

Around 170,000 Cambodian workers have headed home in the past week, according to the International Organization for Migration (IOM), although the exodus is now slowing. Many left after hearing rumors that Thailand’s junta was bent on cracking down on illegal migrants.

Then I saw this tidbit in a Credit Suisse analysis shared on Twitter by Andrew Marshall (emphasis mine):

The coup and martial laws have produced stronger negative impact on Thai tourism, worsening the 2014 earnings outlook and could affect the magnitude of recovery anticipated for 2015…

Based on our conversations with [Airports of Thailand Plc], the coup…appears to have had a stronger impact on its international passenger volumes than the political conflicts… While the coup has restored peace in Bangkok and Thailand, and comforted the Thai people [!], we reckon tourists may take this more negatively and have chosen to go to other destinations.

In a country where international tourism contributes nearly 10 percent of the gross domestic product (here), that impact is a serious issue.

What’s important about both of these reports for the question at hand is the explicit connection they make between the occurrence of the coup and the economy-damaging behavior that followed. To me, these look like consequences rather than coincidences. Neither report definitively proves that the occurrence of a coups usually has an independent, negative effect on a country’s economic growth rate, of course. But they do make me more confident that the effect I saw in my statistical analysis is not just an artifact of some deeper forces I failed to consider.

China and Russia and What Could Have Happened

Twenty five years ago, I was strolling down Leningrad’s main drag, Nevsky Prospekt, with a clutch of other American undergraduates who had recently arrived for two months of intensive language study when Professor Edna Andrews dashed up to us with the news. “They’re shooting them,” she said (or something like it—who can trust a 25-year-old memory of a speech fragment?) with obvious agitation. “They’re shooting the students in Tiananmen Square!”

Had Edna not given us that news, we probably wouldn’t have heard it, or at least not until we got home. In 1989, glasnost’ had already come to the USSR, but that didn’t mean speech was free. State newspapers were still the only ones around, at least for those of us without connections to the world of samizdat. Some of those newspapers were more informative than others, but the limits of political conversation were still clearly proscribed. The Internet didn’t exist, and international calls could only be made by appointment from state-run locations with plastic phones in cubicle-like spaces and who-knows who listening while you talked. Trustworthy information still only trickled through a public sphere mostly bifurcated between propaganda and silence.

What’s striking to me in retrospect is how differently things could have turned out in both countries. When she gave us the news about Tiananmen, Edna was surely agitated because it involved students like the ones she taught being slaughtered. I suspect she was also distressed, though, because at the time it was still easy to imagine something similar happening in the USSR, perhaps even to people she knew personally.

In 1989, politics had already started to move in the Soviet Union, but neither democratization nor disintegration was a foregone conclusion. That spring, citizens had picked delegates to the inaugural session of the Congress of People’s Deputies in elections that were, at the time, the freest the USSR had ever held. The new Congress’ sessions were shown on live television, and their content was stunning. “Deputies from around the country railed against every scandal and shortcoming of the Soviet system that could be identified,” Thomas Skallerup and James P. Nichol describe in their chapter for the Library of Congress’ Russia country study. “Speakers spared neither Gorbachev, the KGB, nor the military.”

But the outspokenness of those reformist deputies belied their formal power. More than 80 percent of the Congress’ deputies were Communist Party members, and the new legislative body the deputies elected that summer, the Supreme Soviet of the USSR, was stuffed with “old-style party apparatchiks.” Two years later, reactionaries inside the government mounted a coup attempt in which President Gorbachev was arrested and detained for a few days and tanks were deployed on the streets of Moscow.

Tank near Red Square on 19 August 1991. © Anatoly Sapronyenkov/AFP/Getty Images

That August Putsch looks a bit clowny with hindsight, but it didn’t have to fail. Likewise, the brutal suppression of China’s 1989 uprising didn’t have to happen, or to succeed when it did. In a story published this week in the New York Times, Andrew Jacobs and Chris Buckley describe the uncertainty of Chinese policy toward the uprising and the disunity of the armed forces tasked with executing it—and, eventually, the protesters in Tiananmen Square.

“At the time,” Jacobs and Buckley write, “few in the military wanted to take direct responsibility for the decision to fire on civilians. Even as troops pressed into Beijing, they were given vague, confusing instructions about what to do, and some commanders sought reassurances that they would not be required to shoot.” Seven senior commanders signed a petition calling on political leaders to withdraw the troops. Those leaders responded by disconnecting many of the special phones those commanders used to communicate with each other. When troops were finally given orders to retake the square “at any cost,” some commanders ignored them. At least one pretended that his battalion’s radio had malfunctioned.

As Erica Chenoweth and Maria Stephan show in their study of civil resistance, nonviolent uprisings are much more likely to succeed when they prompt defections by security forces. The Tiananmen uprising was crushed, but history could have slipped in many other directions. And it still can.

Conflict Events, Coup Forecasts, and Data Prospecting

Last week, for an upcoming post to the interim blog of the atrocities early-warning project I direct, I got to digging around in ACLED’s conflict event data for the first time. Once I had the data processed, I started wondering if they might help improve forecasts of coup attempts, too. That train of thought led to the preliminary results I’ll describe here, and to a general reminder of the often-frustrating nature of applied statistical forecasting.

ACLED is the Armed Conflict Location & Event Data Project, a U.S. Department of Defense–funded, multi-year endeavor to capture information about instances of political violence in sub-Saharan Africa from 1997 to the present.ACLED’s coders scan an array of print and broadcast sources, identifiy relevant events from them, and then record those events’ date, location, and form (battle, violence against civilians, or riots/protests); the types of actors involved; whether or not territory changed hands; and the number of fatalities that occurred. Researchers can download all of the project’s data in various formats and structures from the Data page, one of the better ones I’ve seen in political science.

I came to ACLED last week because I wanted to see if violence against civilians in Somalia had waxed, waned, or held steady in recent months. Trying to answer that question with their data meant:

  • Downloading two Excel spreadsheets, Version 4 of the data for 1997-2013 and the Realtime Data file covering (so far) the first five months of this year;
  • Processing and merging those two files, which took a little work because my software had trouble reading the original spreadsheets and the labels and formats differed a bit across them; and
  • Subsetting and summarizing the data on violence against civilians in Somalia, which also took some care because there was an extra space at the end of the relevant label in some of the records.

Once I had done these things, it was easy to generalize it to the entire data set, producing tables with monthly counts of fatalities and events by type  for all African countries over the past 13 years. And, once I had those country-month counts of conflict events, it was easy to imagine using them to try to help forecast of coup attempts in the world’s most coup-prone region. Other things being equal, variations across countries and over time in the frequency of conflict events might tell us a little more about the state of politics in those countries, and therefore where and when coup attempts are more likely to happen.

Well, in this case, it turns out they don’t tell us much more. The plot below shows ROC curves and the areas under those curves for the out-of-sample predictions from a five-fold cross-validation exercise involving a few country-month models of coup attempts. The Base Model includes: national political regime type (the categorization scheme from PITF’s global instability model applied to Polity 3d, the spell-file version); time since last change in Polity score (in days, logged); infant mortality rate (relative to the annual global median, logged); and an indicator for any coup attempts in the previous 24 months (yes/no). The three other models add logged sums of counts of ACLED events by type—battles, violence against civilians, or riots/protests—in the same country over the previous three, six, or 12 months, respectively. These are all logistic regression models, and the dependent variable is a binary one indicating whether or not any coup attempts (successful or failed) occurred in that country during that month, according to Powell and Thyne.

ROC Curves and AUC Scores from Five-Fold Cross-Validation of Coup Models Without and With ACLED Event Counts

ROC Curves and AUC Scores from Five-Fold Cross-Validation of Coup Models Without and With ACLED Event Counts

As the chart shows, adding the conflict event counts to the base model seems to buy us a smidgen more discriminatory power, but not enough to have confidence that they would routinely lead to more accurate forecasts. Intriguingly, the crossing of the ROC curves suggests that the base model, which emphasizes structural conditions, is actually a little better at identifying the most coup-prone countries. The addition of conflict event counts to the model leads to some under-prediction of coups in that high-risk set, but the balance tips the other way in countries with less structural vulnerability. In the aggregate, though, there is virtually no difference in discriminatory power between the base model and the ones that at the conflict event counts.

There are, of course, many other ways to group and slice ACLED’s data, but the rarity of coups leads me to believe that narrower cuts or alternative operationalizations aren’t likely to produce stronger predictive signals. In Africa since 1997, there are only 36 country-months with coup attempts, according to Powell and Thyne. When the events are this rare and complex and the examples this few, there’s really not much point in going beyond the most direct measures. Under these circumstances, we’re unlikely to discover finer patterns, and if we do, we probably shouldn’t have much confidence in them. There are also other models and techniques to try, but I’m dubious for the same reasons. (FWIW, I did try Random Forests and got virtually identical accuracy.)

So those are the preliminary results from this specific exercise. (The R scripts I used are on Github, here). I think those results are interesting in their own right, but the process involved in getting to them is also a great example of the often-frustrating nature of applied statistical forecasting. I spent a few hours each day for three days straight getting from the thought of exploring ACLED to the results described here. Nearly all of that time was spent processing data; only the last half-hour or so involved any modeling. As is often the case, a lot of that data-processing time was really just me staring at my monitor trying to think of another way to solve some problem I’d already tried and failed to solve.

In my experience, that kind of null result is where nearly all statistical forecasting ideas end. Even when you’re lucky enough to have the data to pursue them, few of your ideas pan out. But panning is the right metaphor, I think. Most of the work is repetitive and frustrating, but every so often you catch a nice nugget. Those nuggets tempt you to keep looking for more, and once in a great while, they can make you rich.

Ripple Effects from Thailand’s Coup

Thailand just had another coup, its first since 2006 but its twelfth since 1932. Here are a few things statistical analysis tells us about how that coup is likely to reverberate through Thailand’s economy and politics for the next few years.

1. Economic growth will probably suffer a bit more. Thailand’s economy was already struggling in 2014, thanks in part to the political instability to which the military leadership was reacting. Still, a statistical analysis I did a few years ago indicates that the coup itself will probably impose yet more drag on the economy. When we compare annual GDP growth rates from countries that suffered coups to similarly susceptible ones that didn’t, we see an average difference of about 2 percentage points in the year of the coup and another 1 percentage point the year after. (See this FiveThirtyEight post for a nice plot and discussion of those results.) Thailand might find its way to the “good” side of the distribution underlying those averages, but the central tendency suggests an additional knock on the country’s economy.

2. The risk of yet another coup will remain elevated for several years. The “coup trap” is real. Countries that have recently suffered successful or failed coup attempts are more likely to get hit again than ones that haven’t. This increase in risk seems to persist for several years, so Thailand will probably stick toward the top of the global watch list for these events until at least 2019.

3. Thailand’s risk of state-led mass killing has nearly tripled…but remains modest. The risk and occurrence of coups and the character of a country’s national political regime feature prominently in the multimodel ensemble we’re using in our atrocities early-warning project to assess risks of onsets of state-led mass killing. When I recently updated those assessments using data from year-end 2013—coming soon to a blog near you!—Thailand remained toward the bottom of the global distribution: 100th of 162 countries, with a predicted probability of just 0.3%. If I alter the inputs to that ensemble to capture the occurrence of this week’s coup and its effect on Thailand’s regime type, the predicted probability jumps to about 0.8%.

That’s a big change in relative risk, but it’s not enough of a change in absolute risk to push the country into the end of the global distribution where the vast majority of these events occur. In the latest assessments, a risk of 0.8% would have placed Thailand about 50th in the world, still essentially indistinguishable from the many other countries in that long, thin tail. Even with changes in these important risk factors and an ongoing insurgency in its southern provinces, Thailand remains in the vast bloc of countries where state-led mass killing is extremely unlikely, thanks (statistically speaking) to its relative wealth, the strength of its connection to the global economy, and the absence of certain markers of atrocities-prone regimes.

4. Democracy will probably be restored within the next few years… As Henk Goemans and Nikolay Marinov show in a paper published last year in the British Journal of Political Science, since the end of the Cold War, most coups have been followed within a few years by competitive elections. The pattern they observe is even stronger in countries that have at least seven years of democratic experience and have held at least two elections, as Thailand does and has. In a paper forthcoming in Foreign Policy Analysis that uses a different measure of coups, Jonathan Powell and Clayton Thyne see that same broad pattern. After the 2006 coup, it took Thailand a little over a year to get back to a competitive elections for a civilian government under a new constitution. If anything, I would expect this junta to move a little faster, and I would be very surprised if the same junta was still ruling in 2016.

5. …but it could wind up right back here again after that. As implied by nos. 1 and 2 above, however, the resumption of democracy wouldn’t mean that Thailand won’t repeat the cycle again. Both statistical and game-theoretic models indicate that prospects for yet another democratic breakdown will stay relatively high as long as Thai politics remains sharply polarized. My knowledge of Thailand is shallow, but the people I read or follow who know the country much better skew pessimistic on the prospects for this polarization ending soon. From afar, I wonder if it’s ultimately a matter of generational change and suspect that Thailand will finally switch to a stable and less contentious equilibrium when today’s conservative leaders start retiring from their jobs in the military and bureaucracy age out of street politics.

Follow

Get every new post delivered to your Inbox.

Join 10,418 other followers

%d bloggers like this: