All posts tagged Monty Marshall

A Research Note on Updating Coup Forecasts

A new year is about to start, and that means it’s time for me to update my coup forecasts (see here and here for the 2013 and 2012 editions, respectively). The forecasts themselves aren’t quite ready yet—I need to wait until mid-January for updates from Freedom House to arrive—but I am making some changes to my forecasting process that I thought I would go ahead and describe now, because the thinking behind them illustrates some important dilemmas and new opportunities for predictions of many kinds of political events.

When it comes time to build a predictive statistical model of some rare political event, it’s usually not the model specification that gives me headaches. For many events of interest, I think we now have a pretty good understanding of which methods and variables are likely to produce more accurate forecasts.

Instead, it’s the data, or really the lack thereof, that sets me to pulling my hair out. As I discussed in a recent post, things we’d like to include in our models fall into a few general classes in this regard:

No data exist (fuggeddaboudit)
Data exist for some historical period, but they aren’t updated (“HA-ha!”)
Data exist and are updated, but they are patchy and not missing at random (so long, some countries)
Data exist and are updated, but not until many months or even years later (Spinning Pinwheel of Death)

In the past, I’ve set aside measures that fall into the first three of those sets but gone ahead and used some from the fourth, if I thought the feature was important enough. To generate forecasts before the original sources updated, I either a) pulled forward the last observed value for each case (if the measure was slow-changing, like a country’s infant mortality rate) or b) hand-coded my own updates (if the measure was liable to change from year to year, like a country’s political regime type).

Now, though, I’ve decided to get out of the “artisanal updating” business, too, for all but the most obvious and uncontroversial things, like which countries recently joined the WTO or held national elections. I’m quitting this business, in part, because it takes a lot of time and the results may be pretty noisy. More important, though, I’m also quitting because it’s not so necessary any more, thanks to timelier updates from some data providers and the arrival of some valuable new data sets.

This commitment to more efficient updating has led me to adopt the following rules of thumb for my 2014 forecasting work:

For structural features that don’t change much from year to year (e.g., population size or infant mortality), include the feature and use the last observed value.
For variables that can change from year to year in hard-to-predict ways, only include them if the data source is updated in near-real time or, if it’s updated annually, if those updates are delivered within the first few weeks of the new year.
In all cases, only use data that are publicly available, to facilitate replication and to encourage more data sharing.

And here are some of the results of applying those rules of thumb to the list of features I’d like to include in my coup forecasting models for 2014.

Use Powell and Thyne’s list of coup events instead of Monty Marshall’s. Powell and Thyne’s list is updated throughout the year as events occur, whereas the publicly available version of Marshall’s list is only updated annually, several months after the start of the year. That wouldn’t matter so much if coups were only the dependent variable, but recent coup activity is also an important predictor, so I need the last year’s updates ASAP.

Use Freedom House’s Freedom in the World (FIW) data instead of Polity IV to measure countries’ political regime type. Polity IV offers more granular measures of political regime type than Freedom in the World, but Polity updates aren’t posted until spring or summer of the following year, usually more than a third of the way into my annual forecasting window.

Use IMF data on economic growth instead of the World Bank’s. The Bank now updates its World Development Indicators a couple of times a year, and there’s a great R package that makes it easy to download the bits you need. That’s wonderful for slow-changing structural features, but it still doesn’t get me data on economic performance as fast as I’d like it. I work around that problem by using the IMF’s World Economic Outlook Database, which include projections for years for which observed data aren’t yet available and forecasts for several years into the future.

Last but not least, use GDELT instead of UCDP/PRIO or Major Episodes of Political Violence (MEPV) to measure civil conflict. Knowing which countries have had civil unrest or violence in the recent past can help us predict coup attempts, but the major publicly available measures of these things are only updated well into the year. GDELT now represents a nice alternative. It covers the whole world, measures lots of different forms of political cooperation and conflict, and is updated daily, so country-year updates are available on January 2. GDELT’s period of observation starts in 1979, so it’s still a stretch to use it models of super-rare events like mass-killing onsets, where the number of available examples since 1979 on which to train is still relatively small. For less-rare events like coup attempts, though, starting the analysis around 1980 is no problem. (Just don’t forget to normalize them!) With some help from John Beieler, I’m already experimenting with adding annual GDELT summaries to my coup forecasting process, and I’m finding that they do improve the model’s out-of-sample predictive power.

In all of the forecasting work I do, my long-term goals are 1) to make the forecasts more dynamic by updating them more frequently (e.g., monthly, weekly, or even daily instead of yearly) and 2) to automate that updating process as much as possible. The changes I’m making to my coup forecasting process for 2014 don’t directly accomplish either of these things, but they do take me a few steps in both directions. For example, once GDELT is in the mix, it’s possible to start thinking about how to switch to monthly or even daily updates that rely on a sliding window of recent GDELT tallies. And once I’ve got a coup data set that updates in near-real time, I can imagine pinging that source each day to update the counts of coup attempts in the past several years. I’m still not where I’d like to be, but I think I’m finally stepping onto a path that can carry me there.

3 Comments

by Jay Ulfelder on December 29, 2013 • Permalink

Posted in Coups, Forecasting, Methods, Statistics

Tagged Clayton Thyne, GDELT, IMF, Jonathan Powell, Monty Marshall, Polity, World Bank

Posted by Jay Ulfelder on December 29, 2013

https://dartthrowingchimp.wordpress.com/2013/12/29/a-research-note-on-updating-coup-forecasts/

Animated Map of Coup Attempts Worldwide, 1946-2013

I’m in the throes of updating my data files to prepare for 2014 forecasts of various forms of political change, including coups d’etat. For the past couple of years, I’ve used the coup event list Monty Marshall produces (here) as my primary source on this topic, and I’ve informally cross-referenced Monty’s accounting with the list produced by Jonathan Powell and Clayton Thyne (here).

This year, I decided to quit trying to pick a favorite or adjudicate between the two and just go ahead and mash them up. The two projects use slightly different definitions, but both are basically looking for the same thing: some faction of political insiders (including but not limited to military leaders) seizes executive power at the national level by unconstitutional means that include the use or threat of force.

After stretching the two data sets into country-year format and merging the results, I created separate indicators for successful and failed coups that are scored 1 if either source reports an event of that type and 0 otherwise. For example, Marshall’s data set doesn’t see the removal of Egyptian president Hosni Mubarak from office in 2011 as a coup, but Powell and Thyne’s does, so in my mashed-up version, Egypt gets a 1 for 2011 on the indicator for any successful coups.* The Marshall data set starts in 1946, but Powell and Thyne don’t start until 1950, so my observations for 1946-1949 are based solely on the former. Powell and Thyne update their file on the go, however, whereas Marshall only updates once a year. This means that Powell and Thyne already have most of 2013 covered, so my observations for this year so far are based solely on their reckoning.

The bar plot below shows what the data from the combined version look like over time. The trend is basically the same one we’d see from either of the constituent sources. The frequency of coup attempts grew noticeably in the 1960s and 1970s; continued apace through the 1980s and 1990s, but with fewer successes; and then fell sharply in the past two decades.

We can see those time trends and the geographic distribution of these events in the GIF below (you may need to click on it to get it to play). As the maps show, coup events were pretty well scattered across the world in the 1960s and 1970s, but in the past 20 years, they’ve mostly struck in Africa and Asia.

A .csv with the mashed-up data is on Google Drive (here), and you can find the R script I used to make these plots on Github (here).

Update: For a new-and-improved version that uses daily data and is interactive, see this follow-up post.

* This sentence corrects an error I made in the original version of this post. In that version, I stated that Marshall did not consider the 3 July 2013 events in Egypt to include a coup. That was incorrect, and I apologize to him for my error.

10 Comments

by Jay Ulfelder on November 22, 2013 • Permalink

Posted in Coups, Statistics

Tagged Clayton Thyne, data, Jonathan Powell, Monty Marshall

Posted by Jay Ulfelder on November 22, 2013

https://dartthrowingchimp.wordpress.com/2013/11/22/animated-map-of-coup-attempts-worldwide-1946-2013/

Yes, That’s a Coup in Egypt

Apparently, some of the protesters who support what the Egyptian army is doing right now claim it isn’t a coup because they believe it expresses the popular will, and the U.S. and the E.U. so far refuse to stick a label on it.

Well, I hate to break it to those people, but in any conventional sense of the term, this is a coup. Here are a few of the definitions used by leading scholars of coups and civil-military relations. First, Monty Marshall, who compiles a data set on coups and coup attempts for the Political Instability Task Force (scroll down to the Polity IV section here):

A coup d’état is defined as a forceful seizure of executive authority and office by a dissident/opposition faction within the country’s ruling or political elites that results in a substantial change in the executive leadership and the policies of the prior regime (although not necessarily in the nature of regime authority or mode of governance).

Now Jonathan Powell and Clayton Thyne from the coding rules for their Coup d’état Dataset:

[Coups d’etat are defined as] overt attempts by the military or other elites within the state apparatus to unseat the sitting head of state using unconstitutional means…there is no minimal death threshold for defining a coup. A coup attempt is defined as successful if the coup perpetrators seize and hold power for at least seven days.

Last but not least, Samuel Huntington from his 1968 classic, Political Order in Changing Societies:

The distinguishing characteristics of the coup coup d’état as a political technique are that: (a) it is the effort by a political coalition illegally to replace the existing governmental leaders by violence or the threat of violence; (b) the violence employed is usually small; (c) the number of people involved is small; (d) the participants already possess institutional bases of power within the political system.

Force deployed? Check. By political insiders? Check. Chief executive replaced? Check. Legal procedures not followed? Check.

That the army’s apparent ouster of President Morsi may be popular doesn’t make it legal or erase the fact that he only “agreed” to go when coerced. That military leaders may not claim executive authority for themselves does not obviate the fact that they are pushing out a sitting president at gunpoint. That the coup could push Egypt onto a more positive trajectory doesn’t change the nature of the initial act.

On that last point, I’ll emphasize the word “could.” It’s impossible to say with confidence what comes next for Egypt. I’ve seen a number of people list infamous coups from the past (Algeria, Argentina, Chile, Iran…) as evidence that military intervention always makes things worse, but I’ve also seen a recent study by Nikolay Marinov and Hein Geomans showing that coups in the post-Cold War period have been less damaging to democratization:

Whereas the vast majority of successful coups before 1991 installed durable rules, the majority of coups after that have been followed by competitive elections… While the coup d’état has been and still is the single most important factor leading to the downfall of democratic government, our findings indicate that the new generation of coups has been far less harmful for democracy than their historical predecessors.

Again, I don’t know what comes next in Egypt, but I think the folks using historical analogies to argue that a coup can only make things worse there are ignoring an important source of bias in their analysis. Maybe coups are bad for the health of the polity, but there’s a selection effect at work here, too. Coups happen in situations that are already crappy, and the set of plausible counterfactuals in these crappy situations rarely includes a sharp turn for the better. A coup in Egypt might delay democratization and further damage the already-reeling economy, but it’s hard to imagine an alternative path from June 30 that is both politically realistic and looks a whole lot better. This is the common tragedy of transitional politics, and Egypt appears to be no exception.

40 Comments

by Jay Ulfelder on July 3, 2013 • Permalink

Posted in Coups, Democratization, Transitions to Democracy

Tagged Egypt, Hein Goemans, Monty Marshall, Nikolay Marinov, Samuel Huntington

Posted by Jay Ulfelder on July 3, 2013

https://dartthrowingchimp.wordpress.com/2013/07/03/yes-thats-a-coup-in-egypt/

Coup Forecasts for 2013

Last January, I posted statistical estimates of coup risk for 2012 that drew some wider interest after they correctly identified Mali as a high-risk case. Now that the year’s almost over, I thought it would be a good time to assess more formally how those 2012 forecasts performed and then update them for 2013.

So, first things first: how did the 2012 forecasts fare on the whole? Pretty well, actually.

For purposes of these forecasts, a coup is defined as “as a forceful seizure of executive authority and office by a dissident/opposition faction within the country’s ruling or political elites that results in a substantial change in the executive leadership and the policies of the prior regime.” That language comes from Monty Marshall’s Center for Systemic Peace, whose data set on coup events serves as the basis for one of the two models used to generate the 2012 forecasts. Those forecasts were meant to assess the risk of any coup attempts at some point during the calendar year, whether those attempts succeed or fail. They were not meant to anticipate civil wars, non-violent uprisings, voluntary transfers of executive authority, autogolpes, or interventions by foreign forces, all of which are better thought of (and modeled) as different forms of political crisis.

Okay, so by that definition, I see two countries where coup attempts occurred in 2012: Mali (in March) and Guinea-Bissau (in April). As it happens, both of those countries ranked in the top 10 in January’s forecasts—Guinea-Bissau at no. 2 and Mali at no. 10—so the models seem to be homing in on the right things. We can get a more rigorous take on the forecasts’ accuracy with a couple of statistics commonly used to assess models that try to predict binary outcomes like these (either a coup attempt happens or it doesn’t):

AUC Score. The estimated area under the Receiver Operating Characteristic (ROC) curve, used as a measure of the ability of a binary classification model to discriminate between positive and negative cases. Specifically, AUC represents the probability that a randomly selected positive case (here, a country-year with coup activity) will have a higher predicted probability than a randomly selected negative case (e.g., country-year with no coup activity). Ranges from 0.5 to 1, with higher values indicating better discrimination.
Brier Score. A general measure of forecast performance, defined as the average squared difference between the predicted and observed values. Ranges from 0 to 1, with lower values indicating more accurate predictions.

Assuming that Mali and Guinea-Bissau were the only countries to see coup activity this year, my 2012 coup forecasts get an AUC score of 0.97 and a Brier score of 0.01. Those are really good numbers. Based on my experience trying to forecast other rare political events around the world, I’m pretty happy with any AUC above the low 0.80s and any Brier score that’s better than an across-the-board base-rate forecast. The 2012 coup forecasts surpass both of those benchmarks.

Of course, with just two events in more than 150 countries, these statistics could be very sensitive to changes in the list of coup attempts. Two possible modifications come from Sudan, where authorities claim to have thwarted coup plots in November and December, and Paraguay, where right-wing legislators pushed leftist President Lugo out of office in June. I didn’t count Sudan because country experts tell me those events were probably just a political ploy President Bashir is using to keep his rivals off balance and not actual coup attempts. I didn’t count Paraguay because President Lugo’s rivals used legal procedures, not force, to oust him in a rushed impeachment. I’m pretty confident that neither of those cases counts as a coup attempt as defined here, but for the sake of argument, it’s worth seeing how the addition of those cases would affect the accuracy assessments.

Sudan ranked 11th in the 2012 forecasts, just behind Mali, so the addition of an event there leaves the accuracy stats essentially unchanged at 0.96 and 0.02, respectively.
Paraguay would definitely count as a surprise. It ranked in the 80s in the 2012 forecasts, and counting its June events as a coup would drop the AUC to 0.80 and the Brier score to 0.02.
If we count both cases as yeses, we get an AUC of 0.84 and a Brier score of 0.02.

All of those are still pretty respectable numbers for true forecasts of rare political events, even if they’re not quite as good as the initial ones. Whatever the exact ground truth, these statistics give me some confidence that the two-model average I’m using here makes a useful forecasting tool.

So, without further ado, what about 2013? The chart below plots estimated coup risk for the coming year for the 30 countries at greatest risk using essentially the same models I used for 2012. (One of the two models differs slightly from last year’s; I cut out a couple of variables that had little effect on the estimates and are especially hard to update.) I picked the top 30 because it’s roughly equivalent to the top quintile, and my experience working with models like these tells me that the top quintile makes a pretty good break point for distinguishing between countries at high and low risk. If a country doesn’t appear in this chart, that means my models think it’s highly unlikely to suffer a coup attempt in the coming year.

2013 Coup Risk Estimates

The broad strokes are very similar to 2012, but I’m also seeing a few changes worth noting.

Consistent with 2012, countries from sub-Saharan Africa continue to dominate the high-risk group. Nine of the top 10 and 22 of the top 30 countries come from that part of the world. One of those 22 is South Sudan, which didn’t get a forecast in early 2012 because I didn’t have the requisite data but now makes an ignominious debut at no. 20. Another is Sudan, which, as Armin Rosen discusses, certainly isn’t getting any more stable. Mali and Guinea-Bissau also both stay near the top of the list, thanks in part to the “coup trap” I discussed in another recent post. Meanwhile, I suspect the models are overestimating the risk of a new coup attempt in Niger, which seems to have landed on firmer footing after its “democratizing” coup in February 2010, but that recent history will leave Niger in the statistical high-risk group until at least 2015.
More surprising to me, Timor-Leste now lands in the top 10. That’s a change from 2012, but only because the data used to generate the 2012 forecasts did not count the assassination attempts of 2008 as a coup try. The latest version of CSP’s coup list does consider those events to be failed coup attempt. Layered on top of Timor-Leste’s high poverty and hybrid political authority patterns, that recent coup activity greatly increases the country’s estimated risk. If Timor-Leste makes it through 2013 without another coup attempt, though, its estimated risk should drop sharply next year.
In Latin America, Haiti and Ecuador both make it into the Top 20. As with Timor-Leste, the changes from 2012 are artifacts of adjustments to the historical data—adding a coup attempt in Ecuador in 2010 and counting Haiti as a partial democracy instead of a state under foreign occupation. Those artifacts mean the change from 2012 isn’t informative, but the presence of those two countries in the top 20 most certainly is.
Syria also pops into the high-risk group at no. 25. That’s not an artifact of data revisions; it’s a reflection of the effects of that country’s devastating state collapse and civil war on several of the risk factors for coups.
Finally, notable for its absence is Egypt, which ranks 48th on the 2013 list and has been a source of coup rumors throughout its seemingly interminable transitional period. It’s worth noting though, that if you consider SCAF’s ouster of Mubarak in 2011 to be a successful coup (CSP doesn’t), Egypt would make its way into the top 30.

As always, if you’re interested in the details of the modeling, please drop me a line at ulfelder@gmail.com and I’ll try to answer your questions as soon as I can.

Update: After a Washington Post blog mapped my Top 30, I produced a map of my own.