All posts tagged Haiti

Coup Forecasts for 2013

Last January, I posted statistical estimates of coup risk for 2012 that drew some wider interest after they correctly identified Mali as a high-risk case. Now that the year’s almost over, I thought it would be a good time to assess more formally how those 2012 forecasts performed and then update them for 2013.

So, first things first: how did the 2012 forecasts fare on the whole? Pretty well, actually.

For purposes of these forecasts, a coup is defined as “as a forceful seizure of executive authority and office by a dissident/opposition faction within the country’s ruling or political elites that results in a substantial change in the executive leadership and the policies of the prior regime.” That language comes from Monty Marshall’s Center for Systemic Peace, whose data set on coup events serves as the basis for one of the two models used to generate the 2012 forecasts. Those forecasts were meant to assess the risk of any coup attempts at some point during the calendar year, whether those attempts succeed or fail. They were not meant to anticipate civil wars, non-violent uprisings, voluntary transfers of executive authority, autogolpes, or interventions by foreign forces, all of which are better thought of (and modeled) as different forms of political crisis.

Okay, so by that definition, I see two countries where coup attempts occurred in 2012: Mali (in March) and Guinea-Bissau (in April). As it happens, both of those countries ranked in the top 10 in January’s forecasts—Guinea-Bissau at no. 2 and Mali at no. 10—so the models seem to be homing in on the right things. We can get a more rigorous take on the forecasts’ accuracy with a couple of statistics commonly used to assess models that try to predict binary outcomes like these (either a coup attempt happens or it doesn’t):

AUC Score. The estimated area under the Receiver Operating Characteristic (ROC) curve, used as a measure of the ability of a binary classification model to discriminate between positive and negative cases. Specifically, AUC represents the probability that a randomly selected positive case (here, a country-year with coup activity) will have a higher predicted probability than a randomly selected negative case (e.g., country-year with no coup activity). Ranges from 0.5 to 1, with higher values indicating better discrimination.
Brier Score. A general measure of forecast performance, defined as the average squared difference between the predicted and observed values. Ranges from 0 to 1, with lower values indicating more accurate predictions.

Assuming that Mali and Guinea-Bissau were the only countries to see coup activity this year, my 2012 coup forecasts get an AUC score of 0.97 and a Brier score of 0.01. Those are really good numbers. Based on my experience trying to forecast other rare political events around the world, I’m pretty happy with any AUC above the low 0.80s and any Brier score that’s better than an across-the-board base-rate forecast. The 2012 coup forecasts surpass both of those benchmarks.

Of course, with just two events in more than 150 countries, these statistics could be very sensitive to changes in the list of coup attempts. Two possible modifications come from Sudan, where authorities claim to have thwarted coup plots in November and December, and Paraguay, where right-wing legislators pushed leftist President Lugo out of office in June. I didn’t count Sudan because country experts tell me those events were probably just a political ploy President Bashir is using to keep his rivals off balance and not actual coup attempts. I didn’t count Paraguay because President Lugo’s rivals used legal procedures, not force, to oust him in a rushed impeachment. I’m pretty confident that neither of those cases counts as a coup attempt as defined here, but for the sake of argument, it’s worth seeing how the addition of those cases would affect the accuracy assessments.

Sudan ranked 11th in the 2012 forecasts, just behind Mali, so the addition of an event there leaves the accuracy stats essentially unchanged at 0.96 and 0.02, respectively.
Paraguay would definitely count as a surprise. It ranked in the 80s in the 2012 forecasts, and counting its June events as a coup would drop the AUC to 0.80 and the Brier score to 0.02.
If we count both cases as yeses, we get an AUC of 0.84 and a Brier score of 0.02.

All of those are still pretty respectable numbers for true forecasts of rare political events, even if they’re not quite as good as the initial ones. Whatever the exact ground truth, these statistics give me some confidence that the two-model average I’m using here makes a useful forecasting tool.

So, without further ado, what about 2013? The chart below plots estimated coup risk for the coming year for the 30 countries at greatest risk using essentially the same models I used for 2012. (One of the two models differs slightly from last year’s; I cut out a couple of variables that had little effect on the estimates and are especially hard to update.) I picked the top 30 because it’s roughly equivalent to the top quintile, and my experience working with models like these tells me that the top quintile makes a pretty good break point for distinguishing between countries at high and low risk. If a country doesn’t appear in this chart, that means my models think it’s highly unlikely to suffer a coup attempt in the coming year.

2013 Coup Risk Estimates

The broad strokes are very similar to 2012, but I’m also seeing a few changes worth noting.

Consistent with 2012, countries from sub-Saharan Africa continue to dominate the high-risk group. Nine of the top 10 and 22 of the top 30 countries come from that part of the world. One of those 22 is South Sudan, which didn’t get a forecast in early 2012 because I didn’t have the requisite data but now makes an ignominious debut at no. 20. Another is Sudan, which, as Armin Rosen discusses, certainly isn’t getting any more stable. Mali and Guinea-Bissau also both stay near the top of the list, thanks in part to the “coup trap” I discussed in another recent post. Meanwhile, I suspect the models are overestimating the risk of a new coup attempt in Niger, which seems to have landed on firmer footing after its “democratizing” coup in February 2010, but that recent history will leave Niger in the statistical high-risk group until at least 2015.
More surprising to me, Timor-Leste now lands in the top 10. That’s a change from 2012, but only because the data used to generate the 2012 forecasts did not count the assassination attempts of 2008 as a coup try. The latest version of CSP’s coup list does consider those events to be failed coup attempt. Layered on top of Timor-Leste’s high poverty and hybrid political authority patterns, that recent coup activity greatly increases the country’s estimated risk. If Timor-Leste makes it through 2013 without another coup attempt, though, its estimated risk should drop sharply next year.
In Latin America, Haiti and Ecuador both make it into the Top 20. As with Timor-Leste, the changes from 2012 are artifacts of adjustments to the historical data—adding a coup attempt in Ecuador in 2010 and counting Haiti as a partial democracy instead of a state under foreign occupation. Those artifacts mean the change from 2012 isn’t informative, but the presence of those two countries in the top 20 most certainly is.
Syria also pops into the high-risk group at no. 25. That’s not an artifact of data revisions; it’s a reflection of the effects of that country’s devastating state collapse and civil war on several of the risk factors for coups.
Finally, notable for its absence is Egypt, which ranks 48th on the 2013 list and has been a source of coup rumors throughout its seemingly interminable transitional period. It’s worth noting though, that if you consider SCAF’s ouster of Mubarak in 2011 to be a successful coup (CSP doesn’t), Egypt would make its way into the top 30.

As always, if you’re interested in the details of the modeling, please drop me a line at ulfelder@gmail.com and I’ll try to answer your questions as soon as I can.

Update: After a Washington Post blog mapped my Top 30, I produced a map of my own.

80 Comments

by Jay Ulfelder on December 21, 2012 • Permalink

Posted in Coups, Forecasting

Tagged Ecuador, Egypt, Guinea-Bissau, Haiti, Mali, Monty Marshall, Niger, South Sudan, Sudan, syria, Timor-Leste

Posted by Jay Ulfelder on December 21, 2012

https://dartthrowingchimp.wordpress.com/2012/12/21/coup-forecasts-for-2013/

“State Failure” Has Failed. How About Giving “State Collapse” a Whirl?

Foreign Policy magazine recently published the 2012 edition of the Fund for Peace‘s Failed States Index (FSI), and the response in the corner of the international-studies blogosphere I inhabit has been harsh. Scholars have been grumbling about the Failed States Index for years, but the chorus of academic and advocacy voices attacking it seems to have grown unusually large and loud this year. In an admirable gesture of of fair play, Foreign Policy ran one of the toughest critiques of the FSI on its own web site, where Elliot Ross of the blog Africa is a Country wrote,

We at Africa is a Country think Foreign Policy and the Fund for Peace should either radically rethink the Failed States Index, which they publish in collaboration each year, or abandon it altogether. We just can’t take it seriously: It’s a failed index.

As Ross and many others argue, the core problem with the FSI is that it defines state failure very broadly, and in a way that seems to privilege certain forms of political stability over other aspects of governance and quality of life that the citizens in those states may prize more highly. In a 2008 critique of the “state failure” concept [PDF] that nicely anticipated all of the recent sturm und drang around the FSI, Chuck Call wrote that

The ‘failed states’ concept—and related terms like ‘failing’, ‘fragile’, ‘stressed’ and ‘troubled’ states—has become more of a liability than an asset. Foundations and think tanks have rushed to fund work on ‘failing’ states, resulting in a proliferation of multiple, divergent and poorly defined uses of the term. Not only does the term ‘failing state’ reflect the schoolmarm’s scorecard according to linear index defined by a univocal Weberian endstate, but it has also grown to encompass states as diverse as Colombia, East Timor, Indonesia, North Korea, Cote d’Ivoire, Haiti, Iraq, and the Sudan.

In that essay, Call advocates abandoning the now-hopelessly-freighted concept of “state failure” in favor of a narrower focus on “state collapse”—that is, situations “where no authority is recognisable either internally to a country’s inhabitants or externally to the international community.” I agree.

In fact, in 2010, while still working as research director for the U.S. Government–funded Political Instability Task Force, I led a small research project that aimed to develop a workable definition of state collapse and coding guidelines that would allow researchers to know it when they see it. The project stopped short of producing a global, historical data set, but the coding guidelines were road-tested and refined, and I think the end results have some value. In light of the FSI brouhaha, I’ve posted the results of that project on the Social Science Research Network (SSRN) in hopes that they might be useful to a broader audience.

In those materials—a concept paper and a set of coding guidelines—I argue that we can get to a more workable concept by moving away from Max Weber’s aspirational vision of modern states as legitimate and orderly bureaucracies. Instead, I think we get further when we recognize that real-world states are a specific kind of political organization associated with a particular realization of global politics. That realization—the “Westphalian order,” or just “the international system”—constitutes states and delegates certain forms of political authority to them, but national governments in the real world vary widely in their ability to exercise that authority. When internationally recognized governments cease to exist, or their actual authority is badly circumscribed, we can say that the state has collapsed. That kind of collapse can happen in two different ways: fragmentation and disintegration.

When the failure to rule involves the national government’s territorial reach, we might call it collapse by fragmentation. The ideal of domestic sovereignty presumes final authority within a specific territory and international recognition of that authority, so situations in which large swaths of a state’s territory are effectively governed by organized political challengers whose authority is not internationally recognized represent a form of collapse. In practical terms, these situations usually arise in one of two ways: either 1) a rebel group violently pushes state agents out of a particular area, or 2) a regional government unilaterally proclaims its autonomy or independence and becomes the de facto sovereign authority in that region. In either situation, the rival group directly and publicly challenges the national government’s claim to sovereignty and effectively becomes the supreme political authority in that space. State military forces may still operate in these areas, but they do so in an attempt to reassert control that has already been lost, as indicated by the primacy of the rival organization in day-to-day governance…

State collapse also occurs when the national government fails to enforce its authority in the absence of a rival claimant to sovereignty. This type of failure might be called state collapse by disintegration. The ideal of domestic sovereignty presumes that a central government is capable not just of making rules but also of enforcing them. Dramatic failures of a state’s enforcement capabilities are indicated by widespread lawlessness and disorder, such as rioting, looting, civil violence, and vigilantism. In the extreme, central governments will sometimes disappear completely, but this rarely occurs. More often, a national government will continue to operate, but its rules will be ignored in some portions of its putative territory.

To distinguish state collapse from other forms of political instability and disorder, we have to establish some arbitrary thresholds beyond which the failure is considered catastrophic. Saying focused on the core dimensions of domestic sovereignty—territory and order—I do this as follows:

A state collapse occurs when a sovereign state fails to provide public order in at least one-half of its territory or in its capital city for at least 30 consecutive days. A sovereign state is regarded as failing to provide public order in a particular area when a) an organized challenger, usually a rebel group or regional government, effectively controls that area; b) lawlessness pervades in that area; or c) both. A state is considered sovereign when it is granted membership in the U.N. General Assembly.

If you’re interested, you can find more specific language on how to assess challenger control and lawlessness in the coding guidelines.

Applying this definition to the world today, I see only a handful of states that are clearly collapsed and just a few more that might be. In the “clearly collapsed” category, I would put Libya, Mali, Somalia, and Yemen. In the “probably collapsed” category, I would put Afghanistan and Democratic Republic of Congo. Those judgments are based on cursory knowledge of those cases, however, and I would be interested to hear what others think about where this label does (Chad? Haiti? Ivory Coast? Sudan? South Sudan?) or does not (Afghanistan? Mali?) fit. Either way, the list is much shorter and, I believe, more coherent than the 20-country sets the Failed States Index identifies as “critical” and “in danger.”

More important, this is a topic that still greatly interests me, so I would love to have this conceptual work critiqued, put to use, or both. Fire away!