On Prediction

It is surprisingly hard to find cogent statements about why prediction is important for developing theory in political science. This is primarily a cultural artifact, I think—political science has mostly eschewed prediction for decades, and many colleagues and reviewers remain openly hostile to it, so it’s not something we spend much time writing and talking about—but there’s a hard philosophy-of-science question lurking there, too. It’s the one Karl Popper implies when he argues in “Prediction and Prophecies in the Social Sciences” that:

Long-term prophecies can be derived from scientific conditional predictions only if they apply to systems which can be described as well-isolated, stationary, and recurrent. These systems are very rare in nature; and modern society is not one of them.

In other words, the causal processes social scientists aim to discover could be moving targets, so the theories we develop through our research will typically be bounded and contingent. If we’re not sure a priori how general we expect our theories to be, then how much predictive power should we expect them to have? To which cases is a theory meant to apply, and therefore to predict? Without an answer to that question, we can’t confidently judge how informative our predictive accuracy or errors are, and we can only answer that question with another layer of theory.

Statistical modeling suggests a practical rationale to prefer out-of-sample to in-sample “prediction” as means of validation, namely, overfitting. That’s what we call it when we build models that chase after the idiosyncrasies in the data on which they’re trained and lose sight along the way of the wider regularities we seek to uncover. As a technical matter, though, the problem of overfitting is specific to the method, and as philosophical point it’s another handwave. In particular, it doesn’t deal with the possibility that a theory works wonderfully for the sample from which it’s derived but poorly elsewhere. That proposition must sound bizarre to biologists or physicists, but as Popper and other would argue, it’s not a crazy idea in political science. Do we really think that the causes of war between states are the same now as they were 100 years ago? Of democratization? Human physiology may not evolve that fast, but human society arguably does, at least recently.

After all that hand-wringing, I land on a pragmatic rationale for emphasizing prediction as a means of validation: it works better than the alternatives. Science is a method for deepening our understanding of the world we inhabit. Deepening implies directional movement. For science to move, we have to try to assess the validity of, and adjudicate between, different ideas. From psychology, we know that the sheer plausibility of a story—and at some level, that’s really what all social-science theories are, whichever “language” we use to represent them—is not a reliable guide to its truth. Just because something makes sense or feels right does not mean that it is, and our brains are terrible about (or excellent at) filtering evidence to confirm our expectations.

Under those circumstances, we need some other way to assess whether an explanation is plausible—or, when more than one explanation is available, to determine which version is more plausible than the others. We can try to do that with reference to the evidence from which the explanation was derived, but the result is a tautology: evidence A suggests theory B; we know theory B is correct because it implies evidence A, and A is what we observe.

The alternative that remains is prediction. To determine whether or not our mental models are getting at something “real,” we have to apply them to situations yet unseen and see if the regularities we thought we had uncovered hold. The results will rarely be binary, but they will still provide new information about the usefulness of the model. Crucially, those results can also be compared to ones from competing theories to help us determine which explanation covers more. That’s the engine of accumulation. Under certain conditions, that new information may even reveal something about the scope conditions of the initial theory. When a model routinely predicts well in some kinds of cases but not others, then we have uncovered a new pattern that we can add to the initial construct and then continue to test in the same way.

For reasons to which Popper alludes, I don’t believe these iterations will reveal laws that consistently explain and anticipate political behavior writ large. Still, we have seen this process produce great advances in other fields, so I prefer it to the alternative of not trying. And, absent some version of this process, the theories we construct about politics are epistemologically indistinguishable from fiction. Fiction can be satisfying to write and read and even be “true” in a fashion, but it is not science because, among other things, it does not aspire to accumulation.

I use the word “aspire” in that last sentence advisedly. Directionality does not necessarily imply eventual arrival, or even reliable navigation. I think it’s perfectly reasonable to establish the accumulation of knowledge at the basic point of the endeavor and still understand and experience science as neuroscientist Stuart Firestein describes it in his wonderful book, Ignorance. Firestein opens the book with a proverb—”It is very difficult to find a black cat in a dark room, especially when there is no cat”—and goes on to say that

[Science] is not facts and rules. It’s black cats in dark rooms. As the Princeton mathematician Andrew Wiles describes it: It’s groping and probing and poking, and some bumbling and bungling, and then a switch is discovered, often by accident, and the light is lit, and everyone says, “Oh, wow, so that’s how it looks,” and then it’s off into the next dark room, looking for the next mysterious black feline. If this all sounds depressing, perhaps some bleak Beckett-like scenario of existential endlessness, it’s not. In fact, it’s somehow exhilarating.

From my own experience, I’d say it can be both depressing and exhilarating, but the point still stands.

China Isn’t Socialist, It’s High Modernist

In today’s New York Times, Ian Johnson reports that

China has announced a sweeping plan to manage the flow of rural residents into cities, promising to promote urbanization but also to solve some of the drastic side effects of this great uprooting…

[The plan] states that “urbanization is modernization” and “urbanization is an inevitable requirement for promoting social progress,” noting that every developed country is urbanized and industrialized.

In certain circles of development studies, it’s become almost cliché to invoke James Scott’s Seeing Like a State: How Certain Schemes to Improve the Human Condition Have FailedI’m going to do it anyway—because the book is that good, but also because Scott’s framework suggests two important predictions about where China’s process of managed urbanization is headed.

For a quick synopsis of Scott’s masterwork, I’ll turn to a 1998 review of it by Cass Sunstein. Sunstein describes Scott’s book as a study of social engineering, or “selective interventions into complex systems,” and the moral of the story is that these interventions rarely end well.

Scott does not deny that some designs are well-motivated, and he acknowledges that plans can sometimes do a lot of good. He is concerned to show that when a government, with its “thin simplifications” of complicated systems, fails to understand how human beings organize (and disorganize) themselves, its plans are doomed from the start. Scott calls some governments practitioners of “high modernism,” a recipe for many natural and social disasters, including tyranny… Left to itself, this ideology is overconfident but benign. [High modernism] becomes authoritarian when it is conjoined to “an authoritarian state that is willing and able to use the full weight of its coercive power to bring these high-modernist designs into being.” This is especially dangerous when it is linked to “a prostrate civil society that lacks the capacity to resist these plans.” Thus the greatest calamities in Scott’s book involve a weak society that cannot adapt to a government’s plans.

The intellectual core of Scott’s book is a theory of incremental state-building, but its moral core is a set of observations about cases where high modernist ideology and authoritarian states have come together to produce especially disastrous social outcomes.

So what is this ideology? As Scott explains (pp. 89-90), high modernism

is best conceived as a strong (one might even say muscle-bound) version of the beliefs in scientific and technical progress that were associated with industrialization in Western Europe and in North America from roughly 1830 until World War I. At its center was a supreme self-confidence about continued linear progress, the development of scientific and technical knowledge, the expansion of production, the rational design of social order, the growing satisfaction of human needs, and, not least, an increasing control over nature (including human nature) commensurate with scientific understanding of natural laws. High modernism is thus a particularly sweeping vision of how the benefits of technical and scientific progress might be applied—usually through the state—in every field of human activity… The high-modernist state began with extensive prescriptions for a new society, and it intended to impose them.

High modernism was on full display in many of the USSR’s grand developmental schemes, from the agricultural collectivization drives that killed millions to the massive river diversion project that was finally abandoned in 1986. High modernism has also afflicted Western state-building efforts in Afghanistan (here), and those efforts have often foundered in the very ways that Scott’s book anticipates (here).

China’s sweeping plans for controlled urbanization strike me as high modernism par excellence. This scheme is arguably the twenty-first century version of agricultural collectivization—the kind of “revolution from above” that Stalin promised, only now the goal is to put people into cities instead of farms, and to harness market forces instead of refuting them. “We are here on the path to modernity,” the thinking seems to go, “and we want to be there. We are a smart and powerful state, so we will meticulously plan this transformation, and then use our might to induce it.”

If Scott is right about these “certain schemes,” though, then two things are liable to happen. First, China’s new plan for managed urbanization will probably fail on its own terms. It will fail because human planners don’t really understand how these processes work, and even if those planners did understand, they still couldn’t control them. This prediction doesn’t imply that China won’t continue to urbanize, or even that city-dwellers’ quality of life won’t continue to improve on average. It just means that those trends will continue in spite of these grand plans instead of because of them. If the American experience in Afghanistan—or, heck, in its own urban centers—is any guide, we should expect many of the housing developments, schools, and transportation infrastructure born of this plan to go underused and eventually to decay. Or, as an economist might put it, the return on investment will probably be poor.

The second prediction of sorts I take from Scott’s book is that the Chinese Communist Party’s plans for socially engineered urbanization will probably produce a lot of conflict and suffering on their way to failure. The capacity of Chinese civil society to resist these schemes is not great, but it also varies a great deal across issues and locales and appears to be strengthening. We see hints of this resistance and its coming intensification in Johnson’s story:

Separately, state television reported on Sunday night that 4.75 million people living in shantytowns would have their housing improved this year. These areas are often villages that have been swallowed up by cities, and at times have been flashpoints of violence between municipal officials who want to demolish them and residents unwilling to move. It is unclear whether the plan will significantly raise relocation compensation for the residents of these areas.

Now, I can think of at least two ways these predictions might not come true. First, the CPC might not really try to implement this plan, or it might abandon the plan if and when conflict arises. I have a hard time imagining that outcome, though, precisely because the Party has now become so publicly invested in high modernist ideology. The Party’s claim to public authority is now lashed to the idea of it as a benevolent and capable modernizer, so any obvious slackening of that commitment would open the door to conflict over what or who should replace it.

Second, these predictions might not come true because the Chinese Communist Party might succeed where all others have failed. So, has the Chinese Communist Party cracked the code on “how human beings organize (and disorganize) themselves”, as Sunstein put it? And has it married that never-before-achieved understanding with an unprecedented capacity for design and implementation? If you don’t say yes to both of those questions, it’s hard to see how this scheme manages to pull off what no other comparable scheme before it has done.

Whither Organized Violence?

The Human Security Research Group has just published the latest in its series of now-annual reports on “trends in organized violence around the world,” and it’s essential reading for anyone deeply interested in armed conflict and other forms of political violence. You can find the PDF here.

The 2013 edition takes Steven Pinker’s Better Angels as its muse and largely concurs with Pinker’s conclusions. I’ll sheepishly admit that I haven’t read Pinker’s book (yet), so I’m not going to engage directly in that debate. Instead, I’ll call attention to what the report’s authors infer from their research about future trends in political violence. Here’s how that bit starts, on p. 18:

The most encouraging data from the modern era come from the post–World War II years. This period includes the dramatic decline in the number and deadliness of international wars since the end of World War II and the reversal of the decades-long increase in civil war numbers that followed the end of the Cold War in the early 1990s.

What are the chances that these positive changes will be sustained? No one really knows. There are too many future unknowns to make predictions with any degree of confidence.

On that point, political scientist Bear Braumoeller would agree. In an interview last year for Popular Science (here), Kelsey Atherton asked Braumoeller about Braumoeller’s assertion in a recent paper (here) that it will take 150 years to know if the downward trend in warfare that Pinker and others have identified is holding. Braumoeller replied:

Some of this literature points to “the long peace” of post-World War II. Obviously we haven’t stopped fighting wars entirely, so what they’re referring to is the absence of really really big wars like World War I and World War II. Those wars would have to be absent for like 70 to 75 more years for us to have confidence that there’s been a change in the baseline rate of really really big wars.

That’s sort of a separate question from how we know whether there are trends in warfare in general. We need to understand that war and peace are both stochastic processes. We need a big enough sample to rule out the historical average, which is about one or two big wars per century. We just haven’t had enough time since World War I and World War II to rule out the possibility that nothing’s changed.

I suspect that the authors of the Human Security Report would not dispute that claim, but after carefully reviewing Pinker’s and their own evidence, they do see causes for cautious optimism. Here I’ll quote at length, because I think it’s important to see the full array of forces taken into consideration to increase our confidence in the validity of the authors’ cautious speculations.

The case for pessimism about the global security future is well rehearsed and has considerable support within the research community. Major sources of concern include the possibility of outbreaks of nuclear terrorism, a massive transnational upsurge of lethal Islamist radicalism, or wars triggered by mass droughts and population movements driven by climate change.

Pinker notes reasons for concern about each of these potential future threats but also skepticism about the more extreme claims of the conflict pessimists. Other possible drivers of global violence include the political crises that could follow the collapse of the international financial system and destabilizing shifts in the global balance of economic and military power—the latter being a major concern of realist scholars worried about the economic and military rise of China.

But focusing exclusively on factors and processes that may increase the risks of large-scale violence around the world, while ignoring those that decrease it, also almost certainly leads to unduly pessimistic conclusions.

In the current era, factors and processes that reduce the risks of violence not only include the enduring impact of the long-term trends identified in Better Angels but also the disappearance of two major drivers of warfare in the post–World War II period—colonialism and the Cold War. Other post–World War II changes that have reduced the risks of war include the entrenchment of the global norm against interstate warfare except in self-defence or with the authority of the UN Security Council; the intensification of economic and financial interdependence that increases the costs and decreases the benefits of cross-border warfare; the spread of stable democracies; and the caution-inducing impact of nuclear weapons on relations between the major powers.

With respect to civil wars, the emergent and still-growing system of global security governance discussed in Chapter 1 has clearly helped reduce the number of intrastate conflicts since the end of the Cold War. And, at what might be called the “structural” level, we have witnessed steady increases in national incomes across the developing world. This is important because one of the strongest findings from econometric research on the causes of war is that the risk of civil wars declines as national incomes—and hence governance and other capacities—increase. Chapter 1 reports on a remarkable recent statistical study by the Peace Research Institute, Oslo (PRIO) that found that if current trends in key structural variables are sustained, the proportion of the world’s countries afflicted by civil wars will halve by 2050.

Such an outcome is far from certain, of course, and for reasons that have yet to be imagined, as well as those canvassed by the conflict pessimists. But, thanks in substantial part to Steven Pinker’s extraordinary research, there are now compelling reasons for believing that the historical decline in violence is both real and remarkably large—and also that the future may well be less violent than the past.

After reading the new Human Security Report, I remain a short-term pessimist and long-term optimist. As I’ve said in a few recent posts (see especially this one), I think we’re currently in the thick of period of systemic instability that will continue to produce mass protests, state collapse, mass killing, and other forms of political instability at higher rates than we’ve seen since the early 1990s for at least the next year or two.

At the same time, I don’t think this local upswing marks a deeper reversal of the long-term trend that Pinker identifies, and that the Human Security Report confirms. Instead, I believe that the global political economy is continuing to evolve in a direction that makes political violence less common and less lethal. This system creep is evident not only in the aforementioned trends in armed violence, but also in concurrent and presumably interconnected trends in democratization, socio-economic development, and global governance. Until we see significant and sustained reversals in most or all of these trends, I will remain optimistic about the directionality of the underlying processes of which these data can give us only glimpses.

How (Not?) To Win the Information War Over Ukraine

In an opinion piece for The Telegraph last Friday (here), writer Anne Applebaum bemoans that Russia is winning the “information war” over the crisis in Ukraine with demonstrable falsehoods.

The crude and shrill nature of the propaganda now being aired on Russian media and especially on Russia Today (RT), the international news channel owned by the Russian state, has surprised me. Until now, the tone has generally been snide and cynical rather than aggressive. With slick, plausible American anchors and some self-styled hip outsiders—Julian Assange had a regular show—it seemed designed to undermine Western arguments, not denounce them. But now it is openly joining an information war being conducted on an unprecedented scale. The bald-faced lie has now become commonplace.

To counter this torrent of lies, Applebaum argues, the U.S. and Europe need to speak more truth louder.

The only response to an all-out information war is an all-out information defence. The West used to be quite good at this: simply by being credible truth-tellers, Radio Free Europe and the BBC language services provided our most effective tools in the struggle against communism. Maybe it’s time to look again at their funding, and to find ways to spread their reach once more.

I’d say that Putin & co. are clearly winning the propaganda war over Ukraine on the domestic front and playing to a draw on the international side. Press freedom is nearly non-existent in Russia (here), and Moscow’s domestic audience skews nationalist anyway (here), so that’s an easy victory. International audiences are more heterogeneous and surely less sympathetic than native ones, but as Applebaum notes, the Russian government doesn’t need to convince everyone that its version of the narrative is true to shape the politics of the response.

Unlike Applebaum, though, I am not confident that her proposed remedy—loud truth-telling—will produce the desired result. In fact, experiments conducted in the past few years by political scientist Brendan Nyhan and several co-authors suggest that, in information wars, frontal assaults sometimes have the opposite of the intended effect. In a 2013 paper entitled “The Hazards of Correcting Myths About Healthcare Reform” (here), the authors describe the results of an experiment “to determine if more aggressive media fact-checking could correct the false belief that the Affordable Care Act would create ‘death panels.’”

Participants from an opt-in Internet panel were randomly assigned to either a control group in which they read an article on Sarah Palin’s claims about “death panels” or an intervention group in which the article also contained corrective information refuting Palin.

Findings: The correction reduced belief in death panels and strong opposition to the reform bill among those who view Palin unfavorably and those who view her favorably but have low political knowledge. However, it backfired among politically knowledgeable Palin supporters, who were more likely to believe in death panels and to strongly oppose reform if they received the correction.

Conclusions: These results underscore the difficulty of reducing misperceptions about health care reform among individuals with the motivation and sophistication to reject corrective information.

Nyhan and his co-authors got similar results in a follow-on study designed “to test the effectiveness of messages designed to reduce vaccine misperceptions and increase vaccination rates” (here). This time,

A Web-based nationally representative 2-wave survey experiment was conducted with 1759 parents age 18 years and older residing in the United States who have children in their household age 17 years or younger (conducted June–July 2011). Parents were randomly assigned to receive 1 of 4 interventions: (1) information explaining the lack of evidence that MMR causes autism from the Centers for Disease Control and Prevention; (2) textual information about the dangers of the diseases prevented by MMR from the Vaccine Information Statement; (3) images of children who have diseases prevented by the MMR vaccine; (4) a dramatic narrative about an infant who almost died of measles from a Centers for Disease Control and Prevention fact sheet; or to a control group.

RESULTS: None of the interventions increased parental intent to vaccinate a future child. Refuting claims of an MMR/autism link successfully reduced misperceptions that vaccines cause autism but nonetheless decreased intent to vaccinate among parents who had the least favorable vaccine attitudes. In addition, images of sick children increased expressed belief in a vaccine/autism link and a dramatic narrative about an infant in danger increased self-reported belief in serious vaccine side effects.

CONCLUSIONS: Current public health communications about vaccines may not be effective. For some parents, they may actually increase misperceptions or reduce vaccination intention. Attempts to increase concerns about communicable diseases or correct false claims about vaccines may be especially likely to be counterproductive.

I see the results of those studies and imagine Russian and other audiences already ambivalent or hostile toward the U.S. as the functional equivalent of those Palin supporters and vaccine skeptics. It’s counter-intuitive and frustrating to admit, but facts don’t automatically defeat falsehoods, and attempts to beat the latter with the former can even encourage some antagonists to dig their heels in deeper. Before the U.S. and Europe crank up the volume on their own propaganda, they should think carefully about the results of these studies.

States Aren’t the Only Mass Killers

We tend to think of mass killing as something that states do, but states do not have a monopoly on this use of force. Many groups employ violence in an attempt to further their political and economic agendas; civilians often suffer the consequences of that violence, and sometimes that suffering reaches breathtaking scale.

This point occurred to me again as I thought about the stunning acts of mass violence that Boko Haram has carried out in northern Nigeria in the past few weeks. The chart below comes from the Council on Foreign Relations’ Nigeria Security Tracker, an online interface for a data set that counts deaths from “violent incidents directed at government property, places of worship, and suicide bombings.” The sharp upward bend at the far right of that red line represents the sudden and brutal end of several hundred lives in the past two months in various towns and villages in a part of the world that surely isn’t as alien to Americans as many of us assume. In Nigeria, too, parents wake up and set about the business of providing for themselves and their families, and many kids toddle off to school to learn and fidget and chatter with friends. Over the past few years, Boko Haram has repeatedly interrupted those daily routines with scores of attacks resulting in thousands of murders.


I suspect the tendency to see mass killing as the purview of states is driven by the extraordinary salience of two archetypal cases—the Holocaust, of course, but also the Rwandan genocide. From those examples, we infer that violence on this scale requires resources, organization, and opportunity on a scale that in “modern” times only states are supposed to possess. The Holocaust took this bureaucratic logic to unique extremes, but many accounts of the Rwandan genocide also emphasize state planning and propaganda as necessary conditions for that episode of mass murder in extremis.

It’s true that resources, organization, and opportunity facilitate mass violence, and that states are much more likely to have them. In some contexts, though, rebel groups and other non-state actors can accumulate enough resources and become well enough organized to kill on a comparable scale. This is especially likely in the same contexts in which states usually perpetrate mass killing, namely, in civil wars. In some wars, rebels manage to establish governance systems of their own, and the apparent logic of the atrocities committed by these quasi-states looks very similar to the logic behind the atrocities perpetrated by their foes: destroy your rival’s base of support, and scare civilians into compliance or complicity.

Rebels don’t need to govern to carry out mass killings, though, a point driven home by groups like the RUF in Sierra Leone, the Seleka and anti-balaka militias in the Central African Republic, and, of course, Boko Haram. Sometimes the states we now expect to protect civilians against such violence are so weak or absent or uncaring that those non-state groups don’t need deep pockets and sprawling organizations to accomplish mass murder. On Boko Haram, CFR’s John Campbell observes that, “Several of the most recent incidents involve government security forces unaccountably not at their posts, allowing Boko Haram freedom of movement. The governor of Borno state publicly said that Boko Haram fighters outgun government forces.” Campbell also notes that those security forces might be shirking their duty because they are poorly paid and equipped, and because they simply fear a group that “has a long tradition of killing any person in the security services that it can.” With a state like that, the resources and organization required to accomplish mass murder are, unfortunately, not so vast. What is required is a degree of ruthlessness that most of us find hard to understand, but that incomprehensibility should not be confused with impossibility.

Acts we conventionally describe as “terrorism” nowadays are also atrocities by another name, and so-called terrorist groups occasionally succeed in their lethal business on an extraordinary scale. Al Qaeda’s attacks on September 11, 2001, certainly qualify as a mass killing as we conventionally define it. Nearly 3,000 noncombatant civilians from a discrete group (Americans) were deliberately killed as part of a wider political conflict, and all in a single day. The torrent of car bombings and other indiscriminate attacks in Iraq in recent months has surely crossed that arbitrary 1,000-death threshold by now, too.

For analytical purposes, it would be useful to have a catalog of episodes in which non-state organizations committed atrocities on such a large scale. That catalog would allow us to try to glean patterns and develop predictive models from their comparison to each other and, more important, to situations in which those episodes did not occur. Even more useful would be a reliable assemblage of data on the incidents comprising those episodes, so we could carefully study how and where they arise and accumulate over time, perhaps with some hope of halting or at least mitigating future episodes as they develop.

Unfortunately, the data we want usually aren’t the data we have, and that’s true here, too. The Uppsala Conflict Data Program (UCDP) has compiled a data set on “one-sided violence,” defined as “intentional attacks on civilians by governments and formally organized armed groups,” that includes low, high, and best estimates of deaths attributed to each perpetrator group in cases where that annual estimate is 25 deaths or more (here). These data are an excellent start, but they only cover years since 1989, so the number of episodes involving non-state groups as perpetrators is still very small. The Armed Conflict Location & Event Data Project (ACLED) compiles detailed data (here) on attacks by non-state groups, among others, but it only covers Africa since 1997. New developments in the automated production of political event data hint at the possibility of analyzing deliberate violence against civilians around the world at a much higher resolution in the not-too-distant future. As I’ve discovered in an ongoing efforts to adapt one of these data sets to this purpose, however, we’re not quite there yet (see here).

In the meantime, we’ll keep seeing accounts of murderous sprees by groups like Boko Haram (here and here, to pick just two) and CAR’s Seleka (here) and anti-balaka (here) alongside the thrum of reporting on atrocities from places like Syria and Sudan. And as we read, we would do well to remember that people, not states, are the the common denominator.

PS. In the discussion of relevant data sets, I somehow forgot to mention that the Political Instability Task Force also funds the continuing collection of data on “atrocities” around the world involving five or more civilian fatalities (here). These data, which run all the way back to January 1995, are carefully compiled under the direction of a master of the craft, but they also suffer from the inevitable problems of reporting bias that plague all such efforts and so must be handled with care (see Will Moore here and here on this subject).

Forecasting Coup-ish Events

This is a guest post written by Matt Reichert, Miguel Garces, Quratul-Ann Malik, and Ian Lustick of Lustick Consulting. Any questions about this post, these models, Lustick Consulting, or agent-based modeling can be directed to info@lustickconsulting.com.

For students of civil-military relations, the ouster of Ukrainian ex-President Viktor Yanukovich  is puzzling—it’s just not quite clear what to call it. Unlike the case of Egypt, where the checklist of coup criteria can be marked down with a single swift sentence, classifying the Ukraine case has required much more thought. With parliament as the chief perpetrator, and without the Ukrainian armed forces or military playing a significant role, this coup feels like a fire without the smoke. It is not so much a coup, as coup-ish.

How do we forecast “coup-ish?” We suggest that the Ukraine case typifies the type of forecasting challenge that is best met by conceptual disaggregation. In his treatment of Ukraine, what Ulfelder has captured is the unique analytic role played by what has been termed “diminished subtypes.” According to David Collier and Steven Levitsky (1997), a diminished subtype appends and subtracts one or more criteria from some root concept, making it related to, but not quite a pure specification of, that concept. It is the root concept “with adjectives.” Their example is ‘illiberal democracy,’ which does not meet the minimal requirements of the root concept ‘democracy,’ and includes additional criteria and a smaller number of cases than the root concept democracy. Yet there is still important analytic utility to defining the concept in connection with democracy.

What happened in Ukraine might be classified not as a pure ‘coup,’ yet for Ulfelder, it is clearly close enough that recognizing it as ‘coup-ish’ in some way is analytically useful. Thus, we propose the diminished subtype ‘parliamentary coup.’

For this to hold, the diminished subtype ‘parliamentary coup’ must satisfy most, break at least one, and add at least one criterion to the existing definitional criteria for the root concept “coup.” To understand how this might be accomplished, it is useful to first re-state the three coup definitions used by Ulfelder – one each from the coding rules for the two datasets used to validate his coup forecasts, and one from Ulfelder’s own commentary on Ukraine – and identify where they are congruent and diverge. Following the example of Powell and Thyne, we break down the definitions into three components.




Powell & Thyne
(Ground truth A)
Chief executive Any elite who is part of the state apparatus Illegal; no minimal death criteria
Marshall & Marshall
(Ground truth B)
Executive authority A dissident/opposition faction within the country’s ruling or political elites A forceful seizure
(Ukraine commentary)
Chief executive Political insiders Do not follow constitutional procedure; involves use or threat of force

There is general agreement here on the target, chief executive, and some agreement on the tactic—it must be extra-legal and in some way forceful. There is disagreement on the perpetrator—whether it must be a state actor, or simply a political insider.

Another way of conceptualizing these definitions and their inter-relationships is that extra-legal seizure of the chief executive office operates as an overarching abstract concept. By specifying the perpetrator, we build the root concept ‘coup:’ an extra-legal seizure of the chief executive office by some state actor. When we adjust the specification of the perpetrator, we get our diminished subtype ‘parliamentary coup:’ an extra-legal seizure of the chief executive office by an unarmed political insider. The diminished subtype shares but also excludes some criteria of the root concept, and both fit comfortably under the over arching abstract concept, as illustrated in the figure below.

coup typology graphic

We included here the subtype ‘military coup’ to distinguish the conventional tactic of specification from the diminished subtype route. While the diminished subtype satisfies some but not all of the criteria of the root concept, the conventional subtype exists entirely within the root concept, and simply drops the level of generality by adding criteria and reducing the number of qualifying cases. Thus, while all military coups are coups, not all coups are military coups – and all parliamentary coups are not quite coups, but still coup-ish.

What does classifying our classification scheme buy us? The added value is more than semantic. How we choose to conceptualize, and being self-aware about those choices, can shape and affect how we analyze, how we model, and also how we forecast. As Giovanni Sartori observed (1970), “concept formation stands prior to quantification.” Here, we will demonstrate how using these different conceptualizations in a model, and taking on the analytic baggage that comes with each, affects not just the forecasts we make, but also the questions we ask about those forecasts.

An agent-based model makes for especially fertile ground for this type of introspective test, for two reasons. First, it is a causal model, which means that the individual chains of implications for each type of concept are available for interrogation. Second, the animating principle behind the agent-based modeling approach is complexity, which incorporates the overlapping chains of causation and nth order effects produced by thousands of individually specified interacting agents. This means that the ultimate effects of specifying a concept one way or another are truly unpredictable at the outset.

Here, we will test the forecasting implications of these alternative conceptualizations of a coup, using the Virtual Strategic Analysis and Forecasting Tool (V-SAFT). V-SAFT, which Lustick Consulting has developed with support from DARPA and ONR, includes a battery of agent-based models representing virtual countries available for interrogation, experimentation, and forecasting. Each model is comprised from (1) a landscape of individual agents heterogeneously characterized to reflect the particular social and political topology of the target country, (2) simple rules of interaction by which agents adopt, discard, and trade politically motivating identities, and (3) broader rules orienting swaths of the landscape toward or away from the political center of power (classified as levels, from ‘dominance’ at the center, to a position of radical opposition at the ‘non-system’ level).

Using our models for Thailand, Pakistan, and Egypt, and keeping with the examples posed above, we operationalized a coup in three ways.

Concept Definitional Criteria Model Operationalization
A Coup
(root concept)
Extra-legal seizure A move from the ‘non-system’ level of the model
Of the chief executive office To a position of political ‘dominance’ in the model
By any state actor By the ‘state’ OR the ‘military’ identity
A Military Coup
Extra-legal seizure A move from the ‘non-system’ level of the model
Of the chief executive office To a position of political ‘dominance’ in the model
By a military actor By the ‘military’ identity exclusively
A Parliamentary Coup
(diminished subtype)
Extra-legal seizure A move from the ‘non-system’ level of the model
Of the chief executive office To a position of political ‘dominance’ in the model
By any unarmed political insider By any ‘political party’ identity, but NOT the state or military identities

Our forecast for each conceptualization of a coup, alongside Ulfelder’s original forecasts, appear in the figure below.

Coup Likelihoods Combined

A first glance tells us that disaggregating the root concept ‘coup’ affects the rank ordering of likelihoods. With the parliamentary coup (diminished subtype) operationalization, the rankings produced by LC’s forecasts match Ulfelder’s. With the simple coup or military coup (subtype) operationalizations, however, Thailand drops in the rankings, behind Pakistan and then Egypt.

Where this kind of concept disaggregation buys us the most explanatory power is where we see forecast divergence. In Ulfelder’s forecast, we see the greatest disagreement between his two models on the forecast for Thailand, indicating a lower level of certainty for forecast accuracy. In LC’s forecasts, we also see the greatest disagreement in Thailand. In other words, how we operationalize and disaggregate the root concept ‘coup’ seems to have the greatest implications in Thailand, compared to Pakistan and Egypt – especially with regard to the diminished subtype. What are the implications in Thailand of such a jump in likelihood for a parliamentary coup, alongside a decrease in the likelihood for a military or traditional coup?

 Attack Likelihood by Coup Type

One of the more puzzling features of the Ukraine case’s coup-ish-ness is its normative implications. If the Ukrainian case is a ‘just coup,’ is that exceptional, or are all parliamentary coups ‘just?’ And would a different type of coup have been unjust? The figure below provides some insight. Here we see the likelihood of subversive anti-state violence—a dependent variable endogenously produced by V-SAFT models—disaggregated by coup type. While the presence of a coup significantly increases the likelihood of anti-state violence, the effect is generally greater for a traditional or military coup. The relationship between coup type and violence is more pronounced in the Thailand case, which is likely due to its larger sample size (as noted in the preceding figure, Thailand generated more parliamentary coups). So, while coups of any kind are rarely peaceful, the diminished subtype that we saw in Ukraine is generally associated with less popular violence. In this way, disaggregating our concept can at times help our analytical thinking inform our moral thinking.

By systematically exploiting theoretically differentiated ABM simulation models, forecasters and analysts can sharpen their self-awareness of concept choice, measure its implications, and ultimately improve the real utility of concept categories for policy makers. To further explore the forecasts produced by V-SAFT’s regular battery of country models (Egypt, Pakistan, Thailand, the Philippines, Bangladesh, Venezuela, Indonesia, Malaysia, and Kenya), please visit this page (registration required) on Lustick Consulting’s web site. For our publications on the application of agent-based modeling approaches to social science, please see this page (no registration required).

The Wisdom of Crowds, Oscars Edition

Forecasts derived from prediction markets did an excellent job predicting last night’s Academy Awards.

PredictWise uses odds from online bookmaker Betfair for its Oscars forecasts, and it nearly ran the table. PredictWise assigned the highest probability to the eventual winner in 21 of 24 awards, and its three “misses” came in less prominent categories (Best Documentary, Best Short, Best Animated Short). Even more telling, its calibration was excellent. The probability assigned to the eventual winner in each category averaged 87 percent, and most winners were correctly identified as nearly sure things.

Inkling Markets also did quite well. This public, play-money prediction market has a lot less liquidity than BetFair, but it still assigned the highest probability of winning to the eventual winner in 17 of the 18 categories is covered—it “missed” on Best Original Song—and for extra credit it correctly identified Gravity as the film that would probably win the most Oscars. Just by eyeballing, it’s clear that Inkling’s calibration wasn’t as good as PredictWise’s, but that’s what we’d expect from a market with a much smaller pool of participants. In any case, you still probably would have one your Oscars pool if you’d relied on it.

This is the umpteen-gajillionth reminder that crowds are powerful forecasters. “When our imperfect judgments are aggregated in the right way,” James Surowiecki wrote (p. xiv), “our collective intelligence is often excellent.” Or, as PredictWise’s David Rothschild said in his live blog last night,

This is another case of pundits and insiders advertising a close event when the proper aggregation of data said it was not. As I noted on Twitter earlier, my acceptance speech is short. I would like to thank prediction markets for efficiently aggregating dispersed and idiosyncratic data.

This Is Not a Drill

Times like these, part of me wishes I studied microbes or aeronautics or modern American fiction.

One of the most significant crises in international relations of the past 20 years is unfolding right now in Ukraine, but it is impossible to talk or write publicly about it without engaging in a political act that can have significant personal and even public consequences. There is no political science in real time, only politics. When analysis overlaps with practice, the former becomes part of the latter. Sometimes the stakes are high, and I’ve found recently that more people are listening that I had anticipated when I started blogging about current events, among other things.

Or, more accurately, I just hadn’t thought that part through. I think I started blogging because I had time to do it, I enjoyed and benefited from the mental exercise, and I hoped it would advance my career. Best I can recall, I did not think much about how it might eventually entangle me in public conversations with significant consequences, and how I would handle those situations if and when they arose.

In case it isn’t obvious, my last post, on Ukraine, is the catalyst for this bout of introspection. That post had ramifications in two spheres.

The first was personal. Shortly after I published it, an acquaintance whose opinion I respect called me out for stating so unequivocally that Yanukovych’s ouster was “just.” His prodding forced me to think more carefully about the issue, and the more I did, the less confident I was in the clarity of that judgment. In retrospect, I think that statement had as much to do with not wanting to be hated by people whose opinions I value as it did with any serious moral reasoning. I knew that some people whose opinions I value would read my calling the ouster a “coup” as a betrayal, and I felt compelled to try to soften that blow by saying that the act was good anyway. That moral argument is there for the making, but I didn’t make it in my post, and to be honest I didn’t even make it clearly in my own head before asserting it.

The other sphere is the political one. I still don’t believe that my opinions carry more than a feather’s weight in the public conversation, if that. Still, this post has forced me to think more carefully about the possibility that it could, and that I won’t control when that happens and what the consequences will be.

Before I wrote the post, I queried two scholars who have studied Ukrainian politics and law and asked them whether or not Yanukovych’s removal from office had followed constitutionally prescribed procedures. Both of them replied, but both also asked me not to make their views public. As one explained in an email I received after I had already published my post, the risk wasn’t in being wrong. Instead, the risk was that publicizing a certain interpretation might abet Russia’s ongoing actions in the region, and that potential political effect was more important to this person than the analytical issues my question covered. Of course, it was impossible for me to read that email and not feel some regret about what I had already written.

One irony here is that lots of political scientists talk about wanting their work to be “policy relevant,” to have policymakers turn to them for understanding on significant issues, but I think many of the scholars who say that don’t fully appreciate this point about the inseparability of analysis and politics (just as I didn’t). Those policymakers aren’t technocratic robots, crunching inputs through smart algorithms in faithful pursuit of the public interest.  When you try to inform their decisions in real time, you step out of the realm of intellectual puzzle-solving and become part of a process of power-wielding. I suppose that’s the point for some, but I’m finding it more unnerving than I’d expected.

If you work in this field and haven’t already done so, I urge you to read Mark Lilla’s The Reckless Mind: Intellectuals in Politics for much deeper consideration of this fraught terrain. I picked up Lilla’s book again this morning and found this passage (p. 211) particularly relevant:

Some tyrannical souls become rulers of cities and nations, and when they do entire peoples are subjugated by the rulers’ erotic madness. But such tyrants are rare and their grip on power is weak. There is another, more common class of tyrannical souls that Socrates considers, those who enter public life not as rulers, but as teachers, orators, poets—what today we would call intellectuals. These men can be dangerous, for they are ‘sunburned’ by ideas. Like Dionysius, this kind of intellectual is passionate about the life of the mind, but unlike the philosopher he cannot master that passion; he dives headlong into political discussion, writing books, giving speeches, offering advice in a frenzy of activity that barely masks his incompetence or irresponsibility. Such men consider themselves to be independent minds, when the truth is that they are a herd driven by their inner demons and thirsty for the approval of a fickle public.

In the 2010s, a lot of oration happens in cyberspace, and a public intellectual is more likely to blog than to give a speech. In other words, scholars who blog about politics in real time must recognize that we are “offering advice,” and must therefore guard against the risk of becoming the “sunburned” intellectuals whose urge to speak drowns out our “incompetence or irresponsibility.”

But what does that mean in practice? Lilla isn’t trying to write a self-help guide for bloggers, but he does go on to say this (p. 212):

The philosopher-king is an ‘ideal,’ not in the modern sense of a legitimate object of thought demanding realization, but what Socrates calls a ‘dream’ that serves to remind us how unlikely it is that the philosophical life and the demands of politics can ever be made to coincide. Reforming a tyranny may not be within our power, but the exercise of intellectual self-control always is. That is why the first responsibility of a philosopher who finds himself surrounded by political and intellectual corruption may be to withdraw.

I do not consider myself a philosopher, but I take his point nonetheless.

Ukraine’s Just Coup

As Ukraine’s newly appointed government confronts a deepening separatist challenge in Crimea, Viktor Yanukovych continues to describe his removal from office as a “coup d’etat” (here). According to a recent poll by a reputable firm, roughly one-quarter of Russians agree. A month earlier, 84 percent of respondents in a similar poll saw the protests against Yanukovich as a coup attempt.

But that’s all spin and propaganda, right? Yanukovych is a friend of Moscow’s, which presumably views his ouster as part of a broader Western plot against it, and state-guided Russian media have been peddling this line from the start of the EuroMaidan protests a few months ago.

Well, pedantically, Yanukovych is correct. Academic definitions of coups d’etat generally include four criteria: 1) they replace the chief executive; 2) they do not follow constitutional procedure; 3) they are led or facilitated by political insiders; and 4) they involve the use or threat of force. Sometimes we attach modifiers to signify which political insiders strike the blow—military, palace, parliamentary, or judicial—and the criterion regarding the use or threat of force is often interpreted broadly to include arrest or even credibly menacing statements. When political outsiders topple a ruler, we call it a successful rebellion, not a coup. When political insiders remove a sitting leader by constitutional means, we call it politics.

Ukraine unambiguously satisfies at least a few of these criteria. The sitting chief executive was removed from office in a vote by parliamentarians, who qualify as political insiders. Those parliamentarians were encouraged by a popular uprising that represents a form of coercion. Even if we assume, as I do, that most participants in that uprising would not have physically harmed Yanukovich had they captured him, their forceful attempts to seize and occupy government buildings and their clashes with state security forces are clearly coercive acts.

And, crucially, the vote to remove Yanukovych doesn’t seem to have followed constitutional procedures. Under Articles 108-112 of Ukraine’s constitution (here), there are four ways a sitting president may leave office between elections: resignation, incapacitation, death, and impeachment. None of the first three happened—early rumors to the contrary, Yanukovych has vehemently denied that he resigned—so that leaves the fourth, impeachment. According to Article 111, impeachment must follow a specific set of procedures: Parliament must vote to impeach and then convene a committee to investigate. That committee must investigate and report back to parliament, which must then vote to bring charges. A final vote to convict may only come after receipt of a judgment from the Constitutional Court that “the acts, of which the President of Ukraine is accused, contain elements of treason or other crime.” Best I can tell, though, those procedures were not followed in this case. Instead, parliament simply voted—380 to 0, in a body with 450 seats—to dismiss Yanukovych and then to hand executive authority on an interim basis to its own speaker (here).

The apparent extra-constitutionality of this process gives us the last of the four criteria listed above. So, technically speaking, Yanukovych’s removal checks all of the boxes for what we would conventionally call a coup. We can quibble about how relevant the threat of force was to this outcome, and thus whether or not the label “parliamentary coup” might fit better than plain old coup, but the basic issue doesn’t seem especially ambiguous.

All of this should sound very familiar to Egyptians. Twice in the past three years, they’ve seen sitting presidents toppled by political insiders while protesters massed nearby. In both instances, the applicability of the “coup” label became a point of intense political debate. People cared, in part, because perceptions affect political outcomes, and what we call an event shapes how people perceive it. We shout over each other until one voice finally drowns out the rest, and what that voice says becomes the history we remember. In a world where “the will of the people” is seen by many as the only legitimate source of state authority, a whiff of illegitimacy hangs about “coup” that doesn’t adhere to “revolution.” In a peculiar twist of logic and semantics, many Egyptians insisted that President Morsi’s removal in July 2013 could not have been a coup because millions of people supported it. The end was right, so the means must have been, too. Coup doesn’t sound right, so it couldn’t have been one of those.

It’s easy to deride that thinking from a distance. It’s even easier with the benefit of a hindsight that can take in all the terrible things Egypt’s ruling junta has done since it seized power last July.

Before we sneer too hard at those gullible Egyptian liberals, though, we might pause to consider how we’re now describing events in Ukraine, and why. Most of the people I know personally or follow on social media believe that Yanukovych was a rotten menace whose removal from office was justified by his corruption and, more recently, his responsibility for the use of disproportionate force against activists massed on the Maidan. I agree, and I’m sure the documents his accomplices dumped in the Dnipro River on the way out of town will only clarify and strengthen that impression. Yanukovych’s election win in 2010 and his continuing popularity among a large (but dwindling) segment of the population weighed in his favor before 19-20 February, but the shooting to death of scores of unarmed or crudely armed protesters undoubtedly qualifies as the sort of crime that should trigger an impeachment and might even win a conviction. That is, those shootings qualify as an impeachable offense, but impeachment is not what happened.

As moral beings, we can recognize all of those things, and we can and should weigh them in our judgments about the justice of what’s transpired in Ukraine in the past week. Moral and analytical thinking aren’t the same thing, however, and they don’t always point in the same direction, or even occur on the same plane. I’d like to believe that, as analytical thinkers, we’re capable of acknowledging the parallels between Yanukovich’s removal from power and the things we usually call coups without presuming that this acknowledgement negates our moral judgment about the righteousness of that turn of events. Those two streams of thought can and should and inevitably will inform each other, but they don’t have to move deterministically together. Let there be such a thing as a just coup, and let this be an instance of it.

PS. For an excellent discussion of the philosophical issues I gloss over in that final declaration, see Zack Beauchamp’s “The Political Theory Behind Egypt’s Coup” (here).

Is the World Boiling Over or Just Getting Back to Normal?

Here’s a plot of observed and “predicted” rates of political instability onset around the world from 1956 to 2012, the most recent year for which I now have data. The dots are the annual rates, and the lines are smoothing curves fitted from those annual rates using local regression (or loess).

  • The observed rates come from the U.S. government-funded Political Instability Task Force (PITF), which identifies political instability through the occurrence of civil war, state collapse, contested state break-up, abrupt declines in democracy, or genocide or politicide. The observed rate is just the number of onsets that occurred that year divided by the number of countries in the world at the time.
  • The “predicted” probabilities come from an approximation of a model the PITF developed to assess risks of instability onset in countries worldwide. That model includes measures of infant mortality, political regime type, state-led communal discrimination, armed conflict in nearby states, and geographic region. (See this 2010 journal article on which I was a co-author for more info.) In the plot, the “predicted” rate (green) is the sum of the predicted probabilities for the year divided by the number of countries with predicted probabilities that year. I put predicted in quotes because these are in-sample estimates and not actual forecasts.
Observed and Predicted Rates of Political Instability Onset Worldwide, 1956-2012

Observed and Predicted Rates of Political Instability Onset Worldwide, 1956-2012

I see a couple of interesting things in that plot.

First, these data suggest that the anomaly we need to work harder to explain isn’t the present but the recent past. As the right-most third of the plot shows, the observed incidence of political instability was unusually low in the 1990s and 2000s. For the previous several decades, the average annual rate of instability onset was about 4 percent. Apart from some big spikes around decolonization and the end of the Cold War, the trend over time was pretty flat. Then came the past 20 years, when the annual rate has hovered around 2 percent, and the peaks have barely reached the Cold War–era average. In the context of the past half-century, then, any upticks we’ve seen in the past few years don’t seem so unusual. To answer the question in this post’s title, it looks like the world isn’t boiling over after all. Instead, it looks more like we’re returning to a state of affairs that was, until recently, normal.

Second, the differences between the observed and “predicted” rates suggest that the recent window of comparative stability can’t be explained by generic trends in the structural factors that best predict instability. If anything, the opposite is true. According to our structural model of instability risk, we should have seen an increase in the rate of these crises in the past 20 years, as more countries moved from dictatorial regimes to various transitional and hybrid forms of government. Instead, we saw the opposite. He or she who can explain why that’s so with a theory that accurately predicts where this trend is now headed deserves a…well, whatever prize political scientists would get if we had our own Fields Medal.

For the latest data on the political instability events PITF tracks, see the Center for Systemic Peace’s data page. For the data and code used to approximate the PITF’s global instability model, see this GitHub repository of mine.


Get every new post delivered to your Inbox.

Join 5,806 other followers

%d bloggers like this: