Last April, the American Economic Review published a fascinating paper by some prominent economists that’s only just come onto my radar screen, thanks to a recent Facebook post from Cullen Hendrix. Here’s the abstract:
We provide evidence that increased political influence, arising from CIA interventions during the Cold War, was used to create a larger foreign market for American products. Following CIA interventions, imports from the US increased dramatically, while total exports to the US were unaffected. The surge in imports was concentrated in industries in which the US had a comparative disadvantage, not a comparative advantage. Our analysis is able to rule out decreased trade costs, changing political ideology, and an increase in US loans and grants as alternative explanations. We provide evidence that the increased imports arose through direct purchases of American products by foreign governments.
That’s quite a result—not because we didn’t think stuff like this happened, but because that “stuff” is rarely discussed in academic work on international political economy and even more rarely makes it into our statistical models. We know that powerful states do lots of things to try to shape the world around them, but those things are often hard to observe and even harder to measure, so we usually just leave those “treatments” out of our statistical abstractions of international and domestic politics and hope for the best. (Hello, omitted variable bias!)
But I digress. Where Berger & co. were primarily interested in the economic effects of those interventions, their paper got me thinking about the political ones. Might the episodes of Cold War meddling the authors identify in their data help explain where and when coup attempts happened, too? Theory says “probably,” but not necessarily in a simple way. As Hein Goemans and Nikolay Marinov have argued (here), during the Cold War
The United States and the former colonial powers in Europe had an ambiguous attitude toward coup plots: sometimes helping, sometimes thwarting, and sometimes doing nothing… Because the world was thought to be a chessboard of West vs. East, attitudes toward both the seizure of power by the military and about whether to pressure for elections varied by which side of the ideological conflict the relevant actors took.
Well, was there a clear pattern? Berger & co. have posted extensive replication files (here), and I’ve already got data and scripts to estimate and compare statistical models of coup risk (here), so let’s have a look, starting with a few words on the measures of foreign meddling.
What Berger & co. have compiled are annual, binary indicators of CIA and KGB influence in domestic politics for all countries of the world except the USA and USSR during the period 1947-1990. By “influence,” they mean “periods in which a leader is installed or supported” by one or the other agency. For example, they tag Chile with a 1 (yes) in the column for U.S. influence from 1964, when the CIA supported Eduardo Frei’s successful election campaign and then went on to back various right-wing groups, until 1970, when the presidency passed to Salvador Allende, of whom the U.S. government was not so fond.
Instead of trying to assess the importance of these influence measures by estimating a model and looking at coefficients and p-values, I took some advice from Mike Ward & co. (here) and asked if these variables could help us predict coup attempts. If these episodes of CIA and KGB meddling had much effect on coup risk, then their addition to a base model of coup risk should noticeably improve that model’s predictive power.
To see if they do, I used a 10-fold cross-validation process to get out-of-sample estimates of coup risk from models without and with Berger & co.’s measures of CIA and KGB influence, then compared the accuracy of those two sets of estimates. The base model is the same one I used in my coup forecasts for 2014, and it covers a lot of ground, from national wealth and colonial legacies to political regime type and recent coup activity. Country-years are the units of observation, and the dependent variable in all cases is a binary indicator of whether or not any coup attempts (successful or failed) occurred during that calendar year. Except for the marker for election years, all covariates—including the measures of U.S. and Soviet influence—were lagged one year.
Here’s a plot of the ROC curves for out-of-sample estimates of coup risk from models without (black) and with (red) those intervention indicators. The numbers reported in the bottom right-hand corner of the plot are the areas under those curves (AUC), and bigger is better. As you can see, the addition of these lagged indicators of periods of CIA and KGB influence has no real effect on the model’s predictive power, suggesting that these variables don’t help explain the location and timing of coups, at least not in the context of this model.
After seeing those results, I wondered if the effects of U.S. and Soviet influence on coup risk might depend on some of the other things in my model, like political regime type or the occurrence of elections. To allow for contingent effects without specifying exactly what those contingencies might be, I re-ran the analysis using Random Forests instead of logistic regression. The plot below shows the results. Again, nothing doing.
It would be easy to look at those plots and conclude that CIA and KGB meddling didn’t play much of a role in coups during the Cold War after all. It’d be easy, but it’d also be wrong.
One of the great things about the paper that set off this whole exercise is that the authors extensively document their data, to include short descriptions of the forms of U.S. and Soviet influence they observe and the sources from which that information came. When we take a closer look at those descriptions (“Summary_of_Interventions.pdf”), it becomes clearer that the source of my null result isn’t the fact that the CIA and KGB weren’t in the business of backing or thwarting coups. Instead, the issue is that the effects often flowed in the opposite direction. That is, coups and other forms of meddling in the selection of national leaders were often what started these episodes of influence in the first place. Influence doesn’t cause coups; coups cause influence.
Once we realize this, it’s less surprising to discover that coups were actually somewhat less likely during periods of CIA or KGB influence than not. When I estimate a logistic regression model using all of the available data (1960-1990), the coefficients on the lagged indicators of CIA and KGB influence are both negative and not tiny: -0.3 and -1.0, respectively, with standard errors of 0.2 and 0.5. If you think those agencies might have helped protect their clients from domestic rivals, that result makes sense.
Ultimately, there are two lessons here, one substantive and one methodological. On the substantive side, this exercise reaffirms the knowledge that the great powers of the Cold War era had significant sway over politics within many other states, including the selection and survival of their leaders. Of course, we didn’t need a statistical analysis to tell us that; we only needed to review the evidence Berger & co. have compiled on the way to their finding about how those efforts affected international trade.
On the methodological side, the counter-intuitive findings from the analysis described in this blog post are a useful reminder that we shouldn’t stop thinking when we hit Enter on our statistical estimations. When interpreting results like these, we have to think carefully about what is and isn’t being measured. The models we can specify with data and methods at hand don’t always match the ideas in our heads, and it’s on us to keep the two straight.