On Prediction

It is surprisingly hard to find cogent statements about why prediction is important for developing theory in political science. This is primarily a cultural artifact, I think—political science has mostly eschewed prediction for decades, and many colleagues and reviewers remain openly hostile to it, so it’s not something we spend much time writing and talking about—but there’s a hard philosophy-of-science question lurking there, too. It’s the one Karl Popper implies when he argues in “Prediction and Prophecies in the Social Sciences” that:

Long-term prophecies can be derived from scientific conditional predictions only if they apply to systems which can be described as well-isolated, stationary, and recurrent. These systems are very rare in nature; and modern society is not one of them.

In other words, the causal processes social scientists aim to discover could be moving targets, so the theories we develop through our research will typically be bounded and contingent. If we’re not sure a priori how general we expect our theories to be, then how much predictive power should we expect them to have? To which cases is a theory meant to apply, and therefore to predict? Without an answer to that question, we can’t confidently judge how informative our predictive accuracy or errors are, and we can only answer that question with another layer of theory.

Statistical modeling suggests a practical rationale to prefer out-of-sample to in-sample “prediction” as means of validation, namely, overfitting. That’s what we call it when we build models that chase after the idiosyncrasies in the data on which they’re trained and lose sight along the way of the wider regularities we seek to uncover. As a technical matter, though, the problem of overfitting is specific to the method, and as philosophical point it’s another handwave. In particular, it doesn’t deal with the possibility that a theory works wonderfully for the sample from which it’s derived but poorly elsewhere. That proposition must sound bizarre to biologists or physicists, but as Popper and other would argue, it’s not a crazy idea in political science. Do we really think that the causes of war between states are the same now as they were 100 years ago? Of democratization? Human physiology may not evolve that fast, but human society arguably does, at least recently.

After all that hand-wringing, I land on a pragmatic rationale for emphasizing prediction as a means of validation: it works better than the alternatives. Science is a method for deepening our understanding of the world we inhabit. Deepening implies directional movement. For science to move, we have to try to assess the validity of, and adjudicate between, different ideas. From psychology, we know that the sheer plausibility of a story—and at some level, that’s really what all social-science theories are, whichever “language” we use to represent them—is not a reliable guide to its truth. Just because something makes sense or feels right does not mean that it is, and our brains are terrible about (or excellent at) filtering evidence to confirm our expectations.

Under those circumstances, we need some other way to assess whether an explanation is plausible—or, when more than one explanation is available, to determine which version is more plausible than the others. We can try to do that with reference to the evidence from which the explanation was derived, but the result is a tautology: evidence A suggests theory B; we know theory B is correct because it implies evidence A, and A is what we observe.

The alternative that remains is prediction. To determine whether or not our mental models are getting at something “real,” we have to apply them to situations yet unseen and see if the regularities we thought we had uncovered hold. The results will rarely be binary, but they will still provide new information about the usefulness of the model. Crucially, those results can also be compared to ones from competing theories to help us determine which explanation covers more. That’s the engine of accumulation. Under certain conditions, that new information may even reveal something about the scope conditions of the initial theory. When a model routinely predicts well in some kinds of cases but not others, then we have uncovered a new pattern that we can add to the initial construct and then continue to test in the same way.

For reasons to which Popper alludes, I don’t believe these iterations will reveal laws that consistently explain and anticipate political behavior writ large. Still, we have seen this process produce great advances in other fields, so I prefer it to the alternative of not trying. And, absent some version of this process, the theories we construct about politics are epistemologically indistinguishable from fiction. Fiction can be satisfying to write and read and even be “true” in a fashion, but it is not science because, among other things, it does not aspire to accumulation.

I use the word “aspire” in that last sentence advisedly. Directionality does not necessarily imply eventual arrival, or even reliable navigation. I think it’s perfectly reasonable to establish the accumulation of knowledge at the basic point of the endeavor and still understand and experience science as neuroscientist Stuart Firestein describes it in his wonderful book, Ignorance. Firestein opens the book with a proverb—“It is very difficult to find a black cat in a dark room, especially when there is no cat”—and goes on to say that

[Science] is not facts and rules. It’s black cats in dark rooms. As the Princeton mathematician Andrew Wiles describes it: It’s groping and probing and poking, and some bumbling and bungling, and then a switch is discovered, often by accident, and the light is lit, and everyone says, “Oh, wow, so that’s how it looks,” and then it’s off into the next dark room, looking for the next mysterious black feline. If this all sounds depressing, perhaps some bleak Beckett-like scenario of existential endlessness, it’s not. In fact, it’s somehow exhilarating.

From my own experience, I’d say it can be both depressing and exhilarating, but the point still stands.

Leave a comment

12 Comments

  1. Emmanuel Letouz

     /  March 20, 2014

    Great post..

    Emmanuel Letouz + 1 917-207-7473

    On Mar 20, 2014, at 8:40 AM, Dart-Throwing Chimp wrote:

    WordPress.com dartthrowingchimp posted: “It is surprisingly hard to find cogent statements about why prediction is important for testing theory in political science. This is primarily a cultural artifact, I think—political science has mostly eschewed prediction for decades, and many colleagues”

    Reply
  2. Sounds like you’d like to see more Bayesian statistics in political science. Am I right?

    Reply
    • That’s an easy “yes,” but I’m reluctant to put that front and center because my understanding of statistics in general and the Bayesian vs. frequentist distinction in particular is shallow. I guess that puts me in the “folk Bayesian” camp, but again, I’m not going to advocate loudly for something I can’t confidently explain.

      Reply
  3. Really excellent post. I especially appreciate your point about the problem facing social scientists when we aspire toward the cumulation of knowledge, but the data-generating process changes over time; I’m not really sure how to ‘solve’ (or maybe, ‘resolve’) this problem, but sometimes a careful statement about balancing objectives might be the best we can hope for.

    Reply
  4. Political Science is really behind the times – companies with terabytes of data are making accurate predictions about social behavior. Electrical Engineering and Computer science are eating up statistics because of its focus on theory instead of outcome. Political science is waywardly trailing the statisticians and economists. Good luck, political scientists! If anyone is listening to you.

    Reply
  5. On what grounds do you claim that prediction is better than the alternatives? It seems to me as though there are powerful competing views of validation coming from hermeneutic, relational, and historical sociology. These traditions view the role of social science to be that of meaningfully and usefully ordering the data we have so as to come to certain ethically significant conclusions about it, to undermine historical claims made by public figures, and to provide those studying society with a portable set of conceptual devices that imply mechanisms (rather than law-like concurrences) linking cause and effect.

    Indeed, if you are genuinely convinced that social scientists can basically make no longer-term predictions due to the chaotic nature of social systems, then this is a prima facie reason to dismiss prediction entirely, no? You’ve just established it as a mathematical impossibility absent vast advances in computational power, and provided nothing but fiat for choosing it over other options that may yield better theories if we put time into developing and employing them.

    Reply
  6. ‘And, absent some version of this process, the theories we construct about politics are epistemologically indistinguishable from fiction. Fiction can be satisfying to write and read and even be “true” in a fashion, but it is not science because, among other things, it does not aspire to accumulation.’

    First, it shows a certain ignorance of other methods, to suggest that they do not permit progressive theory building within their methodological borders unless they define progress as ‘finding increasing numbers of robust regularities in types of social phenomena that permit future phenomena to be accurately predicted’. Second, it suggests a definite lack of deeper engagement with the meta-methodological turn in the philosophy of science (eg the big three: Kuhn, Lakatos, and Laudan) and its implications for using progressive aspirations as a demarcation criterion for distinguishing science from non-science.

    Reply
  7. To be perfectly honest, I am a little bit surprised to learn from Adam Elkus and you that there is an active anti-prediction crowd in political science. For a long time I assumed (as I feel a lot of hard science folks do) that political scientists simply haven’t figured out how to make predictions and so aren’t thinking about it. However, after a lot of experience with psychology (although none with Poli Sci) I have come to see a good defense for anti-prediction and so I will sound it out as a devil’s advocate.

    You write:

    [W]e have seen [iterated improvement of predictive power] produce great advances in other fields, so I prefer it to the alternative of not trying. And, absent some version of this process, the theories we construct about politics are epistemologically indistinguishable from fiction. Fiction can be satisfying to write and read and even be “true” in a fashion, but it is not science because, among other things, it does not aspire to accumulation.

    I feel like this view promotes several common misconceptions about the sciences. In particular, that prediction is the only important feature (this is a surprisingly common view among some physicists) that really distinguish the ‘success stories’ of science. However, I think there is an equally (if not more so) important feature: the selection of your terms, language, and what you aim to predict. There is no reason to believe that the folk or common-sense important variables are in fact important, in particular in your post you discuss some of them such as:

    Do we really think that the causes of war between states are the same now as they were 100 years ago? Of democratization?

    When you used that language, you implicitly assumed that what was 100 years ago and was called by common-sense war is the same sort of object as what we have now. It could be that the phenomena as unrelated to each other as peas and black holes. We simply assign the same words to them out of a misguided folk-theory. Of course, you are aware of this, and deliberate on it for some time. However, I want to take it further.

    Some sciences are successful at prediction because they have taken the time to explore the space of languages and select a language for themselves that makes it possible to predict questions asked in that language. It then entrenched that language and only after started the accumulation of ‘progress’. Imagine if physicists were concerned with predicting the shape or the color or the saltiness of an individual electron. From a completely folk-theory it seems like potentially reasonable question to ask, but in reality these are pseudo questions that would just lead physics astray into silly theories.

    I don’t think political science is at the stage where it can recognize pseudo-questions from ‘real’ questions, yet. I think it is still searching for the terms and relevant variables. A way to ask questions that are guaranteed or likely to have answers. When a science is in this early stage, it is better to try to avoid accumulation to avoid climbing a tree to get to the moon. It is better to explore what ideas we can freely until we accidentally stumble on a useful language.

    From this point, the practice of simply explaining existing data parsimoniously and appealing is a good procedure. It allows us to play with what one explains and what one focuses on more so than the pressure that would be there if you had to make policy-relevant predictions. It allows you to escape folk-theory more easily because there isn’t a policy-maker standing over your shoulder demanding to know: “how salty do electrons taste in the West Kongo?”

    Of course, as you noted with the appeal to psychology, there is also a danger with story telling. By judging stories on their ‘elegance’ or ‘compellingness’, we are actually just exploring our own minds and what sort of theories appeal to people, and thus potentially being lead astray into other unproductive folk theories. However, I think this selective pressure is much lower than the pressure that would be present from a demand for prediction. This will hopefully allow one to avoid being stuck at local optima for too long, and let us find the Apollo program instead of eyeing the moon from the top branch of a big bush.

    In the end, I guess I am advocating as Feyerabend does for a plurality of approaches and no restrictions on method or demands for accumulations. At least not until we have reasonable evidence to believe that accumulation will lead us somewhere that we want to be.

    Reply
  1. Demography, Democracy, and Complexity | Dart-Throwing Chimp
  2. Cross-validation in finance, psychology, and political science | Theory, Evolution, and Games Group
  3. Forecasting Round-Up No. 6 | Dart-Throwing Chimp

Leave a Comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: