On Prediction

It is surprisingly hard to find cogent statements about why prediction is important for developing theory in political science. This is primarily a cultural artifact, I think—political science has mostly eschewed prediction for decades, and many colleagues and reviewers remain openly hostile to it, so it’s not something we spend much time writing and talking about—but there’s a hard philosophy-of-science question lurking there, too. It’s the one Karl Popper implies when he argues in “Prediction and Prophecies in the Social Sciences” that:

Long-term prophecies can be derived from scientific conditional predictions only if they apply to systems which can be described as well-isolated, stationary, and recurrent. These systems are very rare in nature; and modern society is not one of them.

In other words, the causal processes social scientists aim to discover could be moving targets, so the theories we develop through our research will typically be bounded and contingent. If we’re not sure a priori how general we expect our theories to be, then how much predictive power should we expect them to have? To which cases is a theory meant to apply, and therefore to predict? Without an answer to that question, we can’t confidently judge how informative our predictive accuracy or errors are, and we can only answer that question with another layer of theory.

Statistical modeling suggests a practical rationale to prefer out-of-sample to in-sample “prediction” as means of validation, namely, overfitting. That’s what we call it when we build models that chase after the idiosyncrasies in the data on which they’re trained and lose sight along the way of the wider regularities we seek to uncover. As a technical matter, though, the problem of overfitting is specific to the method, and as philosophical point it’s another handwave. In particular, it doesn’t deal with the possibility that a theory works wonderfully for the sample from which it’s derived but poorly elsewhere. That proposition must sound bizarre to biologists or physicists, but as Popper and other would argue, it’s not a crazy idea in political science. Do we really think that the causes of war between states are the same now as they were 100 years ago? Of democratization? Human physiology may not evolve that fast, but human society arguably does, at least recently.

After all that hand-wringing, I land on a pragmatic rationale for emphasizing prediction as a means of validation: it works better than the alternatives. Science is a method for deepening our understanding of the world we inhabit. Deepening implies directional movement. For science to move, we have to try to assess the validity of, and adjudicate between, different ideas. From psychology, we know that the sheer plausibility of a story—and at some level, that’s really what all social-science theories are, whichever “language” we use to represent them—is not a reliable guide to its truth. Just because something makes sense or feels right does not mean that it is, and our brains are terrible about (or excellent at) filtering evidence to confirm our expectations.

Under those circumstances, we need some other way to assess whether an explanation is plausible—or, when more than one explanation is available, to determine which version is more plausible than the others. We can try to do that with reference to the evidence from which the explanation was derived, but the result is a tautology: evidence A suggests theory B; we know theory B is correct because it implies evidence A, and A is what we observe.

The alternative that remains is prediction. To determine whether or not our mental models are getting at something “real,” we have to apply them to situations yet unseen and see if the regularities we thought we had uncovered hold. The results will rarely be binary, but they will still provide new information about the usefulness of the model. Crucially, those results can also be compared to ones from competing theories to help us determine which explanation covers more. That’s the engine of accumulation. Under certain conditions, that new information may even reveal something about the scope conditions of the initial theory. When a model routinely predicts well in some kinds of cases but not others, then we have uncovered a new pattern that we can add to the initial construct and then continue to test in the same way.

For reasons to which Popper alludes, I don’t believe these iterations will reveal laws that consistently explain and anticipate political behavior writ large. Still, we have seen this process produce great advances in other fields, so I prefer it to the alternative of not trying. And, absent some version of this process, the theories we construct about politics are epistemologically indistinguishable from fiction. Fiction can be satisfying to write and read and even be “true” in a fashion, but it is not science because, among other things, it does not aspire to accumulation.

I use the word “aspire” in that last sentence advisedly. Directionality does not necessarily imply eventual arrival, or even reliable navigation. I think it’s perfectly reasonable to establish the accumulation of knowledge at the basic point of the endeavor and still understand and experience science as neuroscientist Stuart Firestein describes it in his wonderful book, Ignorance. Firestein opens the book with a proverb—“It is very difficult to find a black cat in a dark room, especially when there is no cat”—and goes on to say that

[Science] is not facts and rules. It’s black cats in dark rooms. As the Princeton mathematician Andrew Wiles describes it: It’s groping and probing and poking, and some bumbling and bungling, and then a switch is discovered, often by accident, and the light is lit, and everyone says, “Oh, wow, so that’s how it looks,” and then it’s off into the next dark room, looking for the next mysterious black feline. If this all sounds depressing, perhaps some bleak Beckett-like scenario of existential endlessness, it’s not. In fact, it’s somehow exhilarating.

From my own experience, I’d say it can be both depressing and exhilarating, but the point still stands.

Posted by Jay Ulfelder on March 20, 2014

https://dartthrowingchimp.wordpress.com/2014/03/20/on-prediction/