My sometimes-colleague Michael Horowitz wrote a great piece for War on the Rocks last week on what “policy relevance” means for political scientists who study international affairs, and the different forms that relevance can take. Among the dimensions of policy relevance he calls out is the idea of “policy actionability”:
Policy actionability refers to a recommendation that is possible to implement for the target of the recommendation. Most academic work is not policy actionable, fundamentally. For example, implications from international relations research are things such as whether countries with high male-to-female ratios are more likely to start military conflicts or that countries that acquire nuclear weapons become harder to coerce.
As Michael notes, most scholarship isn’t “actionable” in this way, and isn’t meant to be. In my experience, though, there is plenty of demand in Washington and elsewhere for policy-actionable research on international affairs, and there is a subset of scholars who, in pursuit of relevance, do try to extract policy prescriptions from their studies.
As an empiricist, I welcome both of those things—in principle. Unfortunately, the recommendations that scholars offer rarely follow directly from their research. Instead, they almost always require some additional, often-heroic assumptions, and those additional assumptions render the whole endeavor deeply problematic. For example, Michael observes that most statistical studies identify average effects—other things being equal, a unit change in x is associated with some amount of change in y—and points out that the effects in any particular case will still be highly uncertain.
That’s true for a lot of what we study, but it’s only the half of it. Even more significant, I think, are the following three assumptions, which implicitly underpin the “policy implications” sections in a lot of the work on international affairs that tries to convert comparative analysis (statistical or not) into policy recommendations:
- Attempts to induce a change in x in the prescribed direction will actually produce the desired change in x;
- Attempts to induce a change in x in the prescribed direction will not produce significant and negative unintended consequences; and
- If it does occur, a change in y induced by the policy actor to whom the scholar is making recommendations will have the same effect as previous changes in y that occurred for various other reasons.
The last assumption isn’t so problematic when the study in question looked specifically at policy actions by that same policy actor, but that’s almost never the case in international relations and other fields using observational data to study macro-political behavior. Instead, we’re more likely to have a study that looked at something like GDP growth rates, female literacy, or the density of “civil society” organizations that the policy audience does not control and does not know how to control. Under these circumstances, all three of those assumptions must hold for the research to be neatly “actionable,” and I bet most social scientists will tell you that at least one and probably two or three of them usually don’t.
With so much uncertainty and so much at stake, I wind up thinking that, unless their research designs have carefully addressed these assumptions, scholars—in their roles as scientists, not as citizens or advocates—should avoid that last mile and leave it to the elected officials and bureaucrats hired for that purpose. That’s hard to do when we care about the policies involved and get asked to offer “expert” advice, but “I don’t know” or “That’s not my area of expertise” will almost always be a more honest answer in these situations.
Dart-Throwing Chimp 
Zach Jones
/ June 21, 2015I’d add to our list of pitfalls that the x measured in the study is not always (often? ever?) the same as the policy manipulation that could be undertaken.
I agree of course. IR scholars should be very reticent about giving policy advice since the epistemic status of their claims is not as different as they might like to think. I think the real impetus to give policy advice is the unshakable feeling that however poorly IR scholars understand IR, they understand it better than decisionmakers. I am not so sure that is obviously true.
dartthrowingchimp
/ June 21, 2015What you say in your first sentence is what I was trying to get at in my penultimate paragraph, and it’s really why the first two of those assumptions I list are so unlikely, or at least uncertain. Often, the question of how policy-makers might try to move x is or could be a whole other field in its own right, and the answer is almost never a strictly and straight-forward technical one.
Rex Brynen
/ June 21, 2015As an academic who has worked on the policy end, I would add a couple of additional observations. First, in my experience, diplomatic policymakers rarely ask academics for input in the hope that their advice holds a research-based answer to a tricky policy challenge, but rather because it offers another perspective on issues that policymakers are already grappling with. Some senior policymakers may also enjoy hearing alternative views to those produced within the institutional machinery of government so as to reduce problems of bureaucratic capture. Second, academics are only partly called upon because they have causal inferences to offer up to inform policy, but even more so because they know lots of often quite empirical, qualitative “stuff”–sometimes more than rotational foreign service officers who have only been on the file a year or two (and not ten or twenty).
dartthrowingchimp
/ June 21, 2015Great points, thanks, Rex.
Total
/ July 1, 2015scholars—in their roles as scientists, not as citizens or advocates—should avoid that last mile and leave it to the elected officials and bureaucrats hired for that purpose
Thanks! That was useful. I was kind of hoping that the millions of dollars of grant money and the thousands of person-hours of research work would buy more than a “Nope, sorry, can’t help” but okay. Have a nice day.
(ignores all social science research for evermore)
(cuts political science funding)
(accidentally starts nuclear war with China)
dartthrowingchimp
/ July 2, 2015Heh.
Seriously, though: I am not arguing that social science is irrelevant to policy, or that social scientists should not involve themselves in policy-making. Instead, I’m arguing that, with most of the research we do, the policy implications do not follow directly from the science, because the research does not directly study policy. Under those circumstances, I believe it’s irresponsible to cloak policy recommendations in the mantle of scientific authority. I would prefer to see scholars inform policy-makers and other “doers” about their research and then leave it to the latter to figure out — sometimes in conversation with those scholars — how that understanding should shape their work. And when scholars wish to involve themselves directly in policy-making or advocacy, they should do so as well-informed citizens without the pretense that their proposals are somehow more scientific. Maybe that’s a subtlety that doesn’t really matter in practice, but I think it matters in principle.
Total
/ July 2, 2015I would prefer to see scholars inform policy-makers and other “doers” about their research and then leave it to the latter to figure out — sometimes in conversation with those scholars — how that understanding should shape their work
The problem with that — applause-worthy as it is — is I think that policymakers simply aren’t going to pay attention. They’re not going to read the academic journals with the research or the working papers and then interpolate how results might apply. They’re simply going to ignore the research and go on their merry way. They’re not going to fill in the gap between empirical research and policy recommendations that you identify; they’re either going to go their own way, or listen to people who are willing to make seemingly scientific recommendations.
alixrgreen
/ July 6, 2015Reblogged this on thehistoricalimperative and commented:
Jay Ulfelder’s recent ‘skeptical note’ on the ‘actionability’ of political science research makes some essential points about the problematic assumptions underpinning policy recommendations. In Britain, the Blairite manifesto pitch ‘what counts is what works’ subdues the complexities of research method that might, at best, conclude ‘what works here’ (with further caveats about target population and other central aspects of the design).
I’m not sure, however, that scholars of any discipline should therefore refrain from proposing recommendations or, even more cautiously, withdraw from offering expert advice.
One of the important problems Ulfelder identifies is the uncertainties that are involved in the space between research and policy. How can a scholar answer the ‘so what?’ question that follows from any finding?
There are two issues that we can unpack here. The first is the inevitability, indeed, the necessity, of uncertainty. Policy is messy, unstable and contested because it involves human beings and their beliefs, habits, commitments, decisions and relationships – in the exercise of power, the exertion of influence, in policy implementation and debate. Instead of searching for the definitive research design to address all the assumptions about the transferability of findings – or indeed, just leaving it to ‘elected officials and bureaucrats’ to do the interpretation – we should be bringing together different disciplines with complementary insights. Given the uncertainties of anything involving human beings, the humanities need to be in there too, rather than ignored as irrelevant, if not ornamental.
The other issue is the ‘so what?’ question. I agree it’s hard for scholars to come up with policy recommendations, but that’s at least in part due to their lack of experience of policymaking in practice. In the UK, there is far less interchange between higher education and government than in the USA and the academic career is still pretty intolerant of periods spent in other settings, something that needs to change. Taking a look over the fence and trying to prescribe policy interventions based on research designed for academic purposes seems foolish at best, if not rather arrogant as well as misguided. Humanities scholars may be largely ignored, but we can often be too concerned to preserve our integrity by not allowing policy concerns to ‘sully’ our work.
This seems a rather self-defeating formula. Policymakers don’t get access to an ecosystem of expertise. Scholars remain on the far side of the fence lamenting the intellectual illiteracy of political rhetoric and decision-making.
But does it need to be this way? I don’t think so. But there’s no easy policy prescription for fixing it, as it involves major shifts in perspective among scholars – towards actively looking for cross-disciplinary approaches – and in policy communities that often have limited conceptions of ‘relevant’ evidence.
Being honest in response to a request for expert advice is not just about admitting the limits of one’s own expertise but also the limits of one’s own discipline. It’s the mix that matters, but it takes confidence for the expert to admit it.