Can we see genocides and other mass atrocities coming? If so, how, and how far in advance? And would public dissemination of those forecasts help policy-makers, advocates, and affected societies prevent those atrocities from occurring?
In October 2011, the U.S. Holocaust Memorial Museum (USHMM) convened a group of advocates and academics for a one-day seminar to ruminate on these questions. These are big and difficult problems, and the event really had a more practical goal at its heart: to help the Museum and other civil-society groups assess the potential for, and value of, a new public early-warning system focused on genocide and other mass atrocities.
Based on that conversation and the recommendations of USHMM Fellow and Dartmouth professor Ben Valentino, the Museum decided that the need and opportunity were sufficient to start considering what such a system might look like and how to build it. In March 2012, the Museum hired me for an eight-month consulting project, to finish in October, that’s meant to push this process forward.
My project has two main parts. First and most important, I’ve been asked to write a prospectus detailing the elements and funding this program would require. Second, I’ve been asked to build a statistical tool that could produce one set of forecasts for this program, if it gets built. Under Ben’s proposal, a second set of forecasts would come from some form of expert survey, and the two could be compared and combined to useful effect.
As I get deeper into the project, I expect to blog occasionally about what I’m working on and where I could use some help. I’ve already had very helpful exchanges with numerous people engaged in related projects, including former Political Instability Task Force colleagues Ted Gurr and Barbara Harff, who produces her own global genocide risk list each year, and Sentinel Project founder Christopher Tuckwood. I’m also slated to present results from a preliminary version of my statistical analysis at NYU’s Northeast Methods Program (NEMP) in early May, and my work will surely benefit from the constructive criticism that esteemed audience can provide.
In the meantime, I wanted to spread the word about the Museum’s interest in this endeavor and invite your reactions and suggestions. If you know of any relevant research or advocacy projects or might be interested in supporting this work in some fashion, please post a comment or drop me a line at ulfelder <at> gmail <dot> com.
Dart-Throwing Chimp 
Rex Brynen
/ April 7, 2012My sense is that work in this area too often focuses on developing methodology and warning indicators at the expense of addressing how such tools might be used (or, as is more often the case, not be used) within policy communities. Having read hundreds of country-specific IC and diplomatic assessments of fragile and conflict-affected states, I’m not sure I can think of a single one that made use of the existing quantitative or mixed-methods early warning indicators.
There are many reasons for this. Some related to the generality of the forecasts. Some relate to the methodological biases of intelligence analysts and foreign service officers. Some have to do with the way early warning projects are “marketed” to the policy community. Bureaucratic politics and standard operating procedures play a role too. (This is al leaving aside, of course, the key issue of political will to do anything about potential mass human rights violations.) Many analysts would likely argue that the error rates (false positives, false negatives, and overall calibration) of most early warning indicators are inferior to existing methods–although given the relative lack of data on the latter in the public domain, its hard to judge.
I think the work is valuable, contributing to the sort of methodological triangulation that is characteristic of the best intelligence assessment and policy analysis. However, my point is that at least as much intellectual attention needs to be devoted to take-up by the target communities as to actually producing the model.
dartthrowingchimp
/ April 7, 2012Thanks for your comment, Rex. Do you think that a lack of confidence in the reliability of the warnings plays any role in this? In other words, do you think that improved accuracy would make those target communities any more likely to act? If so, how reliable do you think the warnings would need to be to get there?
Rex Brynen
/ April 7, 2012Partly I think it is a reliability issue. The current largely qualitative assessment process is c85% accurate at medium term/6 month prediction in general (although that number is misleadingly high for various methodological reasons, and I would suspect that accuracy in predicting mass atrocity is significantly lower still.)
More to the point, however, I think it is a bureaucratic mistake to market the outputs of quantitative projects as “warning indicators.” Instead, they should be seen as suggestive of “perhaps you should take a second look” or “what assumptions are we making that we might want to reconsider?” The latter framing, even if only subtly different, is somewhat less threatening to qualitatively-inclined analysts and diplomats. It also has the added advantage of spurring us to examine why it is a quantitative predictor might be wrong or why analyst might disagree with it: Is there something about the configuration of political power, or the issues in dispute that makes mass atrocity less likely? Any input that causes analysts to periodically reexamine assumptions is a useful one, I think.
My broader point here is that early warning indicator projects are most likely to be useful if they engage the client community (ideally, at the manager and analyst/desk officer level, not the rarefied upper levels) from the outset, and are then designed in (both presentation and contents) a way that maximizes the prospects for take-up.
dartthrowingchimp
/ April 7, 2012Thanks again, that’s very helpful.
Jaroslav Petrik (@jerri81)
/ April 8, 2012Jay, I know you’re planning to use formal models, large-n stats, and all that fancy stuff, but isn’t the Museum basically trying to build another ICG? It struck me that you didn’t include them in your list of prime collaborators among PITF, Sentinel etc., although they’re doing quite a good job incl. on the issues Rex brought up. I know they’re taking a completely different approach (qualitative, context-specific and more focused on conflict in general than mass atrocities in particular), but I believe they now have a decade of experience of what to do with the early warning signals and how to pitch them to people who matter.
dartthrowingchimp
/ April 9, 2012Without claiming to speak for the Museum, which of course I don’t, let me try to put myself in the shoes of someone deciding to pursue this project to answer your question.
First, organizationally speaking, International Crisis Group are generalists, not specialists on genocide and mass atrocities. From my vantage point, I can see room for additional risk assessment and warning focused specifically on that extreme problem. When the Genocide Prevention Task Force took a look at the warning question a few years ago–a process that involved several people from ICG–they didn’t conclude that ICG had it covered and there was no need for additional information.
Second and related, ICG does excellent work, but that doesn’t mean there aren’t parts of it that can’t be done better. As Rex’s comment suggested, the reliability and precision of the warnings makes a difference in how useful they are, and I think the status quo on that score leaves room for improvement.
Helene Lavoix
/ April 10, 2012Hello Jay,
This is great the Holocaust Museum is starting such a project, and even better that you are making a call for looking at previous work on the matter. unfortunately the field of EWS has beenlagued by a very short memory…
Rex’s point above is crucial and delivery of warnings to the right policy-makers or clients in foresight and warning language is indeed very important. Yet having proper indicators is also the first necessary and indispensible condition.
On the first point, and this makes your project even more interesting, considering also citizens as clients – thus remembering that the real ruler in a nation-state and even more so in a democracy is the nation – may be crucial and much improve the delivery of the warning.
Regarding indicators, I am not fully convinced by quantativist approaches, because, from my point of view they fail looking at causes and dynamics that are so crucial for all political phenomena, including genocides. However, if mixed methods could be created, that would especially consider how categories of out groups are constructed, then the chance of building a tool really foreseeing genocide and mass atrocities would be much enhanced. I am sure you ahve already talked with Dr Steve Heder (SOAS) on Cambodia and with Lawrence Woocher now at PITF on Warning about genocide?
As dynamics of genocide was part of my PhD you might also want tp have a look at it, as well as a presentation on warnng about genocide I had put together for the State Department a couple of year ago. You can access and download the two from my academia.edu page, as well as papers on EW systems. http://independent.academia.edu/HeleneLavoix
Also you might find some posts – notably for the clients and warning delivery part – that could interest you on my website http://www.redanalysis.org.
Finally, don’t hesitate to contact me if you want to discuss further.
In the meantime, I wish you the best for developing the detailed proposal :).
Helene
(Dr Helene Lavoix)
dartthrowingchimp
/ April 10, 2012Thanks very much, Helene. I appreciate the advice, and I’ll be sure to look at your work, which sounds very helpful.
Helene Lavoix
/ April 10, 2012You are most welcome 🙂
Helene Lavoix
/ April 10, 2012Also, in case you have not consulted it yet: the online Encyclopedia of Mass violence might be useful to you: http://www.massviolence.org/ at ;east to facilitate access to data, timeline, reviews etc. and the INoGS too (International Network of Genocide Scholars – Nigel Eltringham ) Sorry, if you are already in touch with everyone, it is difficult to know.
ADTS
/ April 10, 2012It is a little unclear to me what you mean by advocacy projects and supporting – are you asking whether there are non-profits/NGOs which may be interested in donating money, providing grants, etc., or do you mean something else?
Thanks
ADTS
dartthrowingchimp
/ April 10, 2012That’s an important question, so thank you for asking. Unfortunately, I don’t have a clear answer for you (yet). What I can say is that the proposal on which my project is based envisions this program being a collaborative effort, perhaps even involving a formal consortium, if it happens. Whether that means bringing money to the table or just people, I can’t say.
Doug Laney (@Doug_Laney)
/ July 7, 2014Fantastic work. I’d like to promote it among our clients and learn of any prelim results. If you could reach out to me via Twitter DM, thanks! –Doug Laney, VP Research, Gartner, Twitter: @doug_laney
Pranav Srinivas
/ July 29, 2014Hello Jay,
I am a high school student interested in learning about conflict and genocide
prediction. I have background in R and some ML tools. I am curious about your
thoughts on using events data in GDELT for conflict/genocide prediction.
Are they useful for genocide prediction? It looks like events data by nature are
of high velocity and may not combine well with slow moving data such as infant
mortality rate, GDP. I am trying to get a grip on what is the right problem to work on:
(a) Predict onset of genocide/conflict at specific time (6 to 18 months) or
(b) Rank nations/regions based on some risk assessment score
Really appreciate all your help.
Best Regards
Pranav Srinivas
Monta Vista High School
Cupertino, CA
dartthrowingchimp
/ July 29, 2014Your question—“Are [GDELT data] useful for genocide prediction?”—is an empirical one to which I don’t know the answer. It’s also one that’s going to be virtually impossible to answer with confidence, because there have been so few onsets of genocide or even mass killing during the time period GDELT covers.
If you’re interested in working on this topic—and it’s great that you apparently are—I would recommend focusing on violent conflict more broadly, so that you have enough instances to train and validate models. I would also recommend starting by pursuing the second of the two approaches you outline (relative risk) and then seeing if you can get some traction on the first (timing) once you’ve got a pretty good structural model.
As for event data, I would also suggest that you keep an eye on what the Open Event Data Alliance is doing. They expect to start releasing data soon, and unlike GDELT, the code used to produce their data will be entirely open source. That means that other users and consumers are likely to treat the data as more valid, because they can understand fully how it is generated, and it gives you the opportunity to customize your event data production if you are so motivated.
Good luck, and please report back if you find anything interesting!
Pranav Srinivas
/ July 29, 2014Thanks Jay. Really appreciate your feedback and insight