All posts tagged Mike Ward

A Bit More on Country-Month Modeling

My family is riding the flu carousel right now, and my turn came this week. So, in lieu of trying to write from scratch, I wanted to pick up where my last post—on moving from country-year to country-month modeling—left off.

As many of you know, this notion is hardly new. For at least the past decade, many political scientists who use statistical tools to study violent conflict have been advocating and sometimes implementing research designs that shrink their units of observation on various dimensions, including time. The Journal of Conflict Resolution published a special issue on “disaggregating civil war” in 2009. At the time, that publication felt (to me) more like the cresting of a wave of new work than the start of one, and it was motivated, in part, by frustration over all the questions that a preceding wave of country-year civil-war modeling had inevitably left unanswered. Over the past several years, Mike Ward and his WardLab collaborators at Duke have been using ICEWS and other higher-resolution data sets to develop predictive models of various kinds of political instability at the country-month level. Their work has used designs that deal thoughtfully with the many challenges this approach entails, including spatial and temporal interdependence and the rarity of the events of interest. So have others.

Meanwhile, sociologists who study protests and social movements have been pushing in this direction even longer. Scholars trying to use statistical methods to help understand the dynamic interplay between mobilization, activism, repression, and change recognized that those processes can take important turns in weeks, days, or even hours. So, researchers in that field started trying to build event data sets that recorded as exactly as possible when and where various actions occurred, and they often use event history models and other methods that “take time seriously” to analyze the results. (One of them sat on my dissertation committee and had a big influence on my work at the time.)

As far as I can tell, there are two main reasons that all research in these fields hasn’t stampeded in the direction of disaggregation, and one of them is a doozy. The first and lesser one is computing power. It’s no simple thing to estimate models of mutually causal processes occurring across many heterogeneous units observed at high frequency. We still aren’t great at it, but accelerating improvements in computational processing, storage, software—and co-evolving improvements in statistical methods—have made it more tractable than it was even five or 10 years ago.

The second, more important, and more persistent impediment to disaggregated analysis is data, or the lack thereof. Data sets used by statistically minded political scientists come in two basic flavors: global, and case– or region-specific. Almost all of the global data sets of which I’m aware have always used, and continue to use, country-years as their units of observation.

That’s partly a function of the research questions they were built to help answer, but it’s also a function of cost. Data sets were (and mostly still are) encoded by hand by people sifting through or poring over relevant documents. All that labor takes a lot of time and therefore costs a lot of money. One can make (or ask RAs to make) a reasonably reliable summary judgment about something like whether or not a civil war was occurring in a particular country during particular year much quicker than one can do that for each month of that year, or each district in that country, or both. This difficulty hasn’t stopped everyone from trying, but the exceptions have been few and often case-specific. In a better world, we could have patched together those case-specific sets to make a larger whole, but they often use idiosyncratic definitions and face different informational constraints, making cross-case comparison difficult.

That’s why I’ve been so excited about the launch of GDELT and Phoenix and now the public release of the ICEWS event data. These are, I think, the leading edge of efforts to solve those data-collection problems in an efficient and durable way. ICEWS data have been available for several years to researchers working on a few contracts, but they haven’t been accessible to most of us until now. At first I thought GDELT had rendered that problem moot, but concerns about its reliability have encouraged me to keep looking. I think Phoenix’s open-source-software approach holds more promise for the long run, but, as its makers describe, it’s still in “beta release” and “under active development.” ICEWS is a more mature project that has tried carefully to solve some of the problems, like event duplication and errors in geolocation, that diminish GDELT’s utility. (Many millions of dollars help.) So, naturally, I and many others have been eager to start exploring it. And now we can. Hooray!

To really open up analysis at this level, though, we’re going to need comparable and publicly (or at least cheaply) available data sets on a lot more of things our theories tell us to care about. As I said in the last post, we have a few of those now, but not many. Some of the work I’ve done over the past couple of years—this, especially—was meant to help fill those gaps, and I’m hoping that work will continue. But it’s just a drop in a leaky bucket. Here’s hoping for a hard turn of the spigot.

2 Comments

by Jay Ulfelder on April 8, 2015 • Permalink

Posted in Measurement, Methods

Tagged GDELT, ICEWS, Mike Ward, Phoenix

Posted by Jay Ulfelder on April 8, 2015

https://dartthrowingchimp.wordpress.com/2015/04/08/a-bit-more-on-country-month-modeling/

Forecasting Round-Up No. 5

This is the latest in a (very) occasional series: No. 1, No. 2, No. 3, and No. 4.

1. Pacific Standard ran a long feature by Graeme Wood last week on the death of Intrade CEO John Delaney and the rise and demise of the prediction market he helped build. The bits about Delaney’s personal life weren’t of interest to me, but the piece is very much worth reading for the intellectual history of prediction markets it offers. Wood concludes with an intriguing argument about which I would like to see more evidence:

More traditional modes of prediction have proved astonishingly bad, yet they continue to run our economic and political worlds, often straight into the ground. Bubbles do occur, and we can all point to examples of markets getting blindsided. But if prediction markets are on balance more accurate and unbiased, they should still be an attractive policy tool, rather than a discarded idea tainted with the odor of unseemliness. As [economist Robin] Hanson asks, “Who wouldn’t want a more accurate source?”

Maybe most people. What motivates us to vote, opine, and prognosticate is often not the desire for efficacy or accuracy in worldly affairs—the things that prediction markets deliver—but instead the desire to send signals to each other about who we are. Humans remain intensely tribal… More than we like accuracy, we like listening to talkers on our side, and identifying them as being on our team—the right team.

2. One thing I think Graeme Wood gets wrong in that article is this prediction: “The chances of another Intrade’s coming into existence are slim.” Au contraire, mon frère. Okay, so for regulatory reasons we many never see another market that looks and works just like Intrade did, but there’s still a lot of action in this area, including the recently-launched American Civics Exchange (ACE), “the first U.S.-based commercial market for political futures.” ACE uses play money (for now), but the exchange pays out real-cash monthly prizes to the most successful traders. I registered when they launched and have focused my trading so far on the 2014 Congressional elections. I don’t have any expectations of winning the prizes they’re offering, but I’m excited to see that the forum even exists at all.

3. Speaking of actual existing prediction markets, have you seen Sean J. Taylor‘s brainchild, Creds? This is a free and open market in which the currency is reputational. Anyone can create statements and make trades, but you have to register to participate so that your name, and thus your credibility, is attached to those trades. The site needs more liquidity (hint, hint) to become really useful as a forecasting resource, but some of the features and functions Sean is experimenting with on the site are novel and very cool.

I’ve been trading on Creds for a little while and recently used it to create a couple of statements about the possibility that Saudi Arabia will acquire nuclear weapons in the next five years (here and here). Those statements were inspired by a tweet from Ian Bremmer and an ensuing report on BBC News. It’s nice to have open venues to quantify our collective beliefs about topics like this one, something we simply couldn’t do not so long ago.

4. Why is quantifying our beliefs so important? I recently had an email exchange with a colleague on this issue. After that colleague wrote a piece on a timely situation that amounted to a “maybe, maybe not” prediction, I pushed him to assign some probabilities to his thinking. He pushed back, saying that any number he produced would “simply be a guess,” and that numeric guesses would smack of “false precision.” In the end, I failed to convince him to offer what I would consider a real forecast.

The next time that debate comes up, I will point my antagonist toward Mike Ward & co.‘s new Predictive Heuristics blog, and in particular to this passage from this post of Mike’s on “Prediction versus Explanation?“:

Pretending that our explanations don’t have to supply accurate predictions—i.e., we are explaining rather than predicting—leads to worse understanding. Rather than ignoring or hiding predictions we should put them front and center so that they may help us in the evaluation of how well our understandings play out in political events and remind us that our understandings are incomplete as well as uncertain… Real understanding will involve both explanation and prediction. Time to get on with it rather than pretending that these two goals are polar opposites. We have a long way to go.