The Steep Slope of the Data Revolution’s Second Derivative

Most of the talk about a social science “data revolution” has emphasized rapid increases in the quantity of data available to us. Some of that talk has also focused on changes in the quality of those data, including new ideas about how to separate the wheat from the chaff in situations where there’s a lot of grain to thresh. So far, though, we seem to be talking much less about the rate of change in those changes, or what calculus calls the second derivative.

Lately, the slope of this second derivative has been pretty steep. It’s not just that we now have much more, and in some cases much better, data. The sources and content of those data sets are often fast-moving targets, too. The whole environment is growing and churning at an accelerating pace, and that’s simultaneously exhilarating and frustrating.

It’s frustrating because data sets that evolve as we use them create a number of analytical problems that we don’t get from stable measurement tools. Most important, evolving data sets make it hard to compare observations across time, and longitudinal analysis is the crux of most social-scientific research. Paul Pierson explains why in his terrific 2004 book, Politics in Time:

Why do social scientists need to focus on how processes unfold over significant stretches of time? First, because many social processes are path dependent, in which case the key causes are temporally removed from their continuing effects… Second, because sequencing—the temporal order of events or processes—can be a crucial determinant of important social outcomes. Third, because many important social causes and outcomes are slow-moving—they take place over quite extended periods of time and are only likely to be adequately explained (or in some cases even observed in the first place) if analysts are specifically attending to that possibility.

When our measurement systems evolve as we use them, changes in the data we receive might reflect shifts in the underlying phenomenon. They also might reflect changes in the methods and mechanisms by which we observe and record information about that phenomenon, however, and it’s often impossible to tease the one out from the other.

recent study by David Lazer, Gary King, Ryan Kennedy, and Alessandro Vespignani on what Google Flu Trends (GFT) teaches us about “traps in Big Data analysis” offers a nice case in point. Developed in the late 2000s by Google engineers and researchers at the Centers for Disease Control and Prevention, GFT uses data on Google search queries to help detect flu epidemics (see this paper). As Lazer and his co-authors describe, GFT initially showed great promise as a forecasting tool, and its success spurred excitement about the power of new data streams to shed light on important social processes. For the past few years, though, the tool has worked poorly on its own, and Lazer & co. believe of changes in Google’s search software are the reason. The problem—for researchers, anyway—is that

The Google search algorithm is not a static entity—the company is constantly testing and improving search. For example, the official Google search blog reported 86 changes in June and July 2012 alone (SM). Search patterns are the result of thousands of decisions made by the company’s programmers in various sub-units and by millions of consumers worldwide.

Google keeps tinkering with its search software because that’s what its business entails, but we can expect to see more frequent changes in some data sets specific to social science, too. One of the developments about which I’m most excited is the recent formation of the Open Event Data Alliance (OEDA) and the initial release of the machine-coded political event data it plans to start producing soon, hopefully this summer. As its name implies, OEDA plans to make not just its data but also its code freely available to the public in order to grow a community of users who can help improve and expand the software. That crowdsourcing will surely accelerate the development of the scraping and coding machinery, but it also ensures that the data OEDA produces will be a moving target for a while in ways that will complicate attempts to analyze it.

If these accelerated changes are challenging for basic researchers, they’re even tougher on applied researchers, who have to show and use their work in real time. So what’s an applied researcher to do when your data-gathering instruments are frequently changing, and often in opaque and unpredictable ways?

First, it seems prudent to build systems that are modular, so that a failure in one part of the system can be identified and corrected without having to rebuild the whole edifice. In the atrocities early-warning system I’m helping to build right now, we’re doing this by creating a few subsystems with some overlap in their functions. If one part doesn’t pan out or suddenly breaks, we can lean on the others while we repair or retool.

Second, it’s also a good idea to embed those technical systems in organizational procedures that emphasize frequent checking and fast adaptation. One way to do this is to share your data and code and to discuss your work often with outsiders as you go, so you can catch mistakes, spot alternatives, and see these changes coming before you get too far down any one path. Using open-source statistical software like R is also helpful in this regard, because it lets you take advantage of new features and crowd fixes as they bubble up.

Last and fuzziest, I think it helps to embrace the idea that you’re work doesn’t really belong to you or your organization but is just one tiny part of a larger ecosystem that you’re hoping to see evolve in a particular direction. What worked one month might not work the next, and you’ll never know exactly what effect you’re having, but that’s okay if you recognize that it’s not really supposed to be about you. Just keep up as best you can, don’t get too heavily invested in any one approach or idea, and try to enjoy the ride.

211 Years of Political Evolution in 60 Seconds — New and Improved!!

The heat maps used in the animation I posted yesterday plotted change over time in counts of countries in each cell of a two-dimensional space representing different kinds of politcal institutions. Over the 211 years in question, however, the number of countries in the world has grown dramatically, from about 50 in 1800 to well over 150 in 2011. For that reason, a couple of commenters wondered whether we would see something different if we plotted proportions instead of counts, using the size of the total population as a denominator in each cell. Proportions better fit the ideas behind a fitness landscape, so I added a line to my code and gave it a whirl. Here’s what I got:

To my eye, there aren’t any big differences in the patterns we see here compared with the ones based on counts. Re-watching the animation today, though, here are a few other things that caught my attention:

  • The predominance in the mid-1800s of intermediate forms combining authoritarian selection with highly polarized political participation—what Polity calls “factionalism.” This peak in the middle left of the heat maps shows how popular mobilization generally led to competitive elections, and not the other way around. As historian Sean Wilenz wrote, “Democracy is never a gift bestowed…It must always be fought for.” It also reminds us that popular mobilization was initially quite polarized in the “developed” world (ha!), just as it often is poorer countries today.
  • The wide variety of intermediate forms present in the early 1900s. Here we see a bunch of cases in the upper left-hand quadrant, combining authoritarian selection procedures with open and well-regulated participation. This is a combination we almost never see nowadays. It looks like there were some interesting experiments occurring in the wake of the industrial explosion that occurred in richer countries in the latter half of the nineteenth century.
  • The sharp bifurcation of the fitness landscape after World War II. Before the war, the peak in the lower left-hand corner representing closed dictatorships had shrunken, and there seemed to be more action in the upper left and lower right quadrants. After the war, the peak in the lower left rose again and remained there until around 1990. This pattern makes clearer that the evolution of the past two centuries has not been a steady march toward democracy. It’s interesting—and potentially chilling—to contemplate how much the fitness landscape of the past 70 years might have differed had World War II taken different turns.

211 Years of Political Evolution in 60 Seconds

The GIF below—click on it to make it play—animates a series of 211 heat maps summarizing annual data on national political regimes around the world from 1800 to 2010. The space in the heat maps represents two of the “concept” variables from the Polity IV data set—executive recruitment and political competition—that roughly correspond to the dimensions of contestation and participation Robert Dahl uses to define modern regime types. In the animated maps, the lower left is least democratic, and the upper right is most democratic. The darker the grey, the higher the number of cases in that cell. [NB. For a version that uses proportions instead of raw counts and some additional thoughts on patterns over time, see this short follow-up post.]

[Fellow propeller-heads: I built this in R with helpful suggestions from Trey Causey and Tom Parris along the way. The heat maps were made with a function appropriately called 'heatmap', and I used the 'animation' package to compile those images into a .gif. Ping me if you'd like to see the script.]

I made this animation because I think it supports the idea, discussed briefly in my last post, that political development is an evolutionary process. Evolutionary processes feed on diversity and mutation, but the results of evolution are not randomly distributed. Borrowing from Daniel Dennett, we can imagine evolution occurring in a multidimensional design space that contains all possible combinations of a particular set of building blocks. In biology, those building blocks are genes; in politics, they might be simple rules.

For present purposes, let’s imagine that there are only two dimensions in this design space. Those two dimensions suggest a map of the design space that evolutionary biologists call a fitness landscape. The topography of this landscape is determined by the fitness of specific combinations, as indicated by sizes of the relevant populations. That’s what the heat maps in the animation above are showing.

The existence of the system is a matter of chance, but once an evolutionary system emerges, we can expect to see certain patterns. The selection pressures present in any particular environment mean that some combinations will be fitter than others, producing visible and often durable peaks in that fitness landscape. Mutation—and, in the case, of social technologies like government, deliberate tinkering—will keep producing new varieties, but most won’t be fit enough for the environment of the day to survive and spread. As a result, most of the variation will cluster around the existing peaks, because small differences in design will often (but not always!) produce small differences in fitness.

When selection pressures change, however, the designs embodied in the previous peaks will often become less fit, and new designs will emerge as stronger competitors. Importantly, though, that transition from the old peaks to new ones usually won’t be smooth and direct. Instead, as Niles Eldredge and Stephen Jay Gould describe in their model of punctuated equlibrium, we can expect to see bursts of diversity as the evolutionary engine “searches” for new forms that better fit the changing environment. As the selection pressures settle into a new normal, the fitness landscape should also settle back into the familiar pattern of clearer peaks and valleys.

The two Polity variables used here are, of course, gross and conceptually biased simplifications of complex phenomena. Underlying each of these dimensions are a few component variables that are themselves simplifications of complex sets of written and unwritten rules. Still, the Polity data are the best we’ve got right now for observing change in over a long period of time, and it’s pretty hard for us humans to visualize four- or seven- or thirty-dimensional space. So, for now, I’m using these two summary indices to get a very rough map of the design space for modern political institutions.

Maybe it’s confirmation bias at work, but when I watch the animation above, I see the patterns evolutionary theorists tell me I should see. In 1800, the fitness landscape is dominated by a single peak representing highly undemocratic regimes—mostly monarchies with virtually no popular participation. If we could extend the movie back several more centuries, we would see the same pattern holding through the entirety of human civilization since our hunter-gatherer days.

Pretty soon after we drop in to watch, however, things start to move. In the early 1800s, a couple of new lumps rise as popular participation expands in some regimes. Most countries still select their rulers by hereditary lineage or other closed means (the peak in the middle left), but some start using competitive elections to pick their governments. By the late nineteenth century, a second peak has clearly emerged in the upper right-hand corner, where rulers are chosen through competitive elections with broad participation. [NB: I think Polity rushes things a bit here by ignoring the disenfranchisement of women, but we go to publish with the data we've got, not the data we'd like.]

Through most of the twentieth century, the same general pattern holds. There’s a fair amount of variation, but most regimes are concentrated in the same few patches of the design space. At the end of the twentieth and start of the twenty-first centuries, however, we see a burst of diversity. The authoritarian peak shrinks, the democratic peak holds, and large swathes of the design space that have rarely been occupied bubble with activity.

To my eye, this very recent phase looks like one of Eldredge and Gould’s punctuation marks, that is, an episode of heightened diversity caused by a significant shift in selection pressures. Most observers of international politics won’t be surprised to see this pattern, and many of them would probably attribute it to the end of the Cold War. I’m not so sure. I’m more inclined to see the collapse of the Soviet Union and the expansion in the diversity of political forms as twin consequences of deeper changes in the global system that seem to be favoring democratic forms over authoritarian ones. What new peaks we’ll see when the system settles down again—and on what heretofore hidden dimensions of political design space they might draw—is impossible to know, but it sure is fascinating to watch.

Top 5 Albums to Play When Writing Stats Code

I like to listen to music when I’m working, but not all music works equally well for all tasks. When I’m writing prose, for example, I constantly get distracted if I play music with a narrative or clever wordplay in the lyric; the words in the song keep pushing out the words in my head. When I’m writing (or, more often, debugging) stats code, I find that certain pieces of music actually help me get and stay in a nice flow. So, in the brilliant tradition of High Fidelity, here are my Top 5 records to play when writing stats code:

5. Cocteau Twins, Heaven or Las Vegas. This one has words, but god bless you if you can understand what the heck she’s singing. All I hear is a lush run of sound with a fantastic bass line.

4. Beethoven’s piano sonatas. I usually listen to Volume II of the three-volume, many-disc Glenn Gould edition, which gives me a few hours of uninterrupted music. My favorite piece is the allegro section of the opening “Pastorale,” which always makes me feel like I’m meandering down a country lane on horseback on a sunny day.

3. Kronos Quartet Performs Philip Glass. I wonder if Glass’s “repetitive structures” help turn on parts of the brain that deal in math and logic. In any case, this recording of a few of his string quartets feels romantic to me in spite of its modern structures, and it never gets old to my ears.

2. Daft Punk, Tron:Legacy Soundtrack. I know, I know, it’s pretty much a cliché, but if you can tolerate the snippet of Jeff Bridges’ character geeking out about The Grid, this album kills for code-writing.

1. Sviatoslav Richter, The Authorised Recordings of J.S. Bach compositions for piano. Put on good over-ear headphones, and listening to this is like slipping into another world where there’s just one long, unbroken thread of gorgeous sound. It’s like coding in a vacuum.

Has Africa Gone Coup-Crazy in 2012?

Guinea-Bissau’s armed forces violently seized control of the country’s capital yesterday in an apparent coup d’etat. This is the second successful coup in West Africa in the past month–the other happened in Mali in mid-March–and, if my Twitter feed is any indication, this pair of events has a lot of people wondering if 2012 is going to be an unusually “hot” year for coups in that part of the world.

Statistically speaking, the answer seems to be “no”–or “not yet,” anyway, and it still has a ways to go to get there.

To see if 2012 is shaping up to be a weird year for coup activity in Africa, I used the ‘bcp’ package in R to apply a technique called Bayesian change point detection (BCP) to annual counts of successful and failed coup attempts in the region from 1946 through 2012 (so far). BCP treats time-series data as a collection of independent and identically distributed partitions and looks for points in that series where the data’s generative parameters appear to change. My data on coup events come from the Center for Systemic Peace.

The results are shown below. The top half of the chart plots the observed annual counts (the dots) and the posterior means for those annual counts (the line). The real action, though, is in the bottom half, which plots the posterior probabilities of a change point. The higher that number, the more confident we are that a particular year marks a sudden change. In this series, we see evidence of three change points: one in the mid-1960s, a few years after the start of decolonization; another in the early 1990s, after the end of the Cold War; and a third in the late 1990s, when the rate of coups in the region takes a sharp dip. Meanwhile, the pair of events observed so far in 2012 looks perfectly normal, just about average for the past decade and still well below the recent peak of six events in 2008.

If two coup bids in 2012 does not an aberration make, how many would we need to see this year to call it a significant change? I reran the BCP analysis several times using ever-larger counts for 2012, and it took a big jump to start moving the posterior probability of a change point in any appreciable way. At five events, the posterior probability still hadn’t moved much. At six, it finally moved appreciably, but only to around 0.2. In the end, it took eight events to push the posterior probability over 0.5.

In other words, it would take a lot more than two coup bids in 2012 to mark a significant change from the recent past, and what we’ve seen this year so far looks like normal variation in a stochastic process. Event counts are often noisy, but our pattern-seeking brains still try to find meaning in those small variations. It’s also harder to remember less recent events, and our brains tend to confuse that difficulty with infrequency. It helps to remember those biases whenever a new event starts you thinking about a trend.

NOTE: This version of the plot and the scenario analysis corrects an error in the data used in the original post. For the first run, I forgot that my analysis file ended in 2010, so the 0 events shown for 2011 was a mistake. There were actually two failed coups in Africa last year, one in the DRC in February and another in Guinea in July. With those two events added to the data set, the first third of 2012 looks even more typical than it did before.


Get every new post delivered to your Inbox.

Join 6,447 other followers

%d bloggers like this: