Which NFL Teams Are the Biggest Surprises of 2015 So Far?

We’re now 4.0625 weeks into the NFL’s 2015 regular season. (If you don’t know what the NFL is, you should probably stop reading now.) That’s about one-quarter of the whole 256-game shebang, enough to start taking stock of preseason predictions. So I got to wondering: Which teams have been the biggest surprises so far?

To get one answer to this question, I downloaded game results from Pro-Football-Reference.com (here) and compared them to the central tendencies of my preseason predictive simulations (here). The mean error of the predictions for each team so far is plotted below. A positive mean error (blue dots) means the team is over-performing relative to the preseason predictions; a negative mean error (red dots) means it’s under-performing.


Most of those results won’t surprise regular NFL watchers. The New York Football Jets finished 4–12 last year and ranked near the bottom in my preseason wiki survey, but they’re off to a 3–1 start this year with a cumulative net score 40 points in their favor. The Falcons, who went 6–10 in 2014 and garnered a low-middle score in the wiki survey, are undefeated after four weeks. At the other end of the scale, the Dolphins got a high-middle score in the preseason survey, but they have stumbled to a 1–3 start with a cumulative net score of -36.

It’s also interesting (to me, anyway) to note how the team-specific errors are only loosely correlated with differences between predicted and observed records. For example, the Giants are only 2–2 so far this year, but they show up as one of the biggest over-performers of the first four weeks. That’s partly because both of those two losses were close games that could easily have flipped the other way. The Giants were expected to be on the bad side of mediocre, but they’ve been competitive in every game so far. Ditto for the Ravens, who only show up as mild under-performers but have a 1–3 record (sob). At least three of those four games were expected to be close, and all of them turned on late scores; unfortunately, only one of those four late turns broke in Baltimore’s favor.

This exercise is only interesting if the preseason predictions on which we’re basing the calls about over– or under-performance are sound. So far, they look pretty solid. After four weeks, the root mean squared error for the predicted net scores is 12.8, and the mean squared error is 165. Those look large, but I think they’re in line with other preseason score forecasts. If we convert the predicted net scores to binary predicted outcomes, the model is 40–23 after four weeks, or 41–23 if we include last night’s Colts-Texans game. That’s not exactly clairvoyant, but it beats eight of ESPN’s 13 experts and matches one more, and they make their predictions each week with updated information.

A Fictitional But Telling Take on Policy Relevance

I finally read and really enjoyed Todd Moss’s first novel, The Golden Hour. It’s a thriller starring Judd Ryker, a political scientist who gets pulled into service at the State Department to help apply a theory he developed on how to nip coups and civil wars in the bud. Before he’s offered that government job, Ryker comes to Washington to brief a small group at State on his ideas. At that point, Ryker has written about his theory but not really tested it. Here’s how the briefing ends:

“What is driving the results on coups? How can you explain what’s so special about timing? I understand the idea of a Golden Hour, but why does it exist?”

“We don’t really know. We can theorize that it probably has something to do with the dynamics of consolidating power after seizure. The coup makers must line up the rest of the security forces and maybe buy off parliament and other local political leaders before those loyal to the deposed president are able to react and countermove. It’s a race for influence. But these are just hypotheses.”

“What about external intervention? Does it matter if an external force gets involved diplomatically?” asked one staffer.

“Or militarily?” interjected another.

“We don’t have classifications for intervention, so it’s not in there,” replied Judd. “The numbers can’t tell us. So we don’t know. I guess we would—”

Parker interrupted abruptly. “But in your expert opinion, Ryker, does it matter? Would it make a difference? Does the United States need to find ways to intervene more rapidly in emerging crises in the developing world? Can we prevent more wars and coups by reacting more quickly?”

Judd looked around the room at all the eyes locked on him. My numbers don’t answer that question. Isn’t that what you guys are here for?

But instead he stood up straight, turned to look Landon Parker directly in the eyes, and said simply, “Yes.”

I think that passage says more about the true nature of the “policy relevance” dance than most of the blog posts I’ve read on that subject. It’s fiction, of course, but it’s written by someone who knows well both sides of that exchange, and it rang true to me.

As we learn later in the novel, the people Ryker was briefing already had a plan, and Ryker’s theory of a Golden Hour—a short window when emerging crises might still be averted—aligned nicely with their existing agenda. This is true, in part, because Ryker’s theory supports the view that U.S. policy makers can and should play an active role in defusing those crises. If Ryker’s theory had implied that U.S. involvement would only make things worse, he would never have been invited to give that briefing.

Scholars who spend time talking to policy makers joke about how much those audiences don’t like to hear “I don’t know” as an answer to questions about why something is happening. That’s real, but I think those audiences might get even more frustrated at hearing “There’s nothing you can do about it” or “Your efforts will only make things worse” in response to questions about what they should do. I suspect that many of those people pursued or accepted government jobs to try to effect change in the world—to “make a difference”—and they don’t want to sit idly while their short windows of opportunity pop open and slam shut.

Then there is Ryker’s decision to submit to his audience’s agenda. Ryker doesn’t know the answer to Parker’s question, and he knows he doesn’t know. Yet, in the moment, he chooses to feign confidence and say “yes” anyway.

The novel hints that this performance owes something to Ryker’s desire to please a mentor who has encouraged him to go into public service. That feels plausible to me, but I would also suspect a deeper and more generic motive: a desire to be wanted by powerful people, to “matter.” If my own experience is any guide, I’d say that we are flattered by attention, and we are eager to stand out. Having government officials ask for your advice feeds both of those cravings.

In short, selection effects abound. The subset of scholars who choose to pursue policy relevance is not a random sample of all academics, and the subset of that subset whose work resonates with policy audiences is not a not a random sample, either. Both partners in this dance have emotional agendas that draw them to each other and then shape their behavior in ways that don’t always align with their ostensible professional ideals: to advance national interests, and to be true to the evidence.

I won’t spoil the novel by telling you how things turn out in Ryker’s case. Instead, I’ll just invite those of you who ever find yourselves on one side or the other of these exchanges—or hope to land there—to consider why you’re doing what you’re doing, and to consider the alternatives before acting.

Military Coup in Burkina Faso

Yesterday, Burkina Faso suffered its second military coup in less than a year. Just a few weeks before scheduled national elections, members of the presidential guard (or RSP, per its French initials) arrested the interim president and prime minister and dissolved the government those men led. According to Reuters:

“The patriotic forces, grouped together in the National Council for Democracy, have decided today to put an end to the deviant transitional regime,” the military official said on RTB state television.

“The transition has progressively distanced itself from the objectives of refounding our democracy,” he said, adding that a revision of the electoral law that blocked supporters of Compaore from running in the planned Oct. 11 had “created divisions and frustrations amongst the people.”

My knowledge of politics in Burkina Faso is shallow, but if I had to guess why this coup happened now, this, also from Reuters, is what I would spotlight:

Burkina Faso’s powerful presidential guard should be dismantled, according to a commission charged with proposing reforms…

In a report submitted to Prime Minister Yacouba Isaac Zida, himself a former commander in the RSP, the national reconciliation and reform commission on Monday described the 1,200 troop strong unit as “an army within an army”.

It called for the regiment to be broken up and its members redeployed within the framework of a broader reform of the military.

In a July post, I spotlighted regional experts’ concerns about another coup by Burkina Faso’s presidential guard, observing how those concerns encapsulated the dilemma that confronts civilian politicians who wish to deepen democracy—or, more cynically, their own power—by strengthening their control over the military. Stronger civilian control means fewer military prerogatives, and as a general rule, political actors prefer not to cede power. I wonder if the RSP saw that reform commission’s report as a harbinger of its fate under the next batch of elected civilian leaders and decided to act now, against the shallow-rooted interim government.

In this year’s statistical assessments of coup risk, Burkina Faso ranked fifth in the world, in no small part because of the coup it suffered last year. As I discussed in a blog post a few years ago, when Mali got hit by its second coup in a 10-month span, coup attempts amplify uncertainty in ways that can keep a country on edge for years. Whether or not the latest coup attempt sticks and without touching the forecasting algorithm, I can tell you that Burkina Faso will land near the top of the global list in next year’s statistical assessments of coup risk, too.

A First-Person Reminder of How Not to Do Statistics and Science

I recently re-learned a lesson in statistical and scientific thinking by getting—or, really, putting—some egg on my face. I think this experience has some didactic value, so I thought I would share it here.

On Monday, the New York Times ran a story claiming that “cities across the nation are seeing a startling rise in murders after years of declines.” The piece included a chart showing year-over-year change in murder counts in 10 cities, many of them substantial, and it discussed various ideas about why homicide rates are spiking now after years of declines.

I read the piece and thought of claims made in the past decade about the relationship between lead (the metal) and crime. I don’t know the science on that topic, but I read about it in 2013 in Mother Jones, where Kevin Drum wrote:

We now have studies at the international level, the national level, the state level, the city level, and even the individual level. Groups of children have been followed from the womb to adulthood, and higher childhood blood lead levels are consistently associated with higher adult arrest rates for violent crimes. All of these studies tell the same story: Gasoline lead is responsible for a good share of the rise and fall of violent crime over the past half century.

When I read the NYT piece, though, I thought: If murder rates are now spiking in the U.S. but ambient lead levels remain historically low, doesn’t that disprove or at least undercut the claim that lead was responsible for the last crime wave? So I tweeted:

Jordan Wilcox pushed back:

Jordan was right, and I had made two basic mistakes en route to my glib but erroneous conclusion.

First and dumbest, I didn’t check the numbers. The Times only reported statistics from a small and non-representative sample of U.S. cities, and it only compared them across two years. In my experience, that’s not uncommon practice in popular-press trend pieces.

As Bruce Frederick argues in a Marshall Project commentary responding to the same NYT piece, however, that’s not a sound way to look for patterns. When Frederick took a deeper look at the latest police data across a more representative set of cases, he found that almost no U.S. cities appear to be experiencing changes in murder rates outside what we would expect from normal variation around historically low means of recent years. He concludes: “Neither the Times analysis nor my own yields compelling evidence that there has been a pervasive increase in homicides that is substantively meaningful.” On the Washington Post‘s Wonkblog, Max Ehrenfreund showed the same.

Second, even with the flawed statistics I had, I didn’t think carefully about how they related to the Pb-crime hypothesis. Instead, I thought: “We are experiencing a new crime wave and lead levels are still low; therefore lead does not explain the current wave; therefore lead can’t explain the last wave, either.”

In that simple chain of reasoning, I had failed to consider the possibility that different crime waves could have different causes—or really contributing factors, as no one doing careful work on this topic would offer a monocausal explanation of crime. Just as leaded gasoline came and went, other potentially relevant “treatments” that might affect crime rates could come and go, and those subsequent variations would provide little new information about the effects of lead at an earlier time. Imagine that in the near future that smoking is virtually eliminated and yet we still see a new wave of lung-cancer cases; would that new wave disprove the link between smoking and lung cancer? No. It might help produce a sharper estimate of the size of that earlier effect and give us a clearer picture of the causal mechanisms at work, but there’s almost always more than one pathway to the same outcome, and the affirmation of one does not disprove the possibility of another.

After reading more about the crime stats and thinking more about the evidence on lead, I’m back where I started. I believe that rates of homicide and other crimes remain close to historical lows in most U.S. cities, and I believe that lead exposure probably had a significant effect on crime rates in previous decades. That’s not terribly interesting, but it’s truer than the glib and provocative thing I tweeted, and it’s easier to see when I slow down and work more carefully through the basics.

Big Data Doesn’t Automatically Produce Better Predictions

At FiveThirtyEight, Neil Payne and Rob Arthur report on an intriguing puzzle:

In an age of unprecedented baseball data, we somehow appear to be getting worse at knowing which teams are — and will be — good.

Player-level predictions are as good if not better than they used to be, but team-level predictions of performance are getting worse. Payne and Arthur aren’t sure why, but they rank a couple of trends in the industry — significant changes in the age structure of the league’s players and, ironically, the increased use of predictive analytics in team management — among the likely culprits.

This story nicely illustrates a fact that breathless discussions of the power of “Big Data” often elide: more and better data don’t automatically lead to more accurate predictions. Observation and prediction are interrelated, but the latter does not move in lock step with the former. At least two things can weaken the link between those two steps in the analytical process.

First, some phenomena are just inherently difficult or impossible to predict with much accuracy. That’s not entirely true of baseball; as Payne and Arthur show, team-level performance predictions have been pretty good in the past. It is true of many other phenomena or systems, however. Take earthquakes; we can now detect and record these events with tremendous precision, but we’re still pretty lousy at anticipating when they’ll occur and how strong they will be. So far, better observation hasn’t led to big gains in prediction.

Second, the systems we’re observing sometimes change, even as we get better at observing them. This is what Payne and Arthur imply is occurring in baseball when they identify trends in the industry as likely explanations for a decline in the predictive power of models derived from historical data. It’s like trying to develop a cure for a disease that’s evolving rapidly as you work on it; the cure you develop in the lab might work great on the last version you captured, but by the time you deploy it, the disease has evolved further, and the treatment doesn’t have much effect.

I wonder if this is also the trajectory social science will follow over the next few decades. Right now, we’re getting hit by the leading edge of what will probably be a large and sustained flood tide of new data on human behavior.  That inflow is producing some rather optimistic statements about how predictable human behavior in general, and sometimes politics in particular, will become as we discover deeper patterns in those data.

I don’t share that confidence. A lot of human behavior is predictably routine, and a torrent of new behavioral data will almost certainly make us even better at predicting these actions and events. For better or for worse, though, those routines are not especially interesting or important to most political scientists. Political scientists are more inclined to focus on “high” politics, which remains fairly opaque, or on system-level outcomes like wars and revolutions that emerge from changes in individual-level behavior in non-obvious ways. I suspect we’ll get a little better at predicting these things as we accumulate richer data on various parts of those systems, but I am pretty sure we won’t ever get great at it. The processes are too complex, and the systems themselves are constantly evolving, maybe even at an accelerating rate.

Interactive 2015 NFL Forecasts

As promised in my last post, I’ve now built and deployed a web app that lets you poke through my preseason forecasts for the 2015 NFL regular season:

2015 NFL Forecasts

I learned several new tricks in the course of generating these forecasts and building this app, so the exercise served its didactic purpose. (You can find the code for the app here, on GitHub.) I also got lucky with the release of a new R package that solved a crucial problem I was having when I started to work on this project a couple of weeks ago. Open source software can be a wonderful thing.

The forecasts posted right now are based on results of the pairwise wiki survey through the morning of Monday, August 17. At that point, the survey had already logged upwards of 12,000 votes, triple the number cast in last year’s edition. This time around, I posted a link to the survey on the r/nfl subreddit, and that post produced a brief torrent of activity from what I hope was a relatively well-informed crowd.

The regular season doesn’t start until September, and I will update these forecasts at least once more before that happens. With so many votes already cast, though, the results will only change significantly if a) a large number of new votes are cast and b) those votes differ substantially from the ones already cast, and those conditions are highly unlikely to intersect.

One thing these forecasts help to illustrate is how noisy a game professional football is. By noisy, I mean hard to predict with precision. Even in games where one team is much stronger than the other, we still see tremendous variance in the simulated net scores and the associated outcomes. Heavy underdogs will win big every once in a while, and games we’d consider close when watching can produce a wide range of net scores.

Take, for example, the week 1 match-up between the Bears and Packers. Even though Chicago’s the home team, the simulation results (below) favor Green Bay by more than eight points. At the same time, those simulations also include a smattering of outcomes in which the Bears win by multiple touchdowns, and the peak of the distribution of simulations is pretty broad and flat. Some of that variance results from the many imperfections of the model and survey scores, but a lot of it is baked into the game, and plots of the predictive simulations nicely illustrate that noisiness.


The big thing that’s still missing from these forecasts is updating during the season. The statistical model that generates the predictive simulations takes just two inputs for each game — the difference between the two teams’ strength scores and the name of the home team — and, barring catastrophe, only one of those inputs can change as the season passes. I could leave the wiki survey running throughout the season, but the model that turns survey votes into scores doesn’t differentiate between recent and older votes, so updating the forecasts with the latest survey scores is unlikely to move the needle by much.*

I’m now hoping to use this problem as an entry point to learning about Bayesian updating and how to program it in R. Instead of updating the actual survey scores, we could treat the preseason scores as priors and then use observed game scores or outcomes to sequentially update estimates of them. I haven’t figured out how to implement this idea yet, but I’m working on it and will report back if I do.

* The pairwise wiki survey runs on open source software, and I can imagine modifying the instrument to give more weight to recent votes than older ones. Right now, I don’t have the programming skills to make those modifications, but I’m still hoping to find someone who might want to work with me, or just take it upon himself or herself, to do this.

Yes, By Golly, I Am Ready for Some Football

The NFL’s 2015 season sort of got underway last night with the Hall of Fame Game. Real preseason play doesn’t start until this weekend, and the first kickoff of the regular season is still a month away.

No matter, though — I’m taking the Hall of Fame Game as my cue to launch this year’s football forecasting effort. As it has for the past two years (see here and here), the process starts with me asking you to help assess the strength of this year’s teams by voting in a pairwise wiki survey:

In the 2015 NFL season, which team will be better?

That survey produces scores on a scale of 0–100. Those scores will become the crucial inputs into simulations based on a simple statistical model estimated from the past two years’ worth of survey data and game results. Using an R function I wrote, I’ve determined that I should be able to improve the accuracy of my forecasts a bit this year by basing them on a mixed-effects model with random intercepts to account for variation in home-team advantages across the league. Having another season’s worth of predicted and actual outcomes should help, too; with two years on the books, my model-training sample has doubled.

An improvement in accuracy would be great, but I’m also excited about using R Studio’s Shiny to build a web page that will let you explore the forecasts at a few levels: by game, by team, and by week. Here’s a screenshot of the game-level tab from a working version using the 2014 data. It plots the distribution of the net scores (home – visitor) from the 1,000 simulations, and it reports win probabilities for both teams and a line (the median of the simulated scores).


The “By team” tab lets you pick a team to see a plot of the forecasts for all 16 of their games, along with their predicted wins (count of games with win probabilities over 0.5) and expected wins (sum of win probabilities for all games) for the year. The “By week” tab (shown below) lets you pick a week to see the forecasts for all the games happening in that slice of the season. Before, I plan to add annotation to the plot, reporting the lines those forecasts imply (e.g., Texans by 7).


Of course, the quality of the forecasts displayed in that app will depend heavily on participation in the wiki survey. Without a diverse and well-informed set of voters, it will be hard to do much better than guessing that each team will do as well this year as it did last year. So, please vote here; please share this post or the survey link with friends and family who know something about pro football; and please check back in a few weeks for the results.

Finding the Right Statistic

Earlier this week, Think Progress reported that at least five black women have died in police custody in the United States since mid-July. The author of that post, Carimah Townes, wrote that those deaths “[shine] an even brighter spotlight on the plight of black women in the criminal justice system and [fuel] the Black Lives Matter movement.” I saw the story on Facebook, where the friend who posted it inferred that “a disproportionate percentage of those who died in jail are from certain ethnic minorities.”

As a citizen, I strongly support efforts to draw attention to implicit and explicit racism in the U.S. criminal justice system, and in the laws that system is supposed to enforce. The inequality of American justice across racial and ethnic groups is a matter of fact, not opinion, and its personal and social costs are steep.

As a social scientist, though, I wondered how much the number in that Think Progress post — five — really tells us. To infer bias, we need to make comparisons to other groups. How many white women died in police custody during that same time? What about black men and white men? And so on for other subsets of interest.

Answering those questions would still get us only partway there, however. To make the comparisons fair, we would also need to know how many people from each of those groups passed through police custody during that time. In epidemiological jargon, what we want are incidence rates for each group: the number of cases from some period divided by the size of the population during that period. Here, cases are deaths, and the population of interest is the number of people from that group who spent time in police custody.

I don’t have those data for the United States for second half of July, and I doubt that they exist in aggregate at this point. What we do have now, however, is a U.S. Department of Justice report from October 2014 on mortality in local jails and state prisons (PDF). This isn’t exactly what we’re after, but it’s close.

So what do those data say? Here’s an excerpt from Table 6, which reports the “mortality rate per 100,000 local jail inmates by selected decedent characteristics, 2000–2012”:

                    2008     2009     2010     2011     2012
By Sex
Male                 123      129      125      123      129
Female               120      120      124      122      123

Race/Hispanic Origin
White                185      202      202      212      220
Black/Afr. Am.       109      100      102       94      109
Hispanic/Latino       70       71       58       67       60
Other                 41       53       36       28       31

Given what we know about the inequality of American justice, these figures surprised me. According to data assembled by the DOJ, the mortality rate of blacks in local jails in those recent years was about half the rate for whites. For Latinos, it was about one-third the rate for whites.

That table got me wondering why those rates were so different from what I’d expected. Table 8 in the same report offers some clues. It provides death rates by cause for each of those same subgroups for the whole 13-year period. According to that table, white inmates committed suicide in local jails at a much higher rate than blacks and Latinos: 80 per 100,000 versus 14 and 25, respectively. Those figures jibe with ones on suicide rates for the general population. White inmates also died from heart disease and drug and alcohol intoxication at a higher rate than their black and Latino counterparts. In short, it looks like whites are more likely than blacks or Latinos to die while in local jails, mostly because they are much more likely to commit suicide there.

These statistics tell us nothing about whether or not racism or malfeasance played a role in the deaths of any of those five black women mentioned in the Think Progress post. They also provide a woefully incomplete picture of the treatment of different racial and ethnic groups by police and the U.S. criminal justice system. For example and as FiveThirtyEight reported just a few days ago, DOJ statistics also show that the rate of arrest-related deaths by homicide is almost twice as high for blacks as whites — 3.4 per 100,000 compared to 1.8. In many parts of the U.S., blacks convicted of murder are more likely than their white counterparts to get the death penalty, even when controlling for similarities in the crimes involved and especially when the victims were white (see here). A 2013 Pew Research Center Study found that, in 2010, black men were six times as likely as white men to be incarcerated in federal, state and local jails.

Bearing all of that in mind, what I hope those figures do is serve as a simple reminder that, when mustering evidence of a pattern, it’s important to consider the right statistic for the question. Raw counts will rarely be that statistic. If we want to make comparisons across groups, we need to think about differences in group size and other factors that might affect group exposure, too.

Be Vewy, Vewy Quiet

This blog has gone relatively quiet of late, and it will probably stay that way for a while. That’s partly a function of my personal life, but it also reflects a conscious decision to spend more time improving my abilities as a programmer.

I want to get better at scraping, making, munging, summarizing, visualizing, and analyzing data. So, instead of contemplating world affairs, I’ve been starting to learn Python; using questions on Stack Overflow as practice problems for R; writing scripts that force me to expand my programming skills; and building Shiny apps that put those those skills to work. Here’s a screenshot of one app I’ve made—yes, it actually works—that interactively visualizes ACLED’s latest data on violence against civilians in Africa, based partly on this script for scraping ACLED’s website:


When I started on this kick, I didn’t plan to stop writing blog posts about international affairs. As I’ve gotten into it, though, I’ve found that my curiosity about current events has ebbed, and the pilot light for my writing brain has gone out. Normally, writing ideas flare up throughout the day, but especially in the early morning. Lately, I wake up thinking about the coding problems I’m stuck on.

I think it’s a matter of attention, not interest. Programming depends on the tiniest details. All those details quickly clog the brain’s RAM, leaving no room for the unconscious associations that form the kernels of new prose. That clogging happens even faster when other parts of your life are busy, stressful, or off kilter, as they are for many of us, as as they are for me right now.

That’s what I think, anyway. Whatever the cause, though, I know that I’m rarely feeling the impulse to write, and I know that shift has sharply slowed the pace of publishing here. I’m leaving the channel open and hope I can find the mental and temporal space to keep using it, but who knows what tomorrow may bring?


The Armed Conflict Location & Event Data Project, a.k.a. ACLED, produces up-to-date event data on certain kinds of political conflict in Africa and, as of 2015, parts of Asia. In this post, I’m not going to dwell on the project’s sources and methods, which you can read about on ACLED’s About page, in the 2010 journal article that introduced the project, or in the project’s user’s guides. Nor am I going to dwell on the necessity of using all political event data sets, including ACLED, with care—understanding the sources of bias in how they observe events and error in how they code them and interpreting (or, in extreme cases, ignoring) the resulting statistics accordingly.

Instead, my only aim here is to share an R script I’ve written that largely automates the process of downloading and merging ACLED’s historical and current Africa data and then creates a new data frame with counts of events by type at the country-month level. If you use ACLED in R, this script might save you some time and some space on your hard drive.

You can find the R script on GitHub, here.

The chief problem with this script is that the URLs and file names of ACLED’s historical and current data sets change with every update, so the code will need to be modified each time that happens. If the names were modular and the changes to them predictable, it would be easy to rewrite the code to keep up with those changes automatically. Unfortunately, they aren’t, so the best I can do for now is to give step-by-step instructions in comments embedded in the script on how to update the relevant four fields by hand. As long as the basic structure of the .csv files posted by ACLED doesn’t change, though, the rest should keep working.

[UPDATE: I revised the script so it will scrape the link addresses from the ACLED website and parse the file names from them. The new version worked after ACLED updated its real-time file earlier today, when the old version would have broken. Unless ACLED changes its file-naming conventions or the structure of its website, the version should work for the rest of 2015. In case it does fail, instructions on how to hard-code a workaround are included as comments at the bottom of the script.]

It should also be easy to adapt the part of the script that generates country-month event counts to slice the data even more finely, or to count by something other than event type. To do that, you would just need to add variables to the group_by() part of the block of code that produces the object ACLED.cm. For example, if you wanted to get counts of events by type at the level of the state or province, you would revise that line to read group_by(gwno, admin1, year, month, event_type). Or, if you wanted country-month counts of events by the type(s) of actor involved, you could use group_by(gwno, year, month, interaction) and then see this user’s guide to decipher those codes. You get the drift.

The script also shows a couple of examples of how to use ‘gglot2’ to generate time-series plots of those monthly counts. Here’s one I made of monthly counts of battle events by country for the entire period covered by ACLED as of this writing: January 1997–June 2015. A production-ready version of this plot would require some more tinkering with the size of the country names and the labeling of the x-axis, but the kind of small-multiples chart offers a nice way to explore the data before analysis.

Monthly counts of battle events, January 1997-June 2015

Monthly counts of battle events, January 1997-June 2015

If you use the script and find flaws in it or have ideas on how to make it work better or do more, please email me at ulfelder <at> gmail <dot> com.


Get every new post delivered to your Inbox.

Join 13,310 other followers

%d bloggers like this: