A First-Person Reminder of How Not to Do Statistics and Science

I recently re-learned a lesson in statistical and scientific thinking by getting—or, really, putting—some egg on my face. I think this experience has some didactic value, so I thought I would share it here.

On Monday, the New York Times ran a story claiming that “cities across the nation are seeing a startling rise in murders after years of declines.” The piece included a chart showing year-over-year change in murder counts in 10 cities, many of them substantial, and it discussed various ideas about why homicide rates are spiking now after years of declines.

I read the piece and thought of claims made in the past decade about the relationship between lead (the metal) and crime. I don’t know the science on that topic, but I read about it in 2013 in Mother Jones, where Kevin Drum wrote:

We now have studies at the international level, the national level, the state level, the city level, and even the individual level. Groups of children have been followed from the womb to adulthood, and higher childhood blood lead levels are consistently associated with higher adult arrest rates for violent crimes. All of these studies tell the same story: Gasoline lead is responsible for a good share of the rise and fall of violent crime over the past half century.

When I read the NYT piece, though, I thought: If murder rates are now spiking in the U.S. but ambient lead levels remain historically low, doesn’t that disprove or at least undercut the claim that lead was responsible for the last crime wave? So I tweeted:

Jordan Wilcox pushed back:

Jordan was right, and I had made two basic mistakes en route to my glib but erroneous conclusion.

First and dumbest, I didn’t check the numbers. The Times only reported statistics from a small and non-representative sample of U.S. cities, and it only compared them across two years. In my experience, that’s not uncommon practice in popular-press trend pieces.

As Bruce Frederick argues in a Marshall Project commentary responding to the same NYT piece, however, that’s not a sound way to look for patterns. When Frederick took a deeper look at the latest police data across a more representative set of cases, he found that almost no U.S. cities appear to be experiencing changes in murder rates outside what we would expect from normal variation around historically low means of recent years. He concludes: “Neither the Times analysis nor my own yields compelling evidence that there has been a pervasive increase in homicides that is substantively meaningful.” On the Washington Post‘s Wonkblog, Max Ehrenfreund showed the same.

Second, even with the flawed statistics I had, I didn’t think carefully about how they related to the Pb-crime hypothesis. Instead, I thought: “We are experiencing a new crime wave and lead levels are still low; therefore lead does not explain the current wave; therefore lead can’t explain the last wave, either.”

In that simple chain of reasoning, I had failed to consider the possibility that different crime waves could have different causes—or really contributing factors, as no one doing careful work on this topic would offer a monocausal explanation of crime. Just as leaded gasoline came and went, other potentially relevant “treatments” that might affect crime rates could come and go, and those subsequent variations would provide little new information about the effects of lead at an earlier time. Imagine that in the near future that smoking is virtually eliminated and yet we still see a new wave of lung-cancer cases; would that new wave disprove the link between smoking and lung cancer? No. It might help produce a sharper estimate of the size of that earlier effect and give us a clearer picture of the causal mechanisms at work, but there’s almost always more than one pathway to the same outcome, and the affirmation of one does not disprove the possibility of another.

After reading more about the crime stats and thinking more about the evidence on lead, I’m back where I started. I believe that rates of homicide and other crimes remain close to historical lows in most U.S. cities, and I believe that lead exposure probably had a significant effect on crime rates in previous decades. That’s not terribly interesting, but it’s truer than the glib and provocative thing I tweeted, and it’s easier to see when I slow down and work more carefully through the basics.

%d bloggers like this: