I recently re-learned a lesson in statistical and scientific thinking by getting—or, really, putting—some egg on my face. I think this experience has some didactic value, so I thought I would share it here.
On Monday, the New York Times ran a story claiming that “cities across the nation are seeing a startling rise in murders after years of declines.” The piece included a chart showing year-over-year change in murder counts in 10 cities, many of them substantial, and it discussed various ideas about why homicide rates are spiking now after years of declines.
I read the piece and thought of claims made in the past decade about the relationship between lead (the metal) and crime. I don’t know the science on that topic, but I read about it in 2013 in Mother Jones, where Kevin Drum wrote:
We now have studies at the international level, the national level, the state level, the city level, and even the individual level. Groups of children have been followed from the womb to adulthood, and higher childhood blood lead levels are consistently associated with higher adult arrest rates for violent crimes. All of these studies tell the same story: Gasoline lead is responsible for a good share of the rise and fall of violent crime over the past half century.
When I read the NYT piece, though, I thought: If murder rates are now spiking in the U.S. but ambient lead levels remain historically low, doesn’t that disprove or at least undercut the claim that lead was responsible for the last crime wave? So I tweeted:
Jordan Wilcox pushed back:
Jordan was right, and I had made two basic mistakes en route to my glib but erroneous conclusion.
First and dumbest, I didn’t check the numbers. The Times only reported statistics from a small and non-representative sample of U.S. cities, and it only compared them across two years. In my experience, that’s not uncommon practice in popular-press trend pieces.
As Bruce Frederick argues in a Marshall Project commentary responding to the same NYT piece, however, that’s not a sound way to look for patterns. When Frederick took a deeper look at the latest police data across a more representative set of cases, he found that almost no U.S. cities appear to be experiencing changes in murder rates outside what we would expect from normal variation around historically low means of recent years. He concludes: “Neither the Times analysis nor my own yields compelling evidence that there has been a pervasive increase in homicides that is substantively meaningful.” On the Washington Post‘s Wonkblog, Max Ehrenfreund showed the same.
Second, even with the flawed statistics I had, I didn’t think carefully about how they related to the Pb-crime hypothesis. Instead, I thought: “We are experiencing a new crime wave and lead levels are still low; therefore lead does not explain the current wave; therefore lead can’t explain the last wave, either.”
In that simple chain of reasoning, I had failed to consider the possibility that different crime waves could have different causes—or really contributing factors, as no one doing careful work on this topic would offer a monocausal explanation of crime. Just as leaded gasoline came and went, other potentially relevant “treatments” that might affect crime rates could come and go, and those subsequent variations would provide little new information about the effects of lead at an earlier time. Imagine that in the near future that smoking is virtually eliminated and yet we still see a new wave of lung-cancer cases; would that new wave disprove the link between smoking and lung cancer? No. It might help produce a sharper estimate of the size of that earlier effect and give us a clearer picture of the causal mechanisms at work, but there’s almost always more than one pathway to the same outcome, and the affirmation of one does not disprove the possibility of another.
After reading more about the crime stats and thinking more about the evidence on lead, I’m back where I started. I believe that rates of homicide and other crimes remain close to historical lows in most U.S. cities, and I believe that lead exposure probably had a significant effect on crime rates in previous decades. That’s not terribly interesting, but it’s truer than the glib and provocative thing I tweeted, and it’s easier to see when I slow down and work more carefully through the basics.
Dart-Throwing Chimp 
Chas Spain
/ September 5, 2015I do love a good causal association so thus will now follow you on twitter just to check on future logic. Nice post Jay- are heat waves and full moons also an actual association in crime stats?
dartthrowingchimp
/ September 5, 2015Thanks. On heat waves and full moons, I have no idea. I don’t know the criminology literature at all, and that probably made me more susceptible to crummy reasoning here. If the NYT piece had been about coups or civil war or democratization, I could have fact-checked it in my head or at least known where to look for a quick cross-reference. On a topic I didn’t know, though, I was probably more gullible.
Chas Spain
/ September 5, 2015I think it’s great to be up front & to cover the issue – it’s such an easy one to fall into. Interesting subject too.
Texas Cowman
/ September 6, 2015I thought it was my old age that condemned me to make fast
and erroneous conclusions. Nice that it happens to those some
younger.
The murder rate is only one stat. just like the final score can
cause some to ignore football yardages.
The cities in Democratic run states seem to be more likely to have riots.
While a murder now and then may be bad for a community, one
big riot can be equal to a large natural disaster.
I have not kept up with the Baltimore and other riots this year,
but I was expecting more than one riot like Ferguson last year,
simply because biological populations mature and when they
mature over a long time of economic and social suppression,
they bear fruit like riots.
The great advantage that cell phones
has given to these populations is to suddenly cluster. The Westfield, IN
Walmart riot of this last July is an example. Walmart’s desire
to supplant the economic centers of communities has had the
less desirable effect of making the stores the social center,
and when these young people rioted they rioted not in the old
city center, but at Walmart. You could see it coming toward
the general population of Walmarts years ago.
Ellie Kesselman
/ September 12, 2015Texas,
I enjoyed your comment, and confess to chuckling at the thought of Walmart’s aberrantly ambitious corporate strategy resulting in the unusually awful situation of a riot whose epicenter was a Walmart store (and it was no accident, but causally motivated).
“While a murder now and then may be bad for a community, one big riot can be equal to a large natural disaster.”
Yes, this is true. I would posit that one big riot can cause greater long-term damage than a large natural disaster, excluding natural disasters that result in significant loss of life. Here’s why I say that: Let’s say the riot does a similar amount of property damage and injury as the natural disaster. After a natural disaster, the affected community will do repairs, try to put more safeguards in place, and move forward. At best there will be camaraderie among people due to experiencing the same traumatic event, and at worst there might be some looting and people will leave the area. Regardless, there won’t be anyone to blame, excepting any human negligence. In contrast, if the event is a large riot, there will be lingering mistrust and anger toward neighbors, elected officials, police and others. There will be legal repercussions, of a criminal sort, not civil. Blame and anger is much easier to focus on people than an act of nature.
Ellie Kesselman
/ September 12, 2015Lead paint is unequivocally harmful to child development. The author of the post still might be correct. Even though the time of unleaded gasoline has come and gone, exposure to lead paint continues through the present, despite the efforts of public health and safety laws.
Exposure to lead has many negative consequence for humans, but without extensive study or a subject matter expert, I don’t know if levels of lead that remain in things that children come in contact with is less or more detrimental to the amount of lead absorbed by children and adults due to leaded gasoline. Non-infectious disease epidemiology relies on heavily on statistical analysis findings, but not exclusively so. It is complicated! I enjoyed and appreciated the author’s acknowledgement of his mistake. Despite the mistake, I think I would consider his data and evidence-driven conclusions more meaningful than Kevin Drum’s 😉
P.S. Jay Ulfelder, I visited your profile on StackExchange, and up voted some of your answers. What sort of day-to-day work does a “freelance political scientist” (as you described yourself on Stats.SE) do? I suspect that there are plenty of opportunities to use statistical inference, e.g. in analyzing survey data.