All posts tagged FiveThirtyEight

Big Data Doesn’t Automatically Produce Better Predictions

At FiveThirtyEight, Neil Payne and Rob Arthur report on an intriguing puzzle:

In an age of unprecedented baseball data, we somehow appear to be getting worse at knowing which teams are — and will be — good.

Player-level predictions are as good if not better than they used to be, but team-level predictions of performance are getting worse. Payne and Arthur aren’t sure why, but they rank a couple of trends in the industry — significant changes in the age structure of the league’s players and, ironically, the increased use of predictive analytics in team management — among the likely culprits.

This story nicely illustrates a fact that breathless discussions of the power of “Big Data” often elide: more and better data don’t automatically lead to more accurate predictions. Observation and prediction are interrelated, but the latter does not move in lock step with the former. At least two things can weaken the link between those two steps in the analytical process.

First, some phenomena are just inherently difficult or impossible to predict with much accuracy. That’s not entirely true of baseball; as Payne and Arthur show, team-level performance predictions have been pretty good in the past. It is true of many other phenomena or systems, however. Take earthquakes; we can now detect and record these events with tremendous precision, but we’re still pretty lousy at anticipating when they’ll occur and how strong they will be. So far, better observation hasn’t led to big gains in prediction.

Second, the systems we’re observing sometimes change, even as we get better at observing them. This is what Payne and Arthur imply is occurring in baseball when they identify trends in the industry as likely explanations for a decline in the predictive power of models derived from historical data. It’s like trying to develop a cure for a disease that’s evolving rapidly as you work on it; the cure you develop in the lab might work great on the last version you captured, but by the time you deploy it, the disease has evolved further, and the treatment doesn’t have much effect.

I wonder if this is also the trajectory social science will follow over the next few decades. Right now, we’re getting hit by the leading edge of what will probably be a large and sustained flood tide of new data on human behavior. That inflow is producing some rather optimistic statements about how predictable human behavior in general, and sometimes politics in particular, will become as we discover deeper patterns in those data.

I don’t share that confidence. A lot of human behavior is predictably routine, and a torrent of new behavioral data will almost certainly make us even better at predicting these actions and events. For better or for worse, though, those routines are not especially interesting or important to most political scientists. Political scientists are more inclined to focus on “high” politics, which remains fairly opaque, or on system-level outcomes like wars and revolutions that emerge from changes in individual-level behavior in non-obvious ways. I suspect we’ll get a little better at predicting these things as we accumulate richer data on various parts of those systems, but I am pretty sure we won’t ever get great at it. The processes are too complex, and the systems themselves are constantly evolving, maybe even at an accelerating rate.

1 Comment

by Jay Ulfelder on August 20, 2015 • Permalink

Posted in Complexity Politics, Forecasting, Measurement

Tagged baseball, Big Data, FiveThirtyEight

Posted by Jay Ulfelder on August 20, 2015

https://dartthrowingchimp.wordpress.com/2015/08/20/big-data-doesnt-automatically-produce-better-predictions/

Finding the Right Statistic

Earlier this week, Think Progress reported that at least five black women have died in police custody in the United States since mid-July. The author of that post, Carimah Townes, wrote that those deaths “[shine] an even brighter spotlight on the plight of black women in the criminal justice system and [fuel] the Black Lives Matter movement.” I saw the story on Facebook, where the friend who posted it inferred that “a disproportionate percentage of those who died in jail are from certain ethnic minorities.”

As a citizen, I strongly support efforts to draw attention to implicit and explicit racism in the U.S. criminal justice system, and in the laws that system is supposed to enforce. The inequality of American justice across racial and ethnic groups is a matter of fact, not opinion, and its personal and social costs are steep.

As a social scientist, though, I wondered how much the number in that Think Progress post — five — really tells us. To infer bias, we need to make comparisons to other groups. How many white women died in police custody during that same time? What about black men and white men? And so on for other subsets of interest.

Answering those questions would still get us only partway there, however. To make the comparisons fair, we would also need to know how many people from each of those groups passed through police custody during that time. In epidemiological jargon, what we want are incidence rates for each group: the number of cases from some period divided by the size of the population during that period. Here, cases are deaths, and the population of interest is the number of people from that group who spent time in police custody.

I don’t have those data for the United States for second half of July, and I doubt that they exist in aggregate at this point. What we do have now, however, is a U.S. Department of Justice report from October 2014 on mortality in local jails and state prisons (PDF). This isn’t exactly what we’re after, but it’s close.

So what do those data say? Here’s an excerpt from Table 6, which reports the “mortality rate per 100,000 local jail inmates by selected decedent characteristics, 2000–2012”:

                    2008     2009     2010     2011     2012
By Sex
Male                 123      129      125      123      129
Female               120      120      124      122      123

Race/Hispanic Origin
White                185      202      202      212      220
Black/Afr. Am.       109      100      102       94      109
Hispanic/Latino       70       71       58       67       60
Other                 41       53       36       28       31

Given what we know about the inequality of American justice, these figures surprised me. According to data assembled by the DOJ, the mortality rate of blacks in local jails in those recent years was about half the rate for whites. For Latinos, it was about one-third the rate for whites.

That table got me wondering why those rates were so different from what I’d expected. Table 8 in the same report offers some clues. It provides death rates by cause for each of those same subgroups for the whole 13-year period. According to that table, white inmates committed suicide in local jails at a much higher rate than blacks and Latinos: 80 per 100,000 versus 14 and 25, respectively. Those figures jibe with ones on suicide rates for the general population. White inmates also died from heart disease and drug and alcohol intoxication at a higher rate than their black and Latino counterparts. In short, it looks like whites are more likely than blacks or Latinos to die while in local jails, mostly because they are much more likely to commit suicide there.

These statistics tell us nothing about whether or not racism or malfeasance played a role in the deaths of any of those five black women mentioned in the Think Progress post. They also provide a woefully incomplete picture of the treatment of different racial and ethnic groups by police and the U.S. criminal justice system. For example and as FiveThirtyEight reported just a few days ago, DOJ statistics also show that the rate of arrest-related deaths by homicide is almost twice as high for blacks as whites — 3.4 per 100,000 compared to 1.8. In many parts of the U.S., blacks convicted of murder are more likely than their white counterparts to get the death penalty, even when controlling for similarities in the crimes involved and especially when the victims were white (see here). A 2013 Pew Research Center Study found that, in 2010, black men were six times as likely as white men to be incarcerated in federal, state and local jails.

Bearing all of that in mind, what I hope those figures do is serve as a simple reminder that, when mustering evidence of a pattern, it’s important to consider the right statistic for the question. Raw counts will rarely be that statistic. If we want to make comparisons across groups, we need to think about differences in group size and other factors that might affect group exposure, too.

2 Comments

by Jay Ulfelder on July 30, 2015 • Permalink

Posted in Measurement, Statistics

Tagged Black Lives Matter, FiveThirtyEight, ThinkProgress

Posted by Jay Ulfelder on July 30, 2015

https://dartthrowingchimp.wordpress.com/2015/07/30/finding-the-right-statistic/

From Thailand, Evidence of Coups’ Economic Costs

Last year, I used this space to report a statistical analysis I’d done on the impact of successful coups on countries’ economic growth rates. Bottom line: they usually hurt. As summarized in this chart from a post at FiveThirtyEight, my analysis indicated that, on average, coups dent a country’s economy by a couple of percentage points in the year they happen and another point the year after. Those are not trivial effects.

What makes this question so tricky to analyze is the possibility of a two-way relationship between these things. It’s not hard to imagine that coups might damage national economies, but it’s also likely that countries suffering slower growth are generally more susceptible to coups. With country-year data and no experimental controls, we can’t just run a model with growth as the dependent variable and the occurrence of a coup as a predictor and expect to get a reliable estimate of the former on the latter.

In my statistical analysis, I tried to deal with this problem by using coarsened exact matching to focus the comparison on sets of country-years with comparable coup risk in which coups did or did not happen. I believe the results are more informative than what we’d get from a pooled sample of all country-years, but they certainly aren’t the last word. After all, matching does not magically resolve deeper identification problems, even if it can help.

Under these circumstances, a little process-tracing can go a long way. If we look at real-world cases and see processes linking the “treatment” (coups) to the hypothesized effect (economic damage), we bolster our confidence that the effect we saw in our statistical analysis is not ephemeral.

Here, the recent coup in Thailand is serving up some intriguing evidence. In the past week, I have seen two reports identifying specific ways in which the coup itself, and not the instability that preceded and arguably precipitated it, is damaging Thailand’s economy. First, I saw this Reuters story (emphasis mine):

Thai officials said on Tuesday that the mass departure of Cambodian laborers would dent the economy as thousands more migrant workers, fearing reprisals from the new military government, poured across the border.

Around 170,000 Cambodian workers have headed home in the past week, according to the International Organization for Migration (IOM), although the exodus is now slowing. Many left after hearing rumors that Thailand’s junta was bent on cracking down on illegal migrants.

Then I saw this tidbit in a Credit Suisse analysis shared on Twitter by Andrew Marshall (emphasis mine):

The coup and martial laws have produced stronger negative impact on Thai tourism, worsening the 2014 earnings outlook and could affect the magnitude of recovery anticipated for 2015…

Based on our conversations with [Airports of Thailand Plc], the coup…appears to have had a stronger impact on its international passenger volumes than the political conflicts… While the coup has restored peace in Bangkok and Thailand, and comforted the Thai people [!], we reckon tourists may take this more negatively and have chosen to go to other destinations.

In a country where international tourism contributes nearly 10 percent of the gross domestic product (here), that impact is a serious issue.

What’s important about both of these reports for the question at hand is the explicit connection they make between the occurrence of the coup and the economy-damaging behavior that followed. To me, these look like consequences rather than coincidences. Neither report definitively proves that the occurrence of a coups usually has an independent, negative effect on a country’s economic growth rate, of course. But they do make me more confident that the effect I saw in my statistical analysis is not just an artifact of some deeper forces I failed to consider.