“They Said It Was Going to Rain”

Most Saturdays and some Sundays, I hook up with a bike ride that winds out of DC’s Rock Creek Park into semi-rural Maryland and back again over the course of a few hours. I depend on this ride for hard training and a shot of competition, but I’m a wet-weather wimp and will usually stay home and use the trainer in my basement if it’s raining or probably going to rain. So, one of the first things I do when I get up most weekend mornings is check the hourly forecasts at weather.com and Weather Underground. If there’s much risk of rain, I’ll open the radar map again close to my 9:45 departure and run the animated forecast for the next few hours. If that animation shows yellow or orange blobs swarming my regular route when I’m going to be on it, I almost always stay in.

One recent Sunday, the forecast had me hemming and hawing for a bit before I decided to go. The hourly breakout at weather.com pegged the chance of rain at 70 percent for the first couple of hours I’d be out, but it wasn’t raining at 9:30 and the radar map didn’t look bad, either. Updating completed, out I went.

The weather often dominates conversations at the start and finish of the ride, and on that Sunday two themes rang through the chatter I overheard: we’d gotten really lucky, and weather forecasters are idiots. “They said it was going to rain,” the Greek chorus kept repeating.

wet paris roubaix

But, of course, that’s not what “they” said. In point of fact, meteorologists had pegged the odds of rain at about 2:1. According to those forecasts, it was probably going to rain, but the chances that it would stay dry weren’t so bad, either. I wouldn’t bet my mortgage on a probability of 0.3, but I’m okay with occasionally risking a soggy ride on one.

As a weather-wimpy cyclist, I was happy to catch the lucky break that Sunday. As a guy who sometimes forecasts for a living, I was intrigued by the consistent way in which so many people had distorted that probability. In our heads, the quantified uncertainty we saw in the paper or on the web was transformed into a categorical prediction of rain. What the modeler would want to contextualize before assessing—“For all of the hours I said there was a 70-percent chance of rain, how often did rain actually happen?”—the intended audience was fine judging in isolation and declaring, “Wrong!”

That we’re not so great at processing probabilities won’t surprise anyone familiar with psychological research from the past few decades on that subject. Exactly what form that bias takes under what conditions, though, still seems to be something of a mystery. In a New York Times blog post about forecasts of the U.S. presidential election, statistician Andrew Gelman wrote:

What if the weatherman told you there was a 30 percent chance of rain—would you be shocked if it rained that day? No.

Apparently, Gelman hasn’t met the crew from my weekend ride. Gelman goes on to connect his assertion to work by Amos Tversky and Daniel Kahneman on prospect theory, which is based, in part, on the expectation people systematically overestimate the risk of low-probability events and underestimate the risk of high-probability ones. That expectation, in turn, is based on empirical research that has been replicated elsewhere, as the following chart shows:

probability weighting estimates

What’s puzzling to me here is that my fellow riders seemed to be distorting things in the opposite direction. Instead of taking a probability of 0.7 and thinking of it as a toss-up as Gelman and that chart predict they would, they had converted it into a sure thing. That’s still bias, of course—just not the kind I would have expected.

If there’s a moral to this story, it’s that we still have a lot of work left to do in understanding how we cogitate on uncertainty and what that implies about how we should produce and present probabilistic forecasts. In many domains, we’re getting better and better at the forecasting part, but even very accurate forecasts are only as useful as we make them or let them be. To get from the one to the other, we still need to learn a lot more about how we process and act on that information—not just individually, but also organizationally and socially.

Dr. Bayes, or How I Learned to Stop Worrying and Love Updating

Over the past week, I spent a chunk of every morning cycling through the desert around Tucson, Arizona. On Friday, while riding toward my in-laws’ place in the mountains west of town, I heard the roar of a jet overhead. My younger son’s really into flying right now, and Tucson’s home to a bunch of fighter jets, so I reflexively glanced up toward the noise, hoping to spot something that would interest him.

On that first glance, all I saw was an empty patch of deep blue sky. Without effort, my brain immediately retrieved a lesson from middle-school physics, reminding me that the relative speeds of light and sound meant any fast-moving plane would appear ahead of its roar. But which way? Before I glanced up again, I drew on prior knowledge of local patterns to guess that it would almost certainly be to my left, traveling east, and not so far ahead of the sound because it would be flying low as it approached either the airport or the Air Force Base.  Moments after my initial glance, I looked up a second time and immediately spotted the plane where I’d now expected to find it. When I did, I wasn’t surprised to see that it was a commercial jet, not a military one, because most of the air traffic in the area is civilian.

This is Bayesian thinking, and it turns out that we do it all the time.  The essence of Bayesian inference is updating. We humans intuitively form and hold beliefs (estimates) about all kinds of things. Those beliefs are often erroneous, but it turns out that we can make them better by revising (updating) them whenever we encounter new information that pertains to them. Updating is really just a form of learning, but Bayes’ theorem gives us a way to structure that learning that turns out to be very powerful. As cognitive scientists Tom Griffiths and Joshua Tenenbaum summarize in a nice 2006 paper [PDF] called “Statistics and the Bayesian Mind,”

The mathematics of Bayesian belief is set out in the box. The degree to which one should believe in a particular hypothesis h after seeing data d is determined by two factors: the degree to which one believed in it before seeing d, as reflected by the prior probability P(h), and how well it predicts the data d, as reflected in the likelihood, P(d|h).

This might sound like a lot of work or just too arcane to bother, but Griffiths and Tenenbaum argue that we often think that way intuitively. Their paper gives several examples, including predictions about the next result in a series of coin flips and the common tendency to infer causality from clusters that actually arise at random.

The same process appears in my airplane-spotting story. My initial glance is akin to the base rates that are often used as the starting point for Bayesian inference: to see something you hear, look where sound is coming from. When that prediction failed, I went through three rounds of updating before I looked up again—one based on general knowledge about the relative speeds of light and sound, and then a second (direction) and third (commercial vs. military) based on prior observations of local air traffic. My final “prediction” turned out to be right because those local patterns are strong, but even with all that objective information, there was still some uncertainty. Who knows, there could have been an emergency, or a rogue pilot, or an alien invasion…

I’m writing about this because I think it’s interesting, but I also have ulterior motives. A big part of my professional life involves using statistical models to forecast rare political events, and I am deeply frustrated by frequent encounters with people who dismiss statistical forecasts out of hand (see here and here for previous posts on the subject). It’s probably unrealistic of me to think so, but I am hopeful that recognition of the intuitive nature and power of Bayesian updating might make it easier for skeptics to make use of my statistical forecasts and others like them.

I’m a firm believer in the forecasting power of statistical models, so I usually treat a statistical forecast as my initial belief (or prior, in Bayesian jargon) and then only revise that forecast as new information arrives. That strategy is based on another prior, namely, the body of evidence amassed by Phil Tetlock and others that the predictive judgments of individual experts often aren’t very reliable, and that statistical models usually produce more accurate forecasts.

From personal experience I gather that most people, including many analysts and policymakers, don’t share that belief about the power of statistical models for forecasting. Even so, I would like to think those skeptics might still see how Bayes’ rule would allow them to make judicious use of statistical forecasts, even if they trust their own or other experts’ judgments more. After all, to ignore a statistical forecast is equivalent to holding the extreme view that that statistical forecast holds absolutely no useful information. In The Theory that Would Not Die, an entertaining lay history of Bayes’ rule, Sharon Bertsch McGrayne quotes Larry Stone, a statistician who used the theorem to help find a nuclear submarine that went missing in 1968, as saying that, “Discarding one of the pieces of information is in effect making the subjective judgment that its weight is zero and the other weight is one.”

So instead of rejecting the statistical forecast out of hand, why not update in response to it? When the statistical forecast closely accords with your prior belief, it will strengthen your confidence in that judgment, and rightly so. When the statistical forecast diverges from your prior belief, Bayes’ theorem offers a structured but simple way to arrive at a new estimate. Experience shows that this deliberate updating will produce more accurate forecasts than the willful myopia involved in ignoring the new information the statistical model has provided. And, as a kind of bonus, the deliberation involved in estimating the conditional probabilities Bayes’ theorem requires may help to clarify your thinking about the underlying processes involved and the sensitivity of your forecasts to certain assumptions.

PS. For some nice worked examples of Bayesian updating, see Appendix B of The Theory that Would Not Die or Chapter 8 of Nate Silver’s book, The Signal and the Noise. And thanks to Paul Meinshausen for pointing out the paper by Griffiths and Tenenbaum, and to Jay Yonamine for recommending The Theory That Would Not Die.

What Cycling Has Taught Me about Rule of Law (and Other Social-Science Geekery)

I make my living as a political scientist, but my big hobby is cycling. I’ve been riding a bike for fun and for exercise off and on for more than two decades now, ever since I was a teenager and first got hooked on mountain biking in the woods behind my parents’ house in Carrboro, NC. Nowadays, I usually spend 10-15 hours on the bike each week, most of it on the roads in and around Washington, DC. I haven’t raced in a few years–a handful of crashes convinced me to get out of the formal side of the sport–but I still do a lot of fast group rides, some of which draw scores of riders when the weather’s nice.

When I’m on my bike, I try to push my work life to the back of my mind and just concentrate on the ride. That, and the exercise, are usually the point of the thing. Still, I’ve spent my entire adulthood training to think like a social scientist, and road cycling is an inherently social endeavor, so the mental firewall sometimes crumbles. When that happens, one of a few recurring themes usually comes to mind, depending on the situation.

1. Rule of law doesn’t mean that laws rule. Social scientists who study political and economic development talk a lot about the importance of “rule of law,” which boils down to the idea that sensible laws are predictably and fairly enforced. According to many development theorists, rich democracies like the United States have prospered in large part because they transitioned early from arbitrary and capricious regulations to rule of law. Poorer countries will only see their economic growth rates take off and politics stabilize, the thinking goes, when they manage to make the same shift.

Even in the United States, though, rule of law can be startlingly incomplete. When you’re perched on a 17-lb. carbon-fiber sculpture, trying to share the road with streams of 3,500-lb. hurtling steel boxes, you’re constantly reminded that the formal rules only tell a small part of the story. Most drivers exceed the formal speed limit most of the time. Almost no one comes to a complete stop at stop signs, even though that’s what the rules of the road tell you to do. (Cyclists are probably worse about this one than drivers, by the way, often blowing through stop signs where cars are already waiting.) A lot of people in Maryland, where I live, talk on hand-held mobile phones while they drive, despite the fact that the state passed a law last year banning that behavior. Some drivers expect cyclists to clear the road for them, even though the law in most (all?) states instructs cyclists to take the lane when the rider judges that it’s not safe to squeeze onto the shoulder. Significantly, this gap between formal and informal rules doesn’t just happen when the police aren’t around. On numerous occasions, I’ve had police officers tell me to do something not prescribed by law (e.g., avoid this road, stay on that bike path), apparently because they thought it was expedient.

If you tried to survive in this environment by counting on people to follow the formal rules, you’d be toast. Some of this is just ignorance of the law, but some of it–like speeding–is the result of informal practices that dominate the formal rules. Some of those informal practices might be more efficient than their formal counterparts, but surely some are not. So, even in places where “rule of law” supposedly prevails, many of our daily practices are still built around shared expectations based on unwritten and sometimes inefficient rules, and these unwritten rules can be very hard to dislodge when they are widely followed.

These observations have strongly influenced how I think about prescriptions for better governance in “developing” countries that are based on changes to formal rules. Some political scientists and economists place great faith in the idea that desirable social outcomes can be brought about by crafting rules that will give people incentives to behave in the ways we’d like. On paper, that idea makes some sense. In practice, however, this yawning gap between formal and informal institutions on the roads reminds me that real life is a lot more complicated.

2. Some people act as if (your) life is really cheap. There are a lot of bad or distracted drivers out there who unintentionally put themselves and cyclists at risk; whenever I encounter them, I might shake my head, but I’m not all that surprised. What do surprise me are the extraordinarily dangerous things some drivers will do to send cyclists a message when they don’t like how those riders are behaving on the road. As far as I can tell, these people just don’t think my life is worth very much, or they just don’t think about it at all.

A couple of years ago, I was on a big group ride in a semi-rural part of Montgomery County on a Saturday morning when a driver apparently got frustrated with waiting behind us for a safe place to pass. On a fast downhill, where our group had stretched into single file and was travelling at or above the 30 mph speed limit, this guy decided to try to pass, then abruptly pulled his car back to the right, splitting the line of cyclists right in front of me. A couple of seconds later, he hit his brakes hard, even though the riders in front of him were still flying down the hill at the same speed. I swerved just enough to avoid ramming straight into his back bumper, clipping the back-left corner of his car instead. My chin hit the trunk, then I flipped through the air and landed on my bum in the opposite lane. Lucky for me, no cars were coming the other way. Without a word and with barely a pause, the driver sped off to his house, which turned out to be just a half-mile down the same road. He pulled into his garage and stayed inside, even when the police came.

That’s just one of many close calls I’ve had on the road with drivers who seemed to be using their machines to tell me how they felt about my presence or behavior on the road. Sometimes it’s just a yell as they pass, but at least a few times a month it’s more: a swerve that squeezes me to the edge of, or even off, the road; a tailgater who could kill me with just a touch of the gas; a guy a couple of weeks ago who sped by, pulled over, jumped out of his car, and screamed at me to come fight him, apparently because I’d delayed him at the last traffic light. (I’ve been part of that particular scenario a few times now.)

It’s hard to imagine that these drivers would engage in these behaviors if they could think through all the potential costs of their actions. For starters, I’d like to think my life is worth something to them, if only in the abstract sense that most of us see human life as a thing worth protecting. Even in totally selfish terms, though, an incident in which I’m badly hurt or killed would be a huge inconvenience for the driver, too. The police, the insurance, the possibility of courts and even jail time–all of that’s going to be a much bigger hassle than the few extra seconds they might wait for a safe opportunity to pass me.

I sometimes think of these angry drivers when I’m reading theories of civil war and other forms of political violence. In the past couple of decades, a lot of the thinking about why civil wars happen where and when they do has centered on the assumption that violence is an instrument which organizations use to advance their political interests, and that individuals who choose to participate in that violence do so after weighing its expected risks and benefits. I still think both of those assumptions can be useful ones for purposes of theorizing about violence, but my experiences on the road have also taught me that those assumptions have stark limits. Sometimes, people threaten or use violence in ways that just don’t seem to take much account of the consequences, and trying to understand that behavior as the product of cost-benefit analysis can take us pretty far away from reality.

3. We all belong to tribes. Cyclists often ride in packs, and the conversation in those packs often turns to drivers. In those conversations, “we” (riders) are typically described as good people doing good things for bodies and our planet, and “they” (drivers) are often described as careless or even bad people who are thinking only of themselves and denying us our rightful place on the road. We share stories of injustices suffered by ourselves or other riders on Facebook and Twitter, and when one of us is threatened by a driver, others often rally around to protect him or her, even if it’s someone we hardly know. That time I clipped the back of a fast-braking car on my Saturday-morning ride, a dozen guys I’ve never seen off a bike stopped their day to make sure I was okay, then waited for almost a half-hour to talk to the police in hopes of punishing the driver.

Based on my limited knowledge of anthropology, I gather this is standard in-group/out-group behavior. We see ourselves as part of a social collective with a distinct identity and way of life; we identify external threats to that way of life; and we go out of our way to protect members of our collective from those threats, even in situations where it isn’t self-evidently “rational” to do so. This is exactly the kind of us vs. them behavior that political scientists and sociologists often describe when discussing “ethnic” or “tribal” groups, usually in pejorative terms. People in rich countries are supposed to have traded in these traditional identities for more “modern” ones, and that break with tradition is supposed to give them the freedom to make decisions based on efficiency instead of obligation.

In short, cyclists may not be an ethnic group, but they sometimes act like one. That cyclists can act like an ethnic group reinforces my belief that the constructivists are right about the origins and behavior of human communities. Supposedly “modern” humans are just the same old people plunked down in different contexts, “ancient hatreds” can get pretty intense pretty fast, and modernity–whatever that is–is not a cure for these quirks of our nature.

Follow

Get every new post delivered to your Inbox.

Join 6,496 other followers

%d bloggers like this: