Positive Feedback Junkie

Yesterday, while grabbing a last half-cup of coffee after an event about political risk assessment, I met a guy who told me he used to work as a futures trader.

“What’s that like?” I asked him.

“Everyone’s different,” he said, and then described a few of the work routines and trading strategies he and his former colleagues had followed.

As he talked about the lifestyle, I recognized some of my own habits. Right now, I’m actively forecasting on at least five different platforms. Together, three of those—the Early Warning Project’s opinion pool, the Good Judgment Project, and Inkling’s public prediction market—cover an almost-absurd array of events and processes around the world, from political violence to trade agreements, election outcomes, and sporting contests. To try to do well on all of those platforms, I have to follow news from as many sources as I can about all kinds of places and organizations. I also forecast on this blog. Here, the prognostications are mostly annual, but they’re public, too, so the results directly affect my professional reputation. The events I forecast here are also rare, so the reputational consequences of a hit or miss will often linger for weeks or months. The fifth platform—the stock market—requires yet-another information set and involves my own real money.

One of the things the Good Judgment Project has found is that subject-matter expertise isn’t reliably associated with higher forecasting accuracy, but voraciously consuming the news and frequently updating your forecasts are. The term “information junkie” comes to mind, and I think the junkie part may be more relevant than we let on. When you’re trying to anticipate the news, there’s a physiological response, an amped-up feeling you get when events are moving quickly in a situation about which you’ve made a forecast. I recognize that cycle of lulls and rushes from a short flirtation with online, play-money poker more than a decade ago, and I sometimes get it now when a blog post gets a burst of attention. When things are slow and nothing relevant seems to be happening, there’s an edginess that persists and pulls you into searching for new information, new opportunities to forecast, new levers to push and then wait for the treat to drop. I’ve also noticed that this feeling gets amplified by Twitter. There, I can see fresh information roll by in real time, like a stock ticker for geopolitics if you follow the right mix of people. I can also chase little rushes by dropping my own tweets into the mix and then watching for retweets, responses, and favorites.

When I started college, I thought I would major in biology. I had really enjoyed math and science in high school, had done well in them, and imagined making a career out of those interests and what seemed like talents. First semester of freshman year, I took vector calculus and chemistry. I also behaved like a lot of college freshman, not working as hard as I had in high school and doing some other things that weren’t especially good for my cognitive skill and accumulation of knowledge. As the semester rolled by, I found that I wasn’t doing as well as I’d expected in those math and science classes, but I was doing very well in my social-science and Russian-language courses. After freshman year, I didn’t take another math or natural-science class in college, and I graduated three years later with a degree in comparative area studies.

Sometimes I regret my failure to chase that initial idea a little harder. When that happens, I explain that failure to myself as the result of a natural impulse to seek out and stay close to streams of positive feedback. I see the same impulse in my forecasting work, and I see it in my own and other people’s behavior on social media, too. It’s not freedom from stress we’re seeking. The absence of stress is boredom, and I don’t know anyone who can sit comfortably with that feeling for long. What I see instead is addictive behavior, the relentless chase for another hit. We’re okay with a little discomfort, as long as the possibility of the next rush hides behind it, and the rush doesn’t have to involve money to feel rewarding.

After the guy I met yesterday had described some traders’ work routines—most of which would probably sound great to people in lots of other jobs, and certainly to people without jobs—I asked him: “So why’d you leave it?”

“Got tired of always chasin’ the money,” he said.

Early Results from a New Atrocities Early Warning System

For the past couple of years, I have been working as a consultant to the U.S. Holocaust Memorial Museum’s Center for the Prevention of Genocide to help build a new early-warning system for mass atrocities around the world. Six months ago, we started running the second of our two major forecasting streams, a “wisdom of (expert) crowds” platform that aggregates probabilistic forecasts from a pool of topical and area experts on potential events of concern. (See this conference paper for more detail.)

The chart below summarizes the output from that platform on most of the questions we’ve asked so far about potential new episodes of mass killing before 2015. For our early-warning system, we define a mass killing as an episode of sustained violence in which at least 1,000 noncombatant civilians from a discrete group are intentionally killed, usually in a period of a year or less. Each line in the chart shows change over time in the daily average of the inputs from all of the participants who choose to make a forecast on that question. In other words, the line is a mathematical summary of the wisdom of our assembled crowd—now numbering nearly 100—on the risk of a mass killing beginning in each case before the end of 2014. Also:

  • Some of the lines (e.g., South Sudan, Iraq, Pakistan) start further to the right than others because we did not ask about those cases when the system launched but instead added them later, as we continue to do.
  • Two lines—Central African Republic and South Sudan—end early because we saw onsets of mass-killing episodes in those countries. The asterisks indicate the dates on which we made those declarations and therefore closed the relevant questions.
  • Most but not all of these questions ask specifically about state-led mass killings, and some focus on specific target groups (e.g., the Rohingya in Burma) or geographic regions (the North Caucasus in Russia) as indicated.
Crowd-Estimated Probabilities of Mass-Killing Onset Before 1 January 2015

Crowd-Estimated Probabilities of Mass-Killing Onset Before 1 January 2015

I look at that chart and conclude that this process is working reasonably well so far. In the six months since we started running this system, the two countries that have seen onsets of mass killing are both ones that our forecasters promptly and consistently put on the high side of 50 percent. Nearly all of the other cases, where mass killings haven’t yet occurred this year, have stuck on the low end of the scale.

I’m also gratified to see that the system is already generating the kind of dynamic output we’d hoped it would, even with fewer than 100 forecasters in the pool. In the past several weeks, the forecasts for both Burma and Iraq have risen sharply, apparently in response to shifts in relevant policies in the former and the escalation of the civil war in the latter. Meanwhile, the forecast for Uighurs in China has risen steadily over the year as a separatist rebellion in Xinjiang Province has escalated and, with it, concerns about a harsh government response. These inflection points and trends can help identify changes in risk that warrant attention from organizations and individuals concerned about preventing or mitigating these potential atrocities.

Finally, I’m also intrigued to see that our opinion pool seems to be sorting cases into a few clusters that could be construed as distinct tiers of concern. Here’s what I have in mind:

  • Above the 50-percent threshold are the high risk cases, where forecasters assess that mass killing is likely to occur during the specified time frame.  These cases won’t necessarily be surprising. Some observers had been warning on the risk of mass atrocities in CAR and South Sudan for months before those episodes began, and the plight of the Rohingya in Burma has been a focal point for many advocacy groups in the past year. Even in supposedly “obvious” cases, however, this system can help by providing a sharper estimate of that risk and giving a sense of how it is trending over time. In the case of Burma, for example, it is the separation that has happened in the last several weeks that tells the story of a switch from possible to likely and thus adds a degree of urgency to that warning.
  • A little farther down the y-axis are the moderate risk cases—ones that probably won’t suffer mass killing during the period in question but could more readily tip in that direction. In the chart above, Iraq, Sudan, Pakistan, Bangladesh, and Burundi all land in this tier, although Iraq now appears to be sliding into the high risk group.
  • Clustered toward the bottom are the low risk cases where the forecasters seem fairly confident that mass killing will not occur in the near future. In the chart above, Russia, Afghanistan, and Ethiopia are the cases that land firmly in this set. China (Uighurs) remains closer to them than the moderate risk tier, but it appears to be creeping toward the moderate-risk group. We are also running a question about the risk of state-led mass killing in Rwanda before 2015, and it currently lands in this tier, with a forecast of 14 percent.

The system that generates the data behind this chart is password protected, but the point of our project is to make these kinds of forecasts freely available to the global public. We are currently building the web site that will display the forecasts from this opinion pool in real time to all comers and hope to have it ready this fall.

In the meantime, if you think you have relevant knowledge or expertise—maybe you study or work on this topic, or maybe you live or work in parts of the world where risks tend to be higher—and are interested in volunteering as a forecaster, please send an email to us at ewp@ushmm.org.

The Wisdom of Crowds, Oscars Edition

Forecasts derived from prediction markets did an excellent job predicting last night’s Academy Awards.

PredictWise uses odds from online bookmaker Betfair for its Oscars forecasts, and it nearly ran the table. PredictWise assigned the highest probability to the eventual winner in 21 of 24 awards, and its three “misses” came in less prominent categories (Best Documentary, Best Short, Best Animated Short). Even more telling, its calibration was excellent. The probability assigned to the eventual winner in each category averaged 87 percent, and most winners were correctly identified as nearly sure things.

Inkling Markets also did quite well. This public, play-money prediction market has a lot less liquidity than BetFair, but it still assigned the highest probability of winning to the eventual winner in 17 of the 18 categories is covered—it “missed” on Best Original Song—and for extra credit it correctly identified Gravity as the film that would probably win the most Oscars. Just by eyeballing, it’s clear that Inkling’s calibration wasn’t as good as PredictWise’s, but that’s what we’d expect from a market with a much smaller pool of participants. In any case, you still probably would have one your Oscars pool if you’d relied on it.

This is the umpteen-gajillionth reminder that crowds are powerful forecasters. “When our imperfect judgments are aggregated in the right way,” James Surowiecki wrote (p. xiv), “our collective intelligence is often excellent.” Or, as PredictWise’s David Rothschild said in his live blog last night,

This is another case of pundits and insiders advertising a close event when the proper aggregation of data said it was not. As I noted on Twitter earlier, my acceptance speech is short. I would like to thank prediction markets for efficiently aggregating dispersed and idiosyncratic data.

  • Author

  • Follow me on Twitter

  • Follow Dart-Throwing Chimp on WordPress.com
  • Enter your email address to follow this blog and receive notifications of new posts by email.

    Join 13,607 other subscribers
  • Archives

%d bloggers like this: