A Quick Post Mortem on Oscars Forecasting

I was intrigued to see that statistical forecasts of the Academy Awards from PredictWise and FiveThirtyEight performed pretty well this year. Neither nailed it, but they both used sound processes to generate probabilistic estimates that turned out to be fairly accurate.

In the six categories both sites covered, PredictWise assigned very high probabilities to the eventual winner in four: Picture, Actor, Actress, and Supporting Actress. PredictWise didn’t miss by much in one more—Supporting Actor, where winner Christoph Waltz ran a close second to Tommy Lee Jones (40% to 44%). Its biggest miss came in the Best Director category, where PredictWise’s final forecast favored Steven Spielberg (76%) over winner Ang Lee (22%).

At FiveThirtyEight, Nate Silver and co. also gave the best odds to the same four of six eventual winners, but they were a little less confident than PredictWise about a couple of them. FiveThirtyEight also had a bigger miss in the Best Supporting Actor category, putting winner Christoph Waltz neck and neck with Philip Seymour Hoffman and both of them a ways behind Jones. FiveThirtyEight landed closer to the mark than PredictWise in the Best Director category, however, putting Lee just a hair’s breadth behind Spielberg (0.56 to 0.58 on its index).

If this were a showdown, I’d give the edge to PredictWise for three reasons. One, my eyeballing of the results tells me that PredictWise’s forecasts were slightly better calibrated. Both put four of the six winners in front and didn’t miss by much on one more, but PredictWise was more confident in the four they both got “right.” Second, PredictWise expressed its forecasts as probabilities, while FiveThirtyEight used some kind of unitless index that I found harder to understand. Last but not least, PredictWise also gets bonus points for forecasting all 24 of the categories presented on Sunday night, and against that larger list it went an impressive 19 for 24.

It’s also worth noting the two forecasters used different methods. Silver and co. based their index on lists of awards that were given out before the Oscars, treating those results like the pre-election polls they used to accurately forecast the last couple of U.S. general elections. Meanwhile, PredictWise used an algorithm to combine forecasts from a few different prediction markets, which themselves combine the judgments of thousands of traders. PredictWise’s use of prediction markets gave it the added advantage of making its forecasts dynamic; as the prediction markets moved in the weeks before the awards ceremony, its forecasts updated in real time. We don’t have enough data to say yet, but it may also be that prediction markets are better predictors than the other award results, and that’s why PredictWise did a smidgen better.

If I’m looking to handicap the Oscars next year and both of these guys are still in the game, I would probably convert Silver’s index to a probability scale and then average the forecasts from the two of them. That approach wouldn’t have improved on the four-of-six record they each managed this year, but the results would have been better calibrated than either one alone, and that bodes well for future iterations. Again and again, we’re seeing that model averaging just works, so whenever the opportunity presents itself, do it.

UPDATE: Later on Monday, Harry Enten did a broader version of this scan for the Guardian‘s Film Blog and reached a similar conclusion:

A more important point to take away is that there was at least one statistical predictor got it right in all six major categories. That suggests that a key fact about political forecasting holds for the Oscars: averaging of the averages works. You get a better idea looking at multiple models, even if they themselves include multiple factors, than just looking at one.

States Are Like the Millennium Falcon

Last week, artist John Powers wrote a wonderful blog post on how the original Star Wars movie pioneered American cinematic representations of a “used future”—that is, “a future with a past.”

The visual program of Star Wars is unique because it was chronologically stratified. Lucas borrowed from real machine age periods to give his cinematic future an immediately recognizable depth of time. We [are] meant to see that the oldest elements of the film, like Obi Wan Kenobi with his Samurai robes, and Darth Vader…were hold-overs from an older order…Luke’s Landspeeder and C3PO stood in for what was clearly an entire machine age…Luke and his rebel cohorts were flying into battle in the Star Wars equivalent of an old WWII surplus.

I’m a fan of Star Wars—well, of the first two and a half movies, anyway—but I’m also a political scientist, and John’s post got me thinking again about how political scientists think about states.

In American political science, the conventional (Modernist) view of states and their origins are embodied in Lucas’ Death Star. Hierarchically organized communities draw up and execute elaborate plans for a system that performs a clear set of functions. Everything in the whole serves a unique purpose, and each component was presumably built just for that purpose. The engineered system is complicated, but it is not complex. The whole is the sum of its modular parts.  Like Le Corbusier’s radiant city as described by James Scott, the ideal Modernist state is “a lyrical marriage between Cartesian pure forms and the implacable requirements of the machine.”

death star power trench plans

In fact, states are more the Millennium Falcon. States are not built de novo to fill political vacuums. Instead, like the hot rods Lucas celebrates in American Graffiti, states are motley assemblages of formal and informal institutions cobbled together on the go. Many of those parts were designed at some time for some purpose, but not necessarily the one for which they’re actually being used. Change often occurs at the margins in response to specific problems that are solved imperfectly. Instead of a spacecraft carefully designed to execute of series of specific tasks, we get a mostly functional “hunk of junk.”

Millennium Falcon

The builders of these assemblages are more craftsmen than engineers. They don’t have the luxury of time, the know-how, or the resources to build a new ship from the ground up, so they tinker at the margins as they go and hope the whole thing doesn’t disintegrate out from under them. Not trivially, they also have some attachment to the idiosyncrasies and imperfections of the system they’ve cobbled together. Even if they could scrap it for a newer and cleaner model, they probably wouldn’t want to. The Millennium Falcon may be a hunk of junk, but to its pilot, it’s “the fastest hunk of junk in the galaxy.”

The aesthetic of a “used future” George Lucas pioneered for American moviegoers in Star Wars is echoed in Paul Pierson’s call for social scientists to situate politics in time:

Contemporary social scientists typically take a snapshot view of political life, but there is often a strong case to be made for shifting from snapshots to moving pictures. This means systematically situating particular moments (including the present) in a temporal sequence of events and processes stretching over extended periods.

Moving pictures of the Hollywood kind often help us see the world differently. Lucas’ vision of a “used future” reminds us that the present is really an accretion of many pasts, and we shouldn’t ignore those origins in our theories of political development.

Prometheus Bungled

[NB: It’s Saturday morning, and I’m at the tail end of a week’s vacation, so I’m digressing from the usual fare to write about a movie. Apologies for the distraction.]

The “science” part of science fiction is important. Good science fiction employs a reality we recognize, but with specific things tweaked or altered. These alterations serve as a way to explore how those things shape our present reality, and how the future—or some counterfactual past or present—might differ. As Philip K. Dick suggests, good science fiction isn’t fantasy, it’s futurism.

For this to work, the alterations have to be scientifically plausible. Fusion reactors are interesting, because they connect to rules and theories we know and are therefore imaginable. Magic energy cubes, on the other hand—things like the Tesseract in the ho-hum Avengers franchise—are frustrating, because they aren’t connected to our lived existence. (Equally important from a narrative perspective, magic energy cubes don’t impose any constraints on the action that we can recognize ahead of time and therefore can’t serve as sources of tension and suspense.)

The elegance of science lies in its simplicity. The basic components and rules are simple; the complexity we inhabit emerges from their interaction and combination. All the things we see around us are composed of few dozen elements, which are, in turn, composed of a handful of subatomic particles. Combinations of simple geometric elements produce dazzling fractal forms that appear over and over again in nature, often in very different contexts. Sentience emerges from clusterings of relatively simple cells—or, perhaps, circuits.

Starting from these premises, I was frustrated and disappointed by Prometheus, the recently released prequel to Ridley Scott’s 1979 science-fiction masterpiece, Alien. I was only nine years old when Alien first came to theaters, so I didn’t much appreciate it at the time. As a Star Wars fan, I was happy to go to any movie set on a spaceship, but I had to head for the lobby after the “indigestion” scene in the galley and only saw bits of the rest of the film through the doors at the back of the theater.

Going back to Alien as an adult, though, I’ve become a big admirer. I don’t much like horror films. What makes Alien so appealing to me now is its smart application of one of one of science’s most elegant theories—evolution—to answer a profound philosophical question. On Earth, humans seem (to ourselves, anyway) to stand at the top of the pyramid, so powerful that we can build ships to expand our reach into space to scavenge and prey elsewhere. As soon as we do that, though, we also expand the circle of our existence into a wider universe in which other creatures might occupy similar niches on other planets. In this wider circle, predator can become prey, parasite can become host, and natural selection can take new turns. Humans become more evidently animal and are clearly maladapted to many of the new environments in which they might land. The stories we tell about our special place in the universe crumble in our first encounters with species from its other corners. The common denominator for life in this wider perspective is the urge to survive and reproduce, and on this larger scale, we may not be so great at it after all. If the question is “What makes us humans special?”, Scott’s answer is a resounding “Not much.”

And then we get Prometheus. Where Alien relied on the elegance of a simple scientific idea, Prometheus injects narrative complexity and scientific ambiguity in a plot that grabs from biology and evolution but ultimately rejects them. The long evolutionary line connecting humans to primordial goo on Earth is cut, replaced with the quasi-mystical idea of a process begun much more recently by Engineers whose own origins are left unexplained. The simple biological imperative of survival through predation is supplanted by a worn and ambiguous political story about warring societies and weapons of mass destruction.

Good science fiction is really hard to write. Most attempts I’ve seen or read have foundered or failed because they have interwoven so many inventions and ideas that we can’t follow the threads back to our lived existence. The core of science is the experimental method, and one of the guiding principles of that method is control. To answer questions about cause and effect, you focus your gaze by varying the hypothesized cause while holding confounding factors constant.

Good science fiction often does the same. Alien puts humans in deep space, hypothesizes about what might have evolved in those strange (to us) environments, and runs one trial of an experiment to see which species would survive by dominating those encounters. (In repeated iterations of this experiment, my money’s on the aliens, the lazy digressions of the sequels notwithstanding.) Prometheus replaces this elegant design with mysticism and hand-waving more characteristic of religion than science, and it’s a much poorer film for it.

Top 5 Albums to Play When Writing Stats Code

I like to listen to music when I’m working, but not all music works equally well for all tasks. When I’m writing prose, for example, I constantly get distracted if I play music with a narrative or clever wordplay in the lyric; the words in the song keep pushing out the words in my head. When I’m writing (or, more often, debugging) stats code, I find that certain pieces of music actually help me get and stay in a nice flow. So, in the brilliant tradition of High Fidelity, here are my Top 5 records to play when writing stats code:

5. Cocteau Twins, Heaven or Las Vegas. This one has words, but god bless you if you can understand what the heck she’s singing. All I hear is a lush run of sound with a fantastic bass line.

4. Beethoven’s piano sonatas. I usually listen to Volume II of the three-volume, many-disc Glenn Gould edition, which gives me a few hours of uninterrupted music. My favorite piece is the allegro section of the opening “Pastorale,” which always makes me feel like I’m meandering down a country lane on horseback on a sunny day.

3. Kronos Quartet Performs Philip Glass. I wonder if Glass’s “repetitive structures” help turn on parts of the brain that deal in math and logic. In any case, this recording of a few of his string quartets feels romantic to me in spite of its modern structures, and it never gets old to my ears.

2. Daft Punk, Tron:Legacy Soundtrack. I know, I know, it’s pretty much a cliché, but if you can tolerate the snippet of Jeff Bridges’ character geeking out about The Grid, this album kills for code-writing.

1. Sviatoslav Richter, The Authorised Recordings of J.S. Bach compositions for piano. Put on good over-ear headphones, and listening to this is like slipping into another world where there’s just one long, unbroken thread of gorgeous sound. It’s like coding in a vacuum.

“His Colleagues Had Ambition”

More than the American rebels, more than their enemies in Europe, more than the angry commoners demonstrating on the streets, the Prime Minister feared his ministers, the men and women who sat at his table and drank his wine. It was a justified anxiety–his colleagues had ambition.

That’s from Ptolemy’s Gate, the third book in Jonathan Stroud’s Bartimaeus Trilogy, which I’m reading right now with my 12-year-old son. I really like the books, in no small part because–as the quote shows–Stroud gets politics. We might not see a lot of elected heroes in fantasy fiction for kids, but that doesn’t mean we don’t ever get a political education.

Of Kid’s Books and Kings (a.k.a. Where’s the Elected Official?)

Why are the governments in popular fiction for kids and young adults almost never elected?

I have two boys, now in third and sixth grades. My wife and I still read to them every night, so I’m familiar with a lot of the popular kids’ fiction of the past 10 years. In every major book or series I can think of–Harry Potter, the Percy Jackson books, The Chronicles of Narnia, the Gregor the Overlander series–political authority either resides in monarchies or is controlled by some other self-selected group of elites, often with special powers.

A few of the books we’ve read have raised tough questions about these authoritarian arrangements and the injustices they entail. Right now, for example, I’m reading the excellent Bartimaeus trilogy to my sixth grader and appreciating the story of popular resistance against a tyrannical aristocracy of greedy magicians. Most of the time, though, poor governance is implicitly blamed on flaws in the character of individual leaders. Villains bring us down, and heroes make things right. Institutions, it seems, are irrelevant.

As a scholar of democratization and a liberal by political philosophy, I really don’t like the message this pattern sends to my kids. Governance is a very hard and perpetual problem, and the parade of gods, kings, and magicians traipsing through kids’ fiction reinforces the authoritarian fantasy that benevolent dictators offer an elegant solution. I realize that fiction isn’t meant to mimic reality, and I understand how these struggles between powerful beings of good and evil make a terrific scaffolding for storytelling. Still, I can’t help but wonder how this steady diet of government by kings and wizards prepares kids to make sense of the politics they will encounter as they grow up.

  • Author

  • Follow me on Twitter

  • Follow Dart-Throwing Chimp on WordPress.com
  • Enter your email address to follow this blog and receive notifications of new posts by email.

    Join 13,612 other followers

  • Archives

%d bloggers like this: