I’ve been working on a think-piece on the future of futures work. (It’s an expansion of questions I started asking in my piece on design and futures.) It’s organized around a simple question: If you were to invent a discipline of futures and forecasting today, organized to deal with today’s problems, and drawing on current science, what would it look like? Would be it be just like the field today? Would it look for weak signals, produce roadmaps and scenarios, and seek to influence strategy and policy?
I suspect the answer is no. No, I’m confident- using the term as Robert Burton would warn it should be used-- that the answer is no. Now I’m trying to explain where I think the field will, or ought, to go.
One of the things I’m thinking through is the role of expert knowledge and accountability in futures work. We claim to be experts about a bunch of things, most notably about how to think about the future in ways that can better inform the present. But the work of Philip Tetlock (which I’ve mentioned before) suggests that claims of expert knowledge, particularly when it comes to dealing with the future, are highly suspect.
Teltock’s argument is nicely summarized by Louis Menand in a New Yorker review:
It is the somewhat gratifying lesson of Philip Tetlock’s new book, “ Expert Political Judgment: How Good Is It? How Can We Know” (Princeton; $35), that people who make prediction their business—people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtables—are no better than the rest of us. When they’re wrong, they’re rarely held accountable, and they rarely admit it, either. They insist that they were just off on timing, or blindsided by an improbable event, or almost right, or wrong for the right reasons. They have the same repertoire of self-justifications that everyone has, and are no more inclined than anyone else to revise their beliefs about the way the world works, or ought to work, just because they made a mistake. No one is paying you for your gratuitous opinions about other people, but the experts are being paid, and Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert’s predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote. Our system of expertise is completely inside out: it rewards bad judgments over good ones.
Tetlock got a statistical handle on his task by putting most of the forecasting questions into a “three possible futures” form. The respondents were asked to rate the probability of three alternative outcomes: the persistence of the status quo, more of something (political freedom, economic growth), or less of something (repression, recession). And he measured his experts on two dimensions: how good they were at guessing probabilities (did all the things they said had an x per cent chance of happening happen x per cent of the time?), and how accurate they were at predicting specific outcomes. The results were unimpressive. On the first scale, the experts performed worse than they would have if they had simply assigned an equal probability to all three outcomes—if they had given each possible future a thirty-three-per-cent chance of occurring. Human beings who spend their lives studying the state of the world, in other words, are poorer forecasters than dart-throwing monkeys, who would have distributed their picks evenly over the three choices.
Tetlock also found that specialists are not significantly more reliable than non-specialists in guessing what is going to happen in the region they study. Knowing a little might make someone a more reliable forecaster, but Tetlock found that knowing a lot can actually make a person less reliable. “We reach the point of diminishing marginal predictive returns for knowledge disconcertingly quickly,” he reports. “In this age of academic hyperspecialization, there is no reason for supposing that contributors to top journals—distinguished political scientists, area study specialists, economists, and so on—are any better than journalists or attentive readers of the New York Times in ‘reading’ emerging situations.” And the more famous the forecaster the more overblown the forecasts. “Experts in demand,” Tetlock says, “were more overconfident than their colleagues who eked out existences far from the limelight.”
The obvious questions are, how relevant is this work to what we futurists do? And are our current attempts to explain that no, we can’t predict the future but our work is still valuable, sufficient in the light of work like Tetlock’s?
