Via Overcoming Bias, I found this great review by Philip Tetlock in The National Interest of several futures-related books I’ve been meaning to read: Ian Bremmer and Preston Keat’s The Fat Tail; Bruce Bueno de Mesquita’s The Predictioneer’s Game; and George Friedman’s The Next 100 Years. (My ongoing near-obsession with Tetlock’s work is well-documented in this blog, and in other things I’ve written.)

It’s one of those reviews that, yes, talks about the books, but really treats the books as a launching-point for talking about other cool things (in other words, it’s the kind of book review I like to write).

In this case, there are two things that jump out at me. (Incidentally, Tetlock’s verdict is that Frieman and Bremer/Keat aren’t very good, but de Mesquita is worth grappling with.) The first is that it includes a nice precis of one of the core arguments of his 2005 book, Expert Political Judgment.

A good deal of research indicates that some ways of thinking (“cognitive styles”) do translate into somewhat more correct forecasts. When we score the accuracy of thousands of predictions from hundreds of experts across dozens of countries over twenty years, we find the best forecasters tend to be modest about their forecasting skills, eclectic in their ideological and theoretical tastes, and self-critical in their analytical styles. Borrowing from philosopher Isaiah Berlin, I call them foxes—experts who know many things and are not finicky about where they get good ideas. Paraphrasing Deng Xiaoping, they do not care if the cat is white or black, only that it catches mice.

Contrast this with what I call hedgehogs—experts who know one big thing from which likely future trends can be more or less directly deduced. The big thing might be any of a variety of theories: Marxist faith in the class struggle as the driver of history or libertarian faith in the self-correcting power of free markets, or a realist faith in balance-of-power politics or an institutionalist faith in the capacity of the international community to make world politics less ruthlessly anarchic, or an eco-doomster faith in the impending apocalypse or a techno-boomster faith in our ability to make cost-effective substitutes for pretty much anything we might run out of.

What experts think—where they fall along the Left-Right spectrum—is a weak predictor of accuracy. But how experts think is a surprisingly consistent predictor. Relative to foxes who are less encumbered by loyalties to an all-encompassing worldview, hedgehogs offer bolder forecasts and, although they hit occasional grand slams, they strike out a lot and wind up with decidedly poorer batting averages.

The second is his suggestion about how to begin to deal with a problem that’s central to the field: that we don’t really keep track of either how accurate our forecasts are (which is something that clients always want to know).

How then can we produce the most accurate forecasts? The answer is not obvious: right now all we can say confidently is that no one can be 100 percent confident about which approach would win if we were to run a series of level-playing-field forecasting tournaments stretching out to, say, 2020.

But if the market seems largely indifferent to our plight, who might rescue us? There is one potential savior on the horizon: a big institutional purchaser of forecasting services that has the financial clout and technical-support staff ready to run forecasting tournaments that would shed light on the relative performance of competing approaches—a big player that also has powerful incentives to discover superior analytical strategies, for even small improvements in its prediction accuracy can translate into billions of dollars and millions of lives saved. And that player is the Office of the Director of National Intelligence.

Unfortunately, although intelligence agencies have been heavy buyers of forecasting services, they have not used their massive purchasing power to their full advantage. They have allowed the diverse interest groups in the intelligence community to choose freely from private-sector forecasting products. On the one hand, this is commendably open-minded. On the other hand, there has been no integrative effort to assess the relative value added of each product. Indeed, intelligence agencies seem as allergic as private-sector forecasters to being held accountable to public accuracy metrics.

I like Tetlock’s suggestion, but personally I think it’s just as important to try to assess how useful our work is- whether it be delivered in the form of roadmaps, scenarios, provocative pieces in industry magazines, or wherever- as well as how correct we are. Futurists constantly argue that utility is the real metric by which we should be judged: IFTF president emeritus Bob Johansen likes to say that you should never trust anyone who says they can predict the future, especially if they’re from California. But too often we don’t have the fine detail about how our work gets used by clients, how it informs their decisions, and how we can adjust it to be more useful.

This is a shame, because the field is capable of evolving very quickly: when I was at IFTF, we developed all kinds of new tools and media for both doing research and communicating our ideas. But we tended to have to do so with less rigorous knowledge about how our earlier work had been received, interpreted, analyzed, etc. than I would have liked.

The other problem that the “you can be useful without being right” argument is it leaves unresolved two big questions.

First, how wrong can be you be and still be useful? Can you get the future totally wrong and still be useful to clients? To put it less provocatively, could a good futurist take a forecast or scenario that was essentially generated at random, and create something useful for clients? Is there a point at which error overwhelms utility? (Or conversely, could it be that erroneous forecasts are actually more useful? More counterintuitive things have proved true in our time.)

Second, who’s responsible for the work being useful? In one sense, clients are always responsible for generating most of the value from a prediction: if I say something exactly right about the future and a client doesn’t act on it, then they’ve lost the opportunity to create value from my prediction. If utility can be divorced from reality (or future reality), does the burden shift entirely to the client (or reader) to find or create the value in a (right or wrong) forecast?

As a once and kind-of current academic, I’m certainly sympathetic to the idea that our work is like teaching: we can help guide our students or clients, offer access our craft and wisdom, but they have to do the work of understanding and learning themselves. But at the very least, I’d like to better understand how these social contracts are supposed to work, and better yet, understand how they actually do function- to have a better sense of how clients create utility from our work, and what we can do to increase that utility.