This came out a few months ago, but I wanted to point to Clive Thompson's interesting article on Bruce Bueno de Mesquita's Predictioneer's Game, which contains some good observations.
First, it helps his track record that de Mesquita works on very specific problems– e.g., how would Company X respond to a hostile takeover attempt by Company Z, how will Country A respond. As Thompson notes,
It is difficult to verify how accurately Bueno de Mesquita’s model performs in corporate settings because most firms are loath to discuss his work for them. For most of the cases we discussed, Bueno de Mesquita would disclose details of the negotiation but wouldn’t name the firms in question. In other cases, clients would talk to me and praise Bueno de Mesquita’s work for them, but they would not disclose verifiable details of specific negotiations.
While Bueno de Mesquita has published many predictions in academic journals, the vast majority of his forecasts have been done in secret for corporate or government clients, where no independent academics can verify them. “We have no idea if he’s right 9 times out of 10, or 9 times out of a hundred, or 9 times out of a thousand,” Walt says. Walt also isn’t impressed by Stanley Feder’s C.I.A. study showing Bueno de Mesquita’s 90 percent hit rate. “It’s one midlevel C.I.A. bureaucrat saying, ‘This was a useful tool,’ ” Walt says. “It’s not like he’s got Brent Scowcroft saying, ‘Back in the Bush administration, we didn’t make a decision without consulting Bueno de Mesquita.’ ” Other academics point out that rational-actor theory has come under increasing criticism in recent years, as more evidence accumulates that people make many decisions irrationally.
Then there's this observation:
Those who have watched Bueno de Mesquita in action call him an extremely astute observer of people. He needs to be: when conducting his fact-gathering interviews, he must detect when the experts know what they’re talking about and when they don’t. The computer’s advantage over humans is its ability to spy unseen coalitions, but this works only when the relative positions of each player are described accurately in the first place. “Garbage in, garbage out,” Bueno de Mesquita notes. Bueno de Mesquita begins each interview by sitting quietly — “in a slightly closed-up manner,” as Lapthorne told me — but as soon as an interviewee expresses doubt or contradicts himself, Bueno de Mesquita instantly asks for clarification.
“His ability to pick up on body language, to pick up on vocal intonation, to remember what people said and challenge them in nonthreatening ways — he’s a master at it,” says Rose McDermott, a political-science professor at Brown who has watched Bueno de Mesquita conduct interviews. She says she thinks his emotional intelligence, along with his ability to listen, is his true gift, not his mathematical smarts. “The thing is, he doesn’t think that’s his gift,” McDermott says. “He thinks it’s the model. I think the model is, I’m sure, brilliant. But lots of other people are good at math. His gift is in interviewing. I’ve said that flat out to him, and he’s said, ‘Well, anyone can do interviews.’ But they can’t.”
And this caution:
it is not so easy to attract clients. This is partly because most of their clients — especially the C.I.A. — swear them to secrecy. (And perhaps also because, as Roundell says, “Bruce and I are . . . terrible salespeople.”) But they have also faced a barrier that’s almost existential, a skepticism that computer models can truly predict the outcome of negotiations. The C.I.A., for example, built its own replica of Bueno de Mesquita’s original forecast model, but as Feder noted in his report, “the vast majority of analysts” didn’t use it because it seemed too rigid. They thought of analysis as reading and pondering until they had an aha! moment — not feeding data points into a computer model and waiting to see what comes up.