Experts, chimps and prediction

Posted by on Jan 6, 2009 in Uncategorized | 8 Comments

Chimp
Don't know about you but I like a good expert: someone who knows stuff and can tell you what's going to happen

Sadly, as Philip Tetlock's study of several thousand specific predictions over a period of years by 284 individuals who make their living as experts shows, expert predictions are rather less valuable than we generally assume:

"The experts were less accurate in their forecasts than a control
group of chimpanzees choosing entirely randomly would have been. Even
specialists in particular narrow fields were not significantly more
successful than reasonably informed laymen. "We reach the point of
diminishing marginal predictive returns for knowledge disconcertingly
quickly," Tetlock suggested. "In this age of academic
hyperspecialisation, there is no reason for supposing that contributors
to top journals – distinguished political scientists, area study
specialists, economists and so on – are any better than attentive
readers of the New York Times in 'reading' emerging situations."

Further,
the more certain the forecaster was, the more likely his judgment would
be awry, scientific proof that "the best lack all conviction, while the
worst are full of a passionate intensity""

                [from this piece in the Observer]

via Ian @ Marbury

8 Comments

  1. Dan Thornton
    January 6, 2009

    Do you think it makes a difference that the study was concentrated on politics and economy?
    (Not trying to suggest that other experts are more valid, or would be right more necessarily!).
    Does this mean that the role of an expert is actually best suited to explaining the past, rather than predicting the future?
    And rather than predictions, would the experts conform to the idea of the successful entrepreneur as someone who happens to spot failure earlier, rather than someone who can accurately predict what will work?

  2. Libby Anderson
    January 6, 2009

    Wow, that’s both distressing and reassuring. Interesting.

  3. Mark
    January 7, 2009

    Dan – think this is likely to be valid for all areas and that (as you suggest) points to expertise being much better at describing the past than predicting the future.
    Interesting point re entrepreneurs: neuroscientist David Ingvar suggests that these kind of minds are constantly scanning for alternatives to what is and this makes it easier for them to embrace opportunities. Not sure the expert mindset is the same as this.
    What do you think?

  4. Simon T Small
    January 7, 2009

    Hey guys,
    When I land on your site, I get this weird popup asking me for my username/password for twitter, looks like some kind of phishing thing… but probably just some broken javascript or something??
    Here’s the screenshot I got
    http://www.flickr.com/photos/simontsmall/3176684292/

  5. Dan Thornton
    January 8, 2009

    Mark,
    Thanks for the reply, and I definitely think it’s difficult to comment without knowing the methodology and examples of the book….
    I think the findings might vary slightly depending on the experts in question – for instance, how many of the experts picked for television are the most respected experts in their field – and how many are people who have some expertise but are also photogenic for example, or were put on the spot for a soundbite on something they may not have said with more time and thought?
    I think there would probably be a difference if it looked at expertise in programming or chemistry for example, but whether that would be enough to beat the chimps would be another matter!
    And I agree that the entrepreneur mindset is different to the expert mindset – perhaps the two need to be taught in combination, rather than the traditional academic idea that you become and expert and then know everything with certainty?

  6. Dan Thornton
    January 8, 2009

    Also just spotted this about how experts don’t tend to actually follow their own public advice:
    http://online.wsj.com/article/SB123093692433550093.html
    And if experts can’t predict the future – this has pretty big implications for politics?

  7. Mark
    January 8, 2009

    Awesome, Dan.
    And yet we persist in our expertmania, don’t we? In just about every way?

  8. Simon
    January 9, 2009

    It’s nice to think that there is a point of intellectual diminishing returns and I can make judgements from reading the Economist and Newsweek; but with Market research, companies go for their own, primary research, rather than reading relying on secondary research that has wholes and gaps in. Would this suggest companies should be greater risk takers and less primary research dependent?