Commentary | 17 December 2016

How to pick your predictors

Sam Roggeveen writes in the Australian Financial Review on how to choose your pundits. Photo: Flickr/Will Fisher

  • Sam Roggeveen

Sam Roggeveen writes in the Australian Financial Review on how to choose your pundits. Photo: Flickr/Will Fisher

  • Sam Roggeveen

It is customary at this time of year for editors to give political pundits space to offer our predictions for the coming 12 months. You are entitled to ask why. Collectively, after all, us pundits got 2016, the year of Trump and Brexit, badly wrong. And they weren't anomalies. There is solid evidence gathered over many years that predictions by political experts are only slightly more reliable than tossing a coin.

So why the seemingly insatiable appetite for predictions? In 2017, many of the same pundits who messed up 2016 will line up again to tell you who will win the French election, whether Trump will start a trade war with China, whether Assad will survive in Syria, and more. Why does anyone even pay attention?

One reason is that we all crave certainty about the future, and we will look for it even if it is based on flimsy evidence or a poor record of prediction.

Another reason is that, although predictions aren't a reliable guide to the future, they do give you information about the state of the world today. You can draw an analogy with science fiction: the original Star Trek series got a lot wrong about future technology (although set in the 23rd century, much of the technology imagined in the series has already been overtaken by real life), but it tells you a lot about the late-1960s, when Star Trek was made. There's a clear Cold War subtext in the series, and echoes of American debates about sex and race.

Of course, predictions also have a practical purpose for businesses and governments, who have to make big financial decisions with long time horizons. Think of defence equipment, for instance: this year the government announced a project to build 12 new submarines in South Australia – the 12th boat could still be in service by 2070! Or consider Airbus, which in the mid-1990s made what only now looks to be a dicey bet on its super-sized wide-body jet, the A380.

In both cases, massive financial commitments were made on the basis of forecasts. Yet the political scientist Philip Tetlock, who has made a career out of exposing the fallibility of expert predictions and showing how it can be done better, says "there is no evidence that geopolitical or economic forecasters can predict anything 10 years out beyond the excruciatingly obvious". Even five years out, the accuracy of such forecasts "declines towards chance".

How to determine reliability

All of that said, what should you look for in a forecaster to help you determine their reliability? Here's a non-exhaustive list.

Look for pundits prepared to take the professional risk of being ostracised. Picking a Hillary victory was low-risk for a professional pundit, because even if you were wrong, you would be in good company. Wrongly predict a Trump win and you look like a goose before your peers and the world.

Be specific: if a pundit is unwilling to put a date on their prediction, or tell you in some detail about how certain they are, it means they don't have much confidence in their forecast and are looking for an escape hatch.

Beware of news junkies: ignorance does not make for better predictions – good forecasters should look for new data and be open to revising their beliefs when information changes. But be wary of pundits who cite the news media too much, because journalism tends to focus on events which are dramatic, or awful or just new. It's called "the parochialism of the present", and it leads us to put too much weight to recent events over longer trends.

Beware of over-confidence: if a pundit is highly committed to a belief, they will find it harder to revise in light of new evidence.

Past performance is not an indicator of future outcomes: it doesn't matter whether a particular pundit called the US election right, because that could have been a fluke. All that matters is how they got it right. Is there any indication that this pundit has applied specific techniques or thinking skills to get the right answer?

Lastly, the best way to improve the political prediction game is to hold pundits to account. That doesn't mean punishing or shaming pundits who get it wrong, but if they are unwilling to own their mistakes, you should back away slowly. Look instead for pundits who are prepared to admit their errors and work on them.