Economist Brian Easton explains why most forecasts never eventuate, often with cringe-worthy explanations. Part of the issue is the media focus on unqualified sources

This is a re-post of an article originally published on pundit.co.nz. It is here with permission.

The research evidence is that humans, even experts, are poor forecasters.

Noise, a book by three eminent experts, brings together the melancholic evidence. For instance, US judges (who are sentencing people to life-changing circumstances) are very poor predictors of the flight-risk of those applying for bail. Algorithms (mechanical statistical models) with exactly the same data information (they don’t see the bail applicants of course) outperform the judges. Even more extraordinarily, if the algorithms are trained on the judges’ flight-risk predictions, they still outperform the judges on actual outcomes.

The book is replete with such examples together with a dissection of how it happens and how forecasting may be improved. It may be a bit technical for many readers, but I would rank it high on any reading list for statisticians or social scientists using statistical methods.

Most of the researched examples are based on data from panels – often huge ones, 750,000+ cases in the study just mentioned. There is very little on forecasting the economy, which involves time series and fewer data points. (Yet, I would still recommend the book as necessary reading for economists.) But we know from other research that the conclusions apply to economics forecasters too.

In particular, there are simple mechanical statistical (autoregressive econometric) models which usually outperform economic forecasters. Interestingly, when economists add economic theory to the mechanical forecasters they perform worse. The best-known examples are the DSGE (dynamic stochastic general equilibrium) models, which did so badly during the Global Financial Crisis. (Admittedly, the assumptions tend to be neoliberal ones, which I judge implausible.) Yet the DSGE models are so fashionable some economists continue to use them.

Why then maintain expensive forecasting teams (or construct the more sophisticated models) when cheap autoregressive ones will outperform them? There seem to be three reasons.

First, if you are the Reserve Bank or the Treasury you are not merely forecasting the economy but advising on policies which may influence the economy. An autoregressive model has no capacity to do this.

Second, a unique economic crisis is the one time when the autoregressive models do not work well because their data base has not anything in its memory like what is happening. Instead, relying upon practical theory, drawing on past episodes and those from other economies, and making ad hoc adjustments, helps to give a better a better idea of what is happening. Such crises only happen once every decade or so, but one needs a working team when they do. A standing economic forecasting team may be like maintaining a fire brigade when there are no fires.

The third reason can be illustrated by the story of pilotless planes. There is still room in the cockpit for a pilot next to whom sits a dog. The pilot is to give the passengers confidence; the dog is to bite the pilot if they touch the controls.

Similarly, the media wants a ‘pilot’, an economist, to give the impression of someone in charge. Sure, they could have a desktop computer running an autoregressive model, which would regularly update the forecasts, but the audience would lack confidence in the prediction unless there was a human face fronting it.

Unfortunately, there is no dog so we can get weird and misleading predictions. This is well illustrated in a recent RNZ Mediawatch, ‘The problem with economists' forecasts’ which would make much of the economics profession cringe.

(The problem is not entirely an economics professional one. The media often choses those who have not a lot of standing/reputation in the profession, but are presentable. Often they are platitudinous, offering nothing but the conventional wisdom which the journalist already believes.)

Let me add to Mediawatch’s cringe list. On 20 March 2021 Steven Joyce announced in the New Zealand Herald that ‘GDP figures show the true cost of Fortress NZ. Shrinking economy won’t be turned around until we open our doors and reconnect with the world.’

(Joyce is not an economist – he’s a businessman – but he is a retired (National) Minister of Finance. He is the first since Robert Muldoon (who retired from the position 37 years ago) to continue to involve himself in such detailed public comment on the economy. Perhaps, the remaining six knew that while they appeared impressive in the job, they depended upon Treasury advice and without it they would look a bit pathetic. Joyce may feel the public performance of National’s economic leadership team requires a bit of stiffening.)

Joyce’s remarks were a response to the just reported contraction of the economy in the 2020 December quarter of 1 percent. The media was hit by a widespread panic. Wiser heads would have thought ‘the figure was surprising but that the economy was still recovering from the first (March-June 2020) Covid lockdown and one might expect the estimates to be all over the place. Let’s see what happens.’ They would have looked at the previous September 2020 quarter and observed a massive 14.1 percent increase – a recovery from the bad June quarter which was dominated by the Covid lockdown. But that is not ‘news’ (or do we mean entertainment?).

The thoughtful would not have been surprised that the March 2021 quarter figure was a 1.6 percent increase on the December 2020 one. To be frank, the economy is still all over the place; I would have written that before the current (August 2021–) lockdown.

Joyce responded quite differently on 7 August 2021: ‘Overheated heated economy about to burn borrowers. Overdone Covid recovery actions fanning flames of inflation.’ He did not apologise for his earlier remarks about the economy sliding downhill. Possibly he thinks that in the intervening three months Grant Robertson has ‘open[ed] our doors and reconnect[ed] with the world’. One doubts that is how Robertson thinks of his achievements over that time.

The lesson here is the same as from Mediawatch. The commentariat are not just poor predictors of the economy, but their predictions bounce around and they rarely apologise. Sometimes one wonders whether they are grinding a political or ideological axe rather than making a technical assessment. (Shamubeel Eaqub shows wisdom above many of his peers when he announced that he has given up forecasting due to its futility. ‘I don't do forecasting anymore because I know that I don't know how to forecast.’)

In one sense then, public economic forecasting is a part of the entertainment industry. But there is another. As Mediawatch concluded:

The economy is vast, complicated, and confusing, and as a result easy to be wrong about. But [economists’] predictions, often made with unwarranted, unshakeable confidence, have real impacts on people’s financial futures. Given the amount they’re wrong, it’s worth adding a disclaimer before printing or broadcasting their opinions: this could be right, but definitely don’t bet your house on it.

Journalists should read Noise too. Perhaps there would then be less of it in the media.

Noise: A Flaw in Human Judgment by Daniel Kahneman, Olivier Sibony, Cass R. Sunstein.

Brian Easton, an independent scholar, is an economist, social statistician, public policy analyst and historian. He was the Listener economic columnist from 1978 to 2014. This is a re-post of an article originally published on pundit.co.nz. It is here with permission.

* Comments have been temporarily disabled due to a website upgrade to allow the new version to catch up with the rapidly moving comments sections. Comments will be turned back on very soon and we thank you for your patience. - Interest.co.nz team