By Rodney Dickens
To properly review the accuracy of the consensus economic forecasts we need to look at what the economic forecasters were predicting versus what happen over the forecast period.
NZIER gets predictions from the economic forecasters for the current and subsequent March years.
We don’t believe economic forecasters can provide much insight beyond the year ahead, so our review focuses on what the economic forecasters were predicting roughly a year ago and what transpired.
To be useful from a business risk management perspective the economic forecasters need to be able to give advance warnings of upturn and downturns over the subsequent year or so.
The black/thick line in the chart shows rolling annual average GDP growth based on the quarterly GDP data (i.e. the percentage growth in the volume of economic activity or GDP for the last four quarters compared to the same four quarters a year earlier).
Last month Stats NZ reported that annual GDP growth for the 2010/11 March year was 1.5%.
When NZIER surveyed the economic forecasters a little over a year ago in June 2010 the average prediction for GDP growth for the 2010/11 March quarter was 3.2%. The lowest prediction by any of the economic forecasters was 2.5% and the highest prediction for GDP growth in the 2010/11 March year was 3.9%.
The economic forecasters get some marks for getting the direction right (i.e. that there would be positive rather than negative economic growth), but they were on average significantly too positive about economic growth (i.e. the outcome was slightly under half of what they were predicting and was outside the range of what all the forecasters were predicting).
The major earthquakes in Canterbury explain part of the deviation and economic forecasters can’t be expected to predict earthquakes, but their overly optimistic predictions were more because they were overly optimistic about underlying economic growth and especially about residential building prospects, as discussed below. Also see below for the quantified scoring.
Reviewing the Consensus Forecasts for residential building activity
The black/thick line in the chart shows the rolling annual level of residential building activity, which includes new buildings and alterations and additions to old buildings. 
It is measured in dollar terms, but the level of activity is measured in constant price terms (i.e. based on prices in the 1995/96 March year), which means the chart effectively shows the volume of residential building activity.
On average the economic forecasters were in June 2010 predicting $5.8b of residential building activity in the 2010/11 March year (in 1995/96 price terms) when the outcome was 14% lower at $5b.
The most optimistic ofthe forecasters was predicting $6.2b and the most pessimistic was predicting $5.3b. In this instance the economic forecasters didn’t even get the direction right so none of them were much use in providing useful insights for firms and individuals impacted by cycles in residential building (see below for the quantified scoring).
The underperformance of residential building activity compared to what the economic forecasters were predicting had little to do with the negative impact of the Canterbury earthquakes, with it being mainly due to the economic forecasters not understanding what was going on in the housing market and especially not realising the major negative impact of the property tax changes announced in the May 2010 Budget.
The major relevance of the property tax changes to residential building activity and overall economic growth went almost unnoticed by the economic forecasters.
Being overly optimistic about the outlook for residential building played a significant part in the economic forecasters being too optimistic about economic growth prospects. The pivotal role housing market activity and house prices play in economic cycles was covered in Chapter Two of the How the Economy Works booklet (see http://sra.co.nz/pdf/PivotalHousing.pdf).
If the economic forecasters can’t get the direction of residential building activity right they have little chance of providing advance warnings of economic upturns and downturns or of predicting the outlook for interest rates (see below).
By contrast, we provided clients with advanced warnings of the downturn in housing market activity in 2010 and we correctly warned that economic growth would be significantly weaker than predicted by the RBNZ and that interest rates would not increase anywhere near as much as the economic forecasters were predicting, although economic growth turned out even weaker than we warned (see http://sra.co.nz/pdf/EconomicTrackRecord.pdf for a review of our forecasting track record in 2010).
Reviewing the Consensus Forecasts for interest rates
NZIER surveys the economic forecasters’ predictions for the 90-day bank bill yield and the 10-year government bond yield. The 90-day bank bill yield is the benchmark short-term wholesale interest rate that both reflects the market’s assessment of the near-term outlook for the OCR and plays a major part in driving the short-term interest rates faced by borrowers (e.g. the floating and short-term fixed mortgage interest rates and the base business lending rates set by banks).
The 90-day bank bill yield is the only interest rate the RBNZ forecasts, with these predictions being proxies for OCR forecasts.
The 10-year bond yield is of more relevance as a benchmark of the longer interest rates available to investors and as the main measure of the 'risk free' interest rate used in assessing the value of shares and the like.
In general the outlook for the 90-day bank bill yield is of relevance to more people, while it has implications for the outlook for the 10-year bond yield and for medium-term and longer-term interest rates more generally. From one perspective, the market’s expectation of the OCR over the next 10 years determines the current 10-year bond yield. From another perspective, the 10-year bond yield changes in response to changes in the market’s expectation about the OCR and to changes in international and especially US bond yields. But the two perspectives should add up to the same outcome for the 10-year bond yield.
The black/thick line in the chart shows the rolling annual average of the 90- day bank bill yield. The 90-day bank bill yield has largely drifted sideways over the last year at just under 3%. The average prediction of the economic forecasters survey by NZIER in June 2010 was that it would average 3.7% in the 2010/11 March year and 5.3% in the 2011/12 March year (see the blue line).
If I interpolate between these two predictions it implies that the forecasters were on average predicting that the rolling annual average of the 90-day bank bill would be around 4.4% now compared to an outcome of 2.9%. The minimum and maximum forecasts lines in the chart show that all of the forecasters surveyed by NZIER in June 2010 were predicting significant increases in the 90-day bank bill yield over the subsequent two years.
In terms of their 90-day bank bill yield predictions the economic forecasters should get low marks in this instance (see below for the quantified scoring).
The 90-day bank bill yield predictions are roughly consistent with the predictions that economic growth would be robust, which meant an element of internal consistency between the Consensus Forecasts for economic growth and the 90-day bank bill yield. But if the 90-day bank bill yield increased as much as the economic forecasters were predicting it would result in sizeable increases in the cheapest available mortgage interest rates (i.e. the floating and short-term fixed rates), which would in turn result in falling rather than surging residential building activity.
Every past cyclical increase in mortgage interest rates has resulted in a cyclical fall in residential building activity 2-3 quarters later.
What we learn from reviewing the Consensus Forecasts is that the economic forecasters in at least this instance produced forecasts that are internally inconsistent, which is supposed to be a no-no for economic forecasters.
Dr Rod Deane, best known now for his role at Telecom, was the Chief Economist at the RBNZ when I was recruited in 1980. He was the founder of mathematical, economic forecasting models in New Zealand. He taught us novice forecasters that a key role of an economic forecasting model is to ensure internal consistency of the forecasts. Unfortunately, the modern economic forecasters seem to have forgotten this cardinal rule, which will greatly inhibit their ability to provide useful insights.
The black/thick line in the adjacent chart shows the rolling annual average of the 10-year government bond yield. The 10-year bond yield has fallen significantly over the last year, which partly reflects the market adjusting downwards expectations for the OCR and partly reflects falling US bond yields.
Being overly hawkish about the outlook for the 90-day bank bill yield will have played a significant part in the economic forecasters on average predicting a rising 10-year bond yield in June 2010, just in time for it to fall.
Again, the Consensus Forecasts didn’t get the direction right, although one of the forecasters surveyed in June 2010 got very close to accurately predicting the annual average level of the 10-year bond yield in the 2010/11 March year (see the red line following the black line up to March 2011, but thereafter deviating).
Reviewing the Consensus Forecasts for the exchange rate
NZIER survey what the economic forecasters are predicting for the trade-weighted value of the NZD (i.e. the trade-weighted index or TWI). This measure of the exchange rate is relevant to economists and the RBNZ in assessing the impact of the NZD on the economy.
The NZD TWI is based on the value of the NZD against the USD (0.3023), the Euro (0.2797), the AUD (0.2126), the Yen (0.1425) and the Pound Sterling (0.0629). It is 50:50 weighted according to:
· each currency area's share of NZ's merchandise trade (exports plus imports), normalised to total 100 percent; and
· each currency area's share of the combined nominal GDP of the five currency areas.
The weights (in parenthesis above), are updated annually, most recently on 16 December 2010.
In client reports the economic forecasters focus more on the cross rates that are relevant to exporters and importers (e.g. NZD/USD, NZD/AUD etc), but reflecting NZIER’s focus on the economy it surveys the TWI forecasts rather than NZD/USD forecasts.
The black/thick line in the adjacent chart shows the rolling annual average of the NZD TWI. Up until around March this year the TWI increased roughly in line with the June 2010 Consensus Forecasts, although it has since headed a bit higher than the economic forecasters were on average predicting in June 2010.
On this front they appear to deserve some praise. But having reviewed the Consensus Forecasts for the TWI over a number of years I can reveal that the coincidence of the predictions with the outcome was more an accident than good judgement.
The Consensus Forecasts generally predict that the TWI will revert to around the historical average level, so in the rare event this happens the Consensus Forecasts will be reasonably accurate.
But it is more often the case that the TWI ends up spiking significantly above or below the average level.
The other observation about the TWI predictions is that the economic forecasters have almost covered all bases (i.e. there is a wide spread between the maximum and minimum predictions). This means that one of them has a reasonable chance of getting it right, although you are unlikely to know which one in advance, while it is important to note that the forecaster predicting the highest/lowest TWI in one March year may not be the same forecaster who is predicting the highest/lowest TWI in the subsequent March year.
Scoring the accuracy of the Consensus Forecasts
We have adopted a 0-10 scoring system for quantifying the accuracy/usefulness of the Consensus Forecasts. The score is 10 if the Consensus Forecasts were close to the mark, meaning they would have been of great value to end-users (i.e. to business managers and individuals impacted by the economic indicator in question).
A score of zero is awarded if the Consensus Forecasts were so far from the mark that they would have been of little or any use to end-users.
Scores between zero and 10 are awarded based on how far the outcomes were from the Consensus Forecasts.
The scores for the June 2010 Consensus Forecasts are shown in the adjacent chart. Based on the forecasting scoring table we are using, the Consensus Forecasts score 4/10 for economic growth, residential building activity and the 90-day bank bill yield (i.e. well below pass GO and pick up $200 scores). The Consensus Forecasts score 5/10 for the 10-year government bond yield and a respectable 8/10 for the NZD TWI, but this outcome might be more by chance than by good judgement, as discussed above.
Based on the June 2010 Consensus Forecasts the forecasters would be kept back a year.
Some general observations about the forecasting track record
In general the Consensus Forecasts predict that most economic indicators will revert to around average or trend levels. In the 2010 June survey NZIER found that the economic forecasters were on average predicting just over 3% GDP growth for both the 2010/11 and 2012/13 March years.
When it comes to predicting economic growth the forecasters have a strong predisposition to predict around 3% growth, which reflects a somewhat optimistic assessment of the historical average rate of annual GDP growth.
If you look at the Consensus Forecasts for residential building activity, the 90-day bank bill yield and the 10- year bond yield (the respective blue lines in the charts above) you will find that they also tend to 'mean revert'.
This means that on average over the long-term the economic forecasters will be reasonably close to the mark, but in any one year economic growth, residential building activity, interest rates and the exchange rate are more likely to end up some distance from the historical averages. This implies that the economic forecasters are unlikely to provide many useful insights for business risk management purposes.
However, this is jumping the gun based on my knowledge of the past track record of the economic forecasters. If my observations are correct (i.e. the economic forecasters often get the direction of a number of key economic indicators wrong and have a tendency to cling to historical averages) it will be revealed as we review the track record of the economic forecasters in the future.
How the accuracy of the Consensus Forecasts are scored
We are using a 0-10 scoring system for quantifying the accuracy/usefulness of the Consensus Forecasts.
The score is 10 if the deviation between the outcome and the Consensus Forecasts is equal to or less than plus or minus 0.25% for GDP growth, +/- $0.25b for residential building activity, +/- 0.25% for the 90-day bank bill yield, +/- 0.25% for the 10-year government bond yield and +/- 1.5 for the NZD TWI.
If the outcomes deviate from the Consensus Forecasts by more than the following amounts we assess that the forecasts are of little or no help and a score of zero awarded: +/- 3% deviation for economic growth; +/- $1.25b for residential building activity; +/- 2.5% for the 90-day bank bill yield; +/- 1.5% for the 10-year bond
yield; and +/- 10 for the NZD TWI.
The numbers used for each of the five economic indictors reflect how much each indicator has varied in the past, which has implications for how close the Consensus Forecasts need to be to the outcomes to be of value.
The table below shows how the score falls from 10 to 0 based on increasing deviations between the outcomes and the Consensus Forecasts (e.g. if the difference between the outcome for economic growth and the Consensus Forecast is greater than +/- 0.25% but less than +/- 0.525% the score is 9; if the deviation is greater than +/- 0.525% but less than +/- 0.8% the score is 8, etc).
-----------------------------
*Rodney Dickens is the managing director and chief research officer of Strategic Risk Analysis Limited.
23 Comments
I think it's become plain for even non literate observers that the whole academic discipline of economics is deeply broken.
These are the people who didn't see the largest worldwide economic disaster since the great depression coming. Indeed they mocked those who suggested otherwise.
Clearly things don't work in the way they say they do.
Therefore dud forecasting would seem par for the course.
Yes there seems to be something very wrong with the advice that they are producing. When I look through Ecc 101 level texts it all seems pretty sensible. When I have considered the comments and advice offered by the so called experts in the field my impressions have been:
They appear to be trying to preserve the status quo or please the establishment, - who pays their wages.
Apart from a few exceptions, much of their advice seems to depart from basic common sense or even the basic tennants of ecconomics. You could not (or shouldn't) run personal finances the way nations are run, presumably with the quidance of ecconomists. The reality of this are now becomming manifest.
They seem to exhibit a lot of 'reef fish' behaviour. This suggests to me that there output is based more on opinion than facts, figures and method.
Without having studied it, I wonder if the leading edge of the subject has strained the bounds of rigor and strayed into the bounds of fantasy.
I wonder if considerations of morality and social justice are overlooked somewhere in the process.
They would be better adopting the Tony Alexander forcasting method - just keep repeating the same stuff year after year and eventually you'll be right.
Fact is we are in a balance sheet recession with unprecedented central bank and Goverment activty clouding the outlook. The forecasters seem to be expecting the economy to behave like a normal recession and rebound. Not going to happen.
An interesting and important subject, but the analysis appears to grounded on some heroic assumptions:
- Surveyed economic forecasters (not specified) are independent and not spinning their corporate or government marketing messages
- Consensus is what matters - not that some forecasters are better than others or their forecasts more important. Where do the ones that matter e.g. TSY's budget forecasts rank?
- The worst forecasts do no damage: A score of zero is awarded if the Consensus Forecasts were so far from the mark that they would have been of little or any use to end-users. Get real - score them from plus 5 to minus 5 with zero being no damage and negative numbers indicating forecasts that hurt business if followed.
We don’t believe economic forecasters can provide much insight beyond the year ahead. Fair enough, but what does that suggest regards TSY forecasts of a return to surplus with the NZD:USD at 0.56?
I think he made the same mistake I did in underestimating how far the rats would scramble to hide the mess we are in.
Perhaps a look back in history would provide a better blueprint, and of course it does (1929-33).
Do you really think we are out of the woods yet?
For the last 3 years all the Bank Economists have been warning of fast and strong interest rates increases.
It hasn't happened.
Because they have no power or influence over global events that they do not understand. They think that crying 'rate hikes' are going to pressure people to buy now, fix rates, frighten them to ensure their mortgages are paid on time.....
Rates will be sinking as low as our sick global economy --- how much Gold will it cost to buy a loaf of bread soon?
During the noughties, the mainstream economists said that the consumer-driven boom based on a boom on housing based on a boom in borrowing could last forever. It took wise heads like Steve Keen & Nouriel Roubini to see through that.
Now the mainstream economists are saying, OK we've had the crisis, its over with, the economy will move strongly ahead into a new boom cycle. They forget about the huge overhang of debt from the housing boom, combined with renewed govt debt.
In reality the hang-over needs to last a decade at least before it can be worked through. The ageing population will add its own downdraughts to any recovery.
Sorry to be a Gloomy Gus.
Cheers to all
They didnt really forget about the debt, they said that one man's debt is anothers credit so the net difference is zero and then ignored it, fatal mistake....Steven Keen and Minsky said the opposite....
We are 3 years in and not at the bottom yet, when it starts if its like the GD it will take 3 years to hit bottom, but some are suggesting that it wont be that fast, so 5 to 6 years to hit bottom.....we will then be about 8 or 9 years in....so a decade? no I cant see less than 2 and tahts ignoring Peak oil which will be severly impacting us within a decade...
So you think you are gloomy? try my outlook.....30 years of depression....
regards
I agree. This is promising to be the paydown century....with bubbles and crashes galore...huge gains for the 'smart' and the lucky...early exit for losers...serfdom for most...fabulous wealth for the bankers and their mates in the govts.
note: For 'smart', read criminal inside traders and those the SEC decide are too powerful to go after.
Wolly, at what point do the ppl rise up?
Where in socio-economic terms does most of the army and National guard lower ranks come from? and who is being hammered the most economically? We have already seen riots in the UK, I wonder how much longer before the US, think Wisconsin as a start.....oso just how far a revolution could be away....
regards
You are joking right...."rise up"....no way steven...most are too ignorant of what the hell is going on and the rest are out to profit from it. The riots were crime based end of story.
The lesson is to always consider the human stupidity factor before investing...to always expect political moves aimed at protecting those in power...to realise that key people really do manipulate the markets.
Money talks steven!
Psssst .....wanna own some peasants...sell em cheap credit!
Wisconsin, take a look at what is and what has happened there.....
http://www.thenation.com/article/162922/democrats-rural-rebellions
ppl are being ridden over and they are hacked off....and its unlikely the ranting right will slow down in these crazy things....
regards
Steven: Yeah, I remember hearing the "one man's debt is anothers credit" argument. Articulately described by leading conventional economists.
And I also remember being totally incredulour & thinking "what a total pile of s##t!". If credit is all on one side (eg China) and debt all the other (eg US), & will clearly never been repaid, then you have a timebomb waiting to go off. It can only end in tears.
Thanks for comments, a good thread.
We welcome your comments below. If you are not already registered, please register to comment.
Remember we welcome robust, respectful and insightful debate. We don't welcome abusive or defamatory comments and will de-register those repeatedly making such comments. Our current comment policy is here.