Brian Easton says evaluating the impact of social policies will be very difficult but the government does not seem to be doing much real evaluation

This is a re-post of an article originally published on pundit.co.nz. It is here with permission.

A couple of terms that have recently become fashionable are ‘cost-benefit analysis’ (CBA) and ‘social-investment analysis’ (SIA), typically proposed by people who have never done either. They sound good but have their limitations. Providing their limitations, are understood they can assist making better decisions. Too often they are ignored, and the resulting analysis has little value, merely reinforcing the conclusion that the decision-maker who commissioned the analysis was going to make anyway.

CBAs entered public policy in the 1960s evaluating the return on irrigation projects. (Locals, including their MPs, were very keen on the projects because they involved large government subsidies.) By the early 1970s they were being used by the National Roads Board to rank spending on roads. A decade later they had a central role in the Major Projects (‘Think Big’) debate. They were then necessary because the projects depended upon government subsidies and other kinds of public support.

The debate was traumatic; economists even lost their cool. Part of the problem was that we had to develop the analytic frame as we went along. (The only New Zealand manual we had was for irrigation projects – I am not making this up.) We economists made mistakes but any economic advice was usually overruled by the politicians who went ahead with even some known ‘dogs’. (Roger Douglas admitted he made wrong decisions – against advice – over New Zealand Steel.)

The policy conclusion was that government shouldn’t subsidise businesses, which explains many of the more sensible changes under Rogernomics. Unfortunately, politicians remain keen on incentives for political reasons and sometimes government involvement is unavoidable – like the broadband rollout and the current debate on the inter-island ferry service.

Moreover, the government is also involved in many decisions which are not strictly business ones. The social area exists as a public policy issue because it cannot be commercialised.

CBA/SIA has been most extensively used in the health sector. But note that once again the politicians have tried to overrule careful analysis, as when they announce they will fund particular pharmaceuticals. (The economic issue is whether the same funds could be spent more effectively – with better health outcomes – on other treatments; we don’t know unless careful analysis is done.) The application of these techniques in the health area is fraught with difficulties. I know, having worked on a number of evaluations in numerous areas and having been a member of an international working party which developed a manual on how to apply the techniques to the analysis of substance use and abuse (published by WHO).

That a manual is necessary was evident from a strange debate over the last published social cost of alcohol study which paid no attention to the issues the working party sweated over and did not understand them; neither did its critics.

The government has announced it now intends to apply Social Investment Analysis in areas such as education, justice and social services. It has even set up the Social Investment Agency to lead its use.

The idea began under the Key-English Government and was implemented under the Ardern-Hipkins Government as the Social Wellbeing Agency. For all the fanfare, sentiments, promises and expenditure, there have been few results to show SIA’s effectiveness.

Instructively, there is no published manual of how to do an SIA. The unwillingness to learn from past experiences or other disciplines is illustrated by the various contributors to Social Investment: A New Zealand Policy Experiment. So bereft were they of practical experience one is reminded of the joke that a consultant is someone who knows 69 ways to make love but has never kissed a girl.

Will the experiment be successful? The experience is that its success, if any, will be limited and hardly at all if there is no manual. There will be a host of undiscussed problems about how to construct the framework.

For instance, a key point in these analyses is what is the perspective of the study? How is success going to be judged? In the health sector we tell the story of the ‘exploding cigarette’, which is highly addictive, generating lots of tax revenue, but kills the person on the day they retire so the state does not have the subsequent health and retirement costs. From the fiscal perspective an ideal policy, except it has left out all consideration of the person’s welfare.

Before you say ignoring people’s welfare is absurd, the feedback I hear is that the government’s SIA focuses on the fiscal savings to the government. So the perspective is the taxpayer, with the rich benefiting more than the poor. I hope it is not true, for in every health evaluation with which I have been involved, the suggested treatment involved net public outlays in the long run because it was trying to benefit the sick.

I’ve given but one example of the minefields we found when developing the WHO report. They’ll all be there in an SIA ,and probably some more. Often there has been a shortage of quantitative evidence in the health evaluation projects with which I have been involved; sometimes that evidence is key to the assessment.

A major issue is, as the term ‘investment’ indicates, that the evaluation has a strong time element. (No, I am not going to go over the time-discount rate issue again – I think we got it wrong forty years ago, and we are still getting it wrong.) Rather, analysis requires evidence about policy impacts decades later. For instance, good nutrition for pregnant women affects the health of their grandchildren. (The mechanism seems to be that the health of the child in utero is affected, and in the case of a girl child, that will affect her children.) Most of that sort of evidence is not available for social service issues. (The health sector is much better provided with the required evidence than social services will be, but even then it is frequently lacking.)

The temptation is for the advocate – or the consultant advising the advocate – is to make up any missing numbers. Sometimes the policy conclusion hinges on their magnitude; you will not be surprised that there is an optimistic bias in their creativity. That bias will feed the politicians.

CBAs and SIAs only informed decisions. They are rarely decisive. The Coalition Government agreement said it will make decisions that are ‘principled – making decisions based on sound public policy principles, including problem definition, rigorous cost benefit analysis and economic efficiency.’ That was almost certainly written by someone who has never done a CBA.

In practice the government seems to be oblivious to the principle. most evidently in the justice area such three-month boot camps. The impression I get – I am not expert in the area – is that the evidence says they won’t work. Did the politician commission any SIA – let alone a rigorous one – before they directed the policy to be instigated? (Will they direct there be follow-up evaluations?) How many of the current government’s policies proposals have been made with the promised rigour? It would be useful if at each policy announcement a journalist asked to see the ‘rigorous cost-benefit analysis’.

SIAs could make a significant contribution to social policy if they got us to focus of the evidence to inform it. (In the health sector it is called ‘evidenced based medicine’.) I am not optimistic; it involves a massive change in the way we make public policy.

*Brian Easton, an independent scholar, is an economist, social statistician, public policy analyst and historian. He was the Listener economic columnist from 1978 to 2014. This is a re-post of an article originally published on pundit.co.nz. It is here with permission.