Economist Brian Easton says we should try to to evaluate our social policies systematically. But big data has limitations

This is a re-post of an article originally published on pundit.co.nz. It is here with permission.

The National Party has promised to reintroduce the idea of ‘social investment’ analysis when it returns to government.

The basic idea goes back long before Bill English as Minister of Finance popularised it, and it did not die when the Ardern-Robertson Government replaced National. It is there institutionally as a Social Wellbeing Agency which describes itself as working ...

on challenging social-sector problems to improve people’s lives. We focus our efforts on where we can make the greatest impact to lead and shine a light on policy issues that affect the wellbeing of New Zealanders. We collaborate across the system advising on policy issues that fall between the gaps. Ultimately, we want our approaches to achieve sustainable improvements for the wellbeing of New Zealanders.

We shall have to wait to find out what National has in mind to extend the notion (other than to return the institution’s title back to Social Investment Agency?). I am sure English had something much more ambitious in mind than the policy announced at the party’s Annual Conference, to further force the young unemployed to find work; Jacinda Ardern criticised the policy and added that the government was already pursuing it.

One very much gets the impression that ‘social investment’ analysis is to do with fashion rather than innovation. You can find its beginnings at least as early as an 1848 article by Jules Dupuit which evolved into cost-benefit-analysis (CBA), later taken up systematically in government decision-making in the 1960s.

The essence of CBA is a framework which enables one to evaluate a policy proposal rigorously. I have used it in a large number of areas including business investment, energy projects, environmental evaluation, fishing, legal drugs (alcohol and tobacco), medical treatments, pharmaceuticals, regional assessments, and transport infrastructure.

It was once necessary even in commercial decisions, because government interventions distorted the market price system so that they did not reflect ‘social values’ (in a certain rigorous economics sense). However in some areas – in my experience drug control, health and infrastructure – there are a number of practical reasons why the market is nowhere near properly functioning, in which case CBA remains necessary.

It is particularly relevant when the objective is not simply maximising economic output but includes a wellbeing dimension. Were healthcare about output, the most common conclusion would be to terminate most patients’ life as quickly as possible. Instead, the aim is to enhance and prolong the quality of life, something which the market does not do very well by itself.

What can be done is limited by the data base. For instance, one of the findings of my work on alcohol was the gaps in our knowledge of what we measured. But the data base is always improving and the hope is that the huge data bases (collected for other purposes) coming on stream will enable us to investigate educational, employment and social welfare issues.

Unfortunately, big data does not always mean one gets useful results for policy-making. We routinely teach ‘correlation is not causation’ which is a shorthand way of saying that if A and B move together, it does not mean A causes B, nor that B causes A. If A precedes B than B cannot possibly cause A. Moreover, C may be causing them both. Suppose you are targeting on B. It may not be much use trying to manipulate A even though it is correlated with B, especially if they are contemporaneous as occurs in many data bases. Correlation as a guideline to policy formation is limited. 

This is illustrated by the specific example of a Massey University centre for Social and Health Outcomes Research and Evaluation (SHORE) survey on gambling I was involved in. It showed that generally the form of gambling – casinos, housie, Lotto, TAB – had little association with gamblers’ mental health; it was much the same as that for non-gamblers. The exception was that those on the pokies typically had poorer mental health than the rest of the population. Before jumping to the conclusion that pokies cause poor mental health and by closing down pokies we can improve the mental health of the players, we need to consider the possibility that some people with poor mental health play the pokies and even that closing pokies may worsen their mental health. We could not tell from our research results; even had we surveyed the entire population we could not tell. (Sadly, funding ran out before we could further explore the issue.)

The problem of ‘identifying’ causation is quite general, and has been well studied in economics (strictly econometrics) for about seventy years. It can be got round by random controlled experiments, which psychological and medical researchers use a lot (as when a treatment is given to half a sample who are compared with the other half). Alas, such experiments are not easy for the sort of questions that economists and those involved with social investment investigate. (Consider the important question of whether unemployment causes suicide; the random controlled experiment would involve making half the sample unemployed and seeing whether they were more likely to take their own life, compared to those in the employed half of the sample.)

Even so, economists have tried to answer some of their questions by actual experiments where they can (I have described a couple of Nobel Laureates who work in the field here) and sometimes there are natural experiments (a Nobel laureate here).

But this cannot solve all the important questions trying to identify causation. Sadly, larger data sets are not sufficient – a routine result from econometrics. They may result in very precise correlations but they still do not tell you about causal processes.* The integrated data  Infrastructure database (IDI), which holds de-identified micro-data about people and households, includes longitudinal (through-time) data which reduces the research difficulties but does not eliminate them. This is not to say ‘big data’ explorations are useless but they need to be pursued with a caution and humility which is not always evident among the advocates.

What does this say about the promises made by the supporters of social investment analysis? Yes, better data bases mean we can make some progress at systematically exploring policies issues which have been neglected in the past. But we should not be overconfident in the way the fashion for social investment is presented.

Moreover, social investment analysts would do well to pay more attention to the cost benefit analysis framework. Thinking rigorously often produces better policies.

* The technical expression, which those of you who did econometrics may recall, is that the estimator is not ‘consistent’ which means its estimated parameters are biased even for very large (infinite) samples. That means you cannot use them for purposes of unbiased prediction.

*Brian Easton, an independent scholar, is an economist, social statistician, public policy analyst and historian. He was the Listener economic columnist from 1978 to 2014. This is a re-post of an article originally published on pundit.co.nz. It is here with permission.