Tom Coupe on the economics of big data; What it is, benefits, its economic impact, its failures, its use for scary policy, data art and more

Today's Top 10 is a guest post from Tom Coupé, an Associate Professor in the Economics and Finance Department at Canterbury University.

As always, we welcome your additions in the comments below or via email to david.chaston@interest.co.nz.

And if you're interested in contributing the occasional Top 10 yourself, contact gareth.vaughan@interest.co.nz.

See all previous Top 10s here.

The economics of big data.

Big data has become a popular topic over the last years as illustrated by the Google trend graph of search intensity for the search term “big data” in New Zealand. While before 2011 almost nobody in New Zealand was searching for “big data”, since 2011 the search intensity for this term has increased rapidly and significantly.

Not surprisingly, Interest.co.nz now also has a dedicated tag for “big data” articles.

In this top 10, I give an overview of 10 key articles that will help you to get up-to-date about the economics of big data, big data’s successes and failures, and also illustrate how ‘cool’ big data applications can be. 

1. Big data definition. 

If you don’t have a clear idea of what big data exactly is, don’t worry – there isn’t one ‘official’ definition – instead everybody seems to have his/her own definition. Here, for example, you can find 42 different definitions, each of them by a prominent big data expert. 

“To settle the question once and for all, we asked more than 40 thought leaders in publishing, fashion, food, automobiles, medicine, marketing, and every industry in between how exactly they would define the phrase “big data.” Their answers might surprise you! Take a look below to find out what big data is." 

2. Big data benefits.

Despite this confusion about what big data exactly is, many believe big data will bring big benefits. Some even consider big data a new "factor of production" or a new “asset class”

“The big data of this revolution is far more powerful than the analytics that were used in the past. We can measure and therefore manage more precisely than ever before. We can make better predictions and smarter decisions. We can target more-effective interventions, and can do so in areas that so far have been dominated by gut and intuition rather than by data and rigor.”

3. Big data benefits for business. 

One of the key benefits big data is supposed to bring to business is that it allows businesses to know their customers better, and hence to personalize the targeting of customers. One of the classic examples is the US retailer Target targeting future mothers: 

“About a year after Pole created his pregnancy-prediction model, a man walked into a Target outside Minneapolis and demanded to see the manager. He was clutching coupons that had been sent to his daughter, and he was angry, according to an employee who participated in the conversation. 

“My daughter got this in the mail!” he said. “She’s still in high school, and you’re sending her coupons for baby clothes and cribs? Are you trying to encourage her to get pregnant?”  

The manager didn’t have any idea what the man was talking about. He looked at the mailer. Sure enough, it was addressed to the man’s daughter and contained advertisements for maternity clothing, nursery furniture and pictures of smiling infants. The manager apologized and then called a few days later to apologize again. 

On the phone, though, the father was somewhat abashed. “I had a talk with my daughter,” he said. “It turns out there’s been some activities in my house I haven’t been completely aware of. She’s due in August. I owe you an apology.”  

4. A New Zealand example.

Also in New Zealand, there are examples of companies that believe that big data will allow them to gain a competitive advantage:  

“Lyon says premium pricing can be determined using data from the likes of social media, government agencies and big brand companies.

“It’s theoretically possible to be able to compile a whole bunch of information that gives insight into a person’s risk profile. Effectively that’s what insurance is all about – understanding risk and then pricing for it accordingly.”

5. Big data benefits for policy. 

Big data cannot only help business to make better decisions, also government officials can benefit from big data. Using online information, it is possible to create ‘early warning systems’ that show the state of the economy before the official statistics are compiled. But Government itself is also a creator of big data as it gathers, as part of its day-to-day activity, lots of data on lots of people. By analysing these data or by opening its data archives to let others analyse these data, government can learn a lot. For example, it can learn about the effectiveness of its services like this example which uses NZ’s Integrated Data Infrastructure.' 

“Youth Service: Not in Employment, Education or Training (or YS: NEET) is a government programme designed to encourage and assist disadvantaged 16-17 year-olds to stay in education or training and improve their qualification attainment. … We find that YS: NEET raises the educational retention of participants in the first year, by up to nine percentage points at peak. The proportion who complete a level two qualification is slightly raised, by around two percentage points. The programme appears to raise rather than lower participants' subsequent benefit receipt rates, and there is no improvement in their likelihood of being employed.” 

6. Big data for scary policy.

Of course, rather than using big data for better policy decisions, one can use these big data also to control the population in a more comprehensive way as this example from China illustrates well.

“The Chinese government is taking the first steps in an evolving plan to employ big data to establish a nationwide system of mass surveillance of the entire population. This “social-credit system” would mobilize technology to collect information on all citizens and use that information to rate their behavior, including financial creditworthiness and personal conduct. The local experiments have provoked mixed reactions.” 

7. Big data economic impact? 

While the economic advantages of big data seem obvious, sceptics will wonder why, if big data is so important, the economy isn’t continuously booming?

“There is just one tiny problem: the economy is, at best, in the doldrums and has stayed there during the latest surge in Web traffic. The rate of productivity growth, whose steady rise from the 1970s well into the 2000s has been credited to earlier phases in the computer and Internet revolutions, has actually fallen. The overall economic trends are complex, but an argument could be made that the slowdown began around 2005 — just when Big Data began to make its appearance.”

“Those factors have some economists questioning whether Big Data will ever have the impact of the first Internet wave, let alone the industrial revolutions of past centuries. One theory holds that the Big Data industry is thriving more by cannibalizing existing businesses in the competition for customers than by creating fundamentally new opportunities.”

8. Big data, big mistake. 

What’s more, many claims made by big data supporters are less well-founded than these supporters would like everybody to believe. Tim Harford, the undercover economist of the Financial Times, summarizes it well: 

“Recall big data's four articles of faith. Uncanny accuracy is easy to overrate if we simply ignore false positives, as with Target's pregnancy predictor. The claim that causation has been “knocked off its pedestal” is fine if we are making predictions in a stable environment but not if the world is changing (as with Flu Trends) or if we ourselves hope to change it. The promise that “N = All”, and therefore that sampling bias does not matter, is simply not true in most cases that count. As for the idea that “with enough data, the numbers speak for themselves” – that seems hopelessly naive in data sets where spurious patterns vastly outnumber genuine discoveries.” 

9. Big data failures.

By now there are also already a number of examples of big data failures. Predicting flu outbreaks based on Google search terms has been less successful than initially hoped for. Predicting unemployment claims based on social media also turned out to be much less accurate than initially thought. And the Farecast algorithm that predicts when it’s the right time to buy your plane ticket is no longer offered by Microsoft’s Bing Travel. 

“In 2008, Microsoft snapped up Farecast, a company in the business of predicting airfares, for a handy $115 million. A poster child of the Big Data revolution, Farecast analyzed hundreds of billions of airline ticket prices using a machine-learning algorithm and told consumers when to buy. The acquisition helped make price prediction the key differentiator of Bing Travel, a core asset of Microsoft’s new “decision engine.”

But a few weeks ago, right as I started gathering data for this article, the heralded price predictor vanished from Bing Travel. Initial reports suggested executives had made a business decision to focus resources on travelers’ other needs. With Microsoft’s exit from the price-prediction game, it’s time we re-assessed the technology to see if what doomed Farecast was instead a failure of Big Data.”  

10. In any case, big data is cool.

Whether or not big data will make all of us better off and become visible in the aggregate economic measures like GDP, remains to be seen. Whatever the case, big data is already bringing us lots of cool stuff, look for example at this ‘Largest Analysis of Film Dialogue, ever’, at ‘How Music Taste Evolved’ or at these examples of ‘data art’: 

“The objective of data art is to create aesthetic forms and artistic works from the digital nature of the data generated from big data (graphics, simulations, worksheets, statistics, etc.).”