Tom Coupé with 10 examples of business experiments, including dating without pictures, women who make you want to spend, Googling up smaller snacks, Bill Gates & chickens, moving from Tonga to NZ causes stress & more

Let’s start with a fun, insightful, well-written blog post by OKCupid, a dating website, which describes what happened when they removed pictures from their website and what happened if they told people they were great matches even though their internal algorithm, which predicts the quality of matches, suggested the opposite.

“I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out.”

2. This Lingerie Company A/B Tests The World's Hottest Women To See Who Makes You Click "Buy".

Next, a great illustration that there is nothing random (or better, that nothing has to be random) in what you see when browsing through a company catalogue. But also that getting rid of randomness can create a competitive advantage.

 “Sex doesn't sell, so forget the boudoir shot. Blondes don't work. Props distract. Couches are fine. Playing with hair is ideal. Those are some of the insights the lingerie company Adore Me has learned from testing the photos of models wearing its sexy products online. For each bra, Adore Me shoots multiple versions of images to run on its website. The distinctions between the pictures might include different models wearing the same set in the exact same position, or the same model in the same set in a different position, for example. Then, like any good tech company, it tests the options to find out which one sells better.”

3. “How Google Optimized Healthy Office Snacks”.

Companies not only experiment on their clients, companies can also experiment on their employees. Here an example of how Google tried to improve the eating habits of its employees.

“In a field experiment in another Google microkitchen, we targeted the most popular snack item: bulk M&Ms. They had been self-serve from bulk bins into four-ounce cups; most employees filled the cup. After taking a baseline measure of consumption, we replaced loose M&Ms with small, individually wrapped packages. This simple intervention reduced the average serving by 58%, from 308 calories to 130.”

4. “Paid Search Ads Pay Off for Lesser-Known Restaurants”.

Experiments can also be used to answer more general business questions like ‘does advertising work?’. Most businesses hope it does but very few will be able to give you convincing estimates of the return on investment in advertising. Michael Luca of Harvard Business School and Weijia Dai of Lehigh University teamed up with Yelp to see whether restaurants benefit from advertising their services on Yelp. To be able to answer this question, they gave away for free millions of dollars of advertisement space. Maybe interest.co.nz too is interested in finding out how effective advertising on its website is? 

“Luca proposed a large-scale experiment in which Yelp would give away several months of advertising spots for free to randomly selected businesses. “Geoff sort of cringed,” Luca recalls, at the thought that he would effectively be giving away the equivalent of millions of dollars worth advertising exposures.”

5. Bill Gates wants to give the poor chickens. What they need is cash.

It is not only business that can benefit from experiments. Governments can do too, as experiments can be used to answer important public policy questions. In this article, Christopher Blattman, professor at the University of Chicago, explains to Bill Gates why he thinks it is better to give cash to poor people, rather than giving chickens, and what can be done to figure out who has the right answer.

“I’m also writing to say that both of us are counting our chickens before they hatch. (Sorry — I couldn’t resist.) Given the state of current research, neither of us can say definitively whether livestock or cash is better — or under what circumstances one might be better than the other. But the answer is in easy reach if only someone would conduct a thorough experiment to test the two approaches. It is irresponsible of us not to do so.”

6. Using Randomized Trials in Policy: Oliver Hauser on the UK Behavioural Insights Team.

In the United Kingdom, there is even an organization, the Behavioural Insights Team, which started as a government unit but is now co-owned by the UK government, and does experiments to try to improve government policy.

“To illustrate these, Oliver shows us a trial they ran on the "Tax Return Initiative" with HM Revenue and Customs. Famously, they tried to prime people to be more honest on tax payment. They ran an RCT including a link directly to the tax form rather than a link to the website. In a randomized trial of approximately 2,000 people in each group, click-through rates were 23.4% compared to 19.2%. In more recent study of tax forms in Guatemala, they have carried out several million observations.”

7. Just do it? But HOW? 24 productivity experiments I tried, plus a QS time management recap.

Arguably less scientific and fraught with methodological issues, but at least as amazing: self- experimentation. In psychology and medicine, there are many examples of experiments that researchers run on themselves. I am not aware of academics in economics or business doing this but I found this relevant example:

“Some time ago I was asked for the ultimate productivity tip, and instead of giving a straightforward take-away, I said that in the end the answer is “it depends.” That wasn’t a cheap shot because what works for you might not work for the next guy, and vice versa. Sound familiar? It’s the same case for medications, meditation, and most anything else we humans do. That’s why it’s best to experiment, examine your results, and decide based on the data. In other words, quantify!”

8. Why Businesses Don’t Experiment.

While more and more businesses do run experiments, many firms refrain from doing so. Dan Ariely of Duke University discusses some reasons why companies fail to take advantage of the benefits of doing experiments.

“Companies pay amazing amounts of money to get answers from consultants with overdeveloped confidence in their own intuition. Managers rely on focus groups—a dozen people riffing on something they know little about—to set strategies. And yet, companies won’t experiment to find evidence of the right way forward.

I think this irrational behaviour stems from two sources. One is the nature of experiments themselves. As the people at the consumer goods firm pointed out, experiments require short-term losses for long-term gains. Companies (and people) are notoriously bad at making those trade-offs. Second, there’s the false sense of security that heeding experts provides.”

9. How to Design (and Analyze) a Business Experiment.

The rational managers of companies and government agencies interested in doing experiments and benefiting from them can find some practical advice in this article

“This article lays out seven steps to ensure that your experiment delivers. These principles draw on the academic research on field experiments as well as our work with a variety of organizations ranging from Yelp to the UK government.”

10. Natural Experiment Evidence on the Effect of Migration on Blood Pressure and Hypertension.

What about New Zealand? I am sure there are businesses and possibly even government agencies in New Zealand that are running experiments. My Google search didn’t give me any examples, however. (If you know of examples, please leave them in the comments or email me!) but I did find this study of a so-called ‘natural’ experiment created by New Zealand’s immigration system. In New Zealand, a lottery has been used to select some applicant-migrants to receive permanent residency. This lottery acts just like an experiment, and by comparing applicants who won the lottery to those who didn’t one can find out how migrating to New Zealand affects, for example, one’s health. Interestingly, the results show migration from Tonga to New Zealand increases blood pressure and stress.

“In this paper, we use a natural experiment, comparing successful and unsuccessful applicants to a migration lottery to experimentally estimate the impact of migration on measured blood pressure and hypertension. …

 In 2002, another channel was opened up for immigration to New Zealand through the creation of the Pacific Access Category (PAC). This allows for a quota of 250 Tongans to immigrate to New Zealand each year regardless of their skill level or socioeconomic status. …

Many more applications are received than the quota allows, so a ballot is used by the New Zealand Department of Labour (DoL) to randomly select from amongst the registrations”