What Do Nobel Prize and Your Business Have in Common? Natural Experiments

A natural experiment is a type of causal analysis that has been widely adopted by many organizations and research fields. It was such a game-changer that the pioneers of the idea were awarded the 2021 Nobel Prize in economics.

This story will explain how these natural experiments are different from regular experiments and how you can use this idea in your organization.

Let's start with an example.

Taken from Nobel Prize 2021 Press Release

There are many reasons why someone might want to go to university. One of them is the expectation of higher income. The assumption that a few more years of education will lead to higher income becomes even more critical if you need to pay for your education. However, the figure above shows correlation, not causation. In other words, it shows us that education and income are related, but it is not a causal analysis.

So, how can you find out whether getting that master's degree will result in higher income?

When we think about exploring causal relationships, we think about experimental design. Traditionally, experiments include a controlled environment and random assignment of participants. 

You can imagine randomized control trials that are used in drug development. Say, my company is developing a cream for acne, and we want to test whether it works. To do this, we will find people with acne and split them into two groups.

One group will receive our new anti-acne cream, and the other will receive a regular cream without any special ingredients. Additionally, we want the splitting process to be random, i.e. there is no specific criterion for a person to be assigned to either one of the groups. 

No, because once we move from pharmacy and medicine to social issues, it's often impossible to get all the factors under control. However, we still have questions that need some type of causal analysis. Natural experiments are the answer.

Natural experiments, also called quasi-experiments, are built on the same principles as regular experiments, except that everything is under control.

Going back to the education and income question, Joshua Angrist (one of the Nobelists) and Alan Krueger answered it. They compared people born in the first and in the last 3 months of the year, and they noticed that those born earlier in the year spent less time in education on average. They also had lower incomes compared to those born in the last quarter. 

This is a natural experiment as it uses the fact that nobody can decide when they are born - it's random. Additionally, their analysis relies on the US educational policy that allows children to leave school when they turn 16 or 17 (depending on the state). The same study design wouldn't work in countries with years-of-education limits rather than age-limit for leaving school.

If you are familiar with user research, you know that causal analysis is used very often. Imagine you are a media outlet, and you want to determine which title will lead to more clicks. Come up with two titles and assign them to users randomly. Then you calculate the click-through rate for both groups, and you will find out which title is more clickable. This is an example of a regular experiment because we can perform the random assignment of the interventions (i.e. titles). 

However, imagine you are in the leadership of a company in 2019. You are considering the option for the employees to work from home. You know that it works in some companies, but you are still hesitant. This is not something you want to experiment with on a massive scale.

Suppose you tell your workforce to work from home for 2 months. What if your company loses money due to decreased productivity? You might think allowing the most reliable employees to work remotely would reduce the risk and help you answer the question. Yet, this would only lead to biased results. 

Do you remember the COVID-19 pandemic? 

Taken from tenor.com

The pandemic is terrible, of course, but it occurred naturally and created an environment for natural experiments. When the pandemic happened, people whose jobs allowed them to work from home started working from home. This allowed us to understand the benefits and drawbacks of remote work much better.

But how exactly can you leverage the pandemic for causal analysis?

Regarding remote work, your natural experiment could compare the productivity level before the pandemic and when people worked from home. You still want to make the time frames as similar as possible. E.g. comparing the last quarter of the year before the pandemic with a second-quarter during the pandemic is not the best idea.

You can even apply causal analysis to your personal life, but you need to be very careful about your conclusions. Let's say you take vitamins every day for 3 months and you notice you didn't become ill. You might conclude that these vitamins strengthen immunity which protects you from diseases. Even though you may be right, just because something works for you doesn't mean it will work for everyone in the same way. We don't know for sure what would have happened if you didn't take the vitamins. Maybe you wouldn't get ill anyway. 

Every type of causal analysis has a specific weight that can be assigned to a claim. Natural experiments are great, but randomized experiments still offer the strongest proof of a causal relationship.

Now, when you understand causality, you can enjoy a bit of poetry.

Note: One half of the 2021 Nobel prize was awarded to David Card “for his empirical contributions to labour economics” and the second half went to Joshua D. Angrist and Guido W. Imbens “for their methodological contributions to the analysis of causal relationships”.