What Is a Natural Experiment? Finding Randomness Already in the World
By the BrainSnail editorial team. How these articles are written and checked, and how to tell us when one is wrong.
Most questions in economics and social policy cannot be settled by assigning people at random, because nobody will randomise who gets a minimum wage or who is born in a particular year. A natural experiment finds a situation where something outside the researcher's control has divided people in a way that is effectively random, and treats that division as if it had been assigned.
The logic
The whole difficulty in observational social science is that people who experience something differ from those who do not in ways that also affect the outcome. People who attend university differ from those who do not before they attend, so comparing their later earnings measures the difference between the people rather than the effect of the degree. A natural experiment looks for a source of variation that shifts who receives the treatment without being connected to anything else about them. Policy changes that apply on one side of a border and not the other, thresholds that switch eligibility at an arbitrary cut-off, lotteries used to allocate scarce places, the timing of a reform relative to someone's birth date, and genuinely unforeseen events all supply that. The claim being made is specific and testable: that the people just either side of the division were alike beforehand, which researchers check by comparing everything they can measure and by looking at whether outcomes were already diverging before the change.
The standard designs
A small number of research designs do most of the work:
- •Difference in differences, comparing the change over time in a group affected by a policy with the change in a similar group that was not, which removes anything that would have affected both equally
- •Regression discontinuity, exploiting a sharp threshold, since people just above and just below an arbitrary cut-off for a scholarship, a class size rule or a benefit are otherwise nearly identical
- •Instrumental variables, using a factor that influences whether someone receives the treatment while having no other route to the outcome, which is a strong assumption and the one most often disputed
- •Lotteries, where a public authority has already randomised access to school places, housing or visas, which is as close to a real experiment as observational work gets
- •Twin and sibling comparisons, which hold family background and much genetics constant
- •Event studies around unexpected shocks, including natural disasters, sudden policy reversals and court rulings
The findings that changed minds
Several natural experiments produced results that overturned settled views, which is why the approach was recognised with a Nobel prize in 2021. Card and Krueger compared fast food employment in New Jersey, which raised its minimum wage in 1992, with neighbouring Pennsylvania, which did not, and found no employment loss, contradicting the textbook prediction and beginning a debate that continues with far better data on both sides. Studies of the Mariel boatlift, which delivered a sudden large influx of workers to Miami in 1980, examined the effect on local wages and produced findings that are still argued over, with reanalyses reaching different conclusions depending on which workers are compared. Angrist and Krueger used quarter of birth, which interacts with school starting age rules, as an instrument for years of schooling to estimate its effect on earnings. Regression discontinuity around class size rules in Israeli schools, based on a maximum class size that forces a split, produced credible estimates of the effect of smaller classes.
How they go wrong
The method's power depends entirely on the assumption that the division really is as good as random, and that assumption fails in identifiable ways. Anticipation undermines it, since people who know a policy is coming change behaviour before it arrives. Sorting undermines it, since if people can choose which side of a threshold to be on, the comparison is no longer between similar groups, which is why researchers test for bunching just above or below a cut-off. Spillovers undermine it, since if the untreated group is affected by the treatment, through migration, competition or imitation, it is not a valid comparison. Parallel trends, the core assumption of difference in differences, is untestable for the period after the change and is only supported by showing the groups moved together beforehand. And the estimate obtained applies to the people whose behaviour the instrument actually shifted, which may be an unrepresentative subgroup, so a credible local answer is not automatically a general one. The field's response has been increasingly explicit reporting of assumptions and routine testing of each.
The takeaway
A natural experiment uses variation created by something outside the researcher's control, such as a border, a threshold, a lottery or an unexpected event, to compare groups that were alike beforehand. Difference in differences, regression discontinuity and instrumental variables are the standard designs. Findings including the New Jersey minimum wage comparison overturned settled predictions and won a Nobel prize in 2021. The approach fails when people anticipate the change, sort across the threshold or affect each other.