A natural experiment
is an empirical study in which individuals (or clusters of individuals)
are exposed to the experimental and control conditions that are
determined by nature or by other factors outside the control of the
investigators. The process governing the exposures arguably resembles random assignment. Thus, natural experiments are observational studies and are not controlled in the traditional sense of a randomized experiment.
Natural experiments are most useful when there has been a clearly
defined exposure involving a well defined subpopulation (and the absence
of exposure in a similar subpopulation) such that changes in outcomes
may be plausibly attributed to the exposure.
In this sense, the difference between a natural experiment and a
non-experimental observational study is that the former includes a
comparison of conditions that pave the way for causal inference, but the latter does not.
Natural experiments are employed as study designs when controlled experimentation is extremely difficult to implement or unethical, such as in several research areas addressed by epidemiology (like evaluating the health impact of varying degrees of exposure to ionizing radiation in people living near Hiroshima at the time of the atomic blast) and economics (like estimating the economic return on amount of schooling in US adults).
Natural experiments are employed as study designs when controlled experimentation is extremely difficult to implement or unethical, such as in several research areas addressed by epidemiology (like evaluating the health impact of varying degrees of exposure to ionizing radiation in people living near Hiroshima at the time of the atomic blast) and economics (like estimating the economic return on amount of schooling in US adults).
History
One of the best-known early natural experiments was the 1854 Broad Street cholera outbreak in London, England. On 31 August 1854, a major outbreak of cholera struck Soho. Over the next three days, 127 people near Broad Street died. By the end of the outbreak 616 people died. The physician John Snow identified the source of the outbreak as the nearest public water pump, using a map of deaths and illness that revealed a cluster of cases around the pump.
In this example, Snow discovered a strong association between the
use of the water from the pump, and deaths and illnesses due to
cholera. Snow found that the Southwark and Vauxhall Waterworks Company, which supplied water to districts with high attack rates, obtained the water from the Thames downstream from where raw sewage was discharged into the river. By contrast, districts that were supplied water by the Lambeth Waterworks Company,
which obtained water upstream from the points of sewage discharge, had
low attack rates. Given the near-haphazard patchwork development of the
water supply in mid-nineteenth century London, Snow viewed the
developments as "an experiment...on the grandest scale."
Of course, the exposure to the polluted water was not under the control
of any scientist. Therefore, this exposure has been recognized as being
a natural experiment.
Recent examples
Family size
An aim of a study Angrist and Evans (1998)
was to estimate the effect of family size on the labor market outcomes
of the mother. For at least two reasons, the correlations between family
size and various outcomes (e.g., earnings) do not inform us about how
family size causally affects labor market outcomes. First, both labor
market outcomes and family size may be affected by unobserved "third"
variables (e.g., personal preferences). Second, labor market outcomes
themselves may affect family size (called "reverse causality"). For
example, a woman may defer having a child if she gets a raise at work.
The authors observed that two-child families with either two boys or two
girls are substantially more likely to have a third child than
two-child families with one boy and one girl. The sex of the first two
children, then, constitutes a kind of natural experiment: it is as if an
experimenter had randomly assigned some families to have two children
and others to have three. The authors were then able to credibly
estimate the causal effect of having a third child on labor market
outcomes. Angrist and Evans found that childbearing had a greater impact
on poor and less educated women than on highly educated women although
the earnings impact of having a third child tended to disappear by that
child's 13th birthday. They also found that having a third child had
little impact on husbands' earnings.
Game shows
Within
economics, game shows are a frequently studied form of natural
experiment. While game shows might seem to be artificial contexts, they
can be considered natural experiments due to the fact that the context
arises without interference of the scientist. Game shows have been used
to study a wide range of different types of economic behavior, such as
decision making under risk and cooperative behavior.
Smoking ban
In Helena, Montana a smoking ban
was in effect in all public spaces, including bars and restaurants,
during the six-month period from June 2002 to December 2002. Helena is
geographically isolated and served by only one hospital. The
investigators observed that the rate of heart attacks
dropped by 40% while the smoking ban was in effect. Opponents of the
law prevailed in getting the enforcement of the law suspended after six
months, after which the rate of heart attacks went back up. This study was an example of a natural experiment, called a case-crossover experiment,
where the exposure is removed for a time and then returned. The study
also potentially suggests that the inability to control variables in
natural experiments can impede investigators from drawing firm
conclusions. Critics argued that the particularly large percentage
fluctuation in the rate of myocardial infarction was likely due to
chance, given the small population size.
Nuclear weapons testing
Nuclear weapons testing released large quantities of radioactive isotopes into the atmosphere, some of which could be incorporated into biological tissues. The release stopped after the Partial Nuclear Test Ban Treaty in 1963, which prohibited atmospheric nuclear tests. This resembled a large-scale pulse-chase experiment,
but could not have been performed as a regular experiment in humans due
to scientific ethics. Several types of observations were made possible
(in people born before 1963), such as determination of the rate of
replacement for cells in different human tissues.
Vietnam War draft
An
important question in economics research is what determines earnings.
Angrist (1990) evaluated the effects of military service on lifetime
earnings. Using statistical methods developed in econometrics, Angrist capitalized on the approximate random assignment of the Vietnam War draft lottery, and used it as an instrumental variable
associated with eligibility (or non-eligibility) for military service.
Because many factors might predict whether someone serves in the
military, the draft lottery frames a natural experiment whereby those
drafted into the military can be compared against those not drafted
because the two groups should not differ substantially prior to military
service. Angrist found that the earnings of veterans were, on average,
about 15 percent less than the earnings of non-veterans.
Industrial melanism
With the Industrial Revolution in the nineteenth century, many species of moth, including the well-studied peppered moth, responded to the atmospheric pollution of sulphur dioxide and soot around cities with industrial melanism,
a dramatic increase in the frequency of dark forms over the formerly
abundant pale, speckled forms. In the twentieth century, as regulation
improved and pollution fell, providing the conditions for a large-scale
natural experiment, the trend towards industrial melanism was reversed,
and melanic forms quickly became scarce. The effect led the evolutionary
biologists L. M. Cook and J. R. G. Turner to conclude that "natural selection is the only credible explanation for the overall decline".
