Evidence-based policy is an idea in public policy proposing that policy decisions should be based on, or informed by, rigorously established objective evidence. The implied contrast is with policymaking based on ideology, 'common sense,' anecdotes, and intuitions. It is the government equivalent of the effective altruism movement. Evidence-based policy uses a thorough research method, such as randomized controlled trials (RCT). Good data, analytical skills, and political support to the use of scientific information are typically seen as the crucial elements of an evidence-based approach.
Some have promoted particular types of evidence as 'best' for policymakers to consider, including scientifically rigorous evaluation studies such as randomized controlled trials to identify programs and practices capable of improving policy-relevant outcomes. However, some areas of policy-relevant knowledge may not be well served by quantitative research. This has led to a debate about the type of evidence to use. For instance, policies concerned with human rights, public acceptability, or social justice may require proof other than randomized trials provide. Also, policy evaluation may require moral philosophical reasoning in addition to considerations of evidence of intervention effect (which randomized trials are principally designed to provide). The purpose of evidence-based policy is to use scientific evidence rigorously and comprehensively to inform decisions rather than to allow political processes to use them in a piecemeal, manipulated, or cherry-picked manner.
Some policy scholars now avoid using the term evidence-based policy, using others such as evidence-informed. This language shift allows continued thinking about the underlying desire to improve evidence use in terms of its rigor or quality while avoiding some of the key limitations or reductionist ideas at times seen with the evidence-based language. Still, the language of evidence-based policy is widely used and, as such, can be interpreted to reflect a desire for evidence to be used well or appropriately in one way or another—such as by ensuring systematic consideration of rigorous and high-quality policy-relevant evidence, or by avoiding biased and erroneous applications of evidence for political ends.
A related group is the rationalist community.
History
The move towards modern evidence-based policy has its roots in the larger movement towards evidence-based practice, which was prompted by the rise of evidence-based medicine in the 1980s. However, the term 'Evidence-based policy' didn't see use in medicine until the 1990s. The term wasn't used in social policy until the early 2000s. The earliest example of evidence-based policy was tariff-making in Australia. The legislation required tariffs to be educated by a public report issued by the Tariff Board. These reported on the tariff, industrial, and economic impacts.
History of evidence-based medicine
The phrase evidence-based medicine (EBM) was coined by Gordon Guyatt. However earlier example of EBM trace to the early 1900s. Some argue that the earliest form of EBM occurred in the 11th century, when Ben Cao Tu Jing from Song Dynasty said, "In order to evaluate the efficacy of ginseng, find two people and let one eat ginseng and run, the other run without ginseng. The one that did not eat ginseng will develop shortness of breath sooner." Many scholars see the term evidence-based policy as evolving from "evidence-based medicine", in which research findings are used as the support for clinical decisions and evidence is gathered by randomized controlled trials (RCTs), which is comparing a treatment group with a placebo group to measure results. Even though the earliest published RCTs in medicine were during WWII and post-war era: 1940s and 1905s, the word 'evidence-based medicine did not appear in published medical research until 1993. In 1993, the Cochrane Collaboration was established in the UK, and works to keep all RCTs up-to-date and provides "Cochrane reviews" which provides primary research in human health and health policy. The evolution of the appearance of the keyword EBM has increased since the 2000s and the effect of EBM has seen significant expansion to the field of medicine.
History of evidence-based policy making
Randomized Controlled Trials were late to appear in the social policy compared to the medical field. Although evidence-based approach can be traced as far back as the fourteenth century, it was more recently popularized by the Blair Government in the United Kingdom. The Blair Government said they wanted to end the ideological led-based decision-making for policy making. For example, a UK Government white paper published in 1999 ("Modernising Government") noted that Government must "produce policies that really deal with problems, that are forward-looking and shaped by evidence rather than a response to short-term pressures; that tackle causes not symptoms". There was then an increase in research and policy activists pushing for more evidence-based policy-making which led to the formation of the sister organization to Cochrane Collaboration, the Campbell Collaboration in 1999. The Campbell Collaboration conducts reviews on the best evidence that analyzes the effects of social and educational policies and practices.
The Economic and Social Research Council (ESRC) became involved in the push for more evidence-based policymaking with its 1.3 million pound grant to the Evidence Network in 1999. The Evidence Network is a center for evidence-based policy and practice and is similar to both the Campbell and Cochrane Collaboration. More recently the Alliance for Useful Evidence has been established, with funding from ESRC, Big Lottery and Nesta, to champion the use of evidence in social policy and practice. The Alliance is a UK-wide network that promotes the use of high-quality evidence to inform decisions on strategy, policy and practice through advocacy, publishing research, sharing ideas and advice, and holding events and training.
People practice evidence-based policy in different ways. For example, Michael Kremer and Rachel Glennerster had many theories about what would work best to improve students' test scores. Therefore, they conducted randomized controlled trials in Kenya. They tried new textbooks and flip charts, as well as smaller class sizes. However, they found that the only intervention that raised school attendance was treating intestinal worms in children. Based on their findings, they started the Deworm the World Initiative, which is rated by GiveWell as one of the best charities in the world for cost-effectiveness.
Recently questions have been raised about the conflicts of interest inherent to evidence-based decision-making used in public policy development. In a study of vocational education in prisons operated by the California Department of Corrections, Andrew J. Dick, William Rich, and Tony Waters found that political considerations inevitably intruded into “evidence-based decisions” which were ostensibly technocratic. They point out that this is particularly true where evidence is paid for by policymakers who have a vested interest in having past political judgments confirmed, evidence-based research is likely to be corrupted, leading to policy-based evidence making.
Methodology
There are many methodologies for evidence-based policy, however they all share the following characteristics:
- Tests a theory as to why the policy will be effective and what the impacts of the policy will be if it is successful
- Includes a counterfactual: what would have occurred if the policy had not been implemented
- Incorporates some measurement of the impact
- Examines both direct and indirect effects that occur because of the policy
- Separates the uncertainties and controls for other influences outside of the policy that may affect the outcome
- Should be able to be tested and replicated by a third party
The form of the methodology used with evidence-based policy fits under the cost-benefit framework. It is created to estimate a net payoff if the policy is implemented. Because there is a difficulty quantifying some effects and outcomes of the policy, it is mostly focused on whether benefits will outweigh costs, instead of using specific values.
Types of evidence for evidence-based policy making
All kinds of data can be considered a piece of evidence. The Scientific Method effectively organizes this data into tests to strengthen or weaken specific beliefs or hypotheses. For example, the results of different tests can be more or less convincing to the scientific community, based on blind experiment type (i.e., blind vs. double-blind), sample size, and replication. However, supports of evidence-based policy attempt to combine what citizens want (within Maslow's Hierarchy of needs) with what the scientific method has shown will be the most likely produce it.
Quantitative evidence
Numerical quantities from peer-reviewed journals, data from public surveillance systems, or individual programs are considered quantitative evidence for policymaking. Quantitative data can also be collected by the government or policymakers themselves through surveys. Qualitative evidence is widely used in EBM and evidence-based public health policy constructions.
Qualitative evidence
Qualitative evidence includes nonnumerical data collected by methods that include observations, interviews, or focus groups. Qualitative evidence is widely used to create compelling stories to impact those in decision-making authority. Although the evidence can be divided according to their type, there is no hierarchical weight over qualitative vs. quantitative data. They are both efficient in acting as evidence in certain areas than others. Often, qualitative and quantitative evidence are combined in the process of policymaking.
Cause priorities
Some approach Evidence-based policy with cause neutrality: they first define the goal or human interest and use Evidence-based processes to identify the most effective method.cause neutrality. Examples of causes include providing food for the hungry, protecting endangered species, mitigating climate change, reforming immigration policy, researching cures for illnesses, preventing sexual violence, alleviating poverty, eliminating factory farming, or averting nuclear warfare. Many people in the effective policy movement have prioritized global health and development, animal welfare, and mitigating risks that threaten the future of humanity.
Global health and development
The alleviation of global poverty and neglected tropical diseases has been a focus of some of the earliest and most prominent organizations associated with the movement to use evidence to make decisions.
Charity evaluator GiveWell was founded by Holden Karnofsky and Elie Hassenfeld in 2007 to address poverty. GiveWell has argued that the marginal impact of donations is greatest for attacking global poverty and health. Its leading recommendations have been in these domains: malaria prevention charities Against Malaria Foundation and Malaria Consortium, deworming charities Schistosomiasis Control Initiative and Deworm the World Initiative, and GiveDirectly for direct unconditional cash transfers.
The Life You Can Save, which originated from Singer's book of the same name, works to alleviate global poverty by promoting evidence-backed charities, conducting philanthropy education, and changing the culture of giving in affluent countries.
While much of the initial focus has been on direct strategies such as health interventions and cash transfers, more systematic social, economic, and political reforms meant to facilitate larger long-term poverty reduction have also attracted attention. In 2011, GiveWell announced the creation of GiveWell Labs, which was later renamed the Open Philanthropy Project, for the purpose of research and philanthropic funding of more speculative and diverse causes such as policy reform, global catastrophic risk reduction and scientific research. It is a collaboration between GiveWell and Good Ventures.
Long-term future and global catastrophic risks
Focusing on the long-term future, some believe that the total value of any meaningful metric (wealth, potential for suffering, potential for happiness, etc.) summed up over future generations, far exceeds the value for people living today. Some researchers have found it psychologically difficult to contemplate the trade-off; Toby Ord stated, "Since there is so much work to be done to fix the needless suffering in our present, I was slow to turn to the future." Reasons Ord gave for working on long-term issues include a belief that preventing long-term suffering is "even more neglected" than causes related to current suffering, and that the residents of the future are even more powerless to affect risks caused by current events than are current dispossessed populations".
Philosophically, assessing the suffering of future populations involves multiple considerations. First, humanity (and other animals) may not exist at all, in which cases there is no suffering to alleviate (presuming that the process of eliminating the population does not itself involve suffering). Second, the cost of an incremental reduction in suffering in the future may be higher (e.g., because of increasing healthcare costs) or lower (brought down, e.g., by the ever-crashing cost of computing or renewable energy). Third, the value of a benefit or cost is affected by the time preferences of the recipient and the payer. Fourth, future suffering may be alleviated by current spending, potentially at a lower cost. Fifth, alleviating suffering sooner may have a knock-on effect of reducing/increasing future suffering. Sixth, if investing money produces outsized returns, that may provide the ability to reduce total suffering by more than if the money is instead donated before it can accumulate. Seventh, future populations may be so much wealthier than the current population that, even if a particular reduction in suffering costs more than it does today, the population might still be better off by waiting. Singer argued that existential risk should not be "the dominant public face of the effective altruism movement" because he claimed that doing so would drastically limit the movement's reach.
In particular, the importance of addressing existential risks such as dangers associated with biotechnology and advanced artificial intelligence is often highlighted and the subject of active research. Because it is generally infeasible to use traditional research techniques such as randomized controlled trials to analyze existential risks, researchers such as Nick Bostrom have used methods such as expert opinion elicitation to estimate their importance. Ord offered probability estimates for a number of existential risks in his 2020 book The Precipice.
Organizations that work actively on research and advocacy for improving the long-term future are the Future of Humanity Institute at the University of Oxford, the Centre for the Study of Existential Risk at the University of Cambridge, and the Future of Life Institute. In addition, the Machine Intelligence Research Institute is focused on the more narrow mission of managing advanced artificial intelligence.
Evidence-based policy from non-government organizations
The Overseas Development Institute
The Overseas Development Institute claims that research-based evidence can contribute to policies that dramatically impact lives. Success stories quoted in the UK's Department for International Development's (DFID) new research strategy include a 22% reduction in neonatal mortality in Ghana as a result of helping women begin breastfeeding within one hour of giving birth and a 43% reduction in deaths among HIV positive children using a widely available antibiotic.
After many policy initiatives, the Overseas Development Institute evaluated their evidence-based policy efforts. They identified specific reasons that policy is weakly informed by research-based evidence. Policy processes are complex and rarely linear or logical. Therefore, simply presenting information to policy-makers and expecting them to act upon it is very unlikely to work. These reasons include information gaps, secrecy, the need for speedy responses and slow data, political expediency (what is popular), and the fact that policy-makers are not interested in making the policy more scientific. When a gap is identified between the scientific and political process, those interested in shrinking the gap must choose between making their politicians use scientific techniques or their scientists use more political methods.
The Overseas Development Institute concluded that, with the lack of evidence-based policy progress, those with the data should move into the political and advertising world of emotion and storytelling to influence those in power. They replaced simple tools such as cost–benefit analysis and logical frameworks, with identifying the key players, being good storytellers, synthesizing complex data from their research into simple, compelling stories. The Overseas Development Institute did not advocate for re-making the system to support evidence-based policy but encouraged those with data to jump into the political process.
Further, they concluded that turning someone who 'finds' data into someone who 'uses' data in our current system involves a fundamental reorientation towards policy engagement rather than academic achievement. This focus requires engaging much more with the policy community, developing a research agenda focusing on policy issues rather than academic interests, acquiring new skills or building multidisciplinary teams, establishing new internal systems and incentives, spending much more on communications, producing a different range of outputs, and working more in partnerships and networks.
Based on research conducted in six Asian and African countries, the Future Health Systems consortium has identified a set of critical strategies for improving uptake of evidence into policy, including improving the technical capacity of policy-makers; better packaging of research findings; use of social networks; establishment of fora to assist in linking evidence with policy outcomes.
The Pew Charitable Trust
The Pew Charitable Trust is a non-governmental organization that has attempts to use data, science, and facts to serve the public good. Pew has a Results First initiative that works with the different US states to implement evidence-based policymaking in the development of their laws. This initiative has developed a framework may be seen as an example of how to implement evidence based policy.
Pew's 5 key components of evidence-based policy are:
- Program assessment. Systematically review available evidence on the effectiveness of public programs.
- Develop an inventory of funded programs.
- Categorize programs by their evidence of effectiveness.
- Identify programs’ potential return on investment.
- Budget development. Incorporate evidence of program
effectiveness into budget and policy decisions, giving funding priority
to those that deliver a high return on investment of public funds.
- Integrate program performance information into the budget development process.
- Present information to policymakers in user-friendly formats that facilitate decision-making.
- Include relevant studies in budget hearings and committee meetings.
- Establish incentives for implementing evidence-based programs and practices.
- Build performance requirements into grants and contracts.
- Implementation oversight. Ensure that programs are effectively delivered and are faithful to their intended design.
- Establish quality standards to govern program implementation.
- Build and maintain capacity for ongoing quality improvement and monitoring of fidelity to program design.
- Balance program fidelity requirements with local needs.
- Conduct data-driven reviews to improve program performance.
- Outcome monitoring. Routinely measure and report outcome data to determine whether programs are achieving desired results.
- Develop meaningful outcome measures for programs, agencies, and the community.
- Conduct regular audits of systems for collecting and reporting performance data.
- Regularly report performance data to policymakers.
- Targeted evaluation. Conduct rigorous evaluations of new and untested programs to ensure that they warrant continued funding.
- Leverage available resources to conduct evaluations.
- Target evaluations to high-priority programs.
- Make better use of administrative data—information typically collected for operational and compliance purposes—to enhance program evaluations.
- Require evaluations as a condition for continued funding for new initiatives.
- Develop a centralized repository for program evaluations.
The Coalition for Evidence-Based Policy
The Coalition for Evidence-Based Policy was a nonprofit, nonpartisan organization, whose mission was to increase government effectiveness through the use of rigorous evidence about "what works." Since 2001, the Coalition worked with U.S. Congressional and Executive Branch officials and advanced evidence-based reforms in U.S. social programs, which have been enacted into law and policy. The Coalition claimed to have no affiliation with any programs or program models, and no financial interest in the policy ideas it supported, enabling it to serve as an independent, objective source of expertise to government officials on evidence-based policy.
Major new policy initiatives that were enacted into law with the work of Coalition with congressional and executive branch officials.
- Evidence-Based Home Visitation Program for at-risk families with young children (Department of Health and Human Services – HHS, $1.5 billion over 2010-2014
- Evidence-Based Teen Pregnancy Prevention Program (HHS, $109 million in FY14)
- Investing in Innovation Fund, to fund development and scale-up of evidence-based K-12 educational interventions (Department of Education, $142 million in FY14)
- First in the World Initiative, to fund development and scale-up of evidence-based interventions in postsecondary education (Department of Education, $75 million in FY14)
- Social Innovation Fund, to support public/private investment in evidence-based programs in low-income communities (Corporation for National and Community Service, $70 million in FY14)
- Trade Adjustment Assistance Community College and Career Training Grants Program, to fund development and scale-up of evidence-based education and career training programs for dislocated workers (Department of Labor – DOL, $2 billion over 2011–2014)
- Workforce Innovation Fund, to fund development and scale-up of evidence-based strategies to improve education/employment outcomes for U.S. workers (DOL, $47 million in FY14).
Their website now says "The Coalition wound down its operations in the spring of 2015, and the Coalition’s leadership and core elements of the group’s work have been integrated into the Laura and John Arnold Foundation". In 2003 the Coalition published a guide on educational evidenced-based practices.
Critiques
Several critiques have emerged. Paul Cairney, professor of politics and public policy at the University of Stirling in Scotland, argues that supporters of the idea underestimate the complexity of policy-making and misconstrue the way that policy decisions are usually made. Cartwright and Hardie oppose emphasizing randomized controlled trials (RCTs). They show that the evidence from RCTs is not always sufficient for undertaking decisions. In particular, they argue that extrapolating experimental evidence into policy context requires understanding what necessary conditions were present within the experimental setting and asserting that these factors also operate in the target of considered intervention. Furthermore, considering the prioritization of RCTs, the evidence-based policy can be accused of being preoccupied with narrowly understood ‘interventions’ denoting surgical actions on one causal factor to influence its effect.
The definition of intervention presupposed by the movement of evidence-based policy overlaps with James Woodward’s interventionist theory of causality. However, policy-making encompasses also other types of decisions such as institutional reforms and actions based on predictions. The other types of evidence-based decision-making do not require having at hand evidence for the causal relation to be invariant under intervention. Therefore, mechanistic evidence and observational studies suffice for introducing institutional reforms and undertaking actions that do not modify the causes of a causal claim.
Moreover, evidence has emerged of front-line public servants, like hospital managers, making decisions that actually worsen patients' care in order to hit pre-ordained targets. This argument was put forward by Professor Jerry Muller of the Catholic University of America in a book called The Tyranny of Metrics. According to articles published in Futures, evidence based policy—in the form of cost-based or risk analyses—may entail forms of compression and exclusion of the issues under analysis, also in relation to power asymmetries among different actors in their capacity to produce evidence. A comprehensive list of critiques, including the fact that policies shown to be successful in one place often fail in others, despite reaching a gold standard of evidence, has been compiled by the policy platform Apolitical.
