Likelihood principle

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Likelihood_principle

In statistics, the likelihood principle is the proposition that, given a statistical model, all the evidence in a sample relevant to model parameters is contained in the likelihood function.

A likelihood function arises from a probability density function considered as a function of its distributional parameterization argument. For example, consider a model which gives the probability density function ƒ_X(x | θ) of observable random variable X as a function of a parameter θ. Then for a specific value x of X, the function ${\displaystyle {\mathcal {L}}}$(θ | x) = ƒ_X(x | θ) is a likelihood function of θ: it gives a measure of how "likely" any particular value of θ is, if we know that X has the value x. The density function may be a density with respect to counting measure, i.e. a probability mass function.

Two likelihood functions are equivalent if one is a scalar multiple of the other. The likelihood principle is this: all information from the data that is relevant to inferences about the value of the model parameters is in the equivalence class to which the likelihood function belongs. The strong likelihood principle applies this same criterion to cases such as sequential experiments where the sample of data that is available results from applying a stopping rule to the observations earlier in the experiment.

Example

Suppose

• X is the number of successes in twelve independent Bernoulli trials with probability θ of success on each trial, and
• Y is the number of independent Bernoulli trials needed to get three successes, again with probability θ (= 1/2 for a coin-toss) of success on each trial.

Then the observation that X = 3 induces the likelihood function

${\displaystyle {\mathcal {L}}(\theta \mid X=3)={\binom {12}{3}}\theta ^{3}(1-\theta )^{9}=220\theta ^{3}(1-\theta )^{9},}$

while the observation that Y = 12 induces the likelihood function

${\displaystyle {\mathcal {L}}(\theta \mid Y=12)={\binom {11}{2}}\theta ^{3}(1-\theta )^{9}=55\theta ^{3}(1-\theta )^{9}.}$

The likelihood principle says that, as the data are the same in both cases, the inferences drawn about the value of θ should also be the same. In addition, all the inferential content in the data about the value of θ is contained in the two likelihoods, and is the same if they are proportional to one another. This is the case in the above example, reflecting the fact that the difference between observing X = 3 and observing Y = 12 lies not in the actual data, but merely in the design of the experiment. Specifically, in one case, one has decided in advance to try twelve times; in the other, to keep trying until three successes are observed. The inference about θ should be the same, and this is reflected in the fact that the two likelihoods are proportional to each other.

This is not always the case, however. The use of frequentist methods involving p-values leads to different inferences for the two cases above, showing that the outcome of frequentist methods depends on the experimental procedure, and thus violates the likelihood principle.

The law of likelihood

A related concept is the law of likelihood, the notion that the extent to which the evidence supports one parameter value or hypothesis against another is equal to the ratio of their likelihoods, their likelihood ratio. That is,

${\displaystyle \Lambda ={{\mathcal {L}}(a\mid X=x) \over {\mathcal {L}}(b\mid X=x)}={P(X=x\mid a) \over P(X=x\mid b)}}$

is the degree to which the observation x supports parameter value or hypothesis a against b. If this ratio is 1, the evidence is indifferent; if greater than 1, the evidence supports the value a against b; or if less, then vice versa. 

In Bayesian statistics, this ratio is known as the Bayes factor, and Bayes' rule can be seen as the application of the law of likelihood to inference.

In frequentist inference, the likelihood ratio is used in the likelihood-ratio test, but other non-likelihood tests are used as well. The Neyman–Pearson lemma states the likelihood-ratio test is the most powerful test for comparing two simple hypotheses at a given significance level, which gives a frequentist justification for the law of likelihood.

Combining the likelihood principle with the law of likelihood yields the consequence that the parameter value which maximizes the likelihood function is the value which is most strongly supported by the evidence. This is the basis for the widely used method of maximum likelihood.

History

The likelihood principle was first identified by that name in print in 1962 (Barnard et al., Birnbaum, and Savage et al.), but arguments for the same principle, unnamed, and the use of the principle in applications goes back to the works of R.A. Fisher in the 1920s. The law of likelihood was identified by that name by I. Hacking (1965). More recently the likelihood principle as a general principle of inference has been championed by A. W. F. Edwards. The likelihood principle has been applied to the philosophy of science by R. Royall.

Birnbaum proved that the likelihood principle follows from two more primitive and seemingly reasonable principles, the conditionality principle and the sufficiency principle. The conditionality principle says that if an experiment is chosen by a random process independent of the states of nature ${\displaystyle \theta }$, then only the experiment actually performed is relevant to inferences about ${\displaystyle \theta }$. The sufficiency principle says that if ${\displaystyle T(X)}$ is a sufficient statistic for ${\displaystyle \theta }$, and if in two experiments with data ${\displaystyle x_{1}}$ and ${\displaystyle x_{2}}$ we have ${\displaystyle T(x_{1})=T(x_{2})}$, then the evidence about ${\displaystyle \theta }$ given by the two experiments is the same.

Arguments for and against

Some widely used methods of conventional statistics, for example many significance tests, are not consistent with the likelihood principle.

Let us briefly consider some of the arguments for and against the likelihood principle.

The original Birnbaum argument

Birnbaum's proof of the likelihood principle has been disputed by philosophers of science, including Deborah Mayo and statisticians including Michael Evans. On the other hand, a new proof of the likelihood principle has been provided by Greg Gandenberger.

Experimental design arguments on the likelihood principle

Unrealized events play a role in some common statistical methods. For example, the result of a significance test depends on the p-value, the probability of a result as extreme or more extreme than the observation, and that probability may depend on the design of the experiment. To the extent that the likelihood principle is accepted, such methods are therefore denied.

Some classical significance tests are not based on the likelihood. A commonly cited example is the optional stopping problem. Suppose I tell you that I tossed a coin 12 times and in the process observed 3 heads. You might make some inference about the probability of heads and whether the coin was fair. Suppose now I tell that I tossed the coin until I observed 3 heads, and I tossed it 12 times. Will you now make some different inference? 

The likelihood function is the same in both cases: it is proportional to

${\displaystyle p^{3}(1-p)^{9}.}$

According to the likelihood principle, the inference should be the same in either case. 

Suppose a number of scientists are assessing the probability of a certain outcome (which we shall call 'success') in experimental trials. Conventional wisdom suggests that if there is no bias towards success or failure then the success probability would be one half. Adam, a scientist, conducted 12 trials and obtains 3 successes and 9 failures. Then he left the lab.

Bill, a colleague in the same lab, continued Adam's work and published Adam's results, along with a significance test. He tested the null hypothesis that p, the success probability, is equal to a half, versus p < 0.5. The probability of the observed result that out of 12 trials 3 or something fewer (i.e. more extreme) were successes, if H₀ is true, is

${\displaystyle \left({12 \choose 9}+{12 \choose 10}+{12 \choose 11}+{12 \choose 12}\right)\left({1 \over 2}\right)^{12}}$

which is 299/4096 = 7.3%. Thus the null hypothesis is not rejected at the 5% significance level.

Charlotte, another scientist, reads Bill's paper and writes a letter, saying that it is possible that Adam kept trying until he obtained 3 successes, in which case the probability of needing to conduct 12 or more experiments is given by

${\displaystyle 1-\left({10 \choose 2}\left({1 \over 2}\right)^{11}+{9 \choose 2}\left({1 \over 2}\right)^{10}+\cdots +{2 \choose 2}\left({1 \over 2}\right)^{3}\right)}$

which is 134/4096 = 3.27%. Now the result is statistically significant at the 5% level. Note that there is no contradiction among these two results; both computations are correct. 

To these scientists, whether a result is significant or not depends on the design of the experiment, not on the likelihood (in the sense of the likelihood function) of the parameter value being 1/2.

Results of this kind are considered by some as arguments against the likelihood principle. For others it exemplifies the value of the likelihood principle and is an argument against significance tests.

Similar themes appear when comparing Fisher's exact test with Pearson's chi-squared test.

The voltmeter story

An argument in favor of the likelihood principle is given by Edwards in his book Likelihood. He cites the following story from J.W. Pratt, slightly condensed here. Note that the likelihood function depends only on what actually happened, and not on what could have happened.

An engineer draws a random sample of electron tubes and measures their voltages. The measurements range from 75 to 99 Volts. A statistician computes the sample mean and a confidence interval for the true mean. Later the statistician discovers that the voltmeter reads only as far as 100 Volts, so technically, the population appears to be “censored”. If the statistician is orthodox this necessitates a new analysis. However, the engineer says he has another meter reading to 1000 Volts, which he would have used if any voltage had been over 100. This is a relief to the statistician, because it means the population was effectively uncensored after all. But later, the statistician ascertains that the second meter was not working at the time of the measurements. The engineer informs the statistician that he would not have held up the original measurements until the second meter was fixed, and the statistician informs him that new measurements are required. The engineer is astounded. “Next you'll be asking about my oscilloscope!”

This story can be translated to Adam's stopping rule above, as follows. Adam stopped immediately after 3 successes, because his boss Bill had instructed him to do so. After the publication of the statistical analysis by Bill, Adam realizes that he has missed a second instruction from Bill to conduct 12 trials instead, and that Bill's paper is based on this second instruction. Adam is very glad that he got his 3 successes after exactly 12 trials, and explains to his friend Charlotte that by coincidence he executed the second instruction. Later, he is astonished to hear about Charlotte's letter explaining that now the result is significant.

Misuse of p-values

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Misuse_of_p-values

Misuse of p-values is common in scientific research and scientific education. p-values are often used or interpreted incorrectly; the American Statistical Association states that p-values can indicate how incompatible the data are with a specified statistical model. From a Neyman–Pearson hypothesis testing approach to statistical inferences, the data obtained by comparing the p-value to a significance level will yield one of two results: either the null hypothesis is rejected (which however does not prove that the null hypothesis is false), or the null hypothesis cannot be rejected at that significance level (which however does not prove that the null hypothesis is true). From a Fisherian statistical testing approach to statistical inferences, a low p-value means either that the null hypothesis is true and a highly improbable event has occurred or that the null hypothesis is false.

Clarifications about p-values

The following list clarifies some issues that are commonly misunderstood regarding p-values:

1. The p-value is not the probability that the null hypothesis is true, or the probability that the alternative hypothesis is false. A p-value can indicate the degree of compatibility between a dataset and a particular hypothetical explanation (such as a null hypothesis). Specifically, the p-value can be taken as the prior probability of obtaining an effect that is at least as extreme as the observed effect, given that the null hypothesis is true. This should not be confused with the posterior probability that the null hypothesis is true given the observed effect. In fact, frequentist statistics does not attach probabilities to hypotheses.
2. The p-value is not the probability that the observed effects were produced by random chance alone. The p-value is computed under the assumption that a certain model, usually the null hypothesis, is true. This means that the p-value is a statement about the relation of the data to that hypothesis.
3. The 0.05 significance level is merely a convention. The 0.05 significance level (alpha level) is often used as the boundary between a statistically significant and a statistically non-significant p-value. However, this does not imply that there is generally a scientific reason to consider results on opposite sides of any threshold as qualitatively different.
4. The p-value does not indicate the size or importance of the observed effect. A small p-value can be observed for an effect that is not meaningful or important. In fact, the larger the sample size, the smaller the minimum effect needed to produce a statistically significant p-value.

Representing probabilities of hypotheses

A frequentist approach rejects the validity of representing probabilities of hypotheses: hypotheses are true or false, not something that can be represented with a probability.

Bayesian statistics actively models the likelihood of hypotheses. The p-value does not in itself allow reasoning about the probabilities of hypotheses, which requires multiple hypotheses or a range of hypotheses, with a prior distribution of likelihoods between them, in which case Bayesian statistics could be used. There, one uses a likelihood function for all possible values of the prior instead of the p-value for a single null hypothesis. The p-value describes a property of data when compared to a specific null hypothesis; it is not a property of the hypothesis itself. For the same reason, p-values do not give the probability that the data were produced by random chance alone.

Multiple comparisons problem

The multiple comparisons problem occurs when one considers a set of statistical inferences simultaneously or infers a subset of parameters selected based on the observed values. It is also known as the look-elsewhere effect. Errors in inference, including confidence intervals that fail to include their corresponding population parameters or hypothesis tests that incorrectly reject the null hypothesis, are more likely to occur when one considers the set as a whole. Several statistical techniques have been developed to prevent this from happening, allowing significance levels for single and multiple comparisons to be directly compared. These techniques generally require a higher significance threshold for individual comparisons, so as to compensate for the number of inferences being made.

The webcomic xkcd satirized misunderstandings of p-values by portraying scientists investigating the claim that eating jellybeans caused acne. After failing to find a significant (p < 0.05) correlation between eating jellybeans and acne, the scientists investigate 20 different colors of jellybeans individually, without adjusting for multiple comparisons. They find one color (green) nominally associated with acne (p < 0.05). The results are then reported by a newspaper as indicating that green jellybeans are linked to acne at a 95% confidence level—as if green were the only color tested. In fact, if 20 independent tests are conducted at the 0.05 significance level and all null hypotheses are true, there is a 64.2% chance of obtaining at least one false positive and the expected number of false positives is 1 (i.e. 0.05 × 20).

In general, the family-wise error rate (FWER)—the probability of obtaining at least one false positive—increases with the number of tests performed. The FWER when all null hypotheses are true for m independent tests, each conducted at significance level α, is:

${\displaystyle {\text{FWER}}=1-(1-\alpha )^{m}}$

Evidence-based toxicology

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Evidence-based_toxicology

The discipline of evidence-based toxicology (EBT) strives to transparently, consistently, and objectively assess available scientific evidence in order to answer questions in toxicology, the study of the adverse effects of chemical, physical, or biological agents on living organisms and the environment, including the prevention and amelioration of such effects. EBT has the potential to address concerns in the toxicological community about the limitations of current approaches to assessing the state of the science. These include concerns related to transparency in decision making, synthesis of different types of evidence, and the assessment of bias and credibility. Evidence-based toxicology has its roots in the larger movement towards evidence-based practices.

By analogy to evidence-based medicine (EBM), the umbrella term evidence-based toxicology (EBT) has been coined to group all approaches intended to better implement the above-mentioned evidence-based principles in toxicology in general and in toxicological decision-making in particular. Besides systematic reviews, the core evidence-based tool, such approaches include inter alia the establishment and universal use of a common ontology, justified design and rigorous conduct of studies, consistently structured and detailed reporting of experimental evidence, probabilistic uncertainty and risk assessment, and the development of synthesis methodology to integrate evidence from diverse evidence streams, e.g. from human observational studies, animal studies, in vitro studies and in silico modeling. A main initial impetus for translating evidence-based approaches to toxicology was the need to improve the performance assessment of toxicological test methods. The U.S. National Research Council (NRC) concurs that new means of assessment are needed to keep pace with recent advances in the development of toxicological test methods, capitalizing on enhanced scientific understanding through modern biochemistry and molecular biology.

A key tool in evidence-based medicine that holds promise for EBT is the systematic review. Historically, authors of reviews assessing the results of toxicological studies on a particular topic have searched, selected, and weighed the scientific evidence in a non-systematic and non-transparent way. Due to their narrative nature, these reviews tend to be subjective, potentially biased, and not readily reproducible. Two examples highlighting these deficiencies are the risk assessments of trichloroethylene and bisphenol A (BPA). Twenty-seven different risk assessments of the evidence that trichloroethylene causes cancer have come to substantially different conclusions. Assessments of BPA range from low risk of harm to the public to potential risks (for some populations), leading to different political decisions. Systematic reviews can help reducing such divergent views. In contrast with narrative reviews, they reflect a highly structured approach to reviewing and synthesizing the scientific literature while limiting bias. The steps to carrying out a systematic review include framing the question to be addressed; identifying and retrieving relevant studies; determining if any retrieved studies should be excluded from the analysis; and appraising the included studies in terms of their methodological quality and risk of bias. Ultimately the data should be synthesized across studies, if possible by a meta-analysis. A protocol of how the review will be conducted is prepared ahead of time and ideally should be registered and/or published.

Scientists have made progress in their efforts to apply the systematic review framework to evaluating the evidence for associations between environmental toxicants and human health risks. To date, researchers have shown that important elements of the framework established in evidence-based medicine can be adapted to toxicology with little change, and some studies have been attempted. Researchers using the systematic review methodology to address toxicological concerns include a group of scientists from government, industry, and academia in North America and the European Union (EU) who have joined together to promote evidence-based approaches to toxicology through the nonprofit Evidence-based Toxicology Collaboration (EBTC). The EBTC brings together the international toxicology community to develop EBT methodology and facilitate the use of EBT to inform regulatory, environmental and public health.

Background

Evidence-based approaches were first conceived as a means of anchoring policy decisions, not to current practices or the beliefs of experts, but to experimental evidence. Evidence-based medicine (EBM) was launched slightly later. Its rise as a distinct discipline is generally credited to the work and advocacy of Scottish epidemiologist Archie Cochrane. The Cochrane Collaboration named in his honor was launched at Oxford University in 1993 to promote evidence-based reviews of clinical medical literature. More recently, EBM expanded to encompass evidence-based health care (EBHC).

EBM/HC involves the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients taking patients' preferences into account. Prior to EBM, medical decisions about diagnosis, prevention, treatment or harm were often made without a rigorous evaluation of the alternatives. Research in the 1970s and 1980s showed that different physicians regularly recommended different treatments and tests for patients with ailments that were essentially the same, and that large proportions of procedures being performed by physicians were considered inappropriate by the standards of medical experts. EBM/HC supporters stress that while evidence always has been important to the practice of medicine, EBM/HC provides an enhanced approach of identifying, assessing, and summarizing evidence. EBT's supporters make a similar argument.

The idea of translating evidence-based approaches from medicine to toxicology has been percolating for two decades, with proponents in both medicine and toxicology. Three research papers published in 2005 and 2006 catalyzed what eventually became known as EBT by suggesting that EBM's established tools and concepts might serve as a prototype of evidence-based decision-making in toxicology.

Process and progress

The First International Forum Toward Evidence-Based Toxicology was held in 2007. The forum was organized by the European Commission and attended by 170 scientists from more than 25 European, American, and Asian countries. The goal was to explore the available concepts of EBT, and to launch an initiative to formally implement evidence-based assessment methods in toxicology. 

The starting point for the discussions were two research papers suggesting that the tools and concepts established in evidence-based medicine could serve as a prototype of evidence-based decision-making for evaluating toxicological data. Apparent fundamental differences between medicine and toxicology were carefully considered during these discussions. Forum participants attempted to bridge the two disciplines in order to make use of the accrued wisdom and apply this approach to toxicology. (See http://www.ebtox.org/resources/evidence-based-toxicology-explained/ .) 

The proceedings of this forum were published as a special issue in Human & Experimental Toxicology.

EBT's proponents include experts in EBM, public health, and toxicology who believe that EBT can help toxicologists to better serve the goals of health protection and safety assurance. They argue that EBT's methodologies for collecting, appraising, and pooling evidence can help ensure that all available information on a given topic is evaluated in a transparent, unbiased, and reproducible manner. They contend that EBT's concept of the systematic review could prove particularly helpful for the standardization and quality assurance of novel methodologies for evaluating toxicity, as well as for their formal validation. In this regard, EBT may prove particularly useful for assessing the performance of newer non-animal “21st century” toxicology tools. EBT can also help scientists integrate new toxicological test methods into test strategies being implemented across the globe.

In 2010, a group of EBT supporters joined together to convene a workshop titled “21st Century Validation for 21st Century Tools.” The session on the potential for evidence-based approaches to assess the performance of the new generation of non-animal test methods inspired the formation of the EBTC. The EBTC was officially launched in the U.S. in 2011 at a Society of Toxicology conference and convened its first workshop in 2012. The EBTC's EU branch was officially opened during the 2012 Eurotox conference.

In 2014, the EBTC hosted a workshop on ‘The Emergence of Systematic Review and Related Evidence-based Approaches in Toxicology“ with speakers representing US and European organizations that are implementing and promoting the use of systematic reviews for toxicological questions. The experts noted that the structured approach of systematic reviews increases objectivity and transparency but also made clear that the approach requires a substantial time investment, which is a challenge to its more widespread adoption. Consequently, the participants called for close collaboration of interested organizations, which they determined to be a pre-requisite for the broad and efficient introduction of systematic reviews in toxicology.

Applications of EBT

Regulatory decision-making

Some scientists and policymakers would like EBT to help them combine information from various sources. Toxicological evidence can be assigned to evidence streams, sets of studies representing the same type or level of evidence, such as human (observational) studies, animal studies, in vitro or mechanistic studies. EBT can be applied both within one evidence stream, and it is especially well-suited to be applied across multiple evidence streams. Regulators often designate one study as “the lead study,” then use later studies as additional information. Many perceive this as unsatisfying, but objective approaches to combine study results are lacking. The EBM concept of the systematic review has promise for this application, and some structured reviews serve as forerunners for this approach.

Evaluating effects of environmental exposures

The U.S. National Toxicology Program's Office of Health Assessment and Translation (OHAT) has started to use systematic review methodology for the program's evaluations. The first systematic review was completed in 2016, reviewing the effects of fluoride on learning and memory in animal studies. OHAT’s approach is tailored to its mandate, but its seems especially appropriate for substances with substantial yet conflicting literature, and hence the need for systematic reviews to sort out somewhat confusing situations.

Causation

One application of EBT focuses on causation. It addresses the challenge of tracing a health effect back to a toxicant, such as lung cancer to smoking. This approach is similar to legal arguments. Some experts warn that this approach could increase the evidence burden for proving causation, and thereby increase the difficulty involved in banning toxic substances.

Clinical toxicology

Practitioners of clinical toxicology, which is concerned with the treatment of patients known to be exposed to toxic substances, are also beginning to use an EBM-style approach. Guidance documents based on this approach have already been published.

21st century toxicology

The National Research Council's (NRC) landmark 2007 publication, Toxicity Testing in the 21st Century, has also been an impetus for EBT. EBT provides new tools for assessing test method performance. Also, as the focus of 21st-century toxicology shifts from animal biology to human biology, EBT provides a method for comparatively evaluating the results gleaned from new methods of investigating the effects of chemical exposure.

Limitations and challenges

The specific differences between toxicology and medicine/health care cause challenges for implementing EBT. Evidence-based methodology of clinical research has been focused on a single type of study—randomized, controlled clinical trials, which are a direct measure of the effectiveness of the health care intervention under scrutiny. In contrast, toxicology employs a variety of different kinds of studies in three distinct evidence streams: human (observational) studies, animal studies, and non-animal studies. Because human evidence is frequently lacking, most evidence is obtained by using animal and non-animal models, which—by definition—is more difficult to generalize and extrapolate to humans. This methodological heterogeneity complicates evidence integration within an evidence stream, such as when inconsistent evidence is obtained from different animal species, but even more so across evidence streams. Adding to the difficulty is the reality that much toxicological evidence, more so than in medicine and health care, is not readily accessible in the literature. Moreover, the role of expert judgment, especially in systematic reviews, needs to be clearly defined, as it is a common misperception that evidence-based approaches leave no room for it. Systematic reviews should strive to make expert judgments clear along with the scientific basis for those judgments in developing conclusions for a systematic review. Further issues to be worked out include exposures to multiple substances, the multitude of outcomes observed in some animal studies, and challenges in improving the experimental designs and reporting of studies.

Evidence-based practice

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Evidence-based_practice

An evidence-based practice (EBP) is any practice that relies on scientific evidence for guidance and decision-making. Practices that are not evidence-based may rely on tradition, intuition, or other unproven methods. Evidence-based practices have been gaining ground since the formal introduction of evidence-based medicine in 1992, and have spread to the allied health professions, education, management, law, public policy, and other fields. In light of studies showing problems in scientific research (such as the replication crisis), there is also a movement to apply evidence-based practices in scientific research itself. Research into the evidence-based practice of science is called metascience.

The movement towards evidence-based practices attempts to encourage, and in some instances to force, professionals and other decision-makers to pay more attention to evidence to inform their decision-making. The goal of evidence-based practice is eliminate unsound or outdated practices in favor of more effective ones by shifting the basis for decision making from tradition, intuition, and unsystematic experience to firmly grounded scientific research.

History

For most of history, professions have based their practices on expertise derived from experience passed down in the form of tradition. Many of these practices have not been justified by evidence, which has sometimes enabled quackery and poor performance. Even when overt quackery is not present, quality and efficiency of tradition-based practices may not be optimal. As the scientific method has become increasingly recognized as a sound means to evaluate practices, evidence-based practices have become increasingly adopted.

One of the earliest proponents of EBP was Archie Cochrane, an epidemiologist who authored the book Effectiveness and Efficiency: Random Reflections on Health Services in 1972. Cochrane's book argued for the importance of properly testing health care strategies, and was foundational to the evidence-based practice of medicine. Cochrane suggested that because resources would always be limited, they should be used to provide forms of health care which had been shown in properly designed evaluations to be effective. Cochrane maintained that the most reliable evidence was that which came from randomised controlled trials.

The term "evidence-based medicine" was introduced in 1992. This marked the first evidence-based practice to be formally established. Some early experiments in evidence-based medicine involved testing primitive medical techniques such as bloodletting, and studying the effectiveness of modern and accepted treatments. There has been a push for evidence-based practices in medicine by insurance providers, which have sometimes refused coverage of practices lacking in systematic evidence of usefulness. It is now expected by most clients that medical professionals should make decisions based on evidence, and stay informed about the most up-to-date information. Since the widespread adoption of evidence-based practices in medicine, the use of evidence-based practices has rapidly spread to other fields.

More recently, there has been a push for evidence-based education. The use of evidence-based learning techniques such as spaced repetition can improve students' rate of learning. Some commentators have suggested that the putative lack of any conspicuous progress in the field of education is attributable to practice resting in the unconnected and noncumulative experience of thousands of individual teachers, each re-inventing the wheel and failing to learn from hard scientific evidence about 'what works'. Opponents of this view argue that hard scientific evidence is a misnomer in education; knowing that a drug works (in medicine) is entirely different from knowing that a teaching method works, for the latter will depend on a host of factors, not least those to do with the style, personality and beliefs of the teacher and the needs of the particular children (Hammersley 2013). Some opponents of EBP in education suggest that teachers need to develop their own personal practice, dependent on personal knowledge garnered through their own experience. Others argue that this must be combined with research evidence, but without the latter being treated as a privileged source.

Vs. tradition

Evidence-based practice is a philosophical approach that is in opposition to tradition. Some degree of reliance on "the way it was always done" can be found in almost every profession, even when those practices are contradicted by new and better information.

Some critics argue that since research is conducted on a population level, results may not generalise to each individual within the population. Therefore, evidence-based practices may fail to provide the best solution to each individual, and traditional practices may better accommodate individual differences. In response, researchers have made an effort to test whether particular practices work better for different subcultures, personality types etc. Some authors have redefined EBP to include practice that incorporates common wisdom, tradition, and personal values in alongside practices based on evidence.

Evaluating evidence

Evaluating scientific research is extremely complex. The process can by greatly simplified with the use of a heuristic that ranks the relative strengths of results obtained from scientific research called a hierarchy of evidence. The design of the study and the endpoints measured (such as survival or quality of life) affect the strength of the evidence. Typically, systematic reviews and meta-analysies rank at the top of the hierarchy while randomized controlled trials rank above observational studies, and expert opinion and case reports rank at the bottom. There is broad agreement on the relative strength of the different types of studies, but there is no single, universally-accepted hierarchy of evidence. More than 80 different hierarchies have been proposed for assessing medical evidence.

Applications of evidence-based practice

Medicine

Evidence-based medicine (EBM) is an approach to medical practice intended to optimize decision-making by emphasizing the use of evidence from well-designed and well-conducted research. Although all medicine based on science has some degree of empirical support, EBM goes further, classifying evidence by its epistemologic strength and requiring that only the strongest types (coming from meta-analyses, systematic reviews, and randomized controlled trials) can yield strong recommendations; weaker types (such as from case-control studies) can yield only weak recommendations. The term was originally used to describe an approach to teaching the practice of medicine and improving decisions by individual physicians about individual patients.^[10] Use of the term rapidly expanded to include a previously described approach that emphasized the use of evidence in the design of guidelines and policies that apply to groups of patients and populations ("evidence-based practice policies").

Whether applied to medical education, decisions about individuals, guidelines and policies applied to populations, or administration of health services in general, evidence-based medicine advocates that to the greatest extent possible, decisions and policies should be based on evidence, not just the beliefs of practitioners, experts, or administrators. It thus tries to assure that a clinician's opinion, which may be limited by knowledge gaps or biases, is supplemented with all available knowledge from the scientific literature so that best practice can be determined and applied. It promotes the use of formal, explicit methods to analyze evidence and makes it available to decision makers. It promotes programs to teach the methods to medical students, practitioners, and policymakers.

A process has been specified that provides a standardised route for those seeking to produce evidence of the effectiveness of interventions. Originally developed to establish processes for the production of evidence in the housing sector, the standard is general in nature and is applicable across a variety of practice areas and potential outcomes of interest.

Mental Health

Not all mental health practitioners receive training in evidence-based approaches, and members of the public are often unaware that evidence-based practices exist. Consequently, patients do not always receive the most effective, safe, and cost-effective treatments available. To improve dissemination of evidence-based practices, the Association for Behavioral and Cognitive Therapies (ABCT) and the Society of Clinical Child and Adolescent Psychology (SCCAP, Division 53 of the American Psychological Association) maintain updated information on their websites on evidence-based practices in psychology for practitioners and the general public. An evidence-based practice consensus statement was developed at a summit on mental healthcare in 2018. As of June 23, 2019, this statement has been endorsed by 36 organizations.

Social policy

There are increasing demands for the whole range of social policy and other decisions and programs run by government and the NGO sector to be based on sound evidence as to their effectiveness. This has seen an increased emphasis on the use of a wide range of Evaluation approaches directed at obtaining evidence about social programs of all types. A research collaboration called the Campbell Collaboration has been set up in the social policy area to provide evidence for evidence-based social policy decision-making. This collaboration follows the approach pioneered by the Cochrane Collaboration in the health sciences. Using an evidence-based approach to social policy has a number of advantages because it has the potential to decrease the tendency to run programs which are socially acceptable (e.g. drug education in schools) but which often prove to be ineffective when evaluated.

Scientific research

As with other fields, many practices in scientific research are rooted in tradition rather than evidence, and are unproven. John Ioannidis 2005 paper "Why most published research findings are false" found evidence that these poor practices regularly result in false findings and enormous waste. The paper was the most downloaded in the Public Library of Science, and has the highest number of Mendeley readers across all science." There has since been a movement for the use of evidence-based practice in conducting scientific research in attempt to address the replication crisis and other major issues affecting scientific research. The application of evidence-based practices to research itself is called metascience. 

Metascience seeks to increase the quality of scientific research while reducing waste. It is also known as "research on research" and "the science of science", as it uses research methods to study how research is done and where improvements can be made. The five main areas of research in metascience are methodology, reporting, reproducibility, evaluation, and incentives. Metascience has produced a number of reforms in science such as the use of study pre-registration and the implementation of reporting guidelines with the goal of bettering scientific research practices.

Metascience

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Metascience

Metascience (also known as meta-research or evidence-based research) is the use of scientific methodology to study science itself. Metascience seeks to increase the quality of scientific research while reducing waste. It is also known as "research on research" and "the science of science", as it uses research methods to study how research is done and where improvements can be made. Metascience concerns itself with all fields of research and has been described as "a bird's eye view of science." In the words of John Ioannidis, "Science is the best thing that has happened to human beings ... but we can do it better."

In 1966, an early meta-research paper examined the statistical methods of 295 papers published in ten high-profile medical journals. It found that, "in almost 73% of the reports read ... conclusions were drawn when the justification for these conclusions was invalid." Meta-research in the following decades found many methodological flaws, inefficiencies, and poor practices in research across numerous scientific fields. Many scientific studies could not be reproduced, particularly in medicine and the soft sciences. The term "replication crisis" was coined in the early 2010s as part of a growing awareness of the problem.

Measures have been implemented to address the issues revealed by metascience. These measures include the pre-registration of scientific studies and clinical trials as well as the founding of organizations such as CONSORT and the EQUATOR Network that issue guidelines for methodology and reporting. There are continuing efforts to reduce the misuse of statistics, to eliminate perverse incentives from academia, to improve the peer review process, to combat bias in scientific literature, and to increase the overall quality and efficiency of the scientific process.

History

In 1966, an early meta-research paper examined the statistical methods of 295 papers published in ten high-profile medical journals. It found that, "in almost 73% of the reports read ... conclusions were drawn when the justification for these conclusions was invalid." Later meta-research identified widespread difficulty in replicating results in many scientific fields, including psychology and medicine. This problem was termed "the replication crisis". Metascience has grown as a reaction to the replication crisis and to concerns about waste in research.

Many prominent publishers are interested in meta-research and in improving the quality of their publications. Top journals such as Science, The Lancet, and Nature, provide ongoing coverage of meta-research and problems with reproducibility. In 2012 PLOS ONE launched a Reproducibility Initiative. In 2015 Biomed Central introduced a minimum-standards-of-reporting checklist to four titles. 

The first international conference in the broad area of meta-research was the Research Waste/EQUATOR conference held in Edinburgh in 2015; the first international conference on peer review was the Peer Review Congress held in 1989. In 2016, Research Integrity and Peer Review was launched. The journal's opening editorial called for "research that will increase our understanding and suggest potential solutions to issues related to peer review, study reporting, and research and publication ethics".

Areas of meta-research

Metascience can be categorize into five major areas of interest: Methods, Reporting, Reproducibility, Evaluation, and Incentives. These correspond, respectively, with how to perform, communicate, verify, evaluate, and reward research.

Methods

Metascience seeks to identify poor research practices, including biases in research, poor study design, abuse of statistics, and to find methods to reduce these practices. Meta-research has identified numerous biases in scientific literature. Of particular note is the widespread misuse of p-values and abuse of statistical significance.

Reporting

Meta-research has identified poor practices in reporting, explaining, disseminating and popularizing research, particularly within the social and health sciences. Poor reporting makes it difficult to accurately interpret the results of scientific studies, to replicate studies, and to identify biases and conflicts of interest in the authors. Solutions include the implementation of reporting standards, and greater transparency in scientific studies (including better requirements for disclosure of conflicts of interest). There is an attempt to standardize reporting of data and methodology through the creation of guidelines by reporting agencies such as CONSORT and the larger EQUATOR Network.

Reproducibility

The replication crisis is an ongoing methodological crisis in which it has been found that many scientific studies are difficult or impossible to replicate. While the crisis has its roots in the meta-research of the mid- to late-1900s, the phrase "replication crisis" was not coined until the early 2010s as part of a growing awareness of the problem. The replication crisis particularly affects psychology (especially social psychology) and medicine. Replication is an essential part of the scientific process, and the widespread failure of replication puts into question the reliability of affected fields.

Moreover, replication of research (or failure to replicate) is considered less influential than original research, and is less likely to be published in many fields. This discourages the reporting of, and even attempts to replicate, studies.

Evaluation

Metascience seeks to create a scientific foundation for peer review. Meta-research evaluates peer review systems including pre-publication peer review, post-publication peer review, and open peer review. It also seeks to develop better research funding criteria.

Incentives

Metascience seeks to promote better research through better incentive systems. This includes studying the accuracy, effectiveness, costs, and benefits of different approaches to ranking and evaluating research and those who perform it. Critics argue that perverse incentives have created a publish-or-perish environment in academia which promotes the production of junk science, low quality research, and false positives. According to Brian Nosek, “The problem that we face is that the incentive system is focused almost entirely on getting research published, rather than on getting research right.” Proponents of reform seek to structure the incentive system to favor higher-quality results.

Reforms

Meta-research identifying flaws in scientific practice has inspired reforms in science. These reforms seek to address and fix problems in scientific practice which lead to low-quality or inefficient research.

Pre-registration

The practice of registering a scientific study before it is conducted is called pre-registration. It arose as a means to address the replication crisis. Pregistration requires the submission of a registered report, which is then accepted for publication or rejected by a journal based on theoretical justification, experimental design, and the proposed statistical analysis. Pre-registration of studies serves to prevent publication bias, reduce data dredging, and increase replicability.

Reporting standards

Studies showing poor consistency and quality of reporting have demonstrated the need for reporting standards and guidelines in science, which has led to the rise of organisations that produce such standards, such as CONSORT (Consolidated Standards of Reporting Trials) and the EQUATOR Network. 

The EQUATOR (Enhancing the QUAlity and Transparency Of health Research) Network is an international initiative aimed at promoting transparent and accurate reporting of health research studies to enhance the value and reliability of medical research literature. The EQUATOR Network was established with the goals of raising awareness of the importance of good reporting of research, assisting in the development, dissemination and implementation of reporting guidelines for different types of study designs, monitoring the status of the quality of reporting of research studies in the health sciences literature, and conducting research relating to issues that impact the quality of reporting of health research studies. The Network acts as an "umbrella" organisation, bringing together developers of reporting guidelines, medical journal editors and peer reviewers, research funding bodies, and other key stakeholders with a mutual interest in improving the quality of research publications and research itself. 

Applications

Medicine

Clinical research in medicine is often of low quality, and many studies cannot be replicated. An estimated 85% of research funding is wasted. Additionally, the presence of bias affects research quality. The pharmaceutical industry exerts substantial influence on the design and execution of medical research. Conflicts of interest are common among authors of medical literature and among editors of medical journals. While almost all medical journals require their authors to disclose conflicts of interest, editors are not required to do so. Financial conflicts of interest have been linked to higher rates of positive study results. In antidepressant trials, pharmaceutical sponsorship is the best predictor of trial outcome.

Blinding is another focus of meta-research, as error caused by poor blinding is a source of experimental bias. Blinding is not well reported in medical literature, and widespread misunderstanding of the subject has resulted in poor implementation of blinding in clinical trials. Furthermore, failure of blinding is rarely measured or reported. Research showing the failure of blinding in antidepressant trials has led some scientists to argue that antidepressants are no better than placebo. In light of meta-research showing failures of blinding, CONSORT standards recommend that all clinical trials assess and report the quality of blinding.

Studies have shown that systematic reviews of existing research evidence are sub-optimally used in planning a new research or summarizing the results. Cumulative meta-analyses of studies evaluating the effectiveness of medical interventions have shown that many clinical trials could have been avoided if a systematic review of existing evidence was done prior to conducting a new trial. For example, Lau et al. analyzed 33 clinical trials (involving 36974 patients) evaluating the effectiveness of intravenous streptokinase for acute myocardial infarction. Their cumulative meta-analysis demonstrated that 25 of 33 trials could have been avoided if a systematic review was conducted prior to conducting a new trial. In other words, randomizing 34542 patients was potentially unnecessary. One study analyzed 1523 clinical trials included in 227 meta-analyses and concluded that "less than one quarter of relevant prior studies" were cited. They also confirmed earlier findings that most clinical trial reports do not present systematic review to justify the research or summarize the results.

Psychology

Metascience has revealed significant problems in psychological research. The field suffers from high bias, low reproducibility, and widespread misuse of statistics. The replication crisis affects psychology more strongly than any other field; as many as two-thirds of highly publicized findings may be impossible to replicate. Meta-research finds that 80-95% of psychological studies support their initial hypotheses, which strongly implies the existence of publication bias.

The replication crisis has led to renewed efforts to re-test important findings. In response to concerns about publication bias and p-hacking, more than 140 psychology journals have adopted result-blind peer review, in which studies are pre-registered and published without regard for their outcome. An analysis of these reforms estimated that 61 percent of result-blind studies produce null results, in contrast with 5 to 20 percent in earlier research. This analysis shows that result-blind peer review substantially reduces publication bias.

Psychologists routinely confuse statistical significance with practical importance, enthusiastically reporting great certainty in unimportant facts. Some psychologists have responded with an increased use of effect size statistics, rather than sole reliance on the p values.

Physics

Richard Feynman noted that estimates of physical constants were closer to published values than would be expected by chance. This was believed to be the result of confirmation bias: results that agreed with existing literature were more likely to be believed, and therefore published. Physicists now implement blinding to prevent this kind of bias.

Associated fields

Journalology

Journalology, also known as publication science, is the scholarly study of all aspects of the academic publishing process. The field seeks to improve the quality of scholarly research by implementing evidence-based practices in academic publishing. The term "journalology" was coined by Stephen Lock, the former editor-in-chief of the BMJ. The first Peer Review Congress, held in 1989 in Chicago, Illinois, is considered a pivotal moment in the founding of journalology as a distinct field. The field of journolology has been influential in pushing for study pre-registration in science, particularly in clinical trials. Clinical-trial registration is now expected in most countries.

Scientometrics

Scientometrics concerns itself with measuring bibliographic data in scientific publications. Major research issues include the measurement of the impact of research papers and academic journals, the understanding of scientific citations, and the use of such measurements in policy and management contexts.

Scientific data science

Scientific data science is the use of data science to analyse research papers. It encompasses both qualitative and quantitative methods. Research in scientific data science includes fraud detection and citation network analysis.