Statistical hypothesis test

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Statistical_hypothesis_test

The above image shows a table with some of the most common test statistics and their corresponding tests or models.

A statistical hypothesis test is a method of statistical inference used to decide whether the data provide sufficient evidence to reject a particular hypothesis. A statistical hypothesis test typically involves a calculation of a test statistic. Then a decision is made, either by comparing the test statistic to a critical value or equivalently by evaluating a p-value computed from the test statistic. Roughly 100 specialized statistical tests are in use and noteworthy.

History

See also: History of probability

While hypothesis testing was popularized early in the 20th century, early forms were used in the 1700s. The first use is credited to John Arbuthnot (1710), followed by Pierre-Simon Laplace (1770s), in analyzing the human sex ratio at birth; see § Human sex ratio.

Choice of null hypothesis

Paul Meehl has argued that the epistemological importance of the choice of null hypothesis has gone largely unacknowledged. When the null hypothesis is predicted by theory, a more precise experiment will be a more severe test of the underlying theory. When the null hypothesis defaults to "no difference" or "no effect", a more precise experiment is a less severe test of the theory that motivated performing the experiment. An examination of the origins of the latter practice may therefore be useful:

1778: Pierre Laplace compares the birthrates of boys and girls in multiple European cities. He states: "it is natural to conclude that these possibilities are very nearly in the same ratio". Thus, the null hypothesis in this case that the birthrates of boys and girls should be equal given "conventional wisdom".

1900: Karl Pearson develops the chi squared test to determine "whether a given form of frequency curve will effectively describe the samples drawn from a given population." Thus the null hypothesis is that a population is described by some distribution predicted by theory. He uses as an example the numbers of five and sixes in the Weldon dice throw data.

1904: Karl Pearson develops the concept of "contingency" in order to determine whether outcomes are independent of a given categorical factor. Here the null hypothesis is by default that two things are unrelated (e.g. scar formation and death rates from smallpox). The null hypothesis in this case is no longer predicted by theory or conventional wisdom, but is instead the principle of indifference that led Fisher and others to dismiss the use of "inverse probabilities".

Modern origins and early controversy

Modern significance testing is largely the product of Karl Pearson (p-value, Pearson's chi-squared test), William Sealy Gosset (Student's t-distribution), and Ronald Fisher ("null hypothesis", analysis of variance, "significance test"), while hypothesis testing was developed by Jerzy Neyman and Egon Pearson (son of Karl). Ronald Fisher began his life in statistics as a Bayesian (Zabell 1992), but Fisher soon grew disenchanted with the subjectivity involved (namely use of the principle of indifference when determining prior probabilities), and sought to provide a more "objective" approach to inductive inference.

Fisher emphasized rigorous experimental design and methods to extract a result from few samples assuming Gaussian distributions. Neyman (who teamed with the younger Pearson) emphasized mathematical rigor and methods to obtain more results from many samples and a wider range of distributions. Modern hypothesis testing is an inconsistent hybrid of the Fisher vs Neyman/Pearson formulation, methods and terminology developed in the early 20th century.

Fisher popularized the "significance test". He required a null-hypothesis (corresponding to a population frequency distribution) and a sample. His (now familiar) calculations determined whether to reject the null-hypothesis or not. Significance testing did not utilize an alternative hypothesis so there was no concept of a Type II error (false negative).

The p-value was devised as an informal, but objective, index meant to help a researcher determine (based on other knowledge) whether to modify future experiments or strengthen one's faith in the null hypothesis. Hypothesis testing (and Type I/II errors) was devised by Neyman and Pearson as a more objective alternative to Fisher's p-value, also meant to determine researcher behaviour, but without requiring any inductive inference by the researcher.

Neyman & Pearson considered a different problem to Fisher (which they called "hypothesis testing"). They initially considered two simple hypotheses (both with frequency distributions). They calculated two probabilities and typically selected the hypothesis associated with the higher probability (the hypothesis more likely to have generated the sample). Their method always selected a hypothesis. It also allowed the calculation of both types of error probabilities.

Fisher and Neyman/Pearson clashed bitterly. Neyman/Pearson considered their formulation to be an improved generalization of significance testing (the defining paper was abstract; Mathematicians have generalized and refined the theory for decades). Fisher thought that it was not applicable to scientific research because often, during the course of the experiment, it is discovered that the initial assumptions about the null hypothesis are questionable due to unexpected sources of error. He believed that the use of rigid reject/accept decisions based on models formulated before data is collected was incompatible with this common scenario faced by scientists and attempts to apply this method to scientific research would lead to mass confusion.

The dispute between Fisher and Neyman–Pearson was waged on philosophical grounds, characterized by a philosopher as a dispute over the proper role of models in statistical inference.

Events intervened: Neyman accepted a position in the University of California, Berkeley in 1938, breaking his partnership with Pearson and separating the disputants (who had occupied the same building). World War II provided an intermission in the debate. The dispute between Fisher and Neyman terminated (unresolved after 27 years) with Fisher's death in 1962. Neyman wrote a well-regarded eulogy. Some of Neyman's later publications reported p-values and significance levels.

Null hypothesis significance testing (NHST)

The modern version of hypothesis testing is generally called the null hypothesis significance testing (NHST) and is a hybrid of the Fisher approach with the Neyman-Pearson approach. In 2000, Raymond S. Nickerson wrote an article stating that NHST was (at the time) "arguably the most widely used method of analysis of data collected in psychological experiments and has been so for about 70 years" and that it was at the same time "very controversial".

This fusion resulted from confusion by writers of statistical textbooks (as predicted by Fisher) beginning in the 1940s (but signal detection, for example, still uses the Neyman/Pearson formulation). Great conceptual differences and many caveats in addition to those mentioned above were ignored. Neyman and Pearson provided the stronger terminology, the more rigorous mathematics and the more consistent philosophy, but the subject taught today in introductory statistics has more similarities with Fisher's method than theirs.

Sometime around 1940, authors of statistical text books began combining the two approaches by using the p-value in place of the test statistic (or data) to test against the Neyman–Pearson "significance level".

A comparison between Fisherian, frequentist (Neyman–Pearson)

#	Fisher's null hypothesis testing	Neyman–Pearson decision theory
1	Set up a statistical null hypothesis. The null need not be a nil hypothesis (i.e., zero difference).	Set up two statistical hypotheses, H1 and H2, and decide about α, β, and sample size before the experiment, based on subjective cost-benefit considerations. These define a rejection region for each hypothesis.
2	Report the exact level of significance (e.g. p = 0.051 or p = 0.049). Do not refer to "accepting" or "rejecting" hypotheses. If the result is "not significant", draw no conclusions and make no decisions, but suspend judgement until further data is available.	If the data falls into the rejection region of H1, accept H2; otherwise accept H1. Accepting a hypothesis does not mean that you believe in it, but only that you act as if it were true.
3	Use this procedure only if little is known about the problem at hand, and only to draw provisional conclusions in the context of an attempt to understand the experimental situation.	The usefulness of the procedure is limited among others to situations where you have a disjunction of hypotheses (e.g. either μ1 = 8 or μ2 = 10 is true) and where you can make meaningful cost-benefit trade-offs for choosing alpha and beta.

Philosophy

Hypothesis testing and philosophy intersect. Inferential statistics, which includes hypothesis testing, is applied probability. Both probability and its application are intertwined with philosophy. Philosopher David Hume wrote, "All knowledge degenerates into probability." Competing practical definitions of probability reflect philosophical differences. The most common application of hypothesis testing is in the scientific interpretation of experimental data, which is naturally studied by the philosophy of science.

Fisher and Neyman opposed the subjectivity of probability. Their views contributed to the objective definitions. The core of their historical disagreement was philosophical.

Many of the philosophical criticisms of hypothesis testing are discussed by statisticians in other contexts, particularly correlation does not imply causation and the design of experiments. Hypothesis testing is of continuing interest to philosophers.

Education

Main article: Statistics education

Statistics is increasingly being taught in schools with hypothesis testing being one of the elements taught Many conclusions reported in the popular press (political opinion polls to medical studies) are based on statistics. Some writers have stated that statistical analysis of this kind allows for thinking clearly about problems involving mass data, as well as the effective reporting of trends and inferences from said data, but caution that writers for a broad public should have a solid understanding of the field in order to use the terms and concepts correctly. An introductory college statistics class places much emphasis on hypothesis testing – perhaps half of the course. Such fields as literature and divinity now include findings based on statistical analysis (see the Bible Analyzer). An introductory statistics class teaches hypothesis testing as a cookbook process. Hypothesis testing is also taught at the postgraduate level. Statisticians learn how to create good statistical test procedures (like z, Student's t, F and chi-squared). Statistical hypothesis testing is considered a mature area within statistics, but a limited amount of development continues.

An academic study states that the cookbook method of teaching introductory statistics leaves no time for history, philosophy or controversy. Hypothesis testing has been taught as received unified method. Surveys showed that graduates of the class were filled with philosophical misconceptions (on all aspects of statistical inference) that persisted among instructors. While the problem was addressed more than a decade ago, and calls for educational reform continue, students still graduate from statistics classes holding fundamental misconceptions about hypothesis testing. Ideas for improving the teaching of hypothesis testing include encouraging students to search for statistical errors in published papers, teaching the history of statistics and emphasizing the controversy in a generally dry subject.

Raymond S. Nickerson commented:

The debate about NHST has its roots in unresolved disagreements among major contributors to the development of theories of inferential statistics on which modern approaches are based. Gigerenzer et al. (1989) have reviewed in considerable detail the controversy between R. A. Fisher on the one hand and Jerzy Neyman and Egon Pearson on the other as well as the disagreements between both of these views and those of the followers of Thomas Bayes. They noted the remarkable fact that little hint of the historical and ongoing controversy is to be found in most textbooks that are used to teach NHST to its potential users. The resulting lack of an accurate historical perspective and understanding of the complexity and sometimes controversial philosophical foundations of various approaches to statistical inference may go a long way toward explaining the apparent ease with which statistical tests are misused and misinterpreted.

Performing a frequentist hypothesis test in practice

The typical steps involved in performing a frequentist hypothesis test in practice are:

1. Define a hypothesis (claim which is testable using data).
2. Select a relevant statistical test with associated test statistic T.
3. Derive the distribution of the test statistic under the null hypothesis from the assumptions. In standard cases this will be a well-known result. For example, the test statistic might follow a Student's t distribution with known degrees of freedom, or a normal distribution with known mean and variance.
4. Select a significance level (α), the maximum acceptable false positive rate. Common values are 5% and 1%.
5. Compute from the observations the observed value t_obs of the test statistic T.
6. Decide to either reject the null hypothesis in favor of the alternative or not reject it. The Neyman-Pearson decision rule is to reject the null hypothesis H₀ if the observed value t_obs is in the critical region, and not to reject the null hypothesis otherwise.

Practical example

The difference in the two processes applied to the radioactive suitcase example (below):

• "The Geiger-counter reading is 10. The limit is 9. Check the suitcase."
• "The Geiger-counter reading is high; 97% of safe suitcases have lower readings. The limit is 95%. Check the suitcase."

The former report is adequate, the latter gives a more detailed explanation of the data and the reason why the suitcase is being checked.

Not rejecting the null hypothesis does not mean the null hypothesis is "accepted" per se (though Neyman and Pearson used that word in their original writings; see the Interpretation section).

The processes described here are perfectly adequate for computation. They seriously neglect the design of experiments considerations.

It is particularly critical that appropriate sample sizes be estimated before conducting the experiment.

The phrase "test of significance" was coined by statistician Ronald Fisher.

Interpretation

When the null hypothesis is true and statistical assumptions are met, the probability that the p-value will be less than or equal to the significance level ${\displaystyle \alpha }$ is at most ${\displaystyle \alpha }$. This ensures that the hypothesis test maintains its specified false positive rate (provided that statistical assumptions are met).

The p-value is the probability that a test statistic which is at least as extreme as the one obtained would occur under the null hypothesis. At a significance level of 0.05, a fair coin would be expected to (incorrectly) reject the null hypothesis (that it is fair) in 1 out of 20 tests on average. The p-value does not provide the probability that either the null hypothesis or its opposite is correct (a common source of confusion).

If the p-value is less than the chosen significance threshold (equivalently, if the observed test statistic is in the critical region), then we say the null hypothesis is rejected at the chosen level of significance. If the p-value is not less than the chosen significance threshold (equivalently, if the observed test statistic is outside the critical region), then the null hypothesis is not rejected at the chosen level of significance.

In the "lady tasting tea" example (below), Fisher required the lady to properly categorize all of the cups of tea to justify the conclusion that the result was unlikely to result from chance. His test revealed that if the lady was effectively guessing at random (the null hypothesis), there was a 1.4% chance that the observed results (perfectly ordered tea) would occur.

Use and importance

Statistics are helpful in analyzing most collections of data. This is equally true of hypothesis testing which can justify conclusions even when no scientific theory exists. In the Lady tasting tea example, it was "obvious" that no difference existed between (milk poured into tea) and (tea poured into milk). The data contradicted the "obvious".

Real world applications of hypothesis testing include:

• Testing whether more men than women suffer from nightmares
• Establishing authorship of documents
• Evaluating the effect of the full moon on behavior
• Determining the range at which a bat can detect an insect by echo
• Deciding whether hospital carpeting results in more infections
• Selecting the best means to stop smoking
• Checking whether bumper stickers reflect car owner behavior
• Testing the claims of handwriting analysts

Statistical hypothesis testing plays an important role in the whole of statistics and in statistical inference. For example, Lehmann (1992) in a review of the fundamental paper by Neyman and Pearson (1933) says: "Nevertheless, despite their shortcomings, the new paradigm formulated in the 1933 paper, and the many developments carried out within its framework continue to play a central role in both the theory and practice of statistics and can be expected to do so in the foreseeable future".

Significance testing has been the favored statistical tool in some experimental social sciences (over 90% of articles in the Journal of Applied Psychology during the early 1990s). Other fields have favored the estimation of parameters (e.g. effect size). Significance testing is used as a substitute for the traditional comparison of predicted value and experimental result at the core of the scientific method. When theory is only capable of predicting the sign of a relationship, a directional (one-sided) hypothesis test can be configured so that only a statistically significant result supports theory. This form of theory appraisal is the most heavily criticized application of hypothesis testing.

Cautions

"If the government required statistical procedures to carry warning labels like those on drugs, most inference methods would have long labels indeed." This caution applies to hypothesis tests and alternatives to them.

The successful hypothesis test is associated with a probability and a type-I error rate. The conclusion might be wrong.

The conclusion of the test is only as solid as the sample upon which it is based. The design of the experiment is critical. A number of unexpected effects have been observed including:

• The clever Hans effect. A horse appeared to be capable of doing simple arithmetic.
• The Hawthorne effect. Industrial workers were more productive in better illumination, and most productive in worse.
• The placebo effect. Pills with no medically active ingredients were remarkably effective.

A statistical analysis of misleading data produces misleading conclusions. The issue of data quality can be more subtle. In forecasting for example, there is no agreement on a measure of forecast accuracy. In the absence of a consensus measurement, no decision based on measurements will be without controversy.

Publication bias: Statistically nonsignificant results may be less likely to be published, which can bias the literature.

Multiple testing: When multiple true null hypothesis tests are conducted at once without adjustment, the overall probability of Type I error is higher than the nominal alpha level.

Those making critical decisions based on the results of a hypothesis test are prudent to look at the details rather than the conclusion alone. In the physical sciences most results are fully accepted only when independently confirmed. The general advice concerning statistics is, "Figures never lie, but liars figure" (anonymous).

Definition of terms

See also: Notation in probability and statistics

The following definitions are mainly based on the exposition in the book by Lehmann and Romano:

• Statistical hypothesis: A statement about the parameters describing a population (not a sample).
• Test statistic: A value calculated from a sample without any unknown parameters, often to summarize the sample for comparison purposes.
• Simple hypothesis: Any hypothesis which specifies the population distribution completely.
• Composite hypothesis: Any hypothesis which does not specify the population distribution completely.
• Null hypothesis (H₀)
• Positive data: Data that enable the investigator to reject a null hypothesis.
• Alternative hypothesis (H₁)

Suppose the data can be realized from an N(0,1) distribution. For example, with a chosen significance level α = 0.05, from the Z-table, a one-tailed critical value of approximately 1.645 can be obtained. The one-tailed critical value C_α ≈ 1.645 corresponds to the chosen significance level. The critical region [C_α, ∞) is realized as the tail of the standard normal distribution.

• Critical values of a statistical test are the boundaries of the acceptance region of the test. The acceptance region is the set of values of the test statistic for which the null hypothesis is not rejected. Depending on the shape of the acceptance region, there can be one or more than one critical value.
  • Region of rejection / Critical region: The set of values of the test statistic for which the null hypothesis is rejected.
• Power of a test (1 − β)
• Size: For simple hypotheses, this is the test's probability of incorrectly rejecting the null hypothesis. The false positive rate. For composite hypotheses this is the supremum of the probability of rejecting the null hypothesis over all cases covered by the null hypothesis. The complement of the false positive rate is termed specificity in biostatistics. ("This is a specific test. Because the result is positive, we can confidently say that the patient has the condition.") See sensitivity and specificity and type I and type II errors for exhaustive definitions.
• Significance level of a test (α)
• p-value
• Statistical significance test: A predecessor to the statistical hypothesis test (see the Origins section). An experimental result was said to be statistically significant if a sample was sufficiently inconsistent with the (null) hypothesis. This was variously considered common sense, a pragmatic heuristic for identifying meaningful experimental results, a convention establishing a threshold of statistical evidence or a method for drawing conclusions from data. The statistical hypothesis test added mathematical rigor and philosophical consistency to the concept by making the alternative hypothesis explicit. The term is loosely used for the modern version which is now part of statistical hypothesis testing.
• Conservative test: A test is conservative if, when constructed for a given nominal significance level, the true probability of incorrectly rejecting the null hypothesis is never greater than the nominal level.
• Exact test

A statistical hypothesis test compares a test statistic (z or t for examples) to a threshold. The test statistic (the formula found in the table below) is based on optimality. For a fixed level of Type I error rate, use of these statistics minimizes Type II error rates (equivalent to maximizing power). The following terms describe tests in terms of such optimality:

• Most powerful test: For a given size or significance level, the test with the greatest power (probability of rejection) for a given value of the parameter(s) being tested, contained in the alternative hypothesis.
• Uniformly most powerful test (UMP)

Nonparametric bootstrap hypothesis testing

Main article: Bootstrapping (statistics)

Bootstrap-based resampling methods can be used for null hypothesis testing. A bootstrap creates numerous simulated samples by randomly resampling (with replacement) the original, combined sample data, assuming the null hypothesis is correct. The bootstrap is very versatile as it is distribution-free and it does not rely on restrictive parametric assumptions, but rather on empirical approximate methods with asymptotic guarantees. Traditional parametric hypothesis tests are more computationally efficient but make stronger structural assumptions. In situations where computing the probability of the test statistic under the null hypothesis is hard or impossible (due to perhaps inconvenience or lack of knowledge of the underlying distribution), the bootstrap offers a viable method for statistical inference.

Examples

Human sex ratio

Main article: Human sex ratio

The earliest use of statistical hypothesis testing is generally credited to the question of whether male and female births are equally likely (null hypothesis), which was addressed in the 1700s by John Arbuthnot (1710), and later by Pierre-Simon Laplace (1770s).

Arbuthnot examined birth records in London for each of the 82 years from 1629 to 1710, and applied the sign test, a simple non-parametric test. In every year, the number of males born in London exceeded the number of females. Considering more male or more female births as equally likely, the probability of the observed outcome is 0.5⁸², or about 1 in 4,836,000,000,000,000,000,000,000; in modern terms, this is the p-value. Arbuthnot concluded that this is too small to be due to chance and must instead be due to divine providence: "From whence it follows, that it is Art, not Chance, that governs." In modern terms, he rejected the null hypothesis of equally likely male and female births at the p = 1/2⁸² significance level.

Laplace considered the statistics of almost half a million births. The statistics showed an excess of boys compared to girls. He concluded by calculation of a p-value that the excess was a real, but unexplained, effect.

Lady tasting tea

Main article: Lady tasting tea

In a famous example of hypothesis testing, known as the Lady tasting tea, Dr. Muriel Bristol, a colleague of Fisher, claimed to be able to tell whether the tea or the milk was added first to a cup. Fisher proposed to give her eight cups, four of each variety, in random order. One could then ask what the probability was for her getting the number she got correct, but just by chance. The null hypothesis was that the Lady had no such ability. The test statistic was a simple count of the number of successes in selecting the 4 cups. The critical region was the single case of 4 successes of 4 possible based on a conventional probability criterion (< 5%). A pattern of 4 successes corresponds to 1 out of 70 possible combinations (p≈ 1.4%). Fisher asserted that no alternative hypothesis was (ever) required. The lady correctly identified every cup, which would be considered a statistically significant result.

Courtroom trial

A statistical test procedure is comparable to a criminal trial; a defendant is considered not guilty as long as his or her guilt is not proven. The prosecutor tries to prove the guilt of the defendant. Only when there is enough evidence for the prosecution is the defendant convicted.

In the start of the procedure, there are two hypotheses ${\displaystyle H_0}$: "the defendant is not guilty", and ${\displaystyle H_1}$: "the defendant is guilty". The first one, ${\displaystyle H_0}$, is called the null hypothesis. The second one, ${\displaystyle H_1}$, is called the alternative hypothesis. It is the alternative hypothesis that one hopes to support.

The hypothesis of innocence is rejected only when an error is very unlikely, because one does not want to convict an innocent defendant. Such an error is called error of the first kind (i.e., the conviction of an innocent person), and the occurrence of this error is controlled to be rare. As a consequence of this asymmetric behaviour, an error of the second kind (acquitting a person who committed the crime), is more common.

	H0 is true Truly not guilty	H1 is true Truly guilty
Do not reject the null hypothesis Acquittal	Right decision	Wrong decision Type II Error
Reject null hypothesis Conviction	Wrong decision Type I Error	Right decision

A criminal trial can be regarded as either or both of two decision processes: guilty vs not guilty or evidence vs a threshold ("beyond a reasonable doubt"). In one view, the defendant is judged; in the other view the performance of the prosecution (which bears the burden of proof) is judged. A hypothesis test can be regarded as either a judgment of a hypothesis or as a judgment of evidence.

Clairvoyant card game

A person (the subject) is tested for clairvoyance. They are shown the back face of a randomly chosen playing card 25 times and asked which of the four suits it belongs to. The number of hits, or correct answers, is called X.

As we try to find evidence of their clairvoyance, for the time being the null hypothesis is that the person is not clairvoyant. The alternative is: the person is (more or less) clairvoyant.

If the null hypothesis is valid, the only thing the test person can do is guess. For every card, the probability (relative frequency) of any single suit appearing is 1/4. If the alternative is valid, the test subject will predict the suit correctly with probability greater than 1/4. We will call the probability of guessing correctly p. The hypotheses, then, are:

• null hypothesis ${\displaystyle \text{:}{H}_0:p=\frac{1}{4}}$     (just guessing)

and

• alternative hypothesis ${\displaystyle \text{:}{H}_1:p>\frac{1}{4}}$    (true clairvoyant).

When the test subject correctly predicts all 25 cards, we will consider them clairvoyant, and reject the null hypothesis. Thus also with 24 or 23 hits. With only 5 or 6 hits, on the other hand, there is no cause to consider them so. But what about 12 hits, or 17 hits? What is the critical number, c, of hits, at which point we consider the subject to be clairvoyant? How do we determine the critical value c? With the choice c=25 (i.e. we only accept clairvoyance when all cards are predicted correctly) we're more critical than with c=10. In the first case almost no test subjects will be recognized to be clairvoyant, in the second case, a certain number will pass the test. In practice, one decides how critical one will be. That is, one decides how often one accepts an error of the first kind – a false positive, or Type I error. With c = 25 the probability of such an error is:

${\displaystyle P(\text{reject }{H}_0\mid H_0\text{ is valid})=P\left(X=25\mid p=\frac{1}{4}\right)={\left(\frac{1}{4}\right)}^{25}\approx {10}^{-15}}$,

and hence, very small. The probability of a false positive is the probability of randomly guessing correctly all 25 times.

Being less critical, with c = 10, gives:

${\displaystyle P(\text{reject }{H}_0\mid H_0\text{ is valid})=P\left(X\ge 10\mid p=\frac{1}{4}\right)=\sum _{k=10}^{25}P\left(X=k\mid p=\frac{1}{4}\right)=\sum _{k=10}^{25}(\genfrac{}{}{0ex}{}{25}{k}){\left(1-\frac{1}{4}\right)}^{25-k}{\left(\frac{1}{4}\right)}^k\approx 0.0713}$.

Thus, c = 10 yields a much greater probability of false positive.

Before the test is actually performed, the maximum acceptable probability of a Type I error (α) is determined. Typically, values in the range of 1% to 5% are selected. (If the maximum acceptable error rate is zero, an infinite number of correct guesses is required.) Depending on this Type 1 error rate, the critical value c is calculated. For example, if we select an error rate of 1%, c is calculated thus:

${\displaystyle P(\text{reject }{H}_0\mid H_0\text{ is valid})=P\left(X\ge c\mid p=\frac{1}{4}\right)\le 0.01}$.

From all the numbers c, with this property, we choose the smallest, in order to minimize the probability of a Type II error, a false negative. For the above example, we select: ${\displaystyle c=13}$.

Variations and sub-classes

Statistical hypothesis testing is a key technique of both frequentist inference and Bayesian inference, although the two types of inference have notable differences. Statistical hypothesis tests define a procedure that controls (fixes) the probability of incorrectly deciding that a default position (null hypothesis) is incorrect. The procedure is based on how likely it would be for a set of observations to occur if the null hypothesis were true. This probability of making an incorrect decision is not the probability that the null hypothesis is true, nor whether any specific alternative hypothesis is true. This contrasts with other possible techniques of decision theory in which the null and alternative hypothesis are treated on a more equal basis.

One naïve Bayesian approach to hypothesis testing is to base decisions on the posterior probability, but this fails when comparing point and continuous hypotheses. Other approaches to decision making, such as Bayesian decision theory, attempt to balance the consequences of incorrect decisions across all possibilities, rather than concentrating on a single null hypothesis. A number of other approaches to reaching a decision based on data are available via decision theory and optimal decisions, some of which have desirable properties. Hypothesis testing, though, is a dominant approach to data analysis in many fields of science. Extensions to the theory of hypothesis testing include the study of the power of tests, i.e. the probability of correctly rejecting the null hypothesis given that it is false. Such considerations can be used for the purpose of sample size determination prior to the collection of data.

Neyman–Pearson hypothesis testing

An example of Neyman–Pearson hypothesis testing (or null hypothesis statistical significance testing) can be made by a change to the radioactive suitcase example. If the "suitcase" is actually a shielded container for the transportation of radioactive material, then a test might be used to select among three hypotheses: no radioactive source present, one present, two (all) present. The test could be required for safety, with actions required in each case. The Neyman–Pearson lemma of hypothesis testing says that a good criterion for the selection of hypotheses is the ratio of their probabilities (a likelihood ratio). A simple method of solution is to select the hypothesis with the highest probability for the Geiger counts observed. The typical result matches intuition: few counts imply no source, many counts imply two sources and intermediate counts imply one source. Notice also that usually there are problems for proving a negative. Null hypotheses should be at least falsifiable.

Neyman–Pearson theory can accommodate both prior probabilities and the costs of actions resulting from decisions. The former allows each test to consider the results of earlier tests (unlike Fisher's significance tests). The latter allows the consideration of economic issues (for example) as well as probabilities. A likelihood ratio remains a good criterion for selecting among hypotheses.

The two forms of hypothesis testing are based on different problem formulations. The original test is analogous to a true/false question; the Neyman–Pearson test is more like multiple choice. In the view of Tukey the former produces a conclusion on the basis of only strong evidence while the latter produces a decision on the basis of available evidence. While the two tests seem quite different both mathematically and philosophically, later developments lead to the opposite claim. Consider many tiny radioactive sources. The hypotheses become 0,1,2,3... grains of radioactive sand. There is little distinction between none or some radiation (Fisher) and 0 grains of radioactive sand versus all of the alternatives (Neyman–Pearson). The major Neyman–Pearson paper of 1933 also considered composite hypotheses (ones whose distribution includes an unknown parameter). An example proved the optimality of the (Student's) t-test, "there can be no better test for the hypothesis under consideration" (p 321). Neyman–Pearson theory was proving the optimality of Fisherian methods from its inception.

Fisher's significance testing has proven a popular flexible statistical tool in application with little mathematical growth potential. Neyman–Pearson hypothesis testing is claimed as a pillar of mathematical statistics, creating a new paradigm for the field. It also stimulated new applications in statistical process control, detection theory, decision theory and game theory. Both formulations have been successful, but the successes have been of a different character.

The dispute over formulations is unresolved. Science primarily uses Fisher's (slightly modified) formulation as taught in introductory statistics. Statisticians study Neyman–Pearson theory in graduate school. Mathematicians are proud of uniting the formulations. Philosophers consider them separately. Learned opinions deem the formulations variously competitive (Fisher vs Neyman), incompatible or complementary. The dispute has become more complex since Bayesian inference has achieved respectability.

The terminology is inconsistent. Hypothesis testing can mean any mixture of two formulations that both changed with time. Any discussion of significance testing vs hypothesis testing is doubly vulnerable to confusion.

Fisher thought that hypothesis testing was a useful strategy for performing industrial quality control, however, he strongly disagreed that hypothesis testing could be useful for scientists. Hypothesis testing provides a means of finding test statistics used in significance testing. The concept of power is useful in explaining the consequences of adjusting the significance level and is heavily used in sample size determination. The two methods remain philosophically distinct. They usually (but not always) produce the same mathematical answer. The preferred answer is context dependent. While the existing merger of Fisher and Neyman–Pearson theories has been heavily criticized, modifying the merger to achieve Bayesian goals has been considered.

Criticism

See also: p-value § Misuse

Criticism of statistical hypothesis testing fills volumes. Much of the criticism can be summarized by the following issues:

• The interpretation of a p-value is dependent upon stopping rule and definition of multiple comparison. The former often changes during the course of a study and the latter is unavoidably ambiguous. (i.e. "p values depend on both the (data) observed and on the other possible (data) that might have been observed but weren't").
• Confusion resulting (in part) from combining the methods of Fisher and Neyman–Pearson which are conceptually distinct.
• Emphasis on statistical significance to the exclusion of estimation and confirmation by repeated experiments.
• Rigidly requiring statistical significance as a criterion for publication, resulting in publication bias. Most of the criticism is indirect. Rather than being wrong, statistical hypothesis testing is misunderstood, overused and misused.
• When used to detect whether a difference exists between groups, a paradox arises. As improvements are made to experimental design (e.g. increased precision of measurement and sample size), the test becomes more lenient. Unless one accepts the absurd assumption that all sources of noise in the data cancel out completely, the chance of finding statistical significance in either direction approaches 100%. However, this absurd assumption that the mean difference between two groups cannot be zero implies that the data cannot be independent and identically distributed (i.i.d.) because the expected difference between any two subgroups of i.i.d. random variates is zero; therefore, the i.i.d. assumption is also absurd.
• Layers of philosophical concerns. The probability of statistical significance is a function of decisions made by experimenters/analysts. If the decisions are based on convention they are termed arbitrary or mindless while those not so based may be termed subjective. To minimize type II errors, large samples are recommended. In psychology practically all null hypotheses are claimed to be false for sufficiently large samples so "...it is usually nonsensical to perform an experiment with the sole aim of rejecting the null hypothesis." "Statistically significant findings are often misleading" in psychology. Statistical significance does not imply practical significance, and correlation does not imply causation. Casting doubt on the null hypothesis is thus far from directly supporting the research hypothesis.
• "[I]t does not tell us what we want to know". Lists of dozens of complaints are available.

Critics and supporters are largely in factual agreement regarding the characteristics of null hypothesis significance testing (NHST): While it can provide critical information, it is inadequate as the sole tool for statistical analysis. Successfully rejecting the null hypothesis may offer no support for the research hypothesis. The continuing controversy concerns the selection of the best statistical practices for the near-term future given the existing practices. However, adequate research design can minimize this issue. Critics would prefer to ban NHST completely, forcing a complete departure from those practices, while supporters suggest a less absolute change.

Controversy over significance testing, and its effects on publication bias in particular, has produced several results. The American Psychological Association has strengthened its statistical reporting requirements after review, medical journal publishers have recognized the obligation to publish some results that are not statistically significant to combat publication bias, and a journal (Journal of Articles in Support of the Null Hypothesis) has been created to publish such results exclusively. Textbooks have added some cautions, and increased coverage of the tools necessary to estimate the size of the sample required to produce significant results. Few major organizations have abandoned use of significance tests although some have discussed doing so. For instance, in 2023, the editors of the Journal of Physiology "strongly recommend the use of estimation methods for those publishing in The Journal" (meaning the magnitude of the effect size (to allow readers to judge whether a finding has practical, physiological, or clinical relevance) and confidence intervals to convey the precision of that estimate), saying "Ultimately, it is the physiological importance of the data that those publishing in The Journal of Physiology should be most concerned with, rather than the statistical significance."

Alternatives

Main article: Estimation statistics

See also: Confidence interval § Statistical hypothesis testing

A unifying position of critics is that statistics should not lead to an accept-reject conclusion or decision, but to an estimated value with an interval estimate; this data-analysis philosophy is broadly referred to as estimation statistics. Estimation statistics can be accomplished with either frequentist or Bayesian methods.

Critics of significance testing have advocated basing inference less on p-values and more on confidence intervals for effect sizes for importance, prediction intervals for confidence, replications and extensions for replicability, meta-analyses for generality :. But none of these suggested alternatives inherently produces a decision. Lehmann said that hypothesis testing theory can be presented in terms of conclusions/decisions, probabilities, or confidence intervals: "The distinction between the ... approaches is largely one of reporting and interpretation."

Bayesian inference is one proposed alternative to significance testing. (Nickerson cited 10 sources suggesting it, including Rozeboom (1960)). For example, Bayesian parameter estimation can provide rich information about the data from which researchers can draw inferences, while using uncertain priors that exert only minimal influence on the results when enough data is available. Psychologist John K. Kruschke has suggested Bayesian estimation as an alternative for the t-test and has also contrasted Bayesian estimation for assessing null values with Bayesian model comparison for hypothesis testing.^[86] Two competing models/hypotheses can be compared using Bayes factors. Bayesian methods could be criticized for requiring information that is seldom available in the cases where significance testing is most heavily used. Neither the prior probabilities nor the probability distribution of the test statistic under the alternative hypothesis are often available in the social sciences.

Advocates of a Bayesian approach sometimes claim that the goal of a researcher is most often to objectively assess the probability that a hypothesis is true based on the data they have collected.^[89][90] Neither Fisher's significance testing, nor Neyman–Pearson hypothesis testing can provide this information, and do not claim to. The probability a hypothesis is true can only be derived from use of Bayes' Theorem, which was unsatisfactory to both the Fisher and Neyman–Pearson camps due to the explicit use of subjectivity in the form of the prior probability. Fisher's strategy is to sidestep this with the p-value (an objective index based on the data alone) followed by inductive inference, while Neyman–Pearson devised their approach of inductive behaviour.

Falsifiability

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Falsifiability 

The belief that "all swans are white" can be falsified by observing a single black swan.

Falsifiability (or refutability) is a deductive standard of evaluation of scientific theories and hypotheses, introduced by the philosopher of science Karl Popper in his book The Logic of Scientific Discovery (1934). A theory or hypothesis is falsifiable if it can be logically contradicted by an empirical test.

Popper emphasized the asymmetry created by the relation of a universal law with basic observation statements and contrasted falsifiability to the intuitively similar concept of verifiability that was then current in logical positivism. He argued that the only way to verify a claim such as "All swans are white" would be if one could theoretically observe all swans, which is not possible. On the other hand, the falsifiability requirement for an anomalous instance, such as the observation of a single black swan, is theoretically reasonable and sufficient to logically falsify the claim.

Popper proposed falsifiability as the cornerstone solution to both the problem of induction and the problem of demarcation. He insisted that, as a logical criterion, his falsifiability is distinct from the related concept "capacity to be proven wrong" discussed in Lakatos's falsificationism. Even being a logical criterion, its purpose is to make the theory predictive and testable, and thus useful in practice.

By contrast, the Duhem–Quine thesis says that definitive experimental falsifications are impossible and that no scientific hypothesis is by itself capable of making predictions, because an empirical test of the hypothesis requires one or more background assumptions.

Popper's response is that falsifiability does not have the Duhem problem because it is a logical criterion. Experimental research has the Duhem problem and other problems, such as the problem of induction, but, according to Popper, statistical tests, which are only possible when a theory is falsifiable, can still be useful within a critical discussion.

As a key notion in the separation of science from non-science and pseudoscience, falsifiability has featured prominently in many scientific controversies and applications, even being used as legal precedent. However, falsifiability is not a sufficient condition for demarcating science as theories have to actually be tested in order to eliminate theories that are wrong. In scientific practice, this can cause theories to change from being falsified back to unfalsified, such as when the once-falsified geocentric world view was restored as a viable reference frame within special relativity. There is ambiguity surrounding the status of theories that cannot currently be tested.

The problem of induction and demarcation

See also: Inductivism, Problem of induction, and Demarcation problem

One of the questions in the scientific method is: how does one move from observations to scientific laws? This is the problem of induction. Suppose we want to put the hypothesis that all swans are white to the test. We come across a white swan. We cannot validly argue (or induce) from "here is a white swan" to "all swans are white"; doing so would require a logical fallacy such as, for example, affirming the consequent.

Popper's idea to solve this problem is that while it is impossible to verify that every swan is white, finding a single black swan shows that not every swan is white. Such falsification uses the valid inference modus tollens: if from a law ${\displaystyle L}$ we logically deduce ${\displaystyle Q}$, but what is observed is ${\displaystyle \mathrm{\neg }Q}$, we infer that the law ${\displaystyle L}$ is false. For example, given the statement ${\displaystyle L=}$ "all swans are white", we can deduce ${\displaystyle Q=}$ "the specific swan here is white", but if what is observed is ${\displaystyle \mathrm{\neg }Q=}$ "the specific swan here is not white" (say black), then "all swans are white" is false. More accurately, the statement ${\displaystyle Q}$ that can be deduced is broken into an initial condition and a prediction as in ${\displaystyle C\Rightarrow P}$ in which ${\displaystyle C=}$ "the thing here is a swan" and ${\displaystyle P=}$ "the thing here is a white swan". If what is observed is C being true while P is false (formally, ${\displaystyle C\wedge \mathrm{\neg }P}$), we can infer that the law is false.

For Popper, induction is actually never needed in science. Instead, in Popper's view, laws are conjectured in a non-logical manner on the basis of expectations and predispositions. This has led David Miller, a student and collaborator of Popper, to write "the mission is to classify truths, not to certify them". In contrast, the logical empiricism movement, which included such philosophers as Moritz Schlick, Rudolf Carnap, Otto Neurath, and A. J. Ayer wanted to formalize the idea that, for a law to be scientific, it must be possible to argue on the basis of observations either in favor of its truth or its falsity. There was no consensus among these philosophers about how to achieve that, but the thought expressed by Mach's dictum that "where neither confirmation nor refutation is possible, science is not concerned" was accepted as a basic precept of critical reflection about science.

Popper said that a demarcation criterion was possible, but we have to use the logical possibility of falsifications, which is falsifiability. He cited his encounter with psychoanalysis in the 1910s. It did not matter what observation was presented, psychoanalysis could explain it. Unfortunately, the reason it could explain everything is that it did not exclude anything also. For Popper, this was a failure, because it meant that it could not make any prediction. From a logical standpoint, if one finds an observation that does not contradict a law, it does not mean that the law is true. A verification has no value in itself. But, if the law makes risky predictions and these are corroborated, Popper says, there is a reason to prefer this law over another law that makes less risky predictions or no predictions at all. In the definition of falsifiability, contradictions with observations are not used to support eventual falsifications, but for logical "falsifications" that show that the law makes risky predictions, which is completely different.

On the basic philosophical side of this issue, Popper said that some philosophers of the Vienna Circle had mixed two different problems, that of meaning and that of demarcation, and had proposed in verificationism a single solution to both: a statement that could not be verified was considered meaningless. In opposition to this view, Popper said that there are meaningful theories that are not scientific, and that, accordingly, a criterion of meaningfulness does not coincide with a criterion of demarcation.

From Hume's problem to non problematic induction

The problem of induction is often called Hume's problem. David Hume studied how human beings obtain new knowledge that goes beyond known laws and observations, including how we can discover new laws. He understood that deductive logic could not explain this learning process and argued in favour of a mental or psychological process of learning that would not require deductive logic. He even argued that this learning process cannot be justified by any general rules, deductive or not. Popper accepted Hume's argument and therefore viewed progress in science as the result of quasi-induction, which does the same as induction, but has no inference rules to justify it. Philip N. Johnson-Laird, professor of psychology, also accepted Hume's conclusion that induction has no justification. For him induction does not require justification and therefore can exist in the same manner as Popper's quasi-induction does.

When Johnson-Laird says that no justification is needed, he does not refer to a general inductive method of justification that, to avoid a circular reasoning, would not itself require any justification. On the contrary, in agreement with Hume, he means that there is no general method of justification for induction and that's ok, because the induction steps do not require justification. Instead, these steps use patterns of induction, which are not expected to have a general justification: they may or may not be applicable depending on the background knowledge. Johnson-Laird wrote: "[P]hilosophers have worried about which properties of objects warrant inductive inferences. The answer rests on knowledge: we don't infer that all the passengers on a plane are male because the first ten off the plane are men. We know that this observation doesn't rule out the possibility of a woman passenger." The reasoning pattern that was not applied here is enumerative induction.

Popper was interested in the overall learning process in science, to quasi-induction, which he also called the "path of science". However, Popper did not show much interest in these reasoning patterns, which he globally referred to as psychologism. He did not deny the possibility of some kind of psychological explanation for the learning process, especially when psychology is seen as an extension of biology, but he felt that these biological explanations were not within the scope of epistemology. Popper proposed an evolutionary mechanism to explain the success of science, which is much in line with Johnson-Laird's view that "induction is just something that animals, including human beings, do to make life possible", but Popper did not consider it a part of his epistemology. He wrote that his interest was mainly in the logic of science and that epistemology should be concerned with logical aspects only. Instead of asking why science succeeds he considered the pragmatic problem of induction. This problem is not how to justify a theory or what is the global mechanism for the success of science but only what methodology do we use to pick one theory among theories that are already conjectured. His methodological answer to the latter question is that we pick the theory that is the most tested with the available technology: "the one, which in the light of our critical discussion, appears to be the best so far". By his own account, because only a negative approach was supported by logic, Popper adopted a negative methodology. The purpose of his methodology is to prevent "the policy of immunizing our theories against refutation". It also supports some "dogmatic attitude" in defending theories against criticism, because this allows the process to be more complete. This negative view of science was much criticized and not only by Johnson-Laird.

In practice, some steps based on observations can be justified under assumptions, which can be very natural. For example, Bayesian inductive logic is justified by theorems that make explicit assumptions. These theorems are obtained with deductive logic, not inductive logic. They are sometimes presented as steps of induction, because they refer to laws of probability, even though they do not go beyond deductive logic. This is yet a third notion of induction, which overlaps with deductive logic in the following sense that it is supported by it. These deductive steps are not really inductive, but the overall process that includes the creation of assumptions is inductive in the usual sense. In a fallibilist perspective, a perspective that is widely accepted by philosophers, including Popper, every logical step of learning only creates an assumption or reinstates one that was doubted—that is all that science logically does.

The elusive distinction between the logic of science and its applied methodology

Popper distinguished between the logic of science and its applied methodology. For example, the falsifiability of Newton's law of gravitation, as defined by Popper, depends purely on the logical relation it has with a statement such as "The brick fell upwards when released". A brick that falls upwards would not alone falsify Newton's law of gravitation. The capacity to verify the absence of conditions such as a hidden string attached to the brick is also needed for this state of affairs to eventually falsify Newton's law of gravitation. However, these applied methodological considerations are irrelevant in falsifiability, because it is a logical criterion. The empirical requirement on the potential falsifier, also called the material requirement, is only that it is observable inter-subjectively with existing technologies. There is no requirement that the potential falsifier can actually show the law to be false. The purely logical contradiction, together with the material requirement, are sufficient. The logical part consists of theories, statements, and their purely logical relationship together with this material requirement, which is needed for a connection with the methodological part.

The methodological part consists, in Popper's view, of informal rules, which are used to guess theories, accept observation statements as factual, etc. These include statistical tests: Popper is aware that observation statements are accepted with the help of statistical methods and that these involve methodological decisions. When this distinction is applied to the term "falsifiability", it corresponds to a distinction between two completely different meanings of the term. The same is true for the term "falsifiable". Popper said that he only uses "falsifiability" or "falsifiable" in reference to the logical side and that, when he refers to the methodological side, he speaks instead of "falsification" and its problems.

Popper said that methodological problems require proposing methodological rules. For example, one such rule is that, if one refuses to go along with falsifications, then one has retired oneself from the game of science. The logical side does not have such methodological problems, in particular with regard to the falsifiability of a theory, because basic statements are not required to be possible. Methodological rules are only needed in the context of actual falsifications.

So observations have two purposes in Popper's view. On the methodological side, observations can be used to show that a law is false, which Popper calls falsification. On the logical side, observations, which are purely logical constructions, do not show a law to be false, but contradict a law to show its falsifiability. Unlike falsifications and free from the problems of falsification, these contradictions establish the value of the law, which may eventually be corroborated.

Popper wrote that an entire literature exists because this distinction between the logical aspect and the methodological aspect was not observed. This is still seen in a more recent literature. For example, in their 2019 article Evidence based medicine as science, Vere and Gibson wrote "[falsifiability has] been considered problematic because theories are not simply tested through falsification but in conjunction with auxiliary assumptions and background knowledge."

Basic statements and the definition of falsifiability

Basic statements

In Popper's view of science, statements of observation can be analyzed within a logical structure independently of any factual observations. The set of all purely logical observations that are considered constitutes the empirical basis. Popper calls them the basic statements or test statements. They are the statements that can be used to show the falsifiability of a theory. Popper says that basic statements do not have to be possible in practice. It is sufficient that they are accepted by convention as belonging to the empirical language, a language that allows intersubjective verifiability: "they must be testable by intersubjective observation (the material requirement)". See the examples in section § Examples of demarcation and applications.

In more than twelve pages of The Logic of Scientific Discovery, Popper discusses informally which statements among those that are considered in the logical structure are basic statements. A logical structure uses universal classes to define laws. For example, in the law "all swans are white" the concept of swans is a universal class. It corresponds to a set of properties that every swan must have. It is not restricted to the swans that exist, existed or will exist. Informally, a basic statement is simply a statement that concerns only a finite number of specific instances in universal classes. In particular, an existential statement such as "there exists a black swan" is not a basic statement, because it is not specific about the instance. On the other hand, "this swan here is black" is a basic statement. Popper says that it is a singular existential statement or simply a singular statement. So, basic statements are singular (existential) statements.

The definition of falsifiability

Thornton says that basic statements are statements that correspond to particular "observation-reports". He then gives Popper's definition of falsifiability:

"A theory is scientific if and only if it divides the class of basic statements into the following two non-empty sub-classes: (a) the class of all those basic statements with which it is inconsistent, or which it prohibits—this is the class of its potential falsifiers (i.e., those statements which, if true, falsify the whole theory), and (b) the class of those basic statements with which it is consistent, or which it permits (i.e., those statements which, if true, corroborate it, or bear it out)."

— Thornton, Stephen, Thornton 2016, at the end of section 3

As in the case of actual falsifiers, decisions must be taken by scientists to accept a logical structure and its associated empirical basis, but these are usually part of a background knowledge that scientists have in common and, often, no discussion is even necessary. The first decision described by Lakatos is implicit in this agreement, but the other decisions are not needed. This agreement, if one can speak of agreement when there is not even a discussion, exists only in principle. This is where the distinction between the logical and methodological sides of science becomes important. When an actual falsifier is proposed, the technology used is considered in detail and, as described in section § Dogmatic falsificationism, an actual agreement is needed. This may require using a deeper empirical basis, hidden within the current empirical basis, to make sure that the properties or values used in the falsifier were obtained correctly (Andersson 2016 gives some examples).

Popper says that despite the fact that the empirical basis can be shaky, more comparable to a swamp than to solid ground, the definition that is given above is simply the formalization of a natural requirement on scientific theories, without which the whole logical process of science would not be possible.

Initial condition and prediction in falsifiers of laws

In his analysis of the scientific nature of universal laws, Popper arrived at the conclusion that laws must "allow us to deduce, roughly speaking, more empirical singular statements than we can deduce from the initial conditions alone." A singular statement that has one part only cannot contradict a universal law. A falsifier of a law has always two parts: the initial condition and the singular statement that contradicts the prediction.

However, there is no need to require that falsifiers have two parts in the definition itself. This removes the requirement that a falsifiable statement must make prediction. In this way, the definition is more general and allows the basic statements themselves to be falsifiable. Criteria that require that a law must be predictive, just as is required by falsifiability (when applied to laws), Popper wrote, "have been put forward as criteria of the meaningfulness of sentences (rather than as criteria of demarcation applicable to theoretical systems) again and again after the publication of my book, even by critics who pooh-poohed my criterion of falsifiability."

Falsifiability in model theory

Scientists such as the Nobel laureate Herbert A. Simon have studied the semantic aspects of the logical side of falsifiability. Here it is proposed that there are two formal requirements for a formally defined and stringent falsifiability that a scientific theory must satisfy to qualify as scientific: that they be finitely and irrevocably testable. These studies were done in the perspective that a logic is a relation between formal sentences in languages and a collection of mathematical structures, each of which is considered a model within model theory. The relation, usually denoted ${\displaystyle \mathfrak{A}\models \varphi }$, says the formal sentence ${\displaystyle \varphi }$ is true when interpreted in the structure ${\displaystyle \mathfrak{A}}$—it provides the semantic of the languages. According to Rynasiewicz, in this semantic perspective, falsifiability as defined by Popper means that in some observation structure (in the collection) there exists a set of observations which refutes the theory. An even stronger notion of falsifiability was considered, which requires, not only that there exists one structure with a contradicting set of observations, but also that all structures in the collection that cannot be expanded to a structure that satisfies ${\displaystyle \varphi }$ contain such a contradicting set of observations.

Examples of demarcation and applications

Newton's theory

Main article: Newton's law of universal gravitation

In response to Lakatos who suggested that Newton's theory was as hard to show falsifiable as Freud's psychoanalytic theory, Popper gave the example of an apple that moves from the ground up to a branch and then starts to dance from one branch to another. Popper thought that it was a basic statement that was a potential falsifier for Newton's theory, because the position of the apple at different times can be measured. Popper's claims on this point are controversial, since Newtonian physics does not deny that there could be forces acting on the apple that are stronger than Earth's gravity.

Einstein's equivalence principle

Main article: Einstein's equivalence principle

Another example of a basic statement is "The inert mass of this object is ten times larger than its gravitational mass." This is a basic statement because the inert mass and the gravitational mass can both be measured separately, even though it never happens that they are different. It is, as described by Popper, a valid falsifier for Einstein's equivalence principle.

Evolution

Industrial melanism

Main article: Industrial melanism

A black-bodied and white-bodied peppered moth

In a discussion of the theory of evolution, Popper mentioned industrial melanism as an example of a falsifiable law. A corresponding basic statement that acts as a potential falsifier is "In this industrial area, the relative fitness of the white-bodied peppered moth is high." Here "fitness" means "reproductive success over the next generation". It is a basic statement, because it is possible to separately determine the kind of environment, industrial vs natural, and the relative fitness of the white-bodied form (relative to the black-bodied form) in an area, even though it never happens that the white-bodied form has a high relative fitness in an industrial area.

Precambrian rabbit

Main article: Precambrian rabbit

A famous example of a basic statement from J. B. S. Haldane is "[These are] fossil rabbits in the Precambrian era." This is a basic statement because it is possible to find a fossil rabbit and to determine that the date of a fossil is in the Precambrian era, even though it never happens that the date of a rabbit fossil is in the Precambrian era. Despite opinions to the contrary, sometimes wrongly attributed to Popper, this shows the scientific character of paleontology or the history of the evolution of life on Earth, because it contradicts the hypothesis in paleontology that all mammals existed in a much more recent era. Richard Dawkins adds that any other modern animal, such as a hippo, would suffice.

Simple examples of unfalsifiable statements

Even if it is accepted that angels exist, "All angels have large wings" is not falsifiable, because no technology exists to identify and observe angels.

A simple example of a non-basic statement is "This angel does not have large wings." It is not a basic statement, because though the absence of large wings can be observed, no technology (independent of the presence of wings) exists to identify angels. Even if it is accepted that angels exist, the sentence "All angels have large wings" is not falsifiable.

Another example from Popper of a non-basic statement is "This human action is altruistic." It is not a basic statement, because no accepted technology allows us to determine whether or not an action is motivated by self-interest. Because no basic statement falsifies it, the statement that "All human actions are egotistic, motivated by self-interest" is thus not falsifiable.

Omphalos hypothesis

Main article: Omphalos hypothesis

Some adherents of young-Earth creationism make an argument (called the Omphalos hypothesis after the Greek word for navel) that the world was created with the appearance of age; e.g., the sudden appearance of a mature chicken capable of laying eggs. This ad hoc hypothesis introduced into young-Earth creationism is unfalsifiable because it says that the time of creation (of a species) measured by the accepted technology is illusory and no accepted technology is proposed to measure the claimed "actual" time of creation. Moreover, if the ad hoc hypothesis says that the world was created as we observe it today without stating further laws, by definition it cannot be contradicted by observations and thus is not falsifiable. This is discussed by Dienes in the case of a variation on the Omphalos hypothesis, which, in addition, specifies that God made the creation in this way to test our faith.

Useful metaphysical statements

Grover Maxwell ^[es] discussed statements such as "All men are mortal." This is not falsifiable, because it does not matter how old a man is, maybe he will die next year. Maxwell said that this statement is nevertheless useful, because it is often corroborated. He coined the term "corroboration without demarcation". Popper's view is that it is indeed useful, because Popper considers that metaphysical statements can be useful, but also because it is indirectly corroborated by the corroboration of the falsifiable law "All men die before the age of 150." For Popper, if no such falsifiable law exists, then the metaphysical law is less useful, because it is not indirectly corroborated. This kind of non-falsifiable statements in science was noticed by Carnap as early as 1937.

Clyde Cowan conducting the neutrino experiment (c. 1956)

Maxwell also used the example "All solids have a melting point." This is not falsifiable, because maybe the melting point will be reached at a higher temperature. The law is falsifiable and more useful if we specify an upper bound on melting points or a way to calculate this upper bound.

Another example from Maxwell is "All beta decays are accompanied with a neutrino emission from the same nucleus." This is also not falsifiable, because maybe the neutrino can be detected in a different manner. The law is falsifiable and much more useful from a scientific point of view, if the method to detect the neutrino is specified. Maxwell said that most scientific laws are metaphysical statements of this kind, which, Popper said, need to be made more precise before they can be indirectly corroborated. In other words, specific technologies must be provided to make the statements inter-subjectively-verifiable, i.e., so that scientists know what the falsification or its failure actually means.

In his critique of the falsifiability criterion, Maxwell considered the requirement for decisions in the falsification of, both, the emission of neutrinos (see § Dogmatic falsificationism) and the existence of the melting point. For example, he pointed out that had no neutrino been detected, it could have been because some conservation law is false. Popper did not argue against the problems of falsification per se. He always acknowledged these problems. Popper's response was at the logical level. For example, he pointed out that, if a specific way is given to trap the neutrino, then, at the level of the language, the statement is falsifiable, because "no neutrino was detected after using this specific way" formally contradicts it (and it is inter-subjectively-verifiable—people can repeat the experiment).

Natural selection

Main article: Survival of the fittest § Tautology

In the 5th and 6th editions of On the Origin of Species, following a suggestion of Alfred Russel Wallace, Darwin used "Survival of the fittest", an expression first coined by Herbert Spencer, as a synonym for "Natural Selection". Popper and others said that, if one uses the most widely accepted definition of "fitness" in modern biology (see subsection § Evolution), namely reproductive success itself, the expression "survival of the fittest" is a tautology.

Darwinist Ronald Fisher worked out mathematical theorems to help answer questions regarding natural selection. But, for Popper and others, there is no (falsifiable) law of Natural Selection in this, because these tools only apply to some rare traits. Instead, for Popper, the work of Fisher and others on Natural Selection is part of an important and successful metaphysical research program.

Mathematics

See also: Mathematical proof, Scientific method § Relationship with mathematics, Mathematics § Mathematics as science, and Philosophy of mathematics § Popper's two senses of number statements

Popper said that not all unfalsifiable statements are useless in science. Mathematical statements are good examples. Like all formal sciences, mathematics is not concerned with the validity of theories based on observations in the empirical world, but rather, mathematics is occupied with the theoretical, abstract study of such topics as quantity, structure, space and change. Methods of the mathematical sciences are, however, applied in constructing and testing scientific models dealing with observable reality. Albert Einstein wrote, "One reason why mathematics enjoys special esteem, above all other sciences, is that its laws are absolutely certain and indisputable, while those of other sciences are to some extent debatable and in constant danger of being overthrown by newly discovered facts."

Historicism

See also: Historicism § Karl Popper

Popper made a clear distinction between the original theory of Marx and what came to be known as Marxism later on. For Popper, the original theory of Marx contained genuine scientific laws. Though they could not make preordained predictions, these laws constrained how changes can occur in society. One of them was that changes in society cannot "be achieved by the use of legal or political means". In Popper's view, this was both testable and subsequently falsified. "Yet instead of accepting the refutations", Popper wrote, "the followers of Marx re-interpreted both the theory and the evidence in order to make them agree. ... They thus gave a 'conventionalist twist' to the theory; and by this stratagem, they destroyed its much advertised claim to scientific status." Popper's attacks were not directed toward Marxism, or Marx's theories, which were falsifiable, but toward Marxists who he considered to have ignored the falsifications which had happened. Popper more fundamentally criticized 'historicism' in the sense of any preordained prediction of history, given what he saw as our right, ability and responsibility to control our own destiny.

Use in courts of law

Falsifiability has been used in the McLean v. Arkansas case (in 1982), the Daubert case (in 1993)^[52] and other cases. A survey of 303 federal judges conducted in 1998 found that "[P]roblems with the nonfalsifiable nature of an expert's underlying theory and difficulties with an unknown or too-large error rate were cited in less than 2% of cases."

McLean v. Arkansas case

In the ruling of the McLean v. Arkansas case, Judge William Overton used falsifiability as one of the criteria to determine that "creation science" was not scientific and should not be taught in Arkansas public schools as such (it can be taught as religion). In his testimony, philosopher Michael Ruse defined the characteristics which constitute science as (see Pennock 2000, p. 5, and Ruse 2010):

• It is guided by natural law;
• It has to be explanatory by reference to natural law;
• It is testable against the empirical world;
• Its conclusions are tentative, i.e., are not necessarily the final word; and
• It is falsifiable.

In his conclusion related to this criterion Judge Overton stated that:

While anybody is free to approach a scientific inquiry in any fashion they choose, they cannot properly describe the methodology as scientific, if they start with the conclusion and refuse to change it regardless of the evidence developed during the course of the investigation.

— William Overton, McLean v. Arkansas 1982, at the end of section IV. (C)

Daubert standard

Main article: Daubert standard

In several cases of the United States Supreme Court, the court described scientific methodology using the five Daubert factors, which include falsifiability. The Daubert result cited Popper and other philosophers of science:

Ordinarily, a key question to be answered in determining whether a theory or technique is scientific knowledge that will assist the trier of fact will be whether it can be (and has been) tested. Scientific methodology today is based on generating hypotheses and testing them to see if they can be falsified; indeed, this methodology is what distinguishes science from other fields of human inquiry. Green 645. See also C. Hempel, Philosophy of Natural Science 49 (1966) ([T]he statements constituting a scientific explanation must be capable of empirical test); K. Popper, Conjectures and Refutations: The Growth of Scientific Knowledge 37 (5th ed. 1989) ([T]he criterion of the scientific status of a theory is its falsifiability, or refutability, or testability) (emphasis deleted).

— Harry Blackmun, Daubert 1993, p. 593

David H. Kaye said that references to the Daubert majority opinion confused falsifiability and falsification and that "inquiring into the existence of meaningful attempts at falsification is an appropriate and crucial consideration in admissibility determinations."

Connections between statistical theories and falsifiability

Considering the specific detection procedure that was used in the neutrino experiment, without mentioning its probabilistic aspect, Popper wrote "it provided a test of the much more significant falsifiable theory that such emitted neutrinos could be trapped in a certain way". In this manner, in his discussion of the neutrino experiment, Popper did not raise at all the probabilistic aspect of the experiment. Together with Maxwell, who raised the problems of falsification in the experiment, he was aware that some convention must be adopted to fix what it means to detect or not a neutrino in this probabilistic context. This is the third kind of decisions mentioned by Lakatos. For Popper and most philosophers, observations are theory impregnated. In this example, the theory that impregnates observations (and justifies that we conventionally accept the potential falsifier "no neutrino was detected") is statistical. In statistical language, the potential falsifier that can be statistically accepted (not rejected to say it more correctly) is typically the null hypothesis, as understood even in popular accounts on falsifiability.

Different ways are used by statisticians to draw conclusions about hypotheses on the basis of available evidence. Fisher, Neyman and Pearson proposed approaches that require no prior probabilities on the hypotheses that are being studied. In contrast, Bayesian inference emphasizes the importance of prior probabilities. But, as far as falsification as a yes/no procedure in Popper's methodology is concerned, any approach that provides a way to accept or not a potential falsifier can be used, including approaches that use Bayes' theorem and estimations of prior probabilities that are made using critical discussions and reasonable assumptions taken from the background knowledge. There is no general rule that considers as falsified an hypothesis with small Bayesian revised probability, because as pointed out by Mayo and argued before by Popper, the individual outcomes described in detail will easily have very small probabilities under available evidence without being genuine anomalies. Nevertheless, Mayo adds, "they can indirectly falsify hypotheses by adding a methodological falsification rule". In general, Bayesian statistic can play a role in critical rationalism in the context of inductive logic, which is said to be inductive because implications are generalized to conditional probabilities. According to Popper and other philosophers such as Colin Howson, Hume's argument precludes inductive logic, but only when the logic makes no use "of additional assumptions: in particular, about what is to be assigned positive prior probability". Inductive logic itself is not precluded, especially not when it is a deductively valid application of Bayes' theorem that is used to evaluate the probabilities of the hypotheses using the observed data and what is assumed about the priors. Gelman and Shalizi mentioned that Bayes' statisticians do not have to disagree with the non-inductivists.

Because statisticians often associate statistical inference with induction, Popper's philosophy is often said to have a hidden form of induction. For example, Mayo wrote "The falsifying hypotheses ... necessitate an evidence-transcending (inductive) statistical inference. This is hugely problematic for Popper". Yet, also according to Mayo, Popper [as a non-inductivist] acknowledged the useful role of statistical inference in the falsification problems: she mentioned that Popper wrote her (in the context of falsification based on evidence) "I regret not studying statistics" and that her thought was then "not as much as I do".

Lakatos's falsificationism

Imre Lakatos divided the problems of falsification in two categories. The first category corresponds to decisions that must be agreed upon by scientists before they can falsify a theory. The other category emerges when one tries to use falsifications and corroborations to explain progress in science. Lakatos described four kind of falsificationisms in view of how they address these problems. Dogmatic falsificationism ignores both types of problems. Methodological falsificationism addresses the first type of problems by accepting that decisions must be taken by scientists. Naive methodological falsificationism or naive falsificationism does not do anything to address the second type of problems. Lakatos used dogmatic and naive falsificationism to explain how Popper's philosophy changed over time and viewed sophisticated falsificationism as his own improvement on Popper's philosophy, but also said that Popper some times appears as a sophisticated falsificationist. Popper responded that Lakatos misrepresented his intellectual history with these terminological distinctions.

Dogmatic falsificationism

A dogmatic falsificationist ignores that every observation is theory-impregnated. Being theory-impregnated means that it goes beyond direct experience. For example, the statement "Here is a glass of water" goes beyond experience, because the concepts of glass and water "denote physical bodies which exhibit a certain law-like behaviour" (Popper). This leads to the critique that it is unclear which theory is falsified. Is it the one that is being studied or the one behind the observation? This is sometimes called the 'Duhem–Quine problem'. An example is Galileo's refutation of the theory that celestial bodies are faultless crystal balls. Many considered that it was the optical theory of the telescope that was false, not the theory of celestial bodies. Another example is the theory that neutrinos are emitted in beta decays. Had they not been observed in the Cowan–Reines neutrino experiment, many would have considered that the strength of the beta-inverse reaction used to detect the neutrinos was not sufficiently high. At the time, Grover Maxwell ^[es] wrote, the possibility that this strength was sufficiently high was a "pious hope".

A dogmatic falsificationist ignores the role of auxiliary hypotheses. The assumptions or auxiliary hypotheses of a particular test are all the hypotheses that are assumed to be accurate in order for the test to work as planned. The predicted observation that is contradicted depends on the theory and these auxiliary hypotheses. Again, this leads to the critique that it cannot be told if it is the theory or one of the required auxiliary hypotheses that is false. Lakatos gives the example of the path of a planet. If the path contradicts Newton's law, we will not know if it is Newton's law that is false or the assumption that no other body influenced the path.

Lakatos says that Popper's solution to these criticisms requires that one relaxes the assumption that an observation can show a theory to be false:

If a theory is falsified [in the usual sense], it is proven false; if it is 'falsified' [in the technical sense], it may still be true.

— Imre Lakatos, Lakatos 1978, p. 24

Methodological falsificationism replaces the contradicting observation in a falsification with a "contradicting observation" accepted by convention among scientists, a convention that implies four kinds of decisions that have these respective goals: the selection of all basic statements (statements that correspond to logically possible observations), selection of the accepted basic statements among the basic statements, making statistical laws falsifiable and applying the refutation to the specific theory (instead of an auxiliary hypothesis). The experimental falsifiers and falsifications thus depend on decisions made by scientists in view of the currently accepted technology and its associated theory.

Naive falsificationism

According to Lakatos, naive falsificationism is the claim that methodological falsifications can by themselves explain how scientific knowledge progresses. Very often a theory is still useful and used even after it is found in contradiction with some observations. Also, when scientists deal with two or more competing theories which are both corroborated, considering only falsifications, it is not clear why one theory is chosen above the other, even when one is corroborated more often than the other. In fact, a stronger version of the Quine-Duhem thesis says that it is not always possible to rationally pick one theory over the other using falsifications. Considering only falsifications, it is not clear why often a corroborating experiment is seen as a sign of progress. Popper's critical rationalism uses both falsifications and corroborations to explain progress in science. How corroborations and falsifications can explain progress in science was a subject of disagreement between many philosophers, especially between Lakatos and Popper.

Popper distinguished between the creative and informal process from which theories and accepted basic statements emerge and the logical and formal process where theories are falsified or corroborated. The main issue is whether the decision to select a theory among competing theories in the light of falsifications and corroborations could be justified using some kind of formal logic. It is a delicate question, because this logic would be inductive: it justifies a universal law in view of instances. Also, falsifications, because they are based on methodological decisions, are useless in a strict justification perspective. The answer of Lakatos and many others to that question is that it should. In contradistinction, for Popper, the creative and informal part is guided by methodological rules, which naturally say to favour theories that are corroborated over those that are falsified, but this methodology can hardly be made rigorous.

Popper's way to analyze progress in science was through the concept of verisimilitude, a way to define how close a theory is to the truth, which he did not consider very significant, except (as an attempt) to describe a concept already clear in practice. Later, it was shown that the specific definition proposed by Popper cannot distinguish between two theories that are false, which is the case for all theories in the history of science. Today, there is still on going research on the general concept of verisimilitude.

From the problem of induction to falsificationism

Hume explained induction with a theory of the mind that was in part inspired by Newton's theory of gravitation. Popper rejected Hume's explanation of induction and proposed his own mechanism: science progresses by trial and error within an evolutionary epistemology. Hume believed that his psychological induction process follows laws of nature, but, for him, this does not imply the existence of a method of justification based on logical rules. In fact, he argued that any induction mechanism, including the mechanism described by his theory, could not be justified logically. Similarly, Popper adopted an evolutionary epistemology, which implies that some laws explain progress in science, but yet insists that the process of trial and error is hardly rigorous and that there is always an element of irrationality in the creative process of science. The absence of a method of justification is a built-in aspect of Popper's trial and error explanation.

As rational as they can be, these explanations that refer to laws, but cannot be turned into methods of justification (and thus do not contradict Hume's argument or its premises), were not sufficient for some philosophers. In particular, Russell once expressed the view that if Hume's problem cannot be solved, “there is no intellectual difference between sanity and insanity” and actually proposed a method of justification. He rejected Hume's premise that there is a need to justify any principle that is itself used to justify induction. It might seem that this premise is hard to reject, but to avoid circular reasoning we do reject it in the case of deductive logic. It makes sense to also reject this premise in the case of principles to justify induction. Lakatos's proposal of sophisticated falsificationism was very natural in that context.

Therefore, Lakatos urged Popper to find an inductive principle behind the trial and error learning process and sophisticated falsificationism was his own approach to address this challenge. Kuhn, Feyerabend, Musgrave and others mentioned and Lakatos himself acknowledged that, as a method of justification, this attempt failed, because there was no normative methodology to justify—Lakatos's methodology was anarchy in disguise.

Falsificationism in Popper's philosophy

Popper's philosophy is sometimes said to fail to recognize the Quine-Duhem thesis, which would make it a form of dogmatic falsificationism. For example, Watkins wrote "apparently forgetting that he had once said 'Duhem is right [...]', Popper set out to devise potential falsifiers just for Newton's fundamental assumptions". But, Popper's philosophy is not always qualified of falsificationism in the pejorative manner associated with dogmatic or naive falsificationism. The problems of falsification are acknowledged by the falsificationists. For example, Chalmers points out that falsificationists freely admit that observation is theory impregnated. Thornton, referring to Popper's methodology, says that the predictions inferred from conjectures are not directly compared with the facts simply because all observation-statements are theory-laden. For the critical rationalists, the problems of falsification are not an issue, because they do not try to make experimental falsifications logical or to logically justify them, nor to use them to logically explain progress in science. Instead, their faith rests on critical discussions around these experimental falsifications. Lakatos made a distinction between a "falsification" (with quotation marks) in Popper's philosophy and a falsification (without quotation marks) that can be used in a systematic methodology where rejections are justified. He knew that Popper's philosophy is not and has never been about this kind of justification, but he felt that it should have been. Sometimes, Popper and other falsificationists say that when a theory is falsified it is rejected, which appears as dogmatic falsificationism, but the general context is always critical rationalism in which all decisions are open to critical discussions and can be revised.

Controversies

Methodless creativity versus inductive methodology

Main article: The problem of induction

As described in section § Naive falsificationism, Lakatos and Popper agreed that universal laws cannot be logically deduced (except from laws that say even more). But unlike Popper, Lakatos felt that if the explanation for new laws cannot be deductive, it must be inductive. He urged Popper explicitly to adopt some inductive principle and sets himself the task to find an inductive methodology. However, the methodology that he found did not offer any exact inductive rules. In a response to Kuhn, Feyerabend and Musgrave, Lakatos acknowledged that the methodology depends on the good judgment of the scientists. Feyerabend wrote in "Against Method" that Lakatos's methodology of scientific research programmes is epistemological anarchism in disguise and Musgrave made a similar comment. In more recent work, Feyerabend says that Lakatos uses rules, but whether or not to follow any of these rules is left to the judgment of the scientists. This is also discussed elsewhere.

Popper also offered a methodology with rules, but these rules are also not-inductive rules, because they are not by themselves used to accept laws or establish their validity. They do that through the creativity or "good judgment" of the scientists only. For Popper, the required non deductive component of science never had to be an inductive methodology. He always viewed this component as a creative process beyond the explanatory reach of any rational methodology, but yet used to decide which theories should be studied and applied, find good problems and guess useful conjectures. Quoting Einstein to support his view, Popper said that this renders obsolete the need for an inductive methodology or logical path to the laws. For Popper, no inductive methodology was ever proposed to satisfactorily explain science.

Ahistorical versus historiographical

Main article: Imre Lakatos § Research programmes

Section § Methodless creativity versus inductive methodology says that both Lakatos's and Popper's methodology are not inductive. Yet Lakatos's methodology extended importantly Popper's methodology: it added a historiographical component to it. This allowed Lakatos to find corroborations for his methodology in the history of science. The basic units in his methodology, which can be abandoned or pursued, are research programmes. Research programmes can be degenerative or progressive and only degenerative research programmes must be abandoned at some point. For Lakatos, this is mostly corroborated by facts in history.

In contradistinction, Popper did not propose his methodology as a tool to reconstruct the history of science. Yet, some times, he did refer to history to corroborate his methodology. For example, he remarked that theories that were considered great successes were also the most likely to be falsified. Zahar's view was that, with regard to corroborations found in the history of science, there was only a difference of emphasis between Popper and Lakatos.

As an anecdotal example, in one of his articles Lakatos challenged Popper to show that his theory was falsifiable: he asked "Under what conditions would you give up your demarcation criterion?". Popper replied "I shall give up my theory if Professor Lakatos succeeds in showing that Newton's theory is no more falsifiable by 'observable states of affairs' than is Freud's." According to David Stove, Lakatos succeeded, since Lakatos showed there is no such thing as a "non-Newtonian" behaviour of an observable object. Stove argued that Popper's counterexamples to Lakatos were either instances of begging the question, such as Popper's example of missiles moving in a "non-Newtonian track", or consistent with Newtonian physics, such as objects not falling to the ground without "obvious" countervailing forces against Earth's gravity.

Normal science versus revolutionary science

Main article: Paradigm shift

Thomas Kuhn analyzed what he calls periods of normal science as well as revolutions from one period of normal science to another, whereas Popper's view is that only revolutions are relevant.^[BZ][CA] For Popper, the role of science, mathematics and metaphysics, actually the role of any knowledge, is to solve puzzles. In the same line of thought, Kuhn observes that in periods of normal science the scientific theories, which represent some paradigm, are used to routinely solve puzzles and the validity of the paradigm is hardly in question. It is only when important new puzzles emerge that cannot be solved by accepted theories that a revolution might occur. This can be seen as a viewpoint on the distinction made by Popper between the informal and formal process in science (see section § Naive falsificationism). In the big picture presented by Kuhn, the routinely solved puzzles are corroborations. Falsifications or otherwise unexplained observations are unsolved puzzles. All of these are used in the informal process that generates a new kind of theory. Kuhn says that Popper emphasizes formal or logical falsifications and fails to explain how the social and informal process works.

Unfalsifiability versus falsity of astrology

Main article: Astrology

Popper often uses astrology as an example of a pseudoscience. He says that it is not falsifiable because both the theory itself and its predictions are too imprecise. Kuhn, as a historian of science, remarked that many predictions made by astrologers in the past were quite precise and they were very often falsified. He also said that astrologers themselves acknowledged these falsifications.

Epistemological anarchism vs the scientific method

Main article: Epistemological anarchism

Paul Feyerabend rejected any prescriptive methodology at all. He rejected Lakatos's argument for ad hoc hypothesis, arguing that science would not have progressed without making use of any and all available methods to support new theories. He rejected any reliance on a scientific method, along with any special authority for science that might derive from such a method. He said that if one is keen to have a universally valid methodological rule, epistemological anarchism or anything goes would be the only candidate. For Feyerabend, any special status that science might have, derives from the social and physical value of the results of science rather than its method.

Sokal and Bricmont

In their book Fashionable Nonsense (from 1997, published in the UK as Intellectual Impostures) the physicists Alan Sokal and Jean Bricmont criticised falsifiability. They include this critique in the "Intermezzo" chapter, where they expose their own views on truth in contrast to the extreme epistemological relativism of postmodernism. Even though Popper is clearly not a relativist, Sokal and Bricmont discuss falsifiability because they see postmodernist epistemological relativism as a reaction to Popper's description of falsifiability, and more generally, to his theory of science.