Hotelling's T-squared distribution

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Student%27s_t-test 

Student's t-test is a statistical test used to test whether the difference between the response of two groups is statistically significant or not. It is any statistical hypothesis test in which the test statistic follows a Student's t-distribution under the null hypothesis. It is most commonly applied when the test statistic would follow a normal distribution if the value of a scaling term in the test statistic were known (typically, the scaling term is unknown and is therefore a nuisance parameter). When the scaling term is estimated based on the data, the test statistic—under certain conditions—follows a Student's t distribution. The t-test's most common application is to test whether the means of two populations are significantly different. In many cases, a Z-test will yield very similar results to a t-test because the latter converges to the former as the size of the dataset increases.

History

William Sealy Gosset, who developed the "t-statistic" and published it under the pseudonym of "Student"

The term "t-statistic" is abbreviated from "hypothesis test statistic". In statistics, the t-distribution was first derived as a posterior distribution in 1876 by Helmert and Lüroth. The t-distribution also appeared in a more general form as Pearson type IV distribution in Karl Pearson's 1895 paper. However, the t-distribution, also known as Student's t-distribution, gets its name from William Sealy Gosset, who first published it in English in 1908 in the scientific journal Biometrika using the pseudonym "Student" because his employer preferred staff to use pen names when publishing scientific papers. Gosset worked at the Guinness Brewery in Dublin, Ireland, and was interested in the problems of small samples – for example, the chemical properties of barley with small sample sizes. Hence a second version of the etymology of the term Student is that Guinness did not want their competitors to know that they were using the t-test to determine the quality of raw material. Although it was William Gosset after whom the term "Student" is penned, it was actually through the work of Ronald Fisher that the distribution became well known as "Student's distribution" and "Student's t-test".

Gosset devised the t-test as an economical way to monitor the quality of stout. The t-test work was submitted to and accepted in the journal Biometrika and published in 1908.

Guinness had a policy of allowing technical staff leave for study (so-called "study leave"), which Gosset used during the first two terms of the 1906–1907 academic year in Professor Karl Pearson's Biometric Laboratory at University College London. Gosset's identity was then known to fellow statisticians and to editor-in-chief Karl Pearson.

Uses

One-sample t-test

A one-sample Student's t-test is a location test of whether the mean of a population has a value specified in a null hypothesis. In testing the null hypothesis that the population mean is equal to a specified value μ₀, one uses the statistic

${\displaystyle t=\frac{\overline{x}-{\mu }_0}{s/\sqrt{n}},}$

where ${\displaystyle \overline{x}}$ is the sample mean, s is the sample standard deviation and n is the sample size. The degrees of freedom used in this test are n − 1. Although the parent population does not need to be normally distributed, the distribution of the population of sample means ${\displaystyle \overline{x}}$ is assumed to be normal.

By the central limit theorem, if the observations are independent and the second moment exists, then ${\displaystyle t}$ will be approximately normal $\mathcal{N}(0,1)$.

Two-sample t-tests

Type I error of unpaired and paired two-sample t-tests as a function of the correlation. The simulated random numbers originate from a bivariate normal distribution with a variance of 1. The significance level is 5% and the number of cases is 60.

Power of unpaired and paired two-sample t-tests as a function of the correlation. The simulated random numbers originate from a bivariate normal distribution with a variance of 1 and a deviation of the expected value of 0.4. The significance level is 5% and the number of cases is 60.

A two-sample location test of the null hypothesis such that the means of two populations are equal. All such tests are usually called Student's t-tests, though strictly speaking that name should only be used if the variances of the two populations are also assumed to be equal; the form of the test used when this assumption is dropped is sometimes called Welch's t-test. These tests are often referred to as unpaired or independent samples t-tests, as they are typically applied when the statistical units underlying the two samples being compared are non-overlapping.

Two-sample t-tests for a difference in means involve independent samples (unpaired samples) or paired samples. Paired t-tests are a form of blocking, and have greater power (probability of avoiding a type II error, also known as a false negative) than unpaired tests when the paired units are similar with respect to "noise factors" (see confounder) that are independent of membership in the two groups being compared. In a different context, paired t-tests can be used to reduce the effects of confounding factors in an observational study.

Independent (unpaired) samples

The independent samples t-test is used when two separate sets of independent and identically distributed samples are obtained, and one variable from each of the two populations is compared. For example, suppose we are evaluating the effect of a medical treatment, and we enroll 100 subjects into our study, then randomly assign 50 subjects to the treatment group and 50 subjects to the control group. In this case, we have two independent samples and would use the unpaired form of the t-test.

Paired samples

Main article: Paired difference test

Paired samples t-tests typically consist of a sample of matched pairs of similar units, or one group of units that has been tested twice (a "repeated measures" t-test).

A typical example of the repeated measures t-test would be where subjects are tested prior to a treatment, say for high blood pressure, and the same subjects are tested again after treatment with a blood-pressure-lowering medication. By comparing the same patient's numbers before and after treatment, we are effectively using each patient as their own control. That way the correct rejection of the null hypothesis (here: of no difference made by the treatment) can become much more likely, with statistical power increasing simply because the random interpatient variation has now been eliminated. However, an increase of statistical power comes at a price: more tests are required, each subject having to be tested twice. Because half of the sample now depends on the other half, the paired version of Student's t-test has only ⁠n/2⁠ − 1 degrees of freedom (with n being the total number of observations). Pairs become individual test units, and the sample has to be doubled to achieve the same number of degrees of freedom. Normally, there are n − 1 degrees of freedom (with n being the total number of observations).

A paired samples t-test based on a "matched-pairs sample" results from an unpaired sample that is subsequently used to form a paired sample, by using additional variables that were measured along with the variable of interest. The matching is carried out by identifying pairs of values consisting of one observation from each of the two samples, where the pair is similar in terms of other measured variables. This approach is sometimes used in observational studies to reduce or eliminate the effects of confounding factors.

Paired samples t-tests are often referred to as "dependent samples t-tests".

Assumptions

Most test statistics have the form t = Z/s, where Z and s are functions of the data.

Z may be sensitive to the alternative hypothesis (i.e., its magnitude tends to be larger when the alternative hypothesis is true), whereas s is a scaling parameter that allows the distribution of t to be determined.

As an example, in the one-sample t-test

${\displaystyle t=\frac{Z}{s}=\frac{\overline{X}-\mu }{\hat{\sigma }/\sqrt{n}},}$

where ${\displaystyle \overline{X}}$ is the sample mean from a sample X₁, X₂, …, X_n, of size n, s is the standard error of the mean, ${\displaystyle \hat{\sigma }=\sqrt{\frac{1}{n-1}\sum _i(X_i-\overline{X}{)}^2}}$ is the estimate of the standard deviation of the population, and μ is the population mean.

The assumptions underlying a t-test in the simplest form above are that:

• X follows a normal distribution with mean μ and variance σ²/n.
• s²(n − 1)/σ² follows a χ² distribution with n − 1 degrees of freedom. This assumption is met when the observations used for estimating s² come from a normal distribution (and i.i.d. for each group).
• Z and s are independent.

In the t-test comparing the means of two independent samples, the following assumptions should be met:

• The means of the two populations being compared should follow normal distributions. Under weak assumptions, this follows in large samples from the central limit theorem, even when the distribution of observations in each group is non-normal.
• If using Student's original definition of the t-test, the two populations being compared should have the same variance (testable using F-test, Levene's test, Bartlett's test, or the Brown–Forsythe test; or assessable graphically using a Q–Q plot). If the sample sizes in the two groups being compared are equal, Student's original t-test is highly robust to the presence of unequal variances. Welch's t-test is insensitive to equality of the variances regardless of whether the sample sizes are similar.
• The data used to carry out the test should either be sampled independently from the two populations being compared or be fully paired. This is in general not testable from the data, but if the data are known to be dependent (e.g. paired by test design), a dependent test has to be applied. For partially paired data, the classical independent t-tests may give invalid results as the test statistic might not follow a t distribution, while the dependent t-test is sub-optimal as it discards the unpaired data.

Most two-sample t-tests are robust to all but large deviations from the assumptions.

For exactness, the t-test and Z-test require normality of the sample means, and the t-test additionally requires that the sample variance follows a scaled χ² distribution, and that the sample mean and sample variance be statistically independent. Normality of the individual data values is not required if these conditions are met. By the central limit theorem, sample means of moderately large samples are often well-approximated by a normal distribution even if the data are not normally distributed. However, the sample size required for the sample means to converge to normality depends on the skewness of the distribution of the original data. The sample can vary from 30 to 100 or higher values depending on the skewness.

For non-normal data, the distribution of the sample variance may deviate substantially from a χ² distribution.

However, if the sample size is large, Slutsky's theorem implies that the distribution of the sample variance has little effect on the distribution of the test statistic. That is, as sample size ${\displaystyle n}$ increases:

${\displaystyle \sqrt{n}(\overline{X}-\mu )\stackrel{d}{\to }N(0,{\sigma }^2)}$ as per the Central limit theorem,

${\displaystyle s^2\stackrel{p}{\to }{\sigma }^2}$ as per the law of large numbers,

${\displaystyle \therefore \frac{\sqrt{n}(\overline{X}-\mu )}{s}\stackrel{d}{\to }N(0,1)}$.

Calculations

Explicit expressions that can be used to carry out various t-tests are given below. In each case, the formula for a test statistic that either exactly follows or closely approximates a t-distribution under the null hypothesis is given. Also, the appropriate degrees of freedom are given in each case. Each of these statistics can be used to carry out either a one-tailed or two-tailed test.

Once the t value and degrees of freedom are determined, a p-value can be found using a table of values from Student's t-distribution. If the calculated p-value is below the threshold chosen for statistical significance (usually the 0.10, the 0.05, or 0.01 level), then the null hypothesis is rejected in favor of the alternative hypothesis.

Slope of a regression line

Suppose one is fitting the model

${\displaystyle Y=\alpha +\beta x+\epsilon ,}$

where x is known, α and β are unknown, ε is a normally distributed random variable with mean 0 and unknown variance σ², and Y is the outcome of interest. We want to test the null hypothesis that the slope β is equal to some specified value β₀ (often taken to be 0, in which case the null hypothesis is that x and y are uncorrelated).

Let

${\displaystyle \begin{array}{rl}\hat{\alpha },\hat{\beta }& =\text{least-squares estimators},\\ S{E}_{\hat{\alpha }},S{E}_{\hat{\beta }}& =\text{the standard errors of least-squares estimators}.\end{array}}$

Then

${\displaystyle t_{\text{score}}=\frac{\hat{\beta }-{\beta }_0}{S{E}_{\hat{\beta }}}\sim {\mathcal{T}}_{n-2}}$

has a t-distribution with n − 2 degrees of freedom if the null hypothesis is true. The standard error of the slope coefficient:

${\displaystyle S{E}_{\hat{\beta }}=\frac{\sqrt{{\displaystyle \frac{1}{n-2}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2}}}{\sqrt{{\displaystyle \sum _{i=1}^n(x_i-\overline{x}{)}^2}}}}$

can be written in terms of the residuals. Let

${\displaystyle \begin{array}{rl}{\hat{\epsilon }}_i& =y_i-{\hat{y}}_i=y_i-(\hat{\alpha }+\hat{\beta }{x}_i)=\text{residuals}=\text{estimated errors},\\ \text{SSR}& =\sum _{i=1}^n{{\hat{\epsilon }}_i}^2=\text{sum of squares of residuals}.\end{array}}$

Then t_score is given by

${\displaystyle t_{\text{score}}=\frac{(\hat{\beta }-{\beta }_0)\sqrt{n-2}}{\sqrt{\frac{SSR}{\sum _{i=1}^n(x_i-\overline{x}{)}^2}}}.}$

Another way to determine the t_score is

${\displaystyle t_{\text{score}}=\frac{r\sqrt{n-2}}{\sqrt{1-r^2}},}$

where r is the Pearson correlation coefficient.

The t_{score, intercept} can be determined from the t_{score, slope}:

${\displaystyle t_{\text{score,intercept}}=\frac{\alpha }{\beta }\frac{t_{\text{score,slope}}}{\sqrt{s_{\text{x}}^2+{\overline{x}}^2}},}$

where s_x² is the sample variance.

Independent two-sample t-test

Equal sample sizes and variance

Given two groups (1, 2), this test is only applicable when:

• the two sample sizes are equal,
• it can be assumed that the two distributions have the same variance.

Violations of these assumptions are discussed below.

The t statistic to test whether the means are different can be calculated as follows:

${\displaystyle t=\frac{{\overline{X}}_1-{\overline{X}}_2}{s_p\sqrt{\frac{2}{n}}},}$

where

${\displaystyle s_p=\sqrt{\frac{s_{X_1}^2+s_{X_2}^2}{2}}.}$

Here s_p is the pooled standard deviation for n = n₁ = n₂, and s ²
_X1 and s ²
_X2 are the unbiased estimators of the population variance. The denominator of t is the standard error of the difference between two means.

For significance testing, the degrees of freedom for this test is 2n − 2, where n is sample size.

Equal or unequal sample sizes, similar variances (⁠1/2⁠ < ⁠s_X1/s_X2⁠ < 2)

This test is used only when it can be assumed that the two distributions have the same variance (when this assumption is violated, see below). The previous formulae are a special case of the formulae below, one recovers them when both samples are equal in size: n = n₁ = n₂.

The t statistic to test whether the means are different can be calculated as follows:

${\displaystyle t=\frac{{\overline{X}}_1-{\overline{X}}_2}{s_p\cdot \sqrt{\frac{1}{n_1}+\frac{1}{n_2}}},}$

where

${\displaystyle s_p=\sqrt{\frac{(n_1-1)s_{X_1}^2+(n_2-1)s_{X_2}^2}{n_1+n_2-2}}}$

is the pooled standard deviation of the two samples: it is defined in this way so that its square is an unbiased estimator of the common variance, whether or not the population means are the same. In these formulae, n_i − 1 is the number of degrees of freedom for each group, and the total sample size minus two (that is, n₁ + n₂ − 2) is the total number of degrees of freedom, which is used in significance testing.

The minimum detectable effect (MDE) is:

${\displaystyle \delta \ge \sqrt{\frac{2S_p^2}{n}}(t_{1-\alpha ,\nu }+t_{1-\beta ,\nu })}$

Equal or unequal sample sizes, unequal variances (s_X1 > 2s_X2 or s_X2 > 2s_X1)

Main article: Welch's t-test

This test, also known as Welch's t-test, is used only when the two population variances are not assumed to be equal (the two sample sizes may or may not be equal) and hence must be estimated separately. The t statistic to test whether the population means are different is calculated as

${\displaystyle t=\frac{{\overline{X}}_1-{\overline{X}}_2}{s_{\overline{\mathrm{\Delta }}}},}$

where

${\displaystyle s_{\overline{\mathrm{\Delta }}}=\sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}.}$

Here s_i² is the unbiased estimator of the variance of each of the two samples with n_i = number of participants in group i (i = 1 or 2). In this case ${\displaystyle (s_{\overline{\mathrm{\Delta }}}{)}^2}$ is not a pooled variance. For use in significance testing, the distribution of the test statistic is approximated as an ordinary Student's t-distribution with the degrees of freedom calculated using

${\displaystyle \text{d.f.}=\frac{{\left(\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}\right)}^2}{\frac{(s_1^2/n_1{)}^2}{n_1-1}+\frac{(s_2^2/n_2{)}^2}{n_2-1}}.}$

This is known as the Welch–Satterthwaite equation. The true distribution of the test statistic actually depends (slightly) on the two unknown population variances (see Behrens–Fisher problem).

Exact method for unequal variances and sample sizes

The test deals with the famous Behrens–Fisher problem, i.e., comparing the difference between the means of two normally distributed populations when the variances of the two populations are not assumed to be equal, based on two independent samples.

The test is developed as an exact test that allows for unequal sample sizes and unequal variances of two populations. The exact property still holds even with extremely small and unbalanced sample sizes (e.g. ${\displaystyle m\equiv n_{\mathsf{X}}=50}$ vs. ${\displaystyle n\equiv n_{\mathsf{Y}}=5}$).

The statistic to test whether the means are different can be calculated as follows:

Let ${\displaystyle X={\left[X_1,X_2,\dots ,X_m\right]}^{\mathrm{\top }}}$ and ${\displaystyle Y={\left[Y_1,Y_2,\dots ,Y_n\right]}^{\mathrm{\top }}}$ be the i.i.d. sample vectors (for ${\displaystyle m\ge n}$) from ${\displaystyle \mathsf{N}\mathsf{o}\mathsf{r}\mathsf{m}\left({\mu }_{\mathsf{X}},{\sigma }_{\mathsf{X}}^2\right)}$ and ${\displaystyle \mathsf{N}\mathsf{o}\mathsf{r}\mathsf{m}\left({\mu }_{\mathsf{Y}},{\sigma }_{\mathsf{Y}}^2\right)}$ separately.

Let ${\displaystyle (P^{\mathrm{\top }}{)}_{n\times n}}$ be an ${\displaystyle n\times n}$ orthogonal matrix whose elements of the first row are all ${\displaystyle \frac{1}{\sqrt{n}},}$ similarly, let ${\displaystyle (Q^{\mathrm{\top }}{)}_{n\times m}}$ be the first ${\displaystyle n}$ rows of an ${\displaystyle m\times m}$ orthogonal matrix (whose elements of the first row are all ${\displaystyle \frac{1}{\sqrt{m}}}$).

Then ${\displaystyle Z\equiv \frac{{\left(Q^{\mathrm{\top }}\right)}_{n\times m}X}{\sqrt{m}}-\frac{{\left(P^{\mathrm{\top }}\right)}_{n\times n}Y}{\sqrt{n}}}$ is an n-dimensional normal random vector:

${\displaystyle Z\sim \mathsf{N}\mathsf{o}\mathsf{r}\mathsf{m}\left({\left[{\mu }_{\mathsf{X}}-{\mu }_{\mathsf{Y}},0,0,\dots ,0\right]}^{\mathrm{\top }},\left(\frac{{\sigma }_{\mathsf{X}}^2}{m}+\frac{{\sigma }_{\mathsf{Y}}^2}{n}\right)I_n\right).}$

From the above distribution we see that the first element of the vector Z is

${\displaystyle Z_1=\overline{X}-\overline{Y}=\frac{1}{m}\sum _{i=1}^m{X}_i-\frac{1}{n}\sum _{j=1}^n{Y}_j,}$

hence the first element is distributed as

${\displaystyle Z_1-\left({\mu }_{\mathsf{X}}-{\mu }_{\mathsf{Y}}\right)\sim \mathsf{N}\mathsf{o}\mathsf{r}\mathsf{m}\left(0,\frac{{\sigma }_{\mathsf{X}}^2}{m}+\frac{{\sigma }_{\mathsf{Y}}^2}{n}\right),}$

and the squares of the remaining elements of Z are chi-squared distributed

${\displaystyle \frac{\sum _{i=2}^n{Z}_i^2}{n-1}\sim \frac{{\chi }_{n-1}^2}{n-1}\times \left(\frac{{\sigma }_{\mathsf{X}}^2}{m}+\frac{{\sigma }_{\mathsf{Y}}^2}{n}\right)}$

and by construction of the orthogonal matricies P and Q we have

${\displaystyle Z_1-\left({\mu }_{\mathsf{X}}-{\mu }_{\mathsf{Y}}\right)\perp \sum _{i=2}^n{Z}_i^2,}$

so Z₁, the first element of Z, is statistically independent of the remaining elements by orthogonality. Finally, take for the test statistic

${\displaystyle T_{\mathsf{e}}\equiv \frac{Z_1-\left({\mu }_{\mathsf{X}}-{\mu }_{\mathsf{Y}}\right)}{\sqrt{\left(\sum _{i=2}^n{Z}_i^2\right)/\left(n-1\right)}}\sim t_{n-1}.}$

Dependent t-test for paired samples

This test is used when the samples are dependent; that is, when there is only one sample that has been tested twice (repeated measures) or when there are two samples that have been matched or "paired". This is an example of a paired difference test. The t statistic is calculated as

${\displaystyle t=\frac{{\overline{X}}_D-{\mu }_0}{s_D/\sqrt{n}},}$

where ${\displaystyle {\overline{X}}_D}$ and ${\displaystyle s_D}$ are the average and standard deviation of the differences between all pairs. The pairs are e.g. either one person's pre-test and post-test scores or between-pairs of persons matched into meaningful groups (for instance, drawn from the same family or age group: see table). The constant μ₀ is zero if we want to test whether the average of the difference is significantly different. The degree of freedom used is n − 1, where n represents the number of pairs.

Example of matched pairs Pair Name Age Test 1 John 35 250 1 Jane 36 340 2 Jimmy 22 460 2 Jessy 21 200

Example of repeated measures Number Name Test 1 Test 2 1 Mike 35% 67% 2 Melanie 50% 46% 3 Melissa 90% 86% 4 Mitchell 78% 91%

Worked examples

Let A₁ denote a set obtained by drawing a random sample of six measurements:

${\displaystyle A_1=\{30.02,29.99,30.11,29.97,30.01,29.99\}}$

and let A₂ denote a second set obtained similarly:

${\displaystyle A_2=\{29.89,29.93,29.72,29.98,30.02,29.98\}}$

These could be, for example, the weights of screws that were manufactured by two different machines.

We will carry out tests of the null hypothesis that the means of the populations from which the two samples were taken are equal.

The difference between the two sample means, each denoted by X_i, which appears in the numerator for all the two-sample testing approaches discussed above, is

${\displaystyle {\overline{X}}_1-{\overline{X}}_2=\mathrm{0.095.}}$

The sample standard deviations for the two samples are approximately 0.05 and 0.11, respectively. For such small samples, a test of equality between the two population variances would not be very powerful. Since the sample sizes are equal, the two forms of the two-sample t-test will perform similarly in this example.

Unequal variances

If the approach for unequal variances (discussed above) is followed, the results are

${\displaystyle \sqrt{\frac{s_1^2}{n_1}+\frac{s_2^2}{n_2}}\approx 0.04849}$

and the degrees of freedom

${\displaystyle \text{d.f.}\approx \mathrm{7.031.}}$

The test statistic is approximately 1.959, which gives a two-tailed test p-value of 0.09077.

Equal variances

If the approach for equal variances (discussed above) is followed, the results are

${\displaystyle s_p\approx 0.08399}$

and the degrees of freedom

${\displaystyle \text{d.f.}=10.}$

The test statistic is approximately equal to 1.959, which gives a two-tailed p-value of 0.07857.

Related statistical tests

Alternatives to the t-test for location problems

The t-test provides an exact test for the equality of the means of two i.i.d. normal populations with unknown, but equal, variances. (Welch's t-test is a nearly exact test for the case where the data are normal but the variances may differ.) For moderately large samples and a one tailed test, the t-test is relatively robust to moderate violations of the normality assumption. In large enough samples, the t-test asymptotically approaches the z-test, and becomes robust even to large deviations from normality.

If the data are substantially non-normal and the sample size is small, the t-test can give misleading results. See Location test for Gaussian scale mixture distributions for some theory related to one particular family of non-normal distributions.

When the normality assumption does not hold, a non-parametric alternative to the t-test may have better statistical power. However, when data are non-normal with differing variances between groups, a t-test may have better type-1 error control than some non-parametric alternatives. Furthermore, non-parametric methods, such as the Mann-Whitney U test discussed below, typically do not test for a difference of means, so should be used carefully if a difference of means is of primary scientific interest. For example, Mann-Whitney U test will keep the type 1 error at the desired level alpha if both groups have the same distribution. It will also have power in detecting an alternative by which group B has the same distribution as A but after some shift by a constant (in which case there would indeed be a difference in the means of the two groups). However, there could be cases where group A and B will have different distributions but with the same means (such as two distributions, one with positive skewness and the other with a negative one, but shifted so to have the same means). In such cases, MW could have more than alpha level power in rejecting the Null hypothesis but attributing the interpretation of difference in means to such a result would be incorrect.

In the presence of an outlier, the t-test is not robust. For example, for two independent samples when the data distributions are asymmetric (that is, the distributions are skewed) or the distributions have large tails, then the Wilcoxon rank-sum test (also known as the Mann–Whitney U test) can have three to four times higher power than the t-test. The nonparametric counterpart to the paired samples t-test is the Wilcoxon signed-rank test for paired samples. For a discussion on choosing between the t-test and nonparametric alternatives, see Lumley, et al. (2002).

One-way analysis of variance (ANOVA) generalizes the two-sample t-test when the data belong to more than two groups.

A design which includes both paired observations and independent observations

When both paired observations and independent observations are present in the two sample design, assuming data are missing completely at random (MCAR), the paired observations or independent observations may be discarded in order to proceed with the standard tests above. Alternatively making use of all of the available data, assuming normality and MCAR, the generalized partially overlapping samples t-test could be used.

Multivariate testing

Main article: Hotelling's T-squared distribution

A generalization of Student's t statistic, called Hotelling's t-squared statistic, allows for the testing of hypotheses on multiple (often correlated) measures within the same sample. For instance, a researcher might submit a number of subjects to a personality test consisting of multiple personality scales (e.g. the Minnesota Multiphasic Personality Inventory). Because measures of this type are usually positively correlated, it is not advisable to conduct separate univariate t-tests to test hypotheses, as these would neglect the covariance among measures and inflate the chance of falsely rejecting at least one hypothesis (Type I error). In this case a single multivariate test is preferable for hypothesis testing. Fisher's Method for combining multiple tests with alpha reduced for positive correlation among tests is one. Another is Hotelling's T² statistic follows a T² distribution. However, in practice the distribution is rarely used, since tabulated values for T² are hard to find. Usually, T² is converted instead to an F statistic.

For a one-sample multivariate test, the hypothesis is that the mean vector (μ) is equal to a given vector (μ₀). The test statistic is Hotelling's t²:

${\displaystyle t^2=n(\overline{\mathbf{x}}-{\mathit{\mu }}_0{)}^{\prime }{\mathbf{S}}^{-1}(\overline{\mathbf{x}}-{\mathit{\mu }}_0)}$

where n is the sample size, x is the vector of column means and S is an m × m sample covariance matrix.

For a two-sample multivariate test, the hypothesis is that the mean vectors (μ₁, μ₂) of two samples are equal. The test statistic is Hotelling's two-sample t²:

${\displaystyle t^2=\frac{n_1{n}_2}{n_1+n_2}{\left({\overline{\mathbf{x}}}_1-{\overline{\mathbf{x}}}_2\right)}^{\prime }{{\mathbf{S}}_{\text{pooled}}}^{-1}\left({\overline{\mathbf{x}}}_1-{\overline{\mathbf{x}}}_2\right).}$

The two-sample t-test is a special case of simple linear regression

The two-sample t-test is a special case of simple linear regression as illustrated by the following example.

A clinical trial examines 6 patients given drug or placebo. Three (3) patients get 0 units of drug (the placebo group). Three (3) patients get 1 unit of drug (the active treatment group). At the end of treatment, the researchers measure the change from baseline in the number of words that each patient can recall in a memory test.

A table of the patients' word recall and drug dose values are shown below.

Patient	drug.dose	word.recall
1	0	1
2	0	2
3	0	3
4	1	5
5	1	6
6	1	7

Data and code are given for the analysis using the R programming language with the t.test and lmfunctions for the t-test and linear regression. Here are the same (fictitious) data above generated in R.

> word.recall.data=data.frame(drug.dose=c(0,0,0,1,1,1), word.recall=c(1,2,3,5,6,7))

Perform the t-test. Notice that the assumption of equal variance, var.equal=T, is required to make the analysis exactly equivalent to simple linear regression.

> with(word.recall.data, t.test(word.recall~drug.dose, var.equal=T))

Running the R code gives the following results.

• The mean word.recall in the 0 drug.dose group is 2.
• The mean word.recall in the 1 drug.dose group is 6.
• The difference between treatment groups in the mean word.recall is 6 – 2 = 4.
• The difference in word.recall between drug doses is significant (p=0.00805).

Perform a linear regression of the same data. Calculations may be performed using the R function lm() for a linear model.

> word.recall.data.lm =  lm(word.recall~drug.dose, data=word.recall.data)
> summary(word.recall.data.lm)

The linear regression provides a table of coefficients and p-values.

Coefficient	Estimate	Std. Error	t value	P-value
Intercept	2	0.5774	3.464	0.02572
drug.dose	4	0.8165	4.899	0.000805

The table of coefficients gives the following results.

• The estimate value of 2 for the intercept is the mean value of the word recall when the drug dose is 0.
• The estimate value of 4 for the drug dose indicates that for a 1-unit change in drug dose (from 0 to 1) there is a 4-unit change in mean word recall (from 2 to 6). This is the slope of the line joining the two group means.
• The p-value that the slope of 4 is different from 0 is p = 0.00805.

The coefficients for the linear regression specify the slope and intercept of the line that joins the two group means, as illustrated in the graph. The intercept is 2 and the slope is 4.

Compare the result from the linear regression to the result from the t-test.

• From the t-test, the difference between the group means is 6-2=4.
• From the regression, the slope is also 4 indicating that a 1-unit change in drug dose (from 0 to 1) gives a 4-unit change in mean word recall (from 2 to 6).
• The t-test p-value for the difference in means, and the regression p-value for the slope, are both 0.00805. The methods give identical results.

This example shows that, for the special case of a simple linear regression where there is a single x-variable that has values 0 and 1, the t-test gives the same results as the linear regression. The relationship can also be shown algebraically.

Recognizing this relationship between the t-test and linear regression facilitates the use of multiple linear regression and multi-way analysis of variance. These alternatives to t-tests allow for the inclusion of additional explanatory variables that are associated with the response. Including such additional explanatory variables using regression or anova reduces the otherwise unexplained variance, and commonly yields greater power to detect differences than do two-sample t-tests.

Statistical hypothesis test

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Statistical_hypothesis_test

The above image shows a table with some of the most common test statistics and their corresponding tests or models.

A statistical hypothesis test is a method of statistical inference used to decide whether the data provide sufficient evidence to reject a particular hypothesis. A statistical hypothesis test typically involves a calculation of a test statistic. Then a decision is made, either by comparing the test statistic to a critical value or equivalently by evaluating a p-value computed from the test statistic. Roughly 100 specialized statistical tests are in use and noteworthy.

History

See also: History of probability

While hypothesis testing was popularized early in the 20th century, early forms were used in the 1700s. The first use is credited to John Arbuthnot (1710), followed by Pierre-Simon Laplace (1770s), in analyzing the human sex ratio at birth; see § Human sex ratio.

Choice of null hypothesis

Paul Meehl has argued that the epistemological importance of the choice of null hypothesis has gone largely unacknowledged. When the null hypothesis is predicted by theory, a more precise experiment will be a more severe test of the underlying theory. When the null hypothesis defaults to "no difference" or "no effect", a more precise experiment is a less severe test of the theory that motivated performing the experiment. An examination of the origins of the latter practice may therefore be useful:

1778: Pierre Laplace compares the birthrates of boys and girls in multiple European cities. He states: "it is natural to conclude that these possibilities are very nearly in the same ratio". Thus, the null hypothesis in this case that the birthrates of boys and girls should be equal given "conventional wisdom".

1900: Karl Pearson develops the chi squared test to determine "whether a given form of frequency curve will effectively describe the samples drawn from a given population." Thus the null hypothesis is that a population is described by some distribution predicted by theory. He uses as an example the numbers of five and sixes in the Weldon dice throw data.

1904: Karl Pearson develops the concept of "contingency" in order to determine whether outcomes are independent of a given categorical factor. Here the null hypothesis is by default that two things are unrelated (e.g. scar formation and death rates from smallpox). The null hypothesis in this case is no longer predicted by theory or conventional wisdom, but is instead the principle of indifference that led Fisher and others to dismiss the use of "inverse probabilities".

Modern origins and early controversy

Modern significance testing is largely the product of Karl Pearson (p-value, Pearson's chi-squared test), William Sealy Gosset (Student's t-distribution), and Ronald Fisher ("null hypothesis", analysis of variance, "significance test"), while hypothesis testing was developed by Jerzy Neyman and Egon Pearson (son of Karl). Ronald Fisher began his life in statistics as a Bayesian (Zabell 1992), but Fisher soon grew disenchanted with the subjectivity involved (namely use of the principle of indifference when determining prior probabilities), and sought to provide a more "objective" approach to inductive inference.

Fisher emphasized rigorous experimental design and methods to extract a result from few samples assuming Gaussian distributions. Neyman (who teamed with the younger Pearson) emphasized mathematical rigor and methods to obtain more results from many samples and a wider range of distributions. Modern hypothesis testing is an inconsistent hybrid of the Fisher vs Neyman/Pearson formulation, methods and terminology developed in the early 20th century.

Fisher popularized the "significance test". He required a null-hypothesis (corresponding to a population frequency distribution) and a sample. His (now familiar) calculations determined whether to reject the null-hypothesis or not. Significance testing did not utilize an alternative hypothesis so there was no concept of a Type II error (false negative).

The p-value was devised as an informal, but objective, index meant to help a researcher determine (based on other knowledge) whether to modify future experiments or strengthen one's faith in the null hypothesis. Hypothesis testing (and Type I/II errors) was devised by Neyman and Pearson as a more objective alternative to Fisher's p-value, also meant to determine researcher behaviour, but without requiring any inductive inference by the researcher.

Neyman & Pearson considered a different problem to Fisher (which they called "hypothesis testing"). They initially considered two simple hypotheses (both with frequency distributions). They calculated two probabilities and typically selected the hypothesis associated with the higher probability (the hypothesis more likely to have generated the sample). Their method always selected a hypothesis. It also allowed the calculation of both types of error probabilities.

Fisher and Neyman/Pearson clashed bitterly. Neyman/Pearson considered their formulation to be an improved generalization of significance testing (the defining paper was abstract; Mathematicians have generalized and refined the theory for decades). Fisher thought that it was not applicable to scientific research because often, during the course of the experiment, it is discovered that the initial assumptions about the null hypothesis are questionable due to unexpected sources of error. He believed that the use of rigid reject/accept decisions based on models formulated before data is collected was incompatible with this common scenario faced by scientists and attempts to apply this method to scientific research would lead to mass confusion.

The dispute between Fisher and Neyman–Pearson was waged on philosophical grounds, characterized by a philosopher as a dispute over the proper role of models in statistical inference.

Events intervened: Neyman accepted a position in the University of California, Berkeley in 1938, breaking his partnership with Pearson and separating the disputants (who had occupied the same building). World War II provided an intermission in the debate. The dispute between Fisher and Neyman terminated (unresolved after 27 years) with Fisher's death in 1962. Neyman wrote a well-regarded eulogy. Some of Neyman's later publications reported p-values and significance levels.

Null hypothesis significance testing (NHST)

The modern version of hypothesis testing is generally called the null hypothesis significance testing (NHST) and is a hybrid of the Fisher approach with the Neyman-Pearson approach. In 2000, Raymond S. Nickerson wrote an article stating that NHST was (at the time) "arguably the most widely used method of analysis of data collected in psychological experiments and has been so for about 70 years" and that it was at the same time "very controversial".

This fusion resulted from confusion by writers of statistical textbooks (as predicted by Fisher) beginning in the 1940s (but signal detection, for example, still uses the Neyman/Pearson formulation). Great conceptual differences and many caveats in addition to those mentioned above were ignored. Neyman and Pearson provided the stronger terminology, the more rigorous mathematics and the more consistent philosophy, but the subject taught today in introductory statistics has more similarities with Fisher's method than theirs.

Sometime around 1940, authors of statistical text books began combining the two approaches by using the p-value in place of the test statistic (or data) to test against the Neyman–Pearson "significance level".

A comparison between Fisherian, frequentist (Neyman–Pearson)

#	Fisher's null hypothesis testing	Neyman–Pearson decision theory
1	Set up a statistical null hypothesis. The null need not be a nil hypothesis (i.e., zero difference).	Set up two statistical hypotheses, H1 and H2, and decide about α, β, and sample size before the experiment, based on subjective cost-benefit considerations. These define a rejection region for each hypothesis.
2	Report the exact level of significance (e.g. p = 0.051 or p = 0.049). Do not refer to "accepting" or "rejecting" hypotheses. If the result is "not significant", draw no conclusions and make no decisions, but suspend judgement until further data is available.	If the data falls into the rejection region of H1, accept H2; otherwise accept H1. Accepting a hypothesis does not mean that you believe in it, but only that you act as if it were true.
3	Use this procedure only if little is known about the problem at hand, and only to draw provisional conclusions in the context of an attempt to understand the experimental situation.	The usefulness of the procedure is limited among others to situations where you have a disjunction of hypotheses (e.g. either μ1 = 8 or μ2 = 10 is true) and where you can make meaningful cost-benefit trade-offs for choosing alpha and beta.

Philosophy

Hypothesis testing and philosophy intersect. Inferential statistics, which includes hypothesis testing, is applied probability. Both probability and its application are intertwined with philosophy. Philosopher David Hume wrote, "All knowledge degenerates into probability." Competing practical definitions of probability reflect philosophical differences. The most common application of hypothesis testing is in the scientific interpretation of experimental data, which is naturally studied by the philosophy of science.

Fisher and Neyman opposed the subjectivity of probability. Their views contributed to the objective definitions. The core of their historical disagreement was philosophical.

Many of the philosophical criticisms of hypothesis testing are discussed by statisticians in other contexts, particularly correlation does not imply causation and the design of experiments. Hypothesis testing is of continuing interest to philosophers.

Education

Main article: Statistics education

Statistics is increasingly being taught in schools with hypothesis testing being one of the elements taught Many conclusions reported in the popular press (political opinion polls to medical studies) are based on statistics. Some writers have stated that statistical analysis of this kind allows for thinking clearly about problems involving mass data, as well as the effective reporting of trends and inferences from said data, but caution that writers for a broad public should have a solid understanding of the field in order to use the terms and concepts correctly. An introductory college statistics class places much emphasis on hypothesis testing – perhaps half of the course. Such fields as literature and divinity now include findings based on statistical analysis (see the Bible Analyzer). An introductory statistics class teaches hypothesis testing as a cookbook process. Hypothesis testing is also taught at the postgraduate level. Statisticians learn how to create good statistical test procedures (like z, Student's t, F and chi-squared). Statistical hypothesis testing is considered a mature area within statistics, but a limited amount of development continues.

An academic study states that the cookbook method of teaching introductory statistics leaves no time for history, philosophy or controversy. Hypothesis testing has been taught as received unified method. Surveys showed that graduates of the class were filled with philosophical misconceptions (on all aspects of statistical inference) that persisted among instructors. While the problem was addressed more than a decade ago, and calls for educational reform continue, students still graduate from statistics classes holding fundamental misconceptions about hypothesis testing. Ideas for improving the teaching of hypothesis testing include encouraging students to search for statistical errors in published papers, teaching the history of statistics and emphasizing the controversy in a generally dry subject.

Raymond S. Nickerson commented:

The debate about NHST has its roots in unresolved disagreements among major contributors to the development of theories of inferential statistics on which modern approaches are based. Gigerenzer et al. (1989) have reviewed in considerable detail the controversy between R. A. Fisher on the one hand and Jerzy Neyman and Egon Pearson on the other as well as the disagreements between both of these views and those of the followers of Thomas Bayes. They noted the remarkable fact that little hint of the historical and ongoing controversy is to be found in most textbooks that are used to teach NHST to its potential users. The resulting lack of an accurate historical perspective and understanding of the complexity and sometimes controversial philosophical foundations of various approaches to statistical inference may go a long way toward explaining the apparent ease with which statistical tests are misused and misinterpreted.

Performing a frequentist hypothesis test in practice

The typical steps involved in performing a frequentist hypothesis test in practice are:

1. Define a hypothesis (claim which is testable using data).
2. Select a relevant statistical test with associated test statistic T.
3. Derive the distribution of the test statistic under the null hypothesis from the assumptions. In standard cases this will be a well-known result. For example, the test statistic might follow a Student's t distribution with known degrees of freedom, or a normal distribution with known mean and variance.
4. Select a significance level (α), the maximum acceptable false positive rate. Common values are 5% and 1%.
5. Compute from the observations the observed value t_obs of the test statistic T.
6. Decide to either reject the null hypothesis in favor of the alternative or not reject it. The Neyman-Pearson decision rule is to reject the null hypothesis H₀ if the observed value t_obs is in the critical region, and not to reject the null hypothesis otherwise.

Practical example

The difference in the two processes applied to the radioactive suitcase example (below):

• "The Geiger-counter reading is 10. The limit is 9. Check the suitcase."
• "The Geiger-counter reading is high; 97% of safe suitcases have lower readings. The limit is 95%. Check the suitcase."

The former report is adequate, the latter gives a more detailed explanation of the data and the reason why the suitcase is being checked.

Not rejecting the null hypothesis does not mean the null hypothesis is "accepted" per se (though Neyman and Pearson used that word in their original writings; see the Interpretation section).

The processes described here are perfectly adequate for computation. They seriously neglect the design of experiments considerations.

It is particularly critical that appropriate sample sizes be estimated before conducting the experiment.

The phrase "test of significance" was coined by statistician Ronald Fisher.

Interpretation

When the null hypothesis is true and statistical assumptions are met, the probability that the p-value will be less than or equal to the significance level ${\displaystyle \alpha }$ is at most ${\displaystyle \alpha }$. This ensures that the hypothesis test maintains its specified false positive rate (provided that statistical assumptions are met).

The p-value is the probability that a test statistic which is at least as extreme as the one obtained would occur under the null hypothesis. At a significance level of 0.05, a fair coin would be expected to (incorrectly) reject the null hypothesis (that it is fair) in 1 out of 20 tests on average. The p-value does not provide the probability that either the null hypothesis or its opposite is correct (a common source of confusion).

If the p-value is less than the chosen significance threshold (equivalently, if the observed test statistic is in the critical region), then we say the null hypothesis is rejected at the chosen level of significance. If the p-value is not less than the chosen significance threshold (equivalently, if the observed test statistic is outside the critical region), then the null hypothesis is not rejected at the chosen level of significance.

In the "lady tasting tea" example (below), Fisher required the lady to properly categorize all of the cups of tea to justify the conclusion that the result was unlikely to result from chance. His test revealed that if the lady was effectively guessing at random (the null hypothesis), there was a 1.4% chance that the observed results (perfectly ordered tea) would occur.

Use and importance

Statistics are helpful in analyzing most collections of data. This is equally true of hypothesis testing which can justify conclusions even when no scientific theory exists. In the Lady tasting tea example, it was "obvious" that no difference existed between (milk poured into tea) and (tea poured into milk). The data contradicted the "obvious".

Real world applications of hypothesis testing include:

• Testing whether more men than women suffer from nightmares
• Establishing authorship of documents
• Evaluating the effect of the full moon on behavior
• Determining the range at which a bat can detect an insect by echo
• Deciding whether hospital carpeting results in more infections
• Selecting the best means to stop smoking
• Checking whether bumper stickers reflect car owner behavior
• Testing the claims of handwriting analysts

Statistical hypothesis testing plays an important role in the whole of statistics and in statistical inference. For example, Lehmann (1992) in a review of the fundamental paper by Neyman and Pearson (1933) says: "Nevertheless, despite their shortcomings, the new paradigm formulated in the 1933 paper, and the many developments carried out within its framework continue to play a central role in both the theory and practice of statistics and can be expected to do so in the foreseeable future".

Significance testing has been the favored statistical tool in some experimental social sciences (over 90% of articles in the Journal of Applied Psychology during the early 1990s). Other fields have favored the estimation of parameters (e.g. effect size). Significance testing is used as a substitute for the traditional comparison of predicted value and experimental result at the core of the scientific method. When theory is only capable of predicting the sign of a relationship, a directional (one-sided) hypothesis test can be configured so that only a statistically significant result supports theory. This form of theory appraisal is the most heavily criticized application of hypothesis testing.

Cautions

"If the government required statistical procedures to carry warning labels like those on drugs, most inference methods would have long labels indeed." This caution applies to hypothesis tests and alternatives to them.

The successful hypothesis test is associated with a probability and a type-I error rate. The conclusion might be wrong.

The conclusion of the test is only as solid as the sample upon which it is based. The design of the experiment is critical. A number of unexpected effects have been observed including:

• The clever Hans effect. A horse appeared to be capable of doing simple arithmetic.
• The Hawthorne effect. Industrial workers were more productive in better illumination, and most productive in worse.
• The placebo effect. Pills with no medically active ingredients were remarkably effective.

A statistical analysis of misleading data produces misleading conclusions. The issue of data quality can be more subtle. In forecasting for example, there is no agreement on a measure of forecast accuracy. In the absence of a consensus measurement, no decision based on measurements will be without controversy.

Publication bias: Statistically nonsignificant results may be less likely to be published, which can bias the literature.

Multiple testing: When multiple true null hypothesis tests are conducted at once without adjustment, the overall probability of Type I error is higher than the nominal alpha level.

Those making critical decisions based on the results of a hypothesis test are prudent to look at the details rather than the conclusion alone. In the physical sciences most results are fully accepted only when independently confirmed. The general advice concerning statistics is, "Figures never lie, but liars figure" (anonymous).

Definition of terms

See also: Notation in probability and statistics

The following definitions are mainly based on the exposition in the book by Lehmann and Romano:

• Statistical hypothesis: A statement about the parameters describing a population (not a sample).
• Test statistic: A value calculated from a sample without any unknown parameters, often to summarize the sample for comparison purposes.
• Simple hypothesis: Any hypothesis which specifies the population distribution completely.
• Composite hypothesis: Any hypothesis which does not specify the population distribution completely.
• Null hypothesis (H₀)
• Positive data: Data that enable the investigator to reject a null hypothesis.
• Alternative hypothesis (H₁)

Suppose the data can be realized from an N(0,1) distribution. For example, with a chosen significance level α = 0.05, from the Z-table, a one-tailed critical value of approximately 1.645 can be obtained. The one-tailed critical value C_α ≈ 1.645 corresponds to the chosen significance level. The critical region [C_α, ∞) is realized as the tail of the standard normal distribution.

• Critical values of a statistical test are the boundaries of the acceptance region of the test. The acceptance region is the set of values of the test statistic for which the null hypothesis is not rejected. Depending on the shape of the acceptance region, there can be one or more than one critical value.
  • Region of rejection / Critical region: The set of values of the test statistic for which the null hypothesis is rejected.
• Power of a test (1 − β)
• Size: For simple hypotheses, this is the test's probability of incorrectly rejecting the null hypothesis. The false positive rate. For composite hypotheses this is the supremum of the probability of rejecting the null hypothesis over all cases covered by the null hypothesis. The complement of the false positive rate is termed specificity in biostatistics. ("This is a specific test. Because the result is positive, we can confidently say that the patient has the condition.") See sensitivity and specificity and type I and type II errors for exhaustive definitions.
• Significance level of a test (α)
• p-value
• Statistical significance test: A predecessor to the statistical hypothesis test (see the Origins section). An experimental result was said to be statistically significant if a sample was sufficiently inconsistent with the (null) hypothesis. This was variously considered common sense, a pragmatic heuristic for identifying meaningful experimental results, a convention establishing a threshold of statistical evidence or a method for drawing conclusions from data. The statistical hypothesis test added mathematical rigor and philosophical consistency to the concept by making the alternative hypothesis explicit. The term is loosely used for the modern version which is now part of statistical hypothesis testing.
• Conservative test: A test is conservative if, when constructed for a given nominal significance level, the true probability of incorrectly rejecting the null hypothesis is never greater than the nominal level.
• Exact test

A statistical hypothesis test compares a test statistic (z or t for examples) to a threshold. The test statistic (the formula found in the table below) is based on optimality. For a fixed level of Type I error rate, use of these statistics minimizes Type II error rates (equivalent to maximizing power). The following terms describe tests in terms of such optimality:

• Most powerful test: For a given size or significance level, the test with the greatest power (probability of rejection) for a given value of the parameter(s) being tested, contained in the alternative hypothesis.
• Uniformly most powerful test (UMP)

Nonparametric bootstrap hypothesis testing

Main article: Bootstrapping (statistics)

Bootstrap-based resampling methods can be used for null hypothesis testing. A bootstrap creates numerous simulated samples by randomly resampling (with replacement) the original, combined sample data, assuming the null hypothesis is correct. The bootstrap is very versatile as it is distribution-free and it does not rely on restrictive parametric assumptions, but rather on empirical approximate methods with asymptotic guarantees. Traditional parametric hypothesis tests are more computationally efficient but make stronger structural assumptions. In situations where computing the probability of the test statistic under the null hypothesis is hard or impossible (due to perhaps inconvenience or lack of knowledge of the underlying distribution), the bootstrap offers a viable method for statistical inference.

Examples

Human sex ratio

Main article: Human sex ratio

The earliest use of statistical hypothesis testing is generally credited to the question of whether male and female births are equally likely (null hypothesis), which was addressed in the 1700s by John Arbuthnot (1710), and later by Pierre-Simon Laplace (1770s).

Arbuthnot examined birth records in London for each of the 82 years from 1629 to 1710, and applied the sign test, a simple non-parametric test. In every year, the number of males born in London exceeded the number of females. Considering more male or more female births as equally likely, the probability of the observed outcome is 0.5⁸², or about 1 in 4,836,000,000,000,000,000,000,000; in modern terms, this is the p-value. Arbuthnot concluded that this is too small to be due to chance and must instead be due to divine providence: "From whence it follows, that it is Art, not Chance, that governs." In modern terms, he rejected the null hypothesis of equally likely male and female births at the p = 1/2⁸² significance level.

Laplace considered the statistics of almost half a million births. The statistics showed an excess of boys compared to girls. He concluded by calculation of a p-value that the excess was a real, but unexplained, effect.

Lady tasting tea

Main article: Lady tasting tea

In a famous example of hypothesis testing, known as the Lady tasting tea, Dr. Muriel Bristol, a colleague of Fisher, claimed to be able to tell whether the tea or the milk was added first to a cup. Fisher proposed to give her eight cups, four of each variety, in random order. One could then ask what the probability was for her getting the number she got correct, but just by chance. The null hypothesis was that the Lady had no such ability. The test statistic was a simple count of the number of successes in selecting the 4 cups. The critical region was the single case of 4 successes of 4 possible based on a conventional probability criterion (< 5%). A pattern of 4 successes corresponds to 1 out of 70 possible combinations (p≈ 1.4%). Fisher asserted that no alternative hypothesis was (ever) required. The lady correctly identified every cup, which would be considered a statistically significant result.

Courtroom trial

A statistical test procedure is comparable to a criminal trial; a defendant is considered not guilty as long as his or her guilt is not proven. The prosecutor tries to prove the guilt of the defendant. Only when there is enough evidence for the prosecution is the defendant convicted.

In the start of the procedure, there are two hypotheses ${\displaystyle H_0}$: "the defendant is not guilty", and ${\displaystyle H_1}$: "the defendant is guilty". The first one, ${\displaystyle H_0}$, is called the null hypothesis. The second one, ${\displaystyle H_1}$, is called the alternative hypothesis. It is the alternative hypothesis that one hopes to support.

The hypothesis of innocence is rejected only when an error is very unlikely, because one does not want to convict an innocent defendant. Such an error is called error of the first kind (i.e., the conviction of an innocent person), and the occurrence of this error is controlled to be rare. As a consequence of this asymmetric behaviour, an error of the second kind (acquitting a person who committed the crime), is more common.

	H0 is true Truly not guilty	H1 is true Truly guilty
Do not reject the null hypothesis Acquittal	Right decision	Wrong decision Type II Error
Reject null hypothesis Conviction	Wrong decision Type I Error	Right decision

A criminal trial can be regarded as either or both of two decision processes: guilty vs not guilty or evidence vs a threshold ("beyond a reasonable doubt"). In one view, the defendant is judged; in the other view the performance of the prosecution (which bears the burden of proof) is judged. A hypothesis test can be regarded as either a judgment of a hypothesis or as a judgment of evidence.

Clairvoyant card game

A person (the subject) is tested for clairvoyance. They are shown the back face of a randomly chosen playing card 25 times and asked which of the four suits it belongs to. The number of hits, or correct answers, is called X.

As we try to find evidence of their clairvoyance, for the time being the null hypothesis is that the person is not clairvoyant. The alternative is: the person is (more or less) clairvoyant.

If the null hypothesis is valid, the only thing the test person can do is guess. For every card, the probability (relative frequency) of any single suit appearing is 1/4. If the alternative is valid, the test subject will predict the suit correctly with probability greater than 1/4. We will call the probability of guessing correctly p. The hypotheses, then, are:

• null hypothesis ${\displaystyle \text{:}{H}_0:p=\frac{1}{4}}$     (just guessing)

and

• alternative hypothesis ${\displaystyle \text{:}{H}_1:p>\frac{1}{4}}$    (true clairvoyant).

When the test subject correctly predicts all 25 cards, we will consider them clairvoyant, and reject the null hypothesis. Thus also with 24 or 23 hits. With only 5 or 6 hits, on the other hand, there is no cause to consider them so. But what about 12 hits, or 17 hits? What is the critical number, c, of hits, at which point we consider the subject to be clairvoyant? How do we determine the critical value c? With the choice c=25 (i.e. we only accept clairvoyance when all cards are predicted correctly) we're more critical than with c=10. In the first case almost no test subjects will be recognized to be clairvoyant, in the second case, a certain number will pass the test. In practice, one decides how critical one will be. That is, one decides how often one accepts an error of the first kind – a false positive, or Type I error. With c = 25 the probability of such an error is:

${\displaystyle P(\text{reject }{H}_0\mid H_0\text{ is valid})=P\left(X=25\mid p=\frac{1}{4}\right)={\left(\frac{1}{4}\right)}^{25}\approx {10}^{-15}}$,

and hence, very small. The probability of a false positive is the probability of randomly guessing correctly all 25 times.

Being less critical, with c = 10, gives:

${\displaystyle P(\text{reject }{H}_0\mid H_0\text{ is valid})=P\left(X\ge 10\mid p=\frac{1}{4}\right)=\sum _{k=10}^{25}P\left(X=k\mid p=\frac{1}{4}\right)=\sum _{k=10}^{25}(\genfrac{}{}{0ex}{}{25}{k}){\left(1-\frac{1}{4}\right)}^{25-k}{\left(\frac{1}{4}\right)}^k\approx 0.0713}$.

Thus, c = 10 yields a much greater probability of false positive.

Before the test is actually performed, the maximum acceptable probability of a Type I error (α) is determined. Typically, values in the range of 1% to 5% are selected. (If the maximum acceptable error rate is zero, an infinite number of correct guesses is required.) Depending on this Type 1 error rate, the critical value c is calculated. For example, if we select an error rate of 1%, c is calculated thus:

${\displaystyle P(\text{reject }{H}_0\mid H_0\text{ is valid})=P\left(X\ge c\mid p=\frac{1}{4}\right)\le 0.01}$.

From all the numbers c, with this property, we choose the smallest, in order to minimize the probability of a Type II error, a false negative. For the above example, we select: ${\displaystyle c=13}$.

Variations and sub-classes

Statistical hypothesis testing is a key technique of both frequentist inference and Bayesian inference, although the two types of inference have notable differences. Statistical hypothesis tests define a procedure that controls (fixes) the probability of incorrectly deciding that a default position (null hypothesis) is incorrect. The procedure is based on how likely it would be for a set of observations to occur if the null hypothesis were true. This probability of making an incorrect decision is not the probability that the null hypothesis is true, nor whether any specific alternative hypothesis is true. This contrasts with other possible techniques of decision theory in which the null and alternative hypothesis are treated on a more equal basis.

One naïve Bayesian approach to hypothesis testing is to base decisions on the posterior probability, but this fails when comparing point and continuous hypotheses. Other approaches to decision making, such as Bayesian decision theory, attempt to balance the consequences of incorrect decisions across all possibilities, rather than concentrating on a single null hypothesis. A number of other approaches to reaching a decision based on data are available via decision theory and optimal decisions, some of which have desirable properties. Hypothesis testing, though, is a dominant approach to data analysis in many fields of science. Extensions to the theory of hypothesis testing include the study of the power of tests, i.e. the probability of correctly rejecting the null hypothesis given that it is false. Such considerations can be used for the purpose of sample size determination prior to the collection of data.

Neyman–Pearson hypothesis testing

An example of Neyman–Pearson hypothesis testing (or null hypothesis statistical significance testing) can be made by a change to the radioactive suitcase example. If the "suitcase" is actually a shielded container for the transportation of radioactive material, then a test might be used to select among three hypotheses: no radioactive source present, one present, two (all) present. The test could be required for safety, with actions required in each case. The Neyman–Pearson lemma of hypothesis testing says that a good criterion for the selection of hypotheses is the ratio of their probabilities (a likelihood ratio). A simple method of solution is to select the hypothesis with the highest probability for the Geiger counts observed. The typical result matches intuition: few counts imply no source, many counts imply two sources and intermediate counts imply one source. Notice also that usually there are problems for proving a negative. Null hypotheses should be at least falsifiable.

Neyman–Pearson theory can accommodate both prior probabilities and the costs of actions resulting from decisions. The former allows each test to consider the results of earlier tests (unlike Fisher's significance tests). The latter allows the consideration of economic issues (for example) as well as probabilities. A likelihood ratio remains a good criterion for selecting among hypotheses.

The two forms of hypothesis testing are based on different problem formulations. The original test is analogous to a true/false question; the Neyman–Pearson test is more like multiple choice. In the view of Tukey the former produces a conclusion on the basis of only strong evidence while the latter produces a decision on the basis of available evidence. While the two tests seem quite different both mathematically and philosophically, later developments lead to the opposite claim. Consider many tiny radioactive sources. The hypotheses become 0,1,2,3... grains of radioactive sand. There is little distinction between none or some radiation (Fisher) and 0 grains of radioactive sand versus all of the alternatives (Neyman–Pearson). The major Neyman–Pearson paper of 1933 also considered composite hypotheses (ones whose distribution includes an unknown parameter). An example proved the optimality of the (Student's) t-test, "there can be no better test for the hypothesis under consideration" (p 321). Neyman–Pearson theory was proving the optimality of Fisherian methods from its inception.

Fisher's significance testing has proven a popular flexible statistical tool in application with little mathematical growth potential. Neyman–Pearson hypothesis testing is claimed as a pillar of mathematical statistics, creating a new paradigm for the field. It also stimulated new applications in statistical process control, detection theory, decision theory and game theory. Both formulations have been successful, but the successes have been of a different character.

The dispute over formulations is unresolved. Science primarily uses Fisher's (slightly modified) formulation as taught in introductory statistics. Statisticians study Neyman–Pearson theory in graduate school. Mathematicians are proud of uniting the formulations. Philosophers consider them separately. Learned opinions deem the formulations variously competitive (Fisher vs Neyman), incompatible or complementary. The dispute has become more complex since Bayesian inference has achieved respectability.

The terminology is inconsistent. Hypothesis testing can mean any mixture of two formulations that both changed with time. Any discussion of significance testing vs hypothesis testing is doubly vulnerable to confusion.

Fisher thought that hypothesis testing was a useful strategy for performing industrial quality control, however, he strongly disagreed that hypothesis testing could be useful for scientists. Hypothesis testing provides a means of finding test statistics used in significance testing. The concept of power is useful in explaining the consequences of adjusting the significance level and is heavily used in sample size determination. The two methods remain philosophically distinct. They usually (but not always) produce the same mathematical answer. The preferred answer is context dependent. While the existing merger of Fisher and Neyman–Pearson theories has been heavily criticized, modifying the merger to achieve Bayesian goals has been considered.

Criticism

See also: p-value § Misuse

Criticism of statistical hypothesis testing fills volumes. Much of the criticism can be summarized by the following issues:

• The interpretation of a p-value is dependent upon stopping rule and definition of multiple comparison. The former often changes during the course of a study and the latter is unavoidably ambiguous. (i.e. "p values depend on both the (data) observed and on the other possible (data) that might have been observed but weren't").
• Confusion resulting (in part) from combining the methods of Fisher and Neyman–Pearson which are conceptually distinct.
• Emphasis on statistical significance to the exclusion of estimation and confirmation by repeated experiments.
• Rigidly requiring statistical significance as a criterion for publication, resulting in publication bias. Most of the criticism is indirect. Rather than being wrong, statistical hypothesis testing is misunderstood, overused and misused.
• When used to detect whether a difference exists between groups, a paradox arises. As improvements are made to experimental design (e.g. increased precision of measurement and sample size), the test becomes more lenient. Unless one accepts the absurd assumption that all sources of noise in the data cancel out completely, the chance of finding statistical significance in either direction approaches 100%. However, this absurd assumption that the mean difference between two groups cannot be zero implies that the data cannot be independent and identically distributed (i.i.d.) because the expected difference between any two subgroups of i.i.d. random variates is zero; therefore, the i.i.d. assumption is also absurd.
• Layers of philosophical concerns. The probability of statistical significance is a function of decisions made by experimenters/analysts. If the decisions are based on convention they are termed arbitrary or mindless while those not so based may be termed subjective. To minimize type II errors, large samples are recommended. In psychology practically all null hypotheses are claimed to be false for sufficiently large samples so "...it is usually nonsensical to perform an experiment with the sole aim of rejecting the null hypothesis." "Statistically significant findings are often misleading" in psychology. Statistical significance does not imply practical significance, and correlation does not imply causation. Casting doubt on the null hypothesis is thus far from directly supporting the research hypothesis.
• "[I]t does not tell us what we want to know". Lists of dozens of complaints are available.

Critics and supporters are largely in factual agreement regarding the characteristics of null hypothesis significance testing (NHST): While it can provide critical information, it is inadequate as the sole tool for statistical analysis. Successfully rejecting the null hypothesis may offer no support for the research hypothesis. The continuing controversy concerns the selection of the best statistical practices for the near-term future given the existing practices. However, adequate research design can minimize this issue. Critics would prefer to ban NHST completely, forcing a complete departure from those practices, while supporters suggest a less absolute change.

Controversy over significance testing, and its effects on publication bias in particular, has produced several results. The American Psychological Association has strengthened its statistical reporting requirements after review, medical journal publishers have recognized the obligation to publish some results that are not statistically significant to combat publication bias, and a journal (Journal of Articles in Support of the Null Hypothesis) has been created to publish such results exclusively. Textbooks have added some cautions, and increased coverage of the tools necessary to estimate the size of the sample required to produce significant results. Few major organizations have abandoned use of significance tests although some have discussed doing so. For instance, in 2023, the editors of the Journal of Physiology "strongly recommend the use of estimation methods for those publishing in The Journal" (meaning the magnitude of the effect size (to allow readers to judge whether a finding has practical, physiological, or clinical relevance) and confidence intervals to convey the precision of that estimate), saying "Ultimately, it is the physiological importance of the data that those publishing in The Journal of Physiology should be most concerned with, rather than the statistical significance."

Alternatives

Main article: Estimation statistics

See also: Confidence interval § Statistical hypothesis testing

A unifying position of critics is that statistics should not lead to an accept-reject conclusion or decision, but to an estimated value with an interval estimate; this data-analysis philosophy is broadly referred to as estimation statistics. Estimation statistics can be accomplished with either frequentist or Bayesian methods.

Critics of significance testing have advocated basing inference less on p-values and more on confidence intervals for effect sizes for importance, prediction intervals for confidence, replications and extensions for replicability, meta-analyses for generality :. But none of these suggested alternatives inherently produces a decision. Lehmann said that hypothesis testing theory can be presented in terms of conclusions/decisions, probabilities, or confidence intervals: "The distinction between the ... approaches is largely one of reporting and interpretation."

Bayesian inference is one proposed alternative to significance testing. (Nickerson cited 10 sources suggesting it, including Rozeboom (1960)). For example, Bayesian parameter estimation can provide rich information about the data from which researchers can draw inferences, while using uncertain priors that exert only minimal influence on the results when enough data is available. Psychologist John K. Kruschke has suggested Bayesian estimation as an alternative for the t-test and has also contrasted Bayesian estimation for assessing null values with Bayesian model comparison for hypothesis testing.^[86] Two competing models/hypotheses can be compared using Bayes factors. Bayesian methods could be criticized for requiring information that is seldom available in the cases where significance testing is most heavily used. Neither the prior probabilities nor the probability distribution of the test statistic under the alternative hypothesis are often available in the social sciences.

Advocates of a Bayesian approach sometimes claim that the goal of a researcher is most often to objectively assess the probability that a hypothesis is true based on the data they have collected.^[89][90] Neither Fisher's significance testing, nor Neyman–Pearson hypothesis testing can provide this information, and do not claim to. The probability a hypothesis is true can only be derived from use of Bayes' Theorem, which was unsatisfactory to both the Fisher and Neyman–Pearson camps due to the explicit use of subjectivity in the form of the prior probability. Fisher's strategy is to sidestep this with the p-value (an objective index based on the data alone) followed by inductive inference, while Neyman–Pearson devised their approach of inductive behaviour.