Strange loop

From Wikipedia, the free encyclopedia

For other uses, see Strange loop (disambiguation).

A strange loop is a cyclic structure that goes through several levels in a hierarchical system. It arises when, by moving only upwards or downwards through the system, one finds oneself back where one started. Strange loops may involve self-reference and paradox. The concept of a strange loop was proposed and extensively discussed by Douglas Hofstadter in Gödel, Escher, Bach, and is further elaborated in Hofstadter's book I Am a Strange Loop, published in 2007.

A tangled hierarchy is a hierarchical system in which a strange loop appears.

Definitions

A strange loop is a hierarchy of levels, each of which is linked to at least one other by some type of relationship. A strange loop hierarchy is "tangled" (Hofstadter refers to this as a "heterarchy"), in that there is no well defined highest or lowest level; moving through the levels, one eventually returns to the starting point, i.e., the original level. Examples of strange loops that Hofstadter offers include: many of the works of M. C. Escher, the Canon 5. a 2 from J.S. Bach's Musical Offering, the information flow network between DNA and enzymes through protein synthesis and DNA replication, and self-referential Gödelian statements in formal systems.

In I Am a Strange Loop, Hofstadter defines strange loops as follows:

And yet when I say "strange loop", I have something else in mind — a less concrete, more elusive notion. What I mean by "strange loop" is — here goes a first stab, anyway — not a physical circuit but an abstract loop in which, in the series of stages that constitute the cycling-around, there is a shift from one level of abstraction (or structure) to another, which feels like an upwards movement in an hierarchy, and yet somehow the successive "upward" shifts turn out to give rise to a closed cycle. That is, despite one's sense of departing ever further from one's origin, one winds up, to one's shock, exactly where one had started out. In short, a strange loop is a paradoxical level-crossing feedback loop. (pp. 101–102)

In cognitive science

According to Hofstadter, strange loops take form in human consciousness as the complexity of active symbols in the brain inevitably leads to the same kind of self-reference which Gödel proved was inherent in any sufficiently complex logical or arithmetical system (that allows for arithmetic by means of the Peano axioms) in his incompleteness theorem. Gödel showed that mathematics and logic contain strange loops: propositions that not only refer to mathematical and logical truths, but also to the symbol systems expressing those truths. This leads to the sort of paradoxes seen in statements such as "This statement is false," wherein the sentence's basis of truth is found in referring to itself and its assertion, causing a logical paradox.

Hofstadter argues that the psychological self arises out of a similar kind of paradox. The brain is not born with an "I" – the ego emerges only gradually as experience shapes the brain's dense web of active symbols into a tapestry rich and complex enough to begin twisting back upon itself. According to this view, the psychological "I" is a narrative fiction, something created only from intake of symbolic data and the brain's ability to create stories about itself from that data. The consequence is that a self-perspective is a culmination of a unique pattern of symbolic activity in the brain, which suggests that the pattern of symbolic activity that makes identity, that constitutes subjectivity, can be replicated within the brains of others, and likely even in artificial brains.

Strangeness

The "strangeness" of a strange loop comes from the brain's perception, because the brain categorizes its input in a small number of "symbols" (by which Hofstadter means groups of neurons standing for something in the outside world). So the difference between the video-feedback loop and the brain's strange loops, is that while the former converts light to the same pattern on a screen, the latter categorizes a pattern and outputs its "essence", so that as the brain gets closer and closer to its "essence", it goes further down its strange loop.

Downward causality

Hofstadter thinks that minds appear to determine the world by way of "downward causality", which refers to effects being viewed in terms of their underlying causes. Hofstadter says this happens in the proof of Gödel's incompleteness theorem:

Merely from knowing the formula's meaning, one can infer its truth or falsity without any effort to derive it in the old-fashioned way, which requires one to trudge methodically "upwards" from the axioms. This is not just peculiar; it is astonishing. Normally, one cannot merely look at what a mathematical conjecture says and simply appeal to the content of that statement on its own to deduce whether the statement is true or false. (pp. 169–170)

Hofstadter claims a similar "flipping around of causality" appears to happen in minds possessing self-consciousness; the mind perceives itself as the cause of certain feelings.

The parallels between downward causality in formal systems and downward causality in brains are explored by Theodor Nenu in 2022, together with other aspects of Hofstadter's metaphysics of mind. Nenu also questions the correctness of the above quote by focusing on the sentence which "says about itself" that it is provable (also known as a Henkin-sentence, named after logician Leon Henkin). It turns out that under suitable meta-mathematical choices (where the Hilbert-Bernays provability conditions do not obtain), one can construct formally undecidable (or even formally refutable) Henkin-sentences for the arithmetical system under investigation. This system might very well be Hofstadter's Typographical Number Theory used in Gödel, Escher, Bach or the more familiar Peano Arithmetic or some other sufficiently rich formal arithmetic. Thus, there are examples of sentences "which say about themselves that they are provable", but they don't exhibit the sort of downward causal powers described in the displayed quote.

Examples

Hofstadter points to Bach's Canon per Tonos, M. C. Escher's drawings Waterfall, Drawing Hands, Ascending and Descending, and the liar paradox as examples that illustrate the idea of strange loops, which is expressed fully in the proof of Gödel's incompleteness theorem.

The "chicken or the egg" paradox is perhaps the best-known strange loop problem.

The "ouroboros", which depicts a dragon eating its own tail, is perhaps one of the most ancient and universal symbolic representations of the reflexive loop concept.

A Shepard tone is another illustrative example of a strange loop. Named after Roger Shepard, it is a sound consisting of a superposition of tones separated by octaves. When played with the base pitch of the tone moving upwards or downwards, it is referred to as the Shepard scale. This creates the auditory illusion of a tone that continually ascends or descends in pitch, yet which ultimately seems to get no higher or lower. In a similar way a sound with seemingly ever increasing tempo can be constructed, as was demonstrated by Jean-Claude Risset.

Visual illusions depicting strange loops include the Penrose stairs and the Barberpole illusion.

A quine in software programming is a program that produces a new version of itself without any input from the outside. A similar concept is metamorphic code.

Efron's dice are four dice that are intransitive under gambler's preference. I.e., the dice are ordered A > B > C > D > A, where x > y means "a gambler prefers x to y".

Individual preferences are always transitive, excluding preferences when given explicit rules such as in Efron's dice or rock-paper-scissors; however, aggregate preferences of a group may be intransitive. This can result in a Condorcet paradox wherein following a path from one candidate across a series of majority preferences may return to the original candidate, leaving no clear preference by the group. In this case, some candidate beats an opponent, who in turn beats another opponent, and so forth, until a candidate is reached who beats the original candidate.

The liar paradox and Russell's paradox also involve strange loops, as does René Magritte's painting The Treachery of Images.

The mathematical phenomenon of polysemy has been observed to be a strange loop. At the denotational level, the term refers to situations where a single entity can be seen to mean more than one mathematical object. See Tanenbaum (1999).

The Stonecutter is an old Japanese fairy tale with a story that explains social and natural hierarchies as a strange loop.

A strange loop can be found by traversing the links in the “See also” sections of the respective English Wikipedia articles. For instance: This article->Mise en abyme->Recursion->this article.

Binomial proportion confidence interval

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval 

In statistics, a binomial proportion confidence interval is a confidence interval for the probability of success calculated from the outcome of a series of success–failure experiments (Bernoulli trials). In other words, a binomial proportion confidence interval is an interval estimate of a success probability ${\displaystyle p}$ when only the number of experiments ${\displaystyle n}$ and the number of successes ${\displaystyle n_{\mathsf{s}}}$ are known.

There are several formulas for a binomial confidence interval, but all of them rely on the assumption of a binomial distribution. In general, a binomial distribution applies when an experiment is repeated a fixed number of times, each trial of the experiment has two possible outcomes (success and failure), the probability of success is the same for each trial, and the trials are statistically independent. Because the binomial distribution is a discrete probability distribution (i.e., not continuous) and difficult to calculate for large numbers of trials, a variety of approximations are used to calculate this confidence interval, all with their own tradeoffs in accuracy and computational intensity.

A simple example of a binomial distribution is the set of various possible outcomes, and their probabilities, for the number of heads observed when a coin is flipped ten times. The observed binomial proportion is the fraction of the flips that turn out to be heads. Given this observed proportion, the confidence interval for the true probability of the coin landing on heads is a range of possible proportions, which may or may not contain the true proportion. A 95% confidence interval for the proportion, for instance, will contain the true proportion 95% of the times that the procedure for constructing the confidence interval is employed.

Problems with using a normal approximation or "Wald interval"

Plotting the normal approximation interval on an arbitrary logistic curve reveals problems of overshoot and zero-width intervals.

A commonly used formula for a binomial confidence interval relies on approximating the distribution of error about a binomially-distributed observation, ${\displaystyle \hat{p},}$ with a normal distribution. The normal approximation depends on the de Moivre–Laplace theorem (the original, binomial-only version of the central limit theorem) and becomes unreliable when it violates the theorems' premises, as the sample size becomes small or the success probability grows close to either 0 or 1 .

Using the normal approximation, the success probability ${\displaystyle p}$ is estimated by

$${\displaystyle p\approx \hat{p}\pm \frac{z_{\alpha }}{\sqrt{n}}\sqrt{\hat{p}\left(1-\hat{p}\right)},}$$

where ${\displaystyle \hat{p}\equiv \frac{n_{\mathsf{s}}}{n}}$ is the proportion of successes in a Bernoulli trial process and an estimator for ${\displaystyle p}$ in the underlying Bernoulli distribution. The equivalent formula in terms of observation counts is

$${\displaystyle p\approx \frac{n_{\mathsf{s}}}{n}\pm \frac{z_{\alpha }}{\sqrt{n}}\sqrt{\frac{n_{\mathsf{s}}}{n}\frac{n_{\mathsf{f}}}{n}},}$$

where the data are the results of ${\displaystyle n}$ trials that yielded ${\displaystyle n_{\mathsf{s}}}$ successes and ${\displaystyle n_{\mathsf{f}}=n-n_{\mathsf{s}}}$ failures. The distribution function argument ${\displaystyle z_{\alpha }}$ is the ${\displaystyle 1-\frac{\alpha }{2}}$ quantile of a standard normal distribution (i.e., the probit) corresponding to the target error rate ${\displaystyle \alpha .}$ For a 95% confidence level, the error ${\displaystyle \alpha =1-0.95=0.05,}$ so that ${\displaystyle 1-\frac{\alpha }{2}=0.975}$ and ${\displaystyle z_{.05}=1.96.}$

When using the Wald formula to estimate ${\displaystyle p,}$ or just considering the possible outcomes of this calculation, two problems immediately become apparent:

• First, for ${\displaystyle \hat{p}}$ approaching either 1 or 0, the interval narrows to zero width (falsely implying certainty).
• Second, for values of ${\displaystyle \hat{p}<\frac{1}{1+n/z_{\alpha }^2}}$ (probability too low / too close to 0), the interval boundaries exceed ${\displaystyle [0,1]}$ (overshoot).

(Another version of the second, overshoot problem, arises when instead ${\displaystyle 1-\hat{p}}$ falls below the same upper bound: probability too high / too close to 1 .)

An important theoretical derivation of this confidence interval involves the inversion of a hypothesis test. Under this formulation, the confidence interval represents those values of the population parameter that would have large p-values if they were tested as a hypothesized population proportion. The collection of values, ${\displaystyle \theta ,}$ for which the normal approximation is valid can be represented as

$${\displaystyle \left\{\theta |y_{\alpha }\le \frac{\hat{p}-\theta }{\sqrt{\frac{1}{n}\hat{p}\left(1-\hat{p}\right)}}\le z_{\alpha }\right\},}$$

where ${\displaystyle y_{\alpha }}$ is the lower ${\displaystyle \frac{\alpha }{2}}$ quantile of a standard normal distribution, vs. ${\displaystyle z_{\alpha },}$ which is the upper ( i.e., ${\displaystyle 1-\frac{\alpha }{2}}$) quantile.

Since the test in the middle of the inequality is a Wald test, the normal approximation interval is sometimes called the Wald interval or Wald method, after Abraham Wald, but it was first described by Laplace (1812).

Bracketing the confidence interval

Extending the normal approximation and Wald-Laplace interval concepts, Michael Short has shown that inequalities on the approximation error between the binomial distribution and the normal distribution can be used to accurately bracket the estimate of the confidence interval around ${\displaystyle p:}$

$${\displaystyle \frac{k+C_{\mathsf{L}\mathsf{1}}-z_{\alpha }\hat{W}}{n+z_{\alpha }^2}\le p\le \frac{k+C_{\mathsf{U}\mathsf{1}}+z_{\alpha }\hat{W}}{n+z_{\alpha }^2}}$$

with $${\displaystyle \hat{W}\equiv \sqrt{\frac{nk-k^2+C_{\mathsf{L}\mathsf{2}}n-C_{\mathsf{L}\mathsf{3}}k+C_{\mathsf{L}\mathsf{4}}}{n}},}$$

and where ${\displaystyle p}$ is again the (unknown) proportion of successes in a Bernoulli trial process (as opposed to ${\displaystyle \hat{p}\equiv \frac{n_{\mathsf{s}}}{n}}$ that estimates it) measured with ${\displaystyle n}$ trials yielding ${\displaystyle k}$ successes, ${\displaystyle z_{\alpha }}$ is the ${\displaystyle 1-\frac{\alpha }{2}}$ quantile of a standard normal distribution (i.e., the probit) corresponding to the target error rate ${\displaystyle \alpha ,}$ and the constants ${\displaystyle C_{\mathsf{L}\mathsf{1}},}$ ${\displaystyle C_{\mathsf{L}\mathsf{2}},}$ ${\displaystyle C_{\mathsf{L}\mathsf{3}},}$ ${\displaystyle C_{\mathsf{L}\mathsf{4}},}$ ${\displaystyle C_{\mathsf{U}\mathsf{1}},}$ ${\displaystyle C_{\mathsf{U}\mathsf{2}},}$ ${\displaystyle C_{\mathsf{U}\mathsf{3}},}$ and ${\displaystyle C_{\mathsf{U}\mathsf{4}}}$ are simple algebraic functions of ${\displaystyle z_{\alpha }.}$  For a fixed ${\displaystyle \alpha }$ (hence fixed ${\displaystyle z_{\alpha }}$), the above inequalities give easily computed one- or two-sided intervals which bracket the exact binomial upper and lower confidence limits corresponding to the error rate ${\displaystyle \alpha .}$

Standard error of a proportion estimation when using weighted data

Let there be a simple random sample ${\displaystyle X_1,\dots ,X_n}$ where each ${\displaystyle X_i}$ is i.i.d from a Bernoulli( p ) distribution and weight ${\displaystyle w_i}$ is the weight for each observation, with the(positive) weights ${\displaystyle w_i}$ normalized so they sum to 1 . The weighted sample proportion is: $\hat{p}=\sum _{i=1}^n{w}_i{X}_i.$ Since each of the ${\displaystyle X_i}$ is independent from all the others, and each one has variance ${\displaystyle \mathrm{var}\{X_i\}=p\left(1-p\right)}$ for every ${\displaystyle i=1,\dots ,n;}$ the sampling variance of the proportion therefore is:

$${\displaystyle \mathrm{var}\left\{\hat{p}\right\}=\sum _{i=1}^n\mathrm{var}\left\{w_i{X}_i\right\}=p\left(1-p\right)\sum _{i=1}^n{w}_i^2.}$$

The standard error of ${\displaystyle \hat{p}}$ is the square root of this quantity. Because we do not know ${\displaystyle p\left(1-p\right),}$ we have to estimate it. Although there are many possible estimators, a conventional one is to use ${\displaystyle \hat{p},}$ the sample mean, and plug this into the formula. That gives:

$${\displaystyle \mathrm{SE}\left\{\hat{p}\right\}\approx \sqrt{\hat{p}\left(1-\hat{p}\right)\sum _{i=1}^n{w}_i^2}.}$$

For otherwise unweighted data, the effective weights are uniform $w_i=\frac{1}{n},$ giving $\sum _{i=1}^n{w}_i^2=\frac{1}{n}.$ The ${\displaystyle \mathrm{SE}}$ becomes $\sqrt{\frac{1}{n}\hat{p}\left(1-\hat{p}\right)},$ leading to the familiar formulas, showing that the calculation for weighted data is a direct generalization of them.

Wilson score interval

Wilson score intervals plotted on a logistic curve, revealing asymmetry and good performance for small n and where p is at or near 0 or 1.

The Wilson score interval was developed by E.B. Wilson (1927). It is an improvement over the normal approximation interval in multiple respects: Unlike the symmetric normal approximation interval (above), the Wilson score interval is asymmetric, and it doesn't suffer from problems of overshoot and zero-width intervals that afflict the normal interval. It can be safely employed with small samples and skewed observations. The observed coverage probability is consistently closer to the nominal value, ${\displaystyle 1-\alpha .}$

Like the normal interval, the interval can be computed directly from a formula.

Wilson started with the normal approximation to the binomial: $${\displaystyle z_{\alpha }\approx \frac{p-\hat{p}}{{\sigma }_n}}$$ where ${\displaystyle z_{\alpha }}$ is the standard normal interval half-width corresponding to the desired confidence ${\displaystyle 1-\alpha .}$ The analytic formula for a binomial sample standard deviation is $${\displaystyle {\sigma }_n=\sqrt{\frac{p\left(1-p\right)}{n}}.}$$ Combining the two, and squaring out the radical, gives an equation that is quadratic in ${\displaystyle p:}$ $${\displaystyle {\left(p-\hat{p}\right)}^2=\frac{z_{\alpha }^2}{n}p\left(1-p\right)}$$ or $${\displaystyle p^2-2p\hat{p}+{\hat{p}}^2=p\frac{z_{\alpha }^2}{n}-p^2\frac{z_{\alpha }^2}{n}.}$$ Transforming the relation into a standard-form quadratic equation for ${\displaystyle p,}$ treating ${\displaystyle \hat{p}}$ and ${\displaystyle n}$ as known values from the sample (see prior section), and using the value of ${\displaystyle z_{\alpha }}$ that corresponds to the desired confidence ${\displaystyle 1-\alpha }$ for the estimate of ${\displaystyle p}$ gives this: $${\displaystyle \left(1+\frac{z_{\alpha }^2}{n}\right)p^2-\left(2\hat{p}+\frac{z_{\alpha }^2}{n}\right)p+({\hat{p}}^2)=0,}$$ where all of the values bracketed by parentheses are known quantities. The solution for ${\displaystyle p}$ estimates the upper and lower limits of the confidence interval for ${\displaystyle p.}$ Hence the probability of success ${\displaystyle p}$ is estimated by ${\displaystyle \hat{p}}$ and with ${\displaystyle 1-\alpha }$ confidence bracketed in the interval $${\displaystyle p{\underset{\approx }{\in }}_{\alpha }\left(w^{-},w^{+}\right)=\frac{1}{1+z_{\alpha }^2/n}\left(\hat{p}+\frac{z_{\alpha }^2}{2n}\pm \frac{z_{\alpha }}{2n}\sqrt{4n\hat{p}\left(1-\hat{p}\right)+z_{\alpha }^2}\right)}$$

where ${\displaystyle {\underset{\approx }{\in }}_{\alpha }}$ is an abbreviation for

$${\displaystyle \mathbb{P}\left\{p\in \left(w^{-},w^{+}\right)\right\}=1-\alpha .}$$

An equivalent expression using the observation counts ${\displaystyle n_{\mathsf{s}}}$ and ${\displaystyle n_{\mathsf{f}}}$ is $${\displaystyle p{\underset{\approx }{\in }}_{\alpha }\frac{n_{\mathsf{s}}+\frac{1}{2}{z}_{\alpha }^2}{n+z_{\alpha }^2}\pm \frac{z_{\alpha }}{n+z_{\alpha }^2}\sqrt{\frac{n_{\mathsf{s}}{n}_{\mathsf{f}}}{n}+\frac{z_{\alpha }^2}{4}},}$$

with the counts as above: ${\displaystyle n_{\mathsf{s}}\equiv }$ the count of observed "successes", ${\displaystyle n_{\mathsf{f}}\equiv }$ the count of observed "failures", and their sum is the total number of observations ${\displaystyle n=n_{\mathsf{s}}+n_{\mathsf{f}}.}$

In practical tests of the formula's results, users find that this interval has good properties even for a small number of trials and / or the extremes of the probability estimate, ${\displaystyle \hat{p}\equiv \frac{n_{\mathsf{s}}}{n}.}$

Intuitively, the center value of this interval is the weighted average of ${\displaystyle \hat{p}}$ and $\frac{1}{2},$ with ${\displaystyle \hat{p}}$ receiving greater weight as the sample size increases. Formally, the center value corresponds to using a pseudocount of ${\displaystyle \frac{1}{2}{z}_{\alpha }^2,}$ the number of standard deviations of the confidence interval: Add this number to both the count of successes and of failures to yield the estimate of the ratio. For the common two standard deviations in each direction interval (approximately 95% coverage, which itself is approximately 1.96 standard deviations), this yields the estimate ${\displaystyle \frac{n_{\mathsf{s}}+2}{n+4},}$ which is known as the "plus four rule".

Although the quadratic can be solved explicitly, in most cases Wilson's equations can also be solved numerically using the fixed-point iteration $${\displaystyle p_{k+1}=\hat{p}\pm z_{\alpha }\sqrt{\frac{1}{n}{p}_k\left(1-p_k\right)}}$$ with ${\displaystyle p_0=\hat{p}.}$

The Wilson interval can also be derived from the single sample z-test or Pearson's chi-squared test with two categories. The resulting interval,

$${\displaystyle \left\{\theta |y_{\alpha }\le \frac{\hat{p}-\theta }{\sqrt{\frac{1}{n}\theta \left(1-\theta \right)}}\le z_{\alpha }\right\},}$$

(with ${\displaystyle y_{\alpha }}$ the lower ${\displaystyle \alpha }$ quantile) can then be solved for ${\displaystyle \theta }$ to produce the Wilson score interval. The test in the middle of the inequality is a score test.

The interval equality principle

The probability density function (pdf) for the Wilson score interval, plus pdfs at interval bounds. Tail areas are equal.

Since the interval is derived by solving from the normal approximation to the binomial, the Wilson score interval ${\displaystyle \left(w^{-},w^{+}\right)}$ has the property of being guaranteed to obtain the same result as the equivalent z-test or chi-squared test.

This property can be visualised by plotting the probability density function for the Wilson score interval (see Wallis). After that, then also plotting a normal pdf across each bound. The tail areas of the resulting Wilson and normal distributions represent the chance of a significant result, in that direction, must be equal.

The continuity-corrected Wilson score interval and the Clopper-Pearson interval are also compliant with this property. The practical import is that these intervals may be employed as significance tests, with identical results to the source test, and new tests may be derived by geometry.

Wilson score interval with continuity correction

The Wilson interval may be modified by employing a continuity correction, in order to align the minimum coverage probability, rather than the average coverage probability, with the nominal value, ${\displaystyle 1-\alpha .}$

Just as the Wilson interval mirrors Pearson's chi-squared test, the Wilson interval with continuity correction mirrors the equivalent Yates' chi-squared test.

The following formulae for the lower and upper bounds of the Wilson score interval with continuity correction ${\displaystyle \left(w_{\mathsf{c}\mathsf{c}}^{-},w_{\mathsf{c}\mathsf{c}}^{+}\right)}$ are derived from Newcombe:

$${\displaystyle \begin{array}{rl}{w}_{\mathsf{c}\mathsf{c}}^{-}& =max\left\{0,\frac{2n\hat{p}+z_{\alpha }^2-\left[z_{\alpha }\sqrt{z_{\alpha }^2-\frac{1}{n}+4n\hat{p}\left(1-\hat{p}\right)+\left(4\hat{p}-2\right)}+1\right]}{2\left(n+z_{\alpha }^2\right)}\right\},\\ w_{\mathsf{c}\mathsf{c}}^{+}& =min\left\{1,\frac{2n\hat{p}+z_{\alpha }^2+\left[z_{\alpha }\sqrt{z_{\alpha }^2-\frac{1}{n}+4n\hat{p}\left(1-\hat{p}\right)-\left(4\hat{p}-2\right)}+1\right]}{2\left(n+z_{\alpha }^2\right)}\right\},\end{array}}$$ for ${\displaystyle \hat{p}\ne 0}$ and ${\displaystyle \hat{p}\ne 1.}$

If ${\displaystyle \hat{p}=0,}$ then ${\displaystyle w_{\mathsf{c}\mathsf{c}}^{-}}$ must instead be set to ${\displaystyle 0;}$ if ${\displaystyle \hat{p}=1,}$ then ${\displaystyle w_{\mathsf{c}\mathsf{c}}^{+}}$ must be instead set to ${\displaystyle 1.}$

Wallis (2021) identifies a simpler method for computing continuity-corrected Wilson intervals that employs a special function based on Wilson's lower-bound formula: In Wallis' notation, for the lower bound, let $${\displaystyle \mathsf{W}\mathsf{i}\mathsf{l}\mathsf{s}\mathsf{o}{\mathsf{n}}_{\mathsf{l}\mathsf{o}\mathsf{w}\mathsf{e}\mathsf{r}}\left(\hat{p},n,\frac{\alpha }{2}\right)\equiv w^{-}=\frac{1}{1+z_{\alpha }^2/n}(\hat{p}+\frac{z_{\alpha }^2}{2n}-\frac{z_{\alpha }}{2n}\sqrt{4n\hat{p}(1-\hat{p})+z_{\alpha }^2}),}$$

where ${\displaystyle \alpha }$ is the selected tolerable error level for ${\displaystyle z_{\alpha }.}$ Then $${\displaystyle w_{\mathsf{c}\mathsf{c}}^{-}=\mathsf{W}\mathsf{i}\mathsf{l}\mathsf{s}\mathsf{o}{\mathsf{n}}_{\mathsf{l}\mathsf{o}\mathsf{w}\mathsf{e}\mathsf{r}}\left(max\left\{\hat{p}-\frac{1}{2n},0\right\},n,\frac{\alpha }{2}\right).}$$

This method has the advantage of being further decomposable.

Jeffreys interval

Jeffreys intervals plotted on a logistic curve, revealing asymmetry and good performance for small n and where p is at or near 0 or 1.

The Jeffreys interval has a Bayesian derivation, but good frequentist properties (outperforming most frequentist constructions). In particular, it has coverage properties that are similar to those of the Wilson interval, but it is one of the few intervals with the advantage of being equal-tailed (e.g., for a 95% confidence interval, the probabilities of the interval lying above or below the true value are both close to 2.5%). In contrast, the Wilson interval has a systematic bias such that it is centred too close to ${\displaystyle p=0.5}$.

The Jeffreys interval is the Bayesian credible interval obtained when using the non-informative Jeffreys prior for the binomial proportion ${\displaystyle p.}$ The Jeffreys prior for this problem is a Beta distribution with parameters ${\displaystyle \left(\frac{1}{2},\frac{1}{2}\right),}$ a conjugate prior. After observing ${\displaystyle x}$ successes in ${\displaystyle n}$ trials, the posterior distribution for ${\displaystyle p}$ is a Beta distribution with parameters ${\displaystyle \left(x+\frac{1}{2},n-x+\frac{1}{2}\right).}$

When ${\displaystyle x\ne 0}$ and ${\displaystyle x\ne n,}$ the Jeffreys interval is taken to be the ${\displaystyle 100\left(1-\alpha \right)\mathrm{\%}}$ equal-tailed posterior probability interval, i.e., the ${\displaystyle \frac{1}{2}\alpha }$ and ${\displaystyle 1-\frac{1}{2}\alpha }$ quantiles of a Beta distribution with parameters ${\displaystyle \left(x+\frac{1}{2},n-x+\frac{1}{2}\right).}$

In order to avoid the coverage probability tending to zero when ${\displaystyle p\to 0}$ or 1 , when ${\displaystyle x=0}$ the upper limit is calculated as before but the lower limit is set to 0 , and when ${\displaystyle x=n}$ the lower limit is calculated as before but the upper limit is set to 1 .

Jeffreys' interval can also be thought of as a frequentist interval based on inverting the p-value from the G-test after applying the Yates correction to avoid a potentially-infinite value for the test statistic.

Clopper–Pearson interval

The Clopper–Pearson interval is an early and very common method for calculating binomial confidence intervals. This is often called an 'exact' method, as it attains the nominal coverage level in an exact sense, meaning that the coverage level is never less than the nominal ${\displaystyle 1-\alpha }$.

The Clopper–Pearson interval can be written as

$${\displaystyle S_{\le }\cap S_{\ge }}$$

or equivalently,

$${\displaystyle \left(inf{S}_{\ge },sup{S}_{\le }\right)}$$

with

$${\displaystyle S_{\le }\equiv \left\{p|\mathbb{P}\left\{\mathrm{Bin}\left(n;p\right)\le x\right\}>\frac{\alpha }{2}\right\}}$$ and $${\displaystyle S_{\ge }\equiv \left\{p|\mathbb{P}\left\{\mathrm{Bin}\left(n;p\right)\ge x\right\}>\frac{\alpha }{2}\right\},}$$

where ${\displaystyle 0\le x\le n}$ is the number of successes observed in the sample and ${\displaystyle \mathsf{B}\mathsf{i}\mathsf{n}\left(n;p\right)}$ is a binomial random variable with ${\displaystyle n}$ trials and probability of success ${\displaystyle p.}$

Equivalently we can say that the Clopper–Pearson interval is $\left(\frac{x}{n}-{\epsilon }_1,\frac{x}{n}+{\epsilon }_2\right)$ with confidence level ${\displaystyle 1-\alpha }$ if ${\displaystyle {\epsilon }_i}$ is the infimum of those such that the following tests of hypothesis succeed with significance $\frac{\alpha }{2}:$

1. H₀: ${\displaystyle p=\frac{x}{n}-{\epsilon }_1}$ with H_A: ${\displaystyle p>\frac{x}{n}-{\epsilon }_1}$
2. H₀: ${\displaystyle p=\frac{x}{n}+{\epsilon }_2}$ with H_A: ${\displaystyle p<\frac{x}{n}+{\epsilon }_2.}$

Because of a relationship between the binomial distribution and the beta distribution, the Clopper–Pearson interval is sometimes presented in an alternate format that uses quantiles from the beta distribution.

$${\displaystyle B\left(\frac{\alpha }{2};x,n-x+1\right)<p<B\left(1-\frac{\alpha }{2};x+1,n-x\right)}$$

where ${\displaystyle x}$ is the number of successes, ${\displaystyle n}$ is the number of trials, and ${\displaystyle B\left(p;v,w\right)}$ is the pth quantile from a beta distribution with shape parameters ${\displaystyle v}$ and ${\displaystyle w.}$

Thus, ${\displaystyle p_{min}<p<p_{max},}$ where: $${\displaystyle \frac{\mathrm{\Gamma }(n+1)}{\mathrm{\Gamma }(x)\mathrm{\Gamma }(n-x+1)}{\int }_0^{p_{min}}{t}^{x-1}(1-t{)}^{n-x}\mathrm{d}t=\frac{\alpha }{2},}$$ $${\displaystyle \frac{\mathrm{\Gamma }(n+1)}{\mathrm{\Gamma }(x+1)\mathrm{\Gamma }(n-x)}{\int }_0^{p_{max}}{t}^x(1-t{)}^{n-x-1}\mathrm{d}t=1-\frac{\alpha }{2}.}$$ The binomial proportion confidence interval is then ${\displaystyle \left(p_{min},p_{max}\right),}$ as follows from the relation between the Binomial distribution cumulative distribution function and the regularized incomplete beta function.

When ${\displaystyle x}$ is either 0 or ${\displaystyle n,}$ closed-form expressions for the interval bounds are available: when ${\displaystyle x=0}$ the interval is $\left(0,1-{\left(\frac{\alpha }{2}\right)}^{1/n}\right)$ and when ${\displaystyle x=n}$ it is $\left({\left(\frac{\alpha }{2}\right)}^{1/n},1\right).$

The beta distribution is, in turn, related to the F-distribution so a third formulation of the Clopper–Pearson interval can be written using F quantiles:

$${\displaystyle {\left(1+\frac{n-x+1}{xF\left[\frac{\alpha }{2};2x,2(n-x+1)\right]}\right)}^{-1}<p<{\left(1+\frac{n-x}{(x+1)F\left[1-\frac{\alpha }{2};2(x+1),2(n-x)\right]}\right)}^{-1}}$$

where ${\displaystyle x}$ is the number of successes, ${\displaystyle n}$ is the number of trials, and ${\displaystyle F\left(c;d_1,d_2\right)}$ is the ${\displaystyle c}$ quantile from an F-distribution with ${\displaystyle d_1}$ and ${\displaystyle d_2}$ degrees of freedom.

The Clopper–Pearson interval is an 'exact' interval, since it is based directly on the binomial distribution rather than any approximation to the binomial distribution. This interval never has less than the nominal coverage for any population proportion, but that means that it is usually conservative. For example, the true coverage rate of a 95% Clopper–Pearson interval may be well above 95%, depending on ${\displaystyle n}$ and ${\displaystyle p.}$ Thus the interval may be wider than it needs to be to achieve 95% confidence, and wider than other intervals. In contrast, it is worth noting that other confidence interval may have coverage levels that are lower than the nominal ${\displaystyle 1-\alpha ,}$ i.e., the normal approximation (or "standard") interval, Wilson interval, Agresti–Coull interval, etc., with a nominal coverage of 95% may in fact cover less than 95%, even for large sample sizes.

The definition of the Clopper–Pearson interval can also be modified to obtain exact confidence intervals for different distributions. For instance, it can also be applied to the case where the samples are drawn without replacement from a population of a known size, instead of repeated draws of a binomial distribution. In this case, the underlying distribution would be the hypergeometric distribution.

The interval boundaries can be computed with numerical functions qbeta in R and scipy.stats.beta.ppf in Python.

from scipy.stats import beta
import numpy as np

k = 20
n = 400
alpha = 0.05
p_u, p_o = beta.ppf([alpha / 2, 1 - alpha / 2], [k, k + 1], [n - k + 1, n - k])
if np.isnan(p_o):
    p_o = 1
if np.isnan(p_u):
    p_u = 0

Agresti–Coull interval

The Agresti–Coull interval is also another approximate binomial confidence interval.

Given ${\displaystyle n_{\mathsf{s}}}$ successes in ${\displaystyle n}$ trials, define $${\displaystyle \tilde{n}\equiv n+z_{\alpha }^2}$$

and $${\displaystyle \tilde{p}=\frac{1}{\tilde{n}}\left(n_{\mathsf{s}}+\frac{z_{\alpha }^2}{2}\right)}$$

Then, a confidence interval for ${\displaystyle p}$ is given by

$${\displaystyle p\approx \tilde{p}\pm z_{\alpha }\sqrt{\frac{\tilde{p}}{\tilde{n}}\left(1-\tilde{p}\right)}}$$

where ${\displaystyle z_{\alpha }={\mathrm{\Phi }}^{\text{-}1}\left(1-\frac{\alpha }{2}\right)}$ is the quantile of a standard normal distribution, as before (for example, a 95% confidence interval requires ${\displaystyle \alpha =0.05,}$ thereby producing ${\displaystyle z_{.05}=1.96}$). According to Brown, Cai, & DasGupta (2001), taking ${\displaystyle z=2}$ instead of 1.96 produces the "add 2 successes and 2 failures" interval previously described by Agresti & Coull.

This interval can be summarised as employing the centre-point adjustment, ${\displaystyle \tilde{p},}$ of the Wilson score interval, and then applying the Normal approximation to this point.

$${\displaystyle \tilde{p}=\frac{\hat{p}+\frac{z_{\alpha }^2}{2n}}{1+\frac{z_{\alpha }^2}{n}}}$$

Arcsine transformation

Main article: Arcsine transformation

Further information: Cohen's h

The arcsine transformation has the effect of pulling out the ends of the distribution. While it can stabilize the variance (and thus confidence intervals) of proportion data, its use has been criticized in several contexts.

Let ${\displaystyle X}$ be the number of successes in ${\displaystyle n}$ trials and let ${\displaystyle p=\frac{1}{n}X.}$ The variance of ${\displaystyle p}$ is

$${\displaystyle \mathrm{var}\{p\}=\frac{1}{n}p(1-p).}$$

Using the arc sine transform, the variance of the arcsine of ${\displaystyle \sqrt{p}}$ is

$${\displaystyle \mathrm{var}\left\{\arcsin \sqrt{p}\right\}\approx \frac{\mathrm{var}\{p\}}{4p(1-p)}=\frac{p(1-p)}{4np(1-p)}=\frac{1}{4n}.}$$

So, the confidence interval itself has the form

$${\displaystyle {\sin }^2\left(-\frac{z_{\alpha }}{2\sqrt{n}}+\arcsin \sqrt{p}\right)<\theta <{\sin }^2\left(+\frac{z_{\alpha }}{2\sqrt{n}}+\arcsin \sqrt{p}\right),}$$

where ${\displaystyle z_{\alpha }}$ is the ${\displaystyle 1-\frac{\alpha }{2}}$ quantile of a standard normal distribution.

This method may be used to estimate the variance of ${\displaystyle p}$ but its use is problematic when ${\displaystyle p}$ is close to 0 or 1 .

t_a transform

Let ${\displaystyle p}$ be the proportion of successes. For ${\displaystyle 0\le a\le 2,}$

$${\displaystyle t_a=\log \left(\frac{p^a}{(1-p{)}^{2-a}}\right)=a\log p-(2-a)\log (1-p)}$$

This family is a generalisation of the logit transform which is a special case with a = 1 and can be used to transform a proportional data distribution to an approximately normal distribution. The parameter a has to be estimated for the data set.

Rule of three — for when no successes are observed

The rule of three is used to provide a simple way of stating an approximate 95% confidence interval for ${\displaystyle p}$, in the special case that no successes (${\displaystyle \hat{p}=0}$) have been observed. The interval is ${\displaystyle \left(0,\frac{3}{n}\right)}$.

By symmetry, in the case of only successes (${\displaystyle \hat{p}=1}$), the interval is ${\displaystyle \left(1-\frac{3}{n},1\right)}$.

Comparison and discussion

There are several research papers that compare these and other confidence intervals for the binomial proportion.

Both Ross (2003) and Agresti & Coull (1998) point out that exact methods such as the Clopper–Pearson interval may not work as well as some approximations. The normal approximation interval and its presentation in textbooks has been heavily criticised, with many statisticians advocating that it not be used. The principal problems are overshoot (bounds exceed [0, 1]), zero-width intervals at ${\displaystyle \hat{p}=0}$ or 1 (falsely implying certainty), and overall inconsistency with significance testing.

Of the approximations listed above, Wilson score interval methods (with or without continuity correction) have been shown to be the most accurate and the most robust, though some prefer Agresti & Coulls' approach for larger sample sizes. Wilson and Clopper–Pearson methods obtain consistent results with source significance tests, and this property is decisive for many researchers.

Many of these intervals can be calculated in R using packages like binom.