Bayesian inference

From Wikipedia, the free encyclopedia
https://en.wikipedia.org/wiki/Bayesian_inference

Bayesian inference (/ˈbeɪziən/ BAY-zee-ən or /ˈbeɪʒən/ BAY-zhən) is a method of statistical inference in which Bayes' theorem is used to calculate a probability of a hypothesis, given prior evidence, and update it as more information becomes available. Fundamentally, Bayesian inference uses a prior distribution to estimate posterior probabilities. Bayesian inference is an important technique in statistics, and especially in mathematical statistics. Bayesian updating is particularly important in the dynamic analysis of a sequence of data. Bayesian inference has found application in a wide range of activities, including science, engineering, philosophy, medicine, sport, and law. In the philosophy of decision theory, Bayesian inference is closely related to subjective probability, often called "Bayesian probability".

Introduction to Bayes' rule

A geometric visualisation of Bayes' theorem. In the table, the values 2, 3, 6 and 9 give the relative weights of each corresponding condition and case. The figures denote the cells of the table involved in each metric, the probability being the fraction of each figure that is shaded. This shows that P(A|B) P(B) = P(B|A) P(A) i.e. P(A|B) = ⁠P(B|A) P(A)/P(B)⁠ . Similar reasoning can be used to show that P(¬A|B) = ⁠P(B|¬A) P(¬A)/P(B)⁠ etc.

Main article: Bayes' theorem

See also: Bayesian probability

Formal explanation

Contingency table

Hypothesis Evidence	Satisfies hypothesis H	Violates hypothesis ¬H		Total
Has evidence E	P(H|E)·P(E) = P(E|H)·P(H)	P(¬H|E)·P(E) = P(E|¬H)·P(¬H)		P(E)
No evidence ¬E	P(H|¬E)·P(¬E) = P(¬E|H)·P(H)	P(¬H|¬E)·P(¬E) = P(¬E|¬H)·P(¬H)		P(¬E) = 1−P(E)
Total	P(H)	P(¬H) = 1−P(H)	1

Bayesian inference derives the posterior probability as a consequence of two antecedents: a prior probability and a "likelihood function" derived from a statistical model for the observed data. Bayesian inference computes the posterior probability according to Bayes' theorem: $${\displaystyle P(H\mid E)=\frac{P(E\mid H)\cdot P(H)}{P(E)},}$$ where

• ${\displaystyle H}$ stands for any hypothesis whose probability may be affected by data (called evidence below). Often there are competing hypotheses, and the task is to determine which is the most probable.
• ${\displaystyle P(H)}$, the prior probability, is the estimate of the probability of the hypothesis ${\displaystyle H}$ before the data ${\displaystyle E}$, the current evidence, is observed.
• ${\displaystyle E}$, the evidence, corresponds to new data that were not used in computing the prior probability.
• ${\displaystyle P(H\mid E)}$, the posterior probability, is the probability of ${\displaystyle H}$ given ${\displaystyle E}$, i.e., after ${\displaystyle E}$ is observed. This is what we want to know: the probability of a hypothesis given the observed evidence.
• ${\displaystyle P(E\mid H)}$ is the probability of observing ${\displaystyle E}$ given ${\displaystyle H}$ and is called the likelihood. As a function of ${\displaystyle E}$ with ${\displaystyle H}$ fixed, it indicates the compatibility of the evidence with the given hypothesis. The likelihood function is a function of the evidence, ${\displaystyle E}$, while the posterior probability is a function of the hypothesis, ${\displaystyle H}$.
• ${\displaystyle P(E)}$ is sometimes termed the marginal likelihood or "model evidence". This factor is the same for all possible hypotheses being considered (as is evident from the fact that the hypothesis ${\displaystyle H}$ does not appear anywhere in the symbol, unlike for all the other factors) and hence does not factor into determining the relative probabilities of different hypotheses.
• ${\displaystyle P(E)>0}$ (Else one has ${\displaystyle 0/0}$.)

For different values of ${\displaystyle H}$, only the factors ${\displaystyle P(H)}$ and ${\displaystyle P(E\mid H)}$, both in the numerator, affect the value of ${\displaystyle P(H\mid E)}$ – the posterior probability of a hypothesis is proportional to its prior probability (its inherent likeliness) and the newly acquired likelihood (its compatibility with the new observed evidence).

In cases where ${\displaystyle \mathrm{\neg }H}$ ("not ${\displaystyle H}$"), the logical negation of ${\displaystyle H}$, is a valid likelihood, Bayes' rule can be rewritten as follows: $${\displaystyle \begin{array}{rl}P(H\mid E)& =\frac{P(E\mid H)P(H)}{P(E)}\\ \\ & =\frac{P(E\mid H)P(H)}{P(E\mid H)P(H)+P(E\mid \mathrm{\neg }H)P(\mathrm{\neg }H)}\\ \\ & =\frac{1}{1+\left(\frac{1}{P(H)}-1\right)\frac{P(E\mid \mathrm{\neg }H)}{P(E\mid H)}}\end{array}}$$ because $${\displaystyle P(E)=P(E\mid H)P(H)+P(E\mid \mathrm{\neg }H)P(\mathrm{\neg }H)}$$ and $${\displaystyle P(H)+P(\mathrm{\neg }H)=1.}$$ This focuses attention on the term $${\displaystyle \left(\frac{1}{P(H)}-1\right)\frac{P(E\mid \mathrm{\neg }H)}{P(E\mid H)}.}$$ If that term is approximately 1, then the probability of the hypothesis given the evidence, ${\displaystyle P(H\mid E)}$, is about ${\displaystyle \frac{1}{2}}$, about 50% likely - equally likely or not likely. If that term is very small, close to zero, then the probability of the hypothesis, given the evidence, ${\displaystyle P(H\mid E)}$ is close to 1 or the conditional hypothesis is quite likely. If that term is very large, much larger than 1, then the hypothesis, given the evidence, is quite unlikely. If the hypothesis (without consideration of evidence) is unlikely, then ${\displaystyle P(H)}$ is small (but not necessarily astronomically small) and ${\displaystyle \frac{1}{P(H)}}$ is much larger than 1 and this term can be approximated as ${\displaystyle \frac{P(E\mid \mathrm{\neg }H)}{P(E\mid H)\cdot P(H)}}$ and relevant probabilities can be compared directly to each other.

One quick and easy way to remember the equation would be to use rule of multiplication: $${\displaystyle P(E\cap H)=P(E\mid H)P(H)=P(H\mid E)P(E).}$$

Alternatives to Bayesian updating

Bayesian updating is widely used and computationally convenient. However, it is not the only updating rule that might be considered rational.

Ian Hacking noted that traditional "Dutch book" arguments did not specify Bayesian updating: they left open the possibility that non-Bayesian updating rules could avoid Dutch books. Hacking wrote: "And neither the Dutch book argument nor any other in the personalist arsenal of proofs of the probability axioms entails the dynamic assumption. Not one entails Bayesianism. So the personalist requires the dynamic assumption to be Bayesian. It is true that in consistency a personalist could abandon the Bayesian model of learning from experience. Salt could lose its savour."

Indeed, there are non-Bayesian updating rules that also avoid Dutch books (as discussed in the literature on "probability kinematics") following the publication of Richard C. Jeffrey's rule, which applies Bayes' rule to the case where the evidence itself is assigned a probability. The additional hypotheses needed to uniquely require Bayesian updating have been deemed to be substantial, complicated, and unsatisfactory.

Inference over exclusive and exhaustive possibilities

If evidence is simultaneously used to update belief over a set of exclusive and exhaustive propositions, Bayesian inference may be thought of as acting on this belief distribution as a whole.

General formulation

Diagram illustrating event space ${\displaystyle \mathrm{\Omega }}$ in general formulation of Bayesian inference. Although this diagram shows discrete models and events, the continuous case may be visualized similarly using probability densities.

Suppose a process is generating independent and identically distributed events ${\displaystyle E_n,n=1,2,3,\dots }$, but the probability distribution is unknown. Let the event space ${\displaystyle \mathrm{\Omega }}$ represent the current state of belief for this process. Each model is represented by event ${\displaystyle M_m}$. The conditional probabilities ${\displaystyle P(E_n\mid M_m)}$ are specified to define the models. ${\displaystyle P(M_m)}$ is the degree of belief in ${\displaystyle M_m}$. Before the first inference step, ${\displaystyle \{P(M_m)\}}$ is a set of initial prior probabilities. These must sum to 1, but are otherwise arbitrary.

Suppose that the process is observed to generate ${\displaystyle E\in \{E_n\}}$. For each ${\displaystyle M\in \{M_m\}}$, the prior ${\displaystyle P(M)}$ is updated to the posterior ${\displaystyle P(M\mid E)}$. From Bayes' theorem:

$${\displaystyle P(M\mid E)=\frac{P(E\mid M)}{\sum _{m}P(E\mid M_m)P(M_m)}\cdot P(M).}$$

Upon observation of further evidence, this procedure may be repeated.

Multiple observations

For a sequence of independent and identically distributed observations ${\displaystyle \mathbf{E}=(e_1,\dots ,e_n)}$, it can be shown by induction that repeated application of the above is equivalent to $${\displaystyle P(M\mid \mathbf{E})=\frac{P(\mathbf{E}\mid M)}{\sum _{m}P(\mathbf{E}\mid M_m)P(M_m)}\cdot P(M),}$$ where $${\displaystyle P(\mathbf{E}\mid M)=\prod _{k}P(e_k\mid M).}$$

Parametric formulation: motivating the formal description

By parameterizing the space of models, the belief in all models may be updated in a single step. The distribution of belief over the model space may then be thought of as a distribution of belief over the parameter space. The distributions in this section are expressed as continuous, represented by probability densities, as this is the usual situation. The technique is, however, equally applicable to discrete distributions.

Let the vector ${\displaystyle \mathit{\theta }}$ span the parameter space. Let the initial prior distribution over ${\displaystyle \mathit{\theta }}$ be ${\displaystyle p(\mathit{\theta }\mid \mathit{\alpha })}$, where ${\displaystyle \mathit{\alpha }}$ is a set of parameters to the prior itself, or hyperparameters. Let ${\displaystyle \mathbf{E}=(e_1,\dots ,e_n)}$ be a sequence of independent and identically distributed event observations, where all ${\displaystyle e_i}$ are distributed as ${\displaystyle p(e\mid \mathit{\theta })}$ for some ${\displaystyle \mathit{\theta }}$. Bayes' theorem is applied to find the posterior distribution over ${\displaystyle \mathit{\theta }}$:

$${\displaystyle \begin{array}{rl}p(\mathit{\theta }\mid \mathbf{E},\mathit{\alpha })& =\frac{p(\mathbf{E}\mid \mathit{\theta },\mathit{\alpha })}{p(\mathbf{E}\mid \mathit{\alpha })}\cdot p(\mathit{\theta }\mid \mathit{\alpha })\\ & =\frac{p(\mathbf{E}\mid \mathit{\theta },\mathit{\alpha })}{\int p(\mathbf{E}\mid \mathit{\theta },\mathit{\alpha })p(\mathit{\theta }\mid \mathit{\alpha })d\mathit{\theta }}\cdot p(\mathit{\theta }\mid \mathit{\alpha }),\end{array}}$$ where $${\displaystyle p(\mathbf{E}\mid \mathit{\theta },\mathit{\alpha })=\prod _{k}p(e_k\mid \mathit{\theta }).}$$

Formal description of Bayesian inference

Definitions

• ${\displaystyle x}$, a data point in general. This may in fact be a vector of values.
• ${\displaystyle \theta }$, the parameter of the data point's distribution, i.e., ${\displaystyle x\sim p(x\mid \theta )}$. This may be a vector of parameters.
• ${\displaystyle \alpha }$, the hyperparameter of the parameter distribution, i.e., ${\displaystyle \theta \sim p(\theta \mid \alpha )}$. This may be a vector of hyperparameters.
• ${\displaystyle \mathbf{X}}$ is the sample, a set of ${\displaystyle n}$ observed data points, i.e., ${\displaystyle x_1,\dots ,x_n}$.
• ${\displaystyle \tilde{x}}$, a new data point whose distribution is to be predicted.

Bayesian inference

• The prior distribution is the distribution of the parameter(s) before any data is observed, i.e. ${\displaystyle p(\theta \mid \alpha )}$ . The prior distribution might not be easily determined; in such a case, one possibility may be to use the Jeffreys prior to obtain a prior distribution before updating it with newer observations.
• The sampling distribution is the distribution of the observed data conditional on its parameters, i.e. ${\displaystyle p(\mathbf{X}\mid \theta )}$. This is also termed the likelihood, especially when viewed as a function of the parameter(s), sometimes written ${\displaystyle \mathrm{L}(\theta \mid \mathbf{X})=p(\mathbf{X}\mid \theta )}$.
• The marginal likelihood (sometimes also termed the evidence) is the distribution of the observed data marginalized over the parameter(s), i.e. $${\displaystyle p(\mathbf{X}\mid \alpha )=\int p(\mathbf{X}\mid \theta )p(\theta \mid \alpha )d\theta .}$$ It quantifies the agreement between data and expert opinion, in a geometric sense that can be made precise.^[6] If the marginal likelihood is 0 then there is no agreement between the data and expert opinion and Bayes' rule cannot be applied.
• The posterior distribution is the distribution of the parameter(s) after taking into account the observed data. This is determined by Bayes' rule, which forms the heart of Bayesian inference: $${\displaystyle p(\theta \mid \mathbf{X},\alpha )=\frac{p(\theta ,\mathbf{X},\alpha )}{p(\mathbf{X},\alpha )}=\frac{p(\mathbf{X}\mid \theta ,\alpha )p(\theta ,\alpha )}{p(\mathbf{X}\mid \alpha )p(\alpha )}=\frac{p(\mathbf{X}\mid \theta ,\alpha )p(\theta \mid \alpha )}{p(\mathbf{X}\mid \alpha )}\propto p(\mathbf{X}\mid \theta ,\alpha )p(\theta \mid \alpha ).}$$ This is expressed in words as "posterior is proportional to likelihood times prior", or sometimes as "posterior = likelihood times prior, over evidence".
• In practice, for almost all complex Bayesian models used in machine learning, the posterior distribution ${\displaystyle p(\theta \mid \mathbf{X},\alpha )}$ is not obtained in a closed form distribution, mainly because the parameter space for ${\displaystyle \theta }$ can be very high, or the Bayesian model retains certain hierarchical structure formulated from the observations ${\displaystyle \mathbf{X}}$ and parameter ${\displaystyle \theta }$. In such situations, we need to resort to approximation techniques.
• General case: Let ${\displaystyle P_Y^x}$ be the conditional distribution of ${\displaystyle Y}$ given ${\displaystyle X=x}$ and let ${\displaystyle P_X}$ be the distribution of ${\displaystyle X}$. The joint distribution is then ${\displaystyle P_{X,Y}(dx,dy)=P_Y^x(dy)P_X(dx)}$. The conditional distribution ${\displaystyle P_X^y}$ of ${\displaystyle X}$ given ${\displaystyle Y=y}$ is then determined by

$${\displaystyle P_X^y(A)=E(1_A(X)|Y=y)}$$Existence and uniqueness of the needed conditional expectation is a consequence of the Radon–Nikodym theorem. This was formulated by Kolmogorov in his famous book from 1933. Kolmogorov underlines the importance of conditional probability by writing "I wish to call attention to ... and especially the theory of conditional probabilities and conditional expectations ..." in the Preface. The Bayes theorem determines the posterior distribution from the prior distribution. Uniqueness requires continuity assumptions. Bayes' theorem can be generalized to include improper prior distributions such as the uniform distribution on the real line. Modern Markov chain Monte Carlo methods have boosted the importance of Bayes' theorem including cases with improper priors.

Bayesian prediction

• The posterior predictive distribution is the distribution of a new data point, marginalized over the posterior: $${\displaystyle p(\tilde{x}\mid \mathbf{X},\alpha )=\int p(\tilde{x}\mid \theta )p(\theta \mid \mathbf{X},\alpha )d\theta }$$
• The prior predictive distribution is the distribution of a new data point, marginalized over the prior: $${\displaystyle p(\tilde{x}\mid \alpha )=\int p(\tilde{x}\mid \theta )p(\theta \mid \alpha )d\theta }$$

Bayesian theory calls for the use of the posterior predictive distribution to do predictive inference, i.e., to predict the distribution of a new, unobserved data point. That is, instead of a fixed point as a prediction, a distribution over possible points is returned. Only this way is the entire posterior distribution of the parameter(s) used. By comparison, prediction in frequentist statistics often involves finding an optimum point estimate of the parameter(s)—e.g., by maximum likelihood or maximum a posteriori estimation (MAP)—and then plugging this estimate into the formula for the distribution of a data point. This has the disadvantage that it does not account for any uncertainty in the value of the parameter, and hence will underestimate the variance of the predictive distribution.

In some instances, frequentist statistics can work around this problem. For example, confidence intervals and prediction intervals in frequentist statistics when constructed from a normal distribution with unknown mean and variance are constructed using a Student's t-distribution. This correctly estimates the variance, due to the facts that (1) the average of normally distributed random variables is also normally distributed, and (2) the predictive distribution of a normally distributed data point with unknown mean and variance, using conjugate or uninformative priors, has a Student's t-distribution. In Bayesian statistics, however, the posterior predictive distribution can always be determined exactly—or at least to an arbitrary level of precision when numerical methods are used.

Both types of predictive distributions have the form of a compound probability distribution (as does the marginal likelihood). In fact, if the prior distribution is a conjugate prior, such that the prior and posterior distributions come from the same family, it can be seen that both prior and posterior predictive distributions also come from the same family of compound distributions. The only difference is that the posterior predictive distribution uses the updated values of the hyperparameters (applying the Bayesian update rules given in the conjugate prior article), while the prior predictive distribution uses the values of the hyperparameters that appear in the prior distribution.

Mathematical properties

Interpretation of factor

$\frac{P(E\mid M)}{P(E)}>1\Rightarrow P(E\mid M)>P(E)$. That is, if the model were true, the evidence would be more likely than is predicted by the current state of belief. The reverse applies for a decrease in belief. If the belief does not change, $\frac{P(E\mid M)}{P(E)}=1\Rightarrow P(E\mid M)=P(E)$. That is, the evidence is independent of the model. If the model were true, the evidence would be exactly as likely as predicted by the current state of belief.

Cromwell's rule

Main article: Cromwell's rule

If ${\displaystyle P(M)=0}$ then ${\displaystyle P(M\mid E)=0}$. If ${\displaystyle P(M)=1}$ and ${\displaystyle P(E)>0}$, then ${\displaystyle P(M|E)=1}$. This can be interpreted to mean that hard convictions are insensitive to counter-evidence.

The former follows directly from Bayes' theorem. The latter can be derived by applying the first rule to the event "not ${\displaystyle M}$" in place of "${\displaystyle M}$", yielding "if ${\displaystyle 1-P(M)=0}$, then ${\displaystyle 1-P(M\mid E)=0}$", from which the result immediately follows.

Asymptotic behaviour of posterior

Consider the behaviour of a belief distribution as it is updated a large number of times with independent and identically distributed trials. For sufficiently nice prior probabilities, the Bernstein-von Mises theorem gives that in the limit of infinite trials, the posterior converges to a Gaussian distribution independent of the initial prior under some conditions firstly outlined and rigorously proven by Joseph L. Doob in 1948, namely if the random variable in consideration has a finite probability space. The more general results were obtained later by the statistician David A. Freedman who published in two seminal research papers in 1963 and 1965 when and under what circumstances the asymptotic behaviour of posterior is guaranteed. His 1963 paper treats, like Doob (1949), the finite case and comes to a satisfactory conclusion. However, if the random variable has an infinite but countable probability space (i.e., corresponding to a die with infinite many faces) the 1965 paper demonstrates that for a dense subset of priors the Bernstein-von Mises theorem is not applicable. In this case there is almost surely no asymptotic convergence. Later in the 1980s and 1990s Freedman and Persi Diaconis continued to work on the case of infinite countable probability spaces. To summarise, there may be insufficient trials to suppress the effects of the initial choice, and especially for large (but finite) systems the convergence might be very slow.

Conjugate priors

Main article: Conjugate prior

In parameterized form, the prior distribution is often assumed to come from a family of distributions called conjugate priors. The usefulness of a conjugate prior is that the corresponding posterior distribution will be in the same family, and the calculation may be expressed in closed form.

Estimates of parameters and predictions

It is often desired to use a posterior distribution to estimate a parameter or variable. Several methods of Bayesian estimation select measurements of central tendency from the posterior distribution.

For one-dimensional problems, a unique median exists for practical continuous problems. The posterior median is attractive as a robust estimator.

If there exists a finite mean for the posterior distribution, then the posterior mean is a method of estimation. $${\displaystyle \tilde{\theta }=\mathrm{E}[\theta ]=\int \theta p(\theta \mid \mathbf{X},\alpha )d\theta }$$

Taking a value with the greatest probability defines maximum a posteriori (MAP) estimates: $${\displaystyle \{{\theta }_{\text{MAP}}\}\subset \arg \underset{\theta }{max}p(\theta \mid \mathbf{X},\alpha ).}$$

There are examples where no maximum is attained, in which case the set of MAP estimates is empty.

There are other methods of estimation that minimize the posterior risk (expected-posterior loss) with respect to a loss function, and these are of interest to statistical decision theory using the sampling distribution ("frequentist statistics").

The posterior predictive distribution of a new observation ${\displaystyle \tilde{x}}$ (that is independent of previous observations) is determined by $${\displaystyle p(\tilde{x}|\mathbf{X},\alpha )=\int p(\tilde{x},\theta \mid \mathbf{X},\alpha )d\theta =\int p(\tilde{x}\mid \theta )p(\theta \mid \mathbf{X},\alpha )d\theta .}$$

Examples

Probability of a hypothesis

Contingency table

Bowl Cookie	#1 H1	#2 H2		Total
Plain, E	30	20		50
Choc, ¬E	10	20		30
Total	40	40		80
P(H1|E) = 30 / 50 = 0.6

Suppose there are two full bowls of cookies. Bowl #1 has 10 chocolate chip and 30 plain cookies, while bowl #2 has 20 of each. Our friend Fred picks a bowl at random, and then picks a cookie at random. We may assume there is no reason to believe Fred treats one bowl differently from another, likewise for the cookies. The cookie turns out to be a plain one. How probable is it that Fred picked it out of bowl #1?

Intuitively, it seems clear that the answer should be more than a half, since there are more plain cookies in bowl #1. The precise answer is given by Bayes' theorem. Let ${\displaystyle H_1}$ correspond to bowl #1, and ${\displaystyle H_2}$ to bowl #2. It is given that the bowls are identical from Fred's point of view, thus ${\displaystyle P(H_1)=P(H_2)}$, and the two must add up to 1, so both are equal to 0.5. The event ${\displaystyle E}$ is the observation of a plain cookie. From the contents of the bowls, we know that ${\displaystyle P(E\mid H_1)=30/40=0.75}$ and ${\displaystyle P(E\mid H_2)=20/40=\mathrm{0.5.}}$ Bayes' formula then yields $${\displaystyle \begin{array}{rl}P(H_1\mid E)& =\frac{P(E\mid H_1)P(H_1)}{P(E\mid H_1)P(H_1)+P(E\mid H_2)P(H_2)}\\ \\ & =\frac{0.75\times 0.5}{0.75\times 0.5+0.5\times 0.5}\\ \\ & =0.6\end{array}}$$

Before we observed the cookie, the probability we assigned for Fred having chosen bowl #1 was the prior probability, ${\displaystyle P(H_1)}$, which was 0.5. After observing the cookie, we must revise the probability to ${\displaystyle P(H_1\mid E)}$, which is 0.6.

Making a prediction

Example results for archaeology example. This simulation was generated using c=15.2.

An archaeologist is working at a site thought to be from the medieval period, between the 11th century to the 16th century. However, it is uncertain exactly when in this period the site was inhabited. Fragments of pottery are found, some of which are glazed and some of which are decorated. It is expected that if the site were inhabited during the early medieval period, then 1% of the pottery would be glazed and 50% of its area decorated, whereas if it had been inhabited in the late medieval period then 81% would be glazed and 5% of its area decorated. How confident can the archaeologist be in the date of inhabitation as fragments are unearthed?

The degree of belief in the continuous variable ${\displaystyle C}$ (century) is to be calculated, with the discrete set of events ${\displaystyle \{GD,G\overline{D},\overline{G}D,\overline{G}\overline{D}\}}$ as evidence. Assuming linear variation of glaze and decoration with time, and that these variables are independent,

$${\displaystyle P(E=GD\mid C=c)=(0.01+\frac{0.81-0.01}{16-11}(c-11))(0.5-\frac{0.5-0.05}{16-11}(c-11))}$$ $${\displaystyle P(E=G\overline{D}\mid C=c)=(0.01+\frac{0.81-0.01}{16-11}(c-11))(0.5+\frac{0.5-0.05}{16-11}(c-11))}$$ $${\displaystyle P(E=\overline{G}D\mid C=c)=((1-0.01)-\frac{0.81-0.01}{16-11}(c-11))(0.5-\frac{0.5-0.05}{16-11}(c-11))}$$ $${\displaystyle P(E=\overline{G}\overline{D}\mid C=c)=((1-0.01)-\frac{0.81-0.01}{16-11}(c-11))(0.5+\frac{0.5-0.05}{16-11}(c-11))}$$

Assume a uniform prior of $f_C(c)=0.2$, and that trials are independent and identically distributed. When a new fragment of type ${\displaystyle e}$ is discovered, Bayes' theorem is applied to update the degree of belief for each ${\displaystyle c}$: $${\displaystyle f_C(c\mid E=e)=\frac{P(E=e\mid C=c)}{P(E=e)}{f}_C(c)=\frac{P(E=e\mid C=c)}{{\int }_{11}^{16}P(E=e\mid C=c)f_C(c)dc}{f}_C(c)}$$

A computer simulation of the changing belief as 50 fragments are unearthed is shown on the graph. In the simulation, the site was inhabited around 1420, or ${\displaystyle c=15.2}$. By calculating the area under the relevant portion of the graph for 50 trials, the archaeologist can say that there is practically no chance the site was inhabited in the 11th and 12th centuries, about 1% chance that it was inhabited during the 13th century, 63% chance during the 14th century and 36% during the 15th century. The Bernstein-von Mises theorem asserts here the asymptotic convergence to the "true" distribution because the probability space corresponding to the discrete set of events ${\displaystyle \{GD,G\overline{D},\overline{G}D,\overline{G}\overline{D}\}}$ is finite (see above section on asymptotic behaviour of the posterior).

In frequentist statistics and decision theory

A decision-theoretic justification of the use of Bayesian inference was given by Abraham Wald, who proved that every unique Bayesian procedure is admissible. Conversely, every admissible statistical procedure is either a Bayesian procedure or a limit of Bayesian procedures.

Wald characterized admissible procedures as Bayesian procedures (and limits of Bayesian procedures), making the Bayesian formalism a central technique in such areas of frequentist inference as parameter estimation, hypothesis testing, and computing confidence intervals. For example:

• "Under some conditions, all admissible procedures are either Bayes procedures or limits of Bayes procedures (in various senses). These remarkable results, at least in their original form, are due essentially to Wald. They are useful because the property of being Bayes is easier to analyze than admissibility."
• "In decision theory, a quite general method for proving admissibility consists in exhibiting a procedure as a unique Bayes solution."
• "In the first chapters of this work, prior distributions with finite support and the corresponding Bayes procedures were used to establish some of the main theorems relating to the comparison of experiments. Bayes procedures with respect to more general prior distributions have played a very important role in the development of statistics, including its asymptotic theory." "There are many problems where a glance at posterior distributions, for suitable priors, yields immediately interesting information. Also, this technique can hardly be avoided in sequential analysis."
• "A useful fact is that any Bayes decision rule obtained by taking a proper prior over the whole parameter space must be admissible"
• "An important area of investigation in the development of admissibility ideas has been that of conventional sampling-theory procedures, and many interesting results have been obtained."

Model selection

Main article: Bayesian model selection

See also: Bayesian information criterion

Bayesian methodology also plays a role in model selection where the aim is to select one model from a set of competing models that represents most closely the underlying process that generated the observed data. In Bayesian model comparison, the model with the highest posterior probability given the data is selected. The posterior probability of a model depends on the evidence, or marginal likelihood, which reflects the probability that the data is generated by the model, and on the prior belief of the model. When two competing models are a priori considered to be equiprobable, the ratio of their posterior probabilities corresponds to the Bayes factor. Since Bayesian model comparison is aimed on selecting the model with the highest posterior probability, this methodology is also referred to as the maximum a posteriori (MAP) selection rule  or the MAP probability rule.

Probabilistic programming

Main article: Probabilistic programming

While conceptually simple, Bayesian methods can be mathematically and numerically challenging. Probabilistic programming languages (PPLs) implement functions to easily build Bayesian models together with efficient automatic inference methods. This helps separate the model building from the inference, allowing practitioners to focus on their specific problems and leaving PPLs to handle the computational details for them.

Applications

Statistical data analysis

See the separate Wikipedia entry on Bayesian statistics, specifically the statistical modeling section in that page.

Computer applications

Bayesian inference has applications in artificial intelligence and expert systems. Bayesian inference techniques have been a fundamental part of computerized pattern recognition techniques since the late 1950s. There is also an ever-growing connection between Bayesian methods and simulation-based Monte Carlo techniques since complex models cannot be processed in closed form by a Bayesian analysis, while a graphical model structure may allow for efficient simulation algorithms like the Gibbs sampling and other Metropolis–Hastings algorithm schemes. Recently Bayesian inference has gained popularity among the phylogenetics community for these reasons; a number of applications allow many demographic and evolutionary parameters to be estimated simultaneously.

As applied to statistical classification, Bayesian inference has been used to develop algorithms for identifying e-mail spam. Applications which make use of Bayesian inference for spam filtering include CRM114, DSPAM, Bogofilter, SpamAssassin, SpamBayes, Mozilla, XEAMS, and others. Spam classification is treated in more detail in the article on the naïve Bayes classifier.

Solomonoff's Inductive inference is the theory of prediction based on observations; for example, predicting the next symbol based upon a given series of symbols. The only assumption is that the environment follows some unknown but computable probability distribution. It is a formal inductive framework that combines two well-studied principles of inductive inference: Bayesian statistics and Occam's Razor. Solomonoff's universal prior probability of any prefix p of a computable sequence x is the sum of the probabilities of all programs (for a universal computer) that compute something starting with p. Given some p and any computable but unknown probability distribution from which x is sampled, the universal prior and Bayes' theorem can be used to predict the yet unseen parts of x in optimal fashion.

Bioinformatics and healthcare applications

Bayesian inference has been applied in different Bioinformatics applications, including differential gene expression analysis. Bayesian inference is also used in a general cancer risk model, called CIRI (Continuous Individualized Risk Index), where serial measurements are incorporated to update a Bayesian model which is primarily built from prior knowledge.

In the courtroom

Main article: Jurimetrics § Bayesian analysis of evidence

Bayesian inference can be used by jurors to coherently accumulate the evidence for and against a defendant, and to see whether, in totality, it meets their personal threshold for "beyond a reasonable doubt". Bayes' theorem is applied successively to all evidence presented, with the posterior from one stage becoming the prior for the next. The benefit of a Bayesian approach is that it gives the juror an unbiased, rational mechanism for combining evidence. It may be appropriate to explain Bayes' theorem to jurors in odds form, as betting odds are more widely understood than probabilities. Alternatively, a logarithmic approach, replacing multiplication with addition, might be easier for a jury to handle.

Adding up evidence

If the existence of the crime is not in doubt, only the identity of the culprit, it has been suggested that the prior should be uniform over the qualifying population. For example, if 1,000 people could have committed the crime, the prior probability of guilt would be 1/1000.

The use of Bayes' theorem by jurors is controversial. In the United Kingdom, a defence expert witness explained Bayes' theorem to the jury in R v Adams. The jury convicted, but the case went to appeal on the basis that no means of accumulating evidence had been provided for jurors who did not wish to use Bayes' theorem. The Court of Appeal upheld the conviction, but it also gave the opinion that "To introduce Bayes' Theorem, or any similar method, into a criminal trial plunges the jury into inappropriate and unnecessary realms of theory and complexity, deflecting them from their proper task."

Gardner-Medwin argues that the criterion on which a verdict in a criminal trial should be based is not the probability of guilt, but rather the probability of the evidence, given that the defendant is innocent (akin to a frequentist p-value). He argues that if the posterior probability of guilt is to be computed by Bayes' theorem, the prior probability of guilt must be known. This will depend on the incidence of the crime, which is an unusual piece of evidence to consider in a criminal trial. Consider the following three propositions:

A – the known facts and testimony could have arisen if the defendant is guilty.

B – the known facts and testimony could have arisen if the defendant is innocent.

C – the defendant is guilty.

Gardner-Medwin argues that the jury should believe both A and not-B in order to convict. A and not-B implies the truth of C, but the reverse is not true. It is possible that B and C are both true, but in this case he argues that a jury should acquit, even though they know that they will be letting some guilty people go free. See also Lindley's paradox.

Bayesian epistemology

Bayesian epistemology is a movement that advocates for Bayesian inference as a means of justifying the rules of inductive logic.

Karl Popper and David Miller have rejected the idea of Bayesian rationalism, i.e. using Bayes rule to make epistemological inferences: It is prone to the same vicious circle as any other justificationist epistemology, because it presupposes what it attempts to justify. According to this view, a rational interpretation of Bayesian inference would see it merely as a probabilistic version of falsification, rejecting the belief, commonly held by Bayesians, that high likelihood achieved by a series of Bayesian updates would prove the hypothesis beyond any reasonable doubt, or even with likelihood greater than 0.

Other

• The scientific method is sometimes interpreted as an application of Bayesian inference. In this view, Bayes' rule guides (or should guide) the updating of probabilities about hypotheses conditional on new observations or experiments. The Bayesian inference has also been applied to treat stochastic scheduling problems with incomplete information by Cai et al. (2009).
• Bayesian search theory is used to search for lost objects.
• Bayesian inference in phylogeny
• Bayesian tool for methylation analysis
• Bayesian approaches to brain function investigate the brain as a Bayesian mechanism.
• Bayesian inference in ecological studies
• Bayesian inference is used to estimate parameters in stochastic chemical kinetic models
• Bayesian inference in econophysics for currency or prediction of trend changes in financial quotations
• Bayesian inference in marketing
• Bayesian inference in motor learning
• Bayesian inference is used in probabilistic numerics to solve numerical problems

Bayes and Bayesian inference

The problem considered by Bayes in Proposition 9 of his essay, "An Essay Towards Solving a Problem in the Doctrine of Chances", is the posterior distribution for the parameter a (the success rate) of the binomial distribution.

History

Main article: History of statistics § Bayesian statistics

The term Bayesian refers to Thomas Bayes (1701–1761), who proved that probabilistic limits could be placed on an unknown event. However, it was Pierre-Simon Laplace (1749–1827) who introduced (as Principle VI) what is now called Bayes' theorem and used it to address problems in celestial mechanics, medical statistics, reliability, and jurisprudence. Early Bayesian inference, which used uniform priors following Laplace's principle of insufficient reason, was called "inverse probability" (because it infers backwards from observations to parameters, or from effects to causes). After the 1920s, "inverse probability" was largely supplanted by a collection of methods that came to be called frequentist statistics.

In the 20th century, the ideas of Laplace were further developed in two different directions, giving rise to objective and subjective currents in Bayesian practice. In the objective or "non-informative" current, the statistical analysis depends on only the model assumed, the data analyzed, and the method assigning the prior, which differs from one objective Bayesian practitioner to another. In the subjective or "informative" current, the specification of the prior depends on the belief (that is, propositions on which the analysis is prepared to act), which can summarize information from experts, previous studies, etc.

In the 1980s, there was a dramatic growth in research and applications of Bayesian methods, mostly attributed to the discovery of Markov chain Monte Carlo methods, which removed many of the computational problems, and an increasing interest in nonstandard, complex applications. Despite growth of Bayesian research, most undergraduate teaching is still based on frequentist statistics. Nonetheless, Bayesian methods are widely accepted and used, such as for example in the field of machine learning.

Entropy (statistical thermodynamics)

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Entropy_(statistical_thermodynamics)

The concept entropy was first developed by German physicist Rudolf Clausius in the mid-nineteenth century as a thermodynamic property that predicts that certain spontaneous processes are irreversible or impossible. In statistical mechanics, entropy is formulated as a statistical property using probability theory. The statistical entropy perspective was introduced in 1870 by Austrian physicist Ludwig Boltzmann, who established a new field of physics that provided the descriptive linkage between the macroscopic observation of nature and the microscopic view based on the rigorous treatment of large ensembles of microscopic states that constitute thermodynamic systems.

Boltzmann's principle

Main article: Boltzmann's entropy formula

Ludwig Boltzmann defined entropy as a measure of the number of possible microscopic states (microstates) of a system in thermodynamic equilibrium, consistent with its macroscopic thermodynamic properties, which constitute the macrostate of the system. A useful illustration is the example of a sample of gas contained in a container. The easily measurable parameters volume, pressure, and temperature of the gas describe its macroscopic condition (state). At a microscopic level, the gas consists of a vast number of freely moving atoms or molecules, which randomly collide with one another and with the walls of the container. The collisions with the walls produce the macroscopic pressure of the gas, which illustrates the connection between microscopic and macroscopic phenomena.

A microstate of the system is a description of the positions and momenta of all its particles. The large number of particles of the gas provides an infinite number of possible microstates for the sample, but collectively they exhibit a well-defined average of configuration, which is exhibited as the macrostate of the system, to which each individual microstate contribution is negligibly small. The ensemble of microstates comprises a statistical distribution of probability for each microstate, and the group of most probable configurations accounts for the macroscopic state. Therefore, the system can be described as a whole by only a few macroscopic parameters, called the thermodynamic variables: the total energy E, volume V, pressure P, temperature T, and so forth. However, this description is relatively simple only when the system is in a state of equilibrium.

Equilibrium may be illustrated with a simple example of a drop of food coloring falling into a glass of water. The dye diffuses in a complicated manner, which is difficult to precisely predict. However, after sufficient time has passed, the system reaches a uniform color, a state much easier to describe and explain.

Boltzmann formulated a simple relationship between entropy and the number of possible microstates of a system, which is denoted by the symbol Ω. The entropy S is proportional to the natural logarithm of this number:

${\displaystyle S=k_{\text{B}}\ln \mathrm{\Omega }}$

The proportionality constant k_B is one of the fundamental constants of physics and is named the Boltzmann constant in honor of its discoverer.

Boltzmann's entropy describes the system when all the accessible microstates are equally likely. It is the configuration corresponding to the maximum of entropy at equilibrium. The randomness or disorder is maximal, and so is the lack of distinction (or information) of each microstate.

Entropy is a thermodynamic property just like pressure, volume, or temperature. Therefore, it connects the microscopic and the macroscopic world view.

Boltzmann's principle is regarded as the foundation of statistical mechanics.

Gibbs entropy formula

The macroscopic state of a system is characterized by a distribution on the microstates. The entropy of this distribution is given by the Gibbs entropy formula, named after J. Willard Gibbs. For a classical system (i.e., a collection of classical particles) with a discrete set of microstates, if ${\displaystyle E_i}$ is the energy of microstate i, and ${\displaystyle p_i}$ is the probability that it occurs during the system's fluctuations, then the entropy of the system is $${\displaystyle S=-k_{\text{B}}\sum _i{p}_i\ln (p_i)}$$

Entropy changes for systems in a canonical state

A system with a well-defined temperature, i.e., one in thermal equilibrium with a thermal reservoir, has a probability of being in a microstate i given by Boltzmann's distribution.

Changes in the entropy caused by changes in the external constraints are then given by: $${\displaystyle \begin{array}{rl}dS& =-k_{\text{B}}\sum _{i}d{p}_i\ln p_i\\ & =-k_{\text{B}}\sum _{i}d{p}_i(-E_i/k_{\text{B}}T-\ln Z)\\ & =\sum _i{E}_{i}d{p}_i/T\\ & =\sum _i[d(E_i{p}_i)-(d{E}_i)p_i]/T\end{array}}$$

where we have twice used the conservation of probability, Σ dp_i = 0.

Now, Σ_i d(E_i p_i) is the expectation value of the change in the total energy of the system.

If the changes are sufficiently slow, so that the system remains in the same microscopic state, but the state slowly (and reversibly) changes, then Σ_i (dE_i) p_i is the expectation value of the work done on the system through this reversible process, dw_rev.

But from the first law of thermodynamics, dE = δw + δq. Therefore, $${\displaystyle dS=\frac{\delta ⟨q_{\text{rev}}⟩}{T}}$$

In the thermodynamic limit, the fluctuation of the macroscopic quantities from their average values becomes negligible; so this reproduces the definition of entropy from classical thermodynamics, given above.

The quantity ${\displaystyle k_{\text{B}}}$ is the Boltzmann constant. The remaining factor of the equation, the entire summation is dimensionless, since the value ${\displaystyle p_i}$ is a probability and therefore dimensionless, and ln is the natural logarithm. Hence the SI derived units on both sides of the equation are same as heat capacity: $${\displaystyle [S]=[k_{\text{B}}]=\frac{\mathrm{J}}{\mathrm{K}}}$$

This definition remains meaningful even when the system is far away from equilibrium. Other definitions assume that the system is in thermal equilibrium, either as an isolated system, or as a system in exchange with its surroundings. The set of microstates (with probability distribution) on which the sum is done is called a statistical ensemble. Each type of statistical ensemble (micro-canonical, canonical, grand-canonical, etc.) describes a different configuration of the system's exchanges with the outside, varying from a completely isolated system to a system that can exchange one or more quantities with a reservoir, like energy, volume or molecules. In every ensemble, the equilibrium configuration of the system is dictated by the maximization of the entropy of the union of the system and its reservoir, according to the second law of thermodynamics (see the statistical mechanics article).

Neglecting correlations (or, more generally, statistical dependencies) between the states of individual particles will lead to an incorrect probability distribution on the microstates and hence to an overestimate of the entropy. Such correlations occur in any system with nontrivially interacting particles, that is, in all systems more complex than an ideal gas.

This S is almost universally called simply the entropy. It can also be called the statistical entropy or the thermodynamic entropy without changing the meaning. Note the above expression of the statistical entropy is a discretized version of Shannon entropy. The von Neumann entropy formula is an extension of the Gibbs entropy formula to the quantum mechanical case.

It has been shown that the Gibbs Entropy is equal to the classical "heat engine" entropy characterized by ${\displaystyle dS=\frac{\delta Q}{T}}$, and the generalized Boltzmann distribution is a sufficient and necessary condition for this equivalence. Furthermore, the Gibbs Entropy is the only entropy that is equivalent to the classical "heat engine" entropy under the following postulates:

1. The probability density function is proportional to some function of the ensemble parameters and random variables.
2. Thermodynamic state functions are described by ensemble averages of random variables.
3. At infinite temperature, all the microstates have the same probability.

Ensembles

The various ensembles used in statistical thermodynamics are linked to the entropy by the following relations: $${\displaystyle S=k_{\text{B}}\ln {\mathrm{\Omega }}_{\mathrm{m}\mathrm{i}\mathrm{c}}=k_{\text{B}}(\ln Z_{\mathrm{c}\mathrm{a}\mathrm{n}}+\beta \overline{E})=k_{\text{B}}(\ln {\mathcal{Z}}_{\mathrm{g}\mathrm{r}}+\beta (\overline{E}-\mu \overline{N}))}$$

• ${\displaystyle {\mathrm{\Omega }}_{\mathrm{m}\mathrm{i}\mathrm{c}}}$ is the microcanonical partition function
• ${\displaystyle Z_{\mathrm{c}\mathrm{a}\mathrm{n}}}$ is the canonical partition function
• ${\displaystyle {\mathcal{Z}}_{\mathrm{g}\mathrm{r}}}$ is the grand canonical partition function

Order through chaos and the second law of thermodynamics

We can think of Ω as a measure of our lack of knowledge about a system. To illustrate this idea, consider a set of 100 coins, each of which is either heads up or tails up. In this example, let us suppose that the macrostates are specified by the total number of heads and tails, while the microstates are specified by the facings of each individual coin (i.e., the exact order in which heads and tails occur). For the macrostates of 100 heads or 100 tails, there is exactly one possible configuration, so our knowledge of the system is complete. At the opposite extreme, the macrostate which gives us the least knowledge about the system consists of 50 heads and 50 tails in any order, for which there are 100891344545564193334812497256 (100 choose 50) ≈ 10²⁹ possible microstates.

Even when a system is entirely isolated from external influences, its microstate is constantly changing. For instance, the particles in a gas are constantly moving, and thus occupy a different position at each moment of time; their momenta are also constantly changing as they collide with each other or with the container walls. Suppose we prepare the system in an artificially highly ordered equilibrium state. For instance, imagine dividing a container with a partition and placing a gas on one side of the partition, with a vacuum on the other side. If we remove the partition and watch the subsequent behavior of the gas, we will find that its microstate evolves according to some chaotic and unpredictable pattern, and that on average these microstates will correspond to a more disordered macrostate than before. It is possible, but extremely unlikely, for the gas molecules to bounce off one another in such a way that they remain in one half of the container. It is overwhelmingly probable for the gas to spread out to fill the container evenly, which is the new equilibrium macrostate of the system.

This is an example illustrating the second law of thermodynamics:

the total entropy of any isolated thermodynamic system tends to increase over time, approaching a maximum value.

Since its discovery, this idea has been the focus of a great deal of thought, some of it confused. A chief point of confusion is the fact that the Second Law applies only to isolated systems. For example, the Earth is not an isolated system because it is constantly receiving energy in the form of sunlight. In contrast, the universe may be considered an isolated system, so that its total entropy is constantly increasing. (Needs clarification. See: Second law of thermodynamics#cite note-Grandy 151-21)

Counting of microstates

In classical statistical mechanics, the number of microstates is actually uncountably infinite, since the properties of classical systems are continuous. For example, a microstate of a classical ideal gas is specified by the positions and momenta of all the atoms, which range continuously over the real numbers. If we want to define Ω, we have to come up with a method of grouping the microstates together to obtain a countable set. This procedure is known as coarse graining. In the case of the ideal gas, we count two states of an atom as the "same" state if their positions and momenta are within δx and δp of each other. Since the values of δx and δp can be chosen arbitrarily, the entropy is not uniquely defined. It is defined only up to an additive constant. (As we will see, the thermodynamic definition of entropy is also defined only up to a constant.)

To avoid coarse graining one can take the entropy as defined by the H-theorem.

${\displaystyle S=-k_{\mathrm{B}}{H}_{\mathrm{B}}:=-k_{\mathrm{B}}\int f(q_i,p_i)\ln f(q_i,p_i)d{q}_{1}d{p}_1\cdots d{q}_{N}d{p}_N}$

However, this ambiguity can be resolved with quantum mechanics. The quantum state of a system can be expressed as a superposition of "basis" states, which can be chosen to be energy eigenstates (i.e. eigenstates of the quantum Hamiltonian). Usually, the quantum states are discrete, even though there may be an infinite number of them. For a system with some specified energy E, one takes Ω to be the number of energy eigenstates within a macroscopically small energy range between E and E + δE. In the thermodynamical limit, the specific entropy becomes independent on the choice of δE.

An important result, known as Nernst's theorem or the third law of thermodynamics, states that the entropy of a system at zero absolute temperature is a well-defined constant. This is because a system at zero temperature exists in its lowest-energy state, or ground state, so that its entropy is determined by the degeneracy of the ground state. Many systems, such as crystal lattices, have a unique ground state, and (since ln(1) = 0) this means that they have zero entropy at absolute zero. Other systems have more than one state with the same, lowest energy, and have a non-vanishing "zero-point entropy". For instance, ordinary ice has a zero-point entropy of 3.41 J/(mol⋅K), because its underlying crystal structure possesses multiple configurations with the same energy (a phenomenon known as geometrical frustration).

The third law of thermodynamics states that the entropy of a perfect crystal at absolute zero (0 K) is zero. This means that nearly all molecular motion should cease. The oscillator equation for predicting quantized vibrational levels shows that even when the vibrational quantum number is 0, the molecule still has vibrational energy:

${\displaystyle E_{\nu }=h{\nu }_0(n+\frac{1}{2})}$

where ${\displaystyle h}$ is the Planck constant, ${\displaystyle {\nu }_0}$ is the characteristic frequency of the vibration, and ${\displaystyle n}$ is the vibrational quantum number. Even when ${\displaystyle n=0}$ (the zero-point energy), ${\displaystyle E_n}$ does not equal 0, in adherence to the Heisenberg uncertainty principle.