Gaussian process

From Wikipedia, the free encyclopedia

https://en.wikipedia.org/wiki/Gaussian_process 

In probability theory and statistics, a Gaussian process is a stochastic process (a collection of random variables indexed by time or space), such that every finite collection of those random variables has a multivariate normal distribution, i.e. every finite linear combination of them is normally distributed. The distribution of a Gaussian process is the joint distribution of all those (infinitely many) random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space.

The concept of Gaussian processes is named after Carl Friedrich Gauss because it is based on the notion of the Gaussian distribution (normal distribution). Gaussian processes can be seen as an infinite-dimensional generalization of multivariate normal distributions.

Gaussian processes are useful in statistical modelling, benefiting from properties inherited from the normal distribution. For example, if a random process is modelled as a Gaussian process, the distributions of various derived quantities can be obtained explicitly. Such quantities include the average value of the process over a range of times and the error in estimating the average using sample values at a small set of times. While exact models often scale poorly as the amount of data increases, multiple approximation methods have been developed which often retain good accuracy while drastically reducing computation time.

Definition

A time continuous stochastic process ${\displaystyle \left\{X_t;t\in T\right\}}$ is Gaussian if and only if for every finite set of indices ${\displaystyle t_1,\dots ,t_k}$ in the index set ${\displaystyle T}$

$${\displaystyle {\mathbf{X}}_{t_1,\dots ,t_k}=(X_{t_1},\dots ,X_{t_k})}$$

is a multivariate Gaussian random variable. That is the same as saying every linear combination of ${\displaystyle (X_{t_1},\dots ,X_{t_k})}$ has a univariate normal (or Gaussian) distribution.

Using characteristic functions of random variables, the Gaussian property can be formulated as follows: ${\displaystyle \left\{X_t;t\in T\right\}}$ is Gaussian if and only if, for every finite set of indices ${\displaystyle t_1,\dots ,t_k}$, there are real-valued ${\displaystyle {\sigma }_{\ell j}}$, ${\displaystyle {\mu }_{\ell }}$ with ${\displaystyle {\sigma }_{jj}>0}$ such that the following equality holds for all ${\displaystyle s_1,s_2,\dots ,s_k\in \mathbb{R}}$

$${\displaystyle \mathrm{E}\left(\exp \left(i\sum _{\ell =1}^k{s}_{\ell }{\mathbf{X}}_{t_{\ell }}\right)\right)=\exp \left(-\frac{1}{2}\sum _{\ell ,j}{\sigma }_{\ell j}{s}_{\ell }{s}_j+i\sum _{\ell }{\mu }_{\ell }{s}_{\ell }\right),}$$

where ${\displaystyle i}$ denotes the imaginary unit such that ${\displaystyle i^2=-1}$.

The numbers ${\displaystyle {\sigma }_{\ell j}}$ and ${\displaystyle {\mu }_{\ell }}$ can be shown to be the covariances and means of the variables in the process.

Variance

The variance of a Gaussian process is finite at any time ${\displaystyle t}$, formally

$${\displaystyle \mathrm{var}[X(t)]=\mathrm{E}\left[{\left|X(t)-\mathrm{E}[X(t)]\right|}^2\right]<\mathrm{\infty }\text{for all }t\in T.}$$

Stationarity

For general stochastic processes strict-sense stationarity implies wide-sense stationarity but not every wide-sense stationary stochastic process is strict-sense stationary. However, for a Gaussian stochastic process the two concepts are equivalent.

A Gaussian stochastic process is strict-sense stationary if, and only if, it is wide-sense stationary.

Example

There is an explicit representation for stationary Gaussian processes. A simple example of this representation is

$${\displaystyle X_t=\cos (at){\xi }_1+\sin (at){\xi }_2}$$

where ${\displaystyle {\xi }_1}$ and ${\displaystyle {\xi }_2}$ are independent random variables with the standard normal distribution.

Covariance functions

Main article: Covariance function

Further information: Variogram

A key fact of Gaussian processes is that they can be completely defined by their second-order statistics. Thus, if a Gaussian process is assumed to have mean zero, defining the covariance function completely defines the process' behaviour. Importantly the non-negative definiteness of this function enables its spectral decomposition using the Karhunen–Loève expansion. Basic aspects that can be defined through the covariance function are the process' stationarity, isotropy, smoothness and periodicity.

Stationarity refers to the process' behaviour regarding the separation of any two points ${\displaystyle x}$ and ${\displaystyle x^{\prime }}$. If the process is stationary, the covariance function depends only on ${\displaystyle x-x^{\prime }}$. For example, the Ornstein–Uhlenbeck process is stationary.

If the process depends only on ${\displaystyle |x-x^{\prime }|}$, the Euclidean distance (not the direction) between ${\displaystyle x}$ and ${\displaystyle x^{\prime }}$, then the process is considered isotropic. A process that is concurrently stationary and isotropic is considered to be homogeneous; in practice these properties reflect the differences (or rather the lack of them) in the behaviour of the process given the location of the observer.

Ultimately Gaussian processes translate as taking priors on functions and the smoothness of these priors can be induced by the covariance function. If we expect that for "near-by" input points ${\displaystyle x}$ and ${\displaystyle x^{\prime }}$ their corresponding output points ${\displaystyle y}$ and ${\displaystyle y^{\prime }}$ to be "near-by" also, then the assumption of continuity is present. If we wish to allow for significant displacement then we might choose a rougher covariance function. Extreme examples of the behaviour is the Ornstein–Uhlenbeck covariance function and the squared exponential where the former is never differentiable and the latter infinitely differentiable.

Periodicity refers to inducing periodic patterns within the behaviour of the process. Formally, this is achieved by mapping the input ${\displaystyle x}$ to a two dimensional vector ${\displaystyle u(x)=\left(\cos (x),\sin (x)\right)}$.

Usual covariance functions

The effect of choosing different kernels on the prior function distribution of the Gaussian process. Left is a squared exponential kernel. Middle is Brownian. Right is quadratic.

There are a number of common covariance functions:

• Constant : ${\displaystyle K_{\mathrm{C}}(x,x^{\prime })=C}$
• Linear: ${\displaystyle K_{\mathrm{L}}(x,x^{\prime })=x^{\mathsf{T}}{x}^{\prime }}$
• white Gaussian noise: ${\displaystyle K_{\mathrm{GN}}(x,x^{\prime })={\sigma }^2{\delta }_{x,x^{\prime }}}$
• Squared exponential: ${\displaystyle K_{\mathrm{SE}}(x,x^{\prime })=\exp \left(-\frac{|d{|}^2}{2{\ell }^2}\right)}$
• Ornstein–Uhlenbeck: ${\displaystyle K_{\mathrm{OU}}(x,x^{\prime })=\exp \left(-\frac{|d|}{\ell }\right)}$
• Matérn: ${\displaystyle K_{\mathrm{Matern}}(x,x^{\prime })=\frac{2^{1-\nu }}{\mathrm{\Gamma }(\nu )}{\left(\frac{\sqrt{2\nu }|d|}{\ell }\right)}^{\nu }{K}_{\nu }\left(\frac{\sqrt{2\nu }|d|}{\ell }\right)}$
• Periodic: ${\displaystyle K_{\mathrm{P}}(x,x^{\prime })=\exp \left(-\frac{2{\sin }^2\left(\frac{d}{2}\right)}{{\ell }^2}\right)}$
• Rational quadratic: ${\displaystyle K_{\mathrm{RQ}}(x,x^{\prime })={\left(1+|d{|}^2\right)}^{-\alpha },\alpha \ge 0}$

Here ${\displaystyle d=x-x^{\prime }}$. The parameter ${\displaystyle \ell }$ is the characteristic length-scale of the process (practically, "how close" two points ${\displaystyle x}$ and ${\displaystyle x^{\prime }}$ have to be to influence each other significantly), ${\displaystyle \delta }$ is the Kronecker delta and ${\displaystyle \sigma }$ the standard deviation of the noise fluctuations. Moreover, ${\displaystyle K_{\nu }}$ is the modified Bessel function of order ${\displaystyle \nu }$ and ${\displaystyle \mathrm{\Gamma }(\nu )}$ is the gamma function evaluated at ${\displaystyle \nu }$. Importantly, a complicated covariance function can be defined as a linear combination of other simpler covariance functions in order to incorporate different insights about the data-set at hand.

The inferential results are dependent on the values of the hyperparameters ${\displaystyle \theta }$ (e.g. ${\displaystyle \ell }$ and ${\displaystyle \sigma }$) defining the model's behaviour. A popular choice for ${\displaystyle \theta }$ is to provide maximum a posteriori (MAP) estimates of it with some chosen prior. If the prior is very near uniform, this is the same as maximizing the marginal likelihood of the process; the marginalization being done over the observed process values ${\displaystyle y}$. This approach is also known as maximum likelihood II, evidence maximization, or empirical Bayes.

Continuity

For a Gaussian process, continuity in probability is equivalent to mean-square continuity, and continuity with probability one is equivalent to sample continuity. The latter implies, but is not implied by, continuity in probability. Continuity in probability holds if and only if the mean and autocovariance are continuous functions. In contrast, sample continuity was challenging even for stationary Gaussian processes (as probably noted first by Andrey Kolmogorov), and more challenging for more general processes. As usual, by a sample continuous process one means a process that admits a sample continuous modification. 

Stationary case

For a stationary Gaussian process ${\displaystyle X=(X_t{)}_{t\in \mathbb{R}},}$ some conditions on its spectrum are sufficient for sample continuity, but fail to be necessary. A necessary and sufficient condition, sometimes called Dudley–Fernique theorem, involves the function ${\displaystyle \sigma }$ defined by

$${\displaystyle \sigma (h)=\sqrt{\mathbb{E}(X(t+h)-X(t){)}^2}}$$

(the right-hand side does not depend on ${\displaystyle t}$ due to stationarity). Continuity of ${\displaystyle X}$ in probability is equivalent to continuity of ${\displaystyle \sigma }$ at ${\displaystyle 0.}$ When convergence of ${\displaystyle \sigma (h)}$ to ${\displaystyle 0}$ (as ${\displaystyle h\to 0}$) is too slow, sample continuity of ${\displaystyle X}$ may fail. Convergence of the following integrals matters:

$${\displaystyle I(\sigma )={\int }_0^1\frac{\sigma (h)}{h\sqrt{\log (1/h)}}dh={\int }_0^{\mathrm{\infty }}2\sigma (e^{-x^2})dx,}$$

these two integrals being equal according to integration by substitution $h=e^{-x^2},$ $x=\sqrt{\log (1/h)}.$ The first integrand need not be bounded as ${\displaystyle h\to 0+,}$ thus the integral may converge (${\displaystyle I(\sigma )<\mathrm{\infty }}$) or diverge (${\displaystyle I(\sigma )=\mathrm{\infty }}$). Taking for example $\sigma (e^{-x^2})=\frac{1}{x^a}$ for large ${\displaystyle x,}$ that is, $\sigma (h)=(\log (1/h){)}^{-a/2}$ for small ${\displaystyle h,}$ one obtains ${\displaystyle I(\sigma )<\mathrm{\infty }}$ when ${\displaystyle a>1,}$ and ${\displaystyle I(\sigma )=\mathrm{\infty }}$ when ${\displaystyle 0<a\le 1.}$ In these two cases the function ${\displaystyle \sigma }$ is increasing on ${\displaystyle [0,\mathrm{\infty }),}$ but generally it is not. Moreover, the condition

(*)   there exists ${\displaystyle \epsilon >0}$ such that ${\displaystyle \sigma }$ is monotone on ${\displaystyle [0,\epsilon ]}$

does not follow from continuity of ${\displaystyle \sigma }$ and the evident relations ${\displaystyle \sigma (h)\ge 0}$ (for all ${\displaystyle h}$) and ${\displaystyle \sigma (0)=0.}$

Theorem 1 — Let ${\displaystyle \sigma }$ be continuous and satisfy (*). Then the condition ${\displaystyle I(\sigma )<\mathrm{\infty }}$ is necessary and sufficient for sample continuity of ${\displaystyle X.}$

Some history. Sufficiency was announced by Xavier Fernique in 1964, but the first proof was published by Richard M. Dudley in 1967. Necessity was proved by Michael B. Marcus and Lawrence Shepp in 1970.

There exist sample continuous processes ${\displaystyle X}$ such that ${\displaystyle I(\sigma )=\mathrm{\infty };}$ they violate condition (*). An example found by Marcus and Shepp  is a random lacunary Fourier series

$${\displaystyle X_t=\sum _{n=1}^{\mathrm{\infty }}{c}_n({\xi }_n\cos {\lambda }_{n}t+{\eta }_n\sin {\lambda }_{n}t),}$$

where ${\displaystyle {\xi }_1,{\eta }_1,{\xi }_2,{\eta }_2,\dots }$ are independent random variables with standard normal distribution; frequencies ${\displaystyle 0<{\lambda }_1<{\lambda }_2<\dots }$ are a fast growing sequence; and coefficients ${\displaystyle c_n>0}$ satisfy $\sum _n{c}_n<\mathrm{\infty }.$ The latter relation implies $\mathbb{E}\sum _n{c}_n(|{\xi }_n|+|{\eta }_n|)=\sum _n{c}_n\mathbb{E}(|{\xi }_n|+|{\eta }_n|)=\text{const}\cdot \sum _n{c}_n<\mathrm{\infty },$ whence $\sum _n{c}_n(|{\xi }_n|+|{\eta }_n|)<\mathrm{\infty }$ almost surely, which ensures uniform convergence of the Fourier series almost surely, and sample continuity of ${\displaystyle X.}$

Autocorrelation of a random lacunary Fourier series

Its autocovariation function

$${\displaystyle \mathbb{E}{X}_t{X}_{t+h}=\sum _{n=1}^{\mathrm{\infty }}{c}_n^2\cos {\lambda }_{n}h}$$

is nowhere monotone (see the picture), as well as the corresponding function ${\displaystyle \sigma ,}$

$${\displaystyle \sigma (h)=\sqrt{2\mathbb{E}{X}_t{X}_t-2\mathbb{E}{X}_t{X}_{t+h}}=2\sqrt{\sum _{n=1}^{\mathrm{\infty }}{c}_n^2{\sin }^2\frac{{\lambda }_{n}h}{2}}.}$$

Brownian motion as the integral of Gaussian processes

A Wiener process (also known as Brownian motion) is the integral of a white noise generalized Gaussian process. It is not stationary, but it has stationary increments.

The Ornstein–Uhlenbeck process is a stationary Gaussian process.

The Brownian bridge is (like the Ornstein–Uhlenbeck process) an example of a Gaussian process whose increments are not independent.

The fractional Brownian motion is a Gaussian process whose covariance function is a generalisation of that of the Wiener process.

Driscoll's zero-one law

Driscoll's zero-one law is a result characterizing the sample functions generated by a Gaussian process.

Let ${\displaystyle f}$ be a mean-zero Gaussian process ${\displaystyle \left\{X_t;t\in T\right\}}$ with non-negative definite covariance function ${\displaystyle K}$. Let ${\displaystyle \mathcal{H}(R)}$ be a Reproducing kernel Hilbert space with positive definite kernel ${\displaystyle R}$.

Then

$${\displaystyle \underset{n\to \mathrm{\infty }}{lim}\mathrm{tr}[K_n{R}_n^{-1}]<\mathrm{\infty },}$$

where ${\displaystyle K_n}$ and ${\displaystyle R_n}$ are the covariance matrices of all possible pairs of ${\displaystyle n}$ points, implies

$${\displaystyle Pr[f\in \mathcal{H}(R)]=1.}$$

Moreover,

$${\displaystyle \underset{n\to \mathrm{\infty }}{lim}\mathrm{tr}[K_n{R}_n^{-1}]=\mathrm{\infty }}$$

implies 

$${\displaystyle Pr[f\in \mathcal{H}(R)]=0.}$$

This has significant implications when ${\displaystyle K=R}$, as

$${\displaystyle \underset{n\to \mathrm{\infty }}{lim}\mathrm{tr}[R_n{R}_n^{-1}]=\underset{n\to \mathrm{\infty }}{lim}\mathrm{tr}[I]=\underset{n\to \mathrm{\infty }}{lim}n=\mathrm{\infty }.}$$

As such, almost all sample paths of a mean-zero Gaussian process with positive definite kernel ${\displaystyle K}$ will lie outside of the Hilbert space ${\displaystyle \mathcal{H}(K)}$.

Linearly constrained Gaussian processes

For many applications of interest some pre-existing knowledge about the system at hand is already given. Consider e.g. the case where the output of the Gaussian process corresponds to a magnetic field; here, the real magnetic field is bound by Maxwell's equations and a way to incorporate this constraint into the Gaussian process formalism would be desirable as this would likely improve the accuracy of the algorithm.

A method on how to incorporate linear constraints into Gaussian processes already exists:

Consider the (vector valued) output function ${\displaystyle f(x)}$ which is known to obey the linear constraint (i.e. ${\displaystyle {\mathcal{F}}_X}$ is a linear operator)

$${\displaystyle {\mathcal{F}}_X(f(x))=0.}$$

Then the constraint ${\displaystyle {\mathcal{F}}_X}$ can be fulfilled by choosing ${\displaystyle f(x)={\mathcal{G}}_X(g(x))}$, where ${\displaystyle g(x)\sim \mathcal{G}\mathcal{P}({\mu }_g,K_g)}$ is modelled as a Gaussian process, and finding ${\displaystyle {\mathcal{G}}_X}$ such that

$${\displaystyle {\mathcal{F}}_X({\mathcal{G}}_X(g))=0\mathrm{\forall }g.}$$

Given ${\displaystyle {\mathcal{G}}_X}$ and using the fact that Gaussian processes are closed under linear transformations, the Gaussian process for ${\displaystyle f}$ obeying constraint ${\displaystyle {\mathcal{F}}_X}$ becomes

$${\displaystyle f(x)={\mathcal{G}}_{X}g\sim \mathcal{G}\mathcal{P}({\mathcal{G}}_X{\mu }_g,{\mathcal{G}}_X{K}_g{\mathcal{G}}_{X^{\prime }}^{\mathsf{T}}).}$$

Hence, linear constraints can be encoded into the mean and covariance function of a Gaussian process.

Applications

An example of Gaussian Process Regression (prediction) compared with other regression models.

A Gaussian process can be used as a prior probability distribution over functions in Bayesian inference. Given any set of N points in the desired domain of your functions, take a multivariate Gaussian whose covariance matrix parameter is the Gram matrix of your N points with some desired kernel, and sample from that Gaussian. For solution of the multi-output prediction problem, Gaussian process regression for vector-valued function was developed. In this method, a 'big' covariance is constructed, which describes the correlations between all the input and output variables taken in N points in the desired domain. This approach was elaborated in detail for the matrix-valued Gaussian processes and generalised to processes with 'heavier tails' like Student-t processes.

Inference of continuous values with a Gaussian process prior is known as Gaussian process regression, or kriging; extending Gaussian process regression to multiple target variables is known as cokriging. Gaussian processes are thus useful as a powerful non-linear multivariate interpolation tool.

Gaussian processes are also commonly used to tackle numerical analysis problems such as numerical integration, solving differential equations, or optimisation in the field of probabilistic numerics.

Gaussian processes can also be used in the context of mixture of experts models, for example. The underlying rationale of such a learning framework consists in the assumption that a given mapping cannot be well captured by a single Gaussian process model. Instead, the observation space is divided into subsets, each of which is characterized by a different mapping function; each of these is learned via a different Gaussian process component in the postulated mixture.

In the natural sciences, Gaussian processes have found use as probabilistic models of astronomical time series and as predictors of molecular properties.

Gaussian process prediction, or Kriging

Further information: Kriging

Gaussian Process Regression (prediction) with a squared exponential kernel. Left plot are draws from the prior function distribution. Middle are draws from the posterior. Right is mean prediction with one standard deviation shaded.

When concerned with a general Gaussian process regression problem (Kriging), it is assumed that for a Gaussian process ${\displaystyle f}$ observed at coordinates ${\displaystyle x}$, the vector of values ${\displaystyle f(x)}$ is just one sample from a multivariate Gaussian distribution of dimension equal to number of observed coordinates ${\displaystyle n}$. Therefore, under the assumption of a zero-mean distribution, ${\displaystyle f(x^{\prime })\sim N(0,K(\theta ,x,x^{\prime }))}$, where ${\displaystyle K(\theta ,x,x^{\prime })}$ is the covariance matrix between all possible pairs ${\displaystyle (x,x^{\prime })}$ for a given set of hyperparameters θ. As such the log marginal likelihood is:

$${\displaystyle \log p(f(x^{\prime })\mid \theta ,x)=-\frac{1}{2}\left(f(x{)}^{\mathsf{T}}K(\theta ,x,x^{\prime }{)}^{-1}f(x^{\prime })+\log det(K(\theta ,x,x^{\prime }))+n\log 2\pi \right)}$$

and maximizing this marginal likelihood towards θ provides the complete specification of the Gaussian process f. One can briefly note at this point that the first term corresponds to a penalty term for a model's failure to fit observed values and the second term to a penalty term that increases proportionally to a model's complexity. Having specified θ, making predictions about unobserved values ${\displaystyle f(x^{\ast })}$ at coordinates x* is then only a matter of drawing samples from the predictive distribution ${\displaystyle p(y^{\ast }\mid x^{\ast },f(x),x)=N(y^{\ast }\mid A,B)}$ where the posterior mean estimate A is defined as

$${\displaystyle A=K(\theta ,x^{\ast },x)K(\theta ,x,x^{\prime }{)}^{-1}f(x)}$$

and the posterior variance estimate B is defined as:

$${\displaystyle B=K(\theta ,x^{\ast },x^{\ast })-K(\theta ,x^{\ast },x)K(\theta ,x,x^{\prime }{)}^{-1}K(\theta ,x^{\ast },x{)}^{\mathsf{T}}}$$

where ${\displaystyle K(\theta ,x^{\ast },x)}$ is the covariance between the new coordinate of estimation x* and all other observed coordinates x for a given hyperparameter vector θ, ${\displaystyle K(\theta ,x,x^{\prime })}$ and ${\displaystyle f(x)}$ are defined as before and ${\displaystyle K(\theta ,x^{\ast },x^{\ast })}$ is the variance at point x* as dictated by θ. It is important to note that practically the posterior mean estimate of ${\displaystyle f(x^{\ast })}$ (the "point estimate") is just a linear combination of the observations ${\displaystyle f(x)}$; in a similar manner the variance of ${\displaystyle f(x^{\ast })}$ is actually independent of the observations ${\displaystyle f(x)}$. A known bottleneck in Gaussian process prediction is that the computational complexity of inference and likelihood evaluation is cubic in the number of points |x|, and as such can become unfeasible for larger data sets. Works on sparse Gaussian processes, that usually are based on the idea of building a representative set for the given process f, try to circumvent this issue. The kriging method can be used in the latent level of a nonlinear mixed-effects model for a spatial functional prediction: this technique is called the latent kriging.

Often, the covariance has the form $K(\theta ,x,x^{\prime })=\frac{1}{{\sigma }^2}\tilde{K}(\theta ,x,x^{\prime })$, where ${\displaystyle {\sigma }^2}$ is a scaling parameter. Examples are the Matérn class covariance functions. If this scaling parameter ${\displaystyle {\sigma }^2}$ is either known or unknown (i.e. must be marginalized), then the posterior probability, ${\displaystyle p(\theta \mid D)}$, i.e. the probability for the hyperparameters ${\displaystyle \theta }$ given a set of data pairs ${\displaystyle D}$ of observations of ${\displaystyle x}$ and ${\displaystyle f(x)}$, admits an analytical expression.

Bayesian neural networks as Gaussian processes

Further information: Neural network Gaussian process

Bayesian neural networks are a particular type of Bayesian network that results from treating deep learning and artificial neural network models probabilistically, and assigning a prior distribution to their parameters. Computation in artificial neural networks is usually organized into sequential layers of artificial neurons. The number of neurons in a layer is called the layer width. As layer width grows large, many Bayesian neural networks reduce to a Gaussian process with a closed form compositional kernel. This Gaussian process is called the Neural Network Gaussian Process (NNGP). It allows predictions from Bayesian neural networks to be more efficiently evaluated, and provides an analytic tool to understand deep learning models.

Computational issues

See also: Gaussian process approximations

In practical applications, Gaussian process models are often evaluated on a grid leading to multivariate normal distributions. Using these models for prediction or parameter estimation using maximum likelihood requires evaluating a multivariate Gaussian density, which involves calculating the determinant and the inverse of the covariance matrix. Both of these operations have cubic computational complexity which means that even for grids of modest sizes, both operations can have a prohibitive computational cost. This drawback led to the development of multiple approximation methods.

Gene duplication

From Wikipedia, the free encyclopedia
https://en.wikipedia.org/wiki/Gene_duplication

Gene duplication (or chromosomal duplication or gene amplification) is a major mechanism through which new genetic material is generated during molecular evolution. It can be defined as any duplication of a region of DNA that contains a gene. Gene duplications can arise as products of several types of errors in DNA replication and repair machinery as well as through fortuitous capture by selfish genetic elements. Common sources of gene duplications include ectopic recombination, retrotransposition event, aneuploidy, polyploidy, and replication slippage.

Mechanisms of duplication

Ectopic recombination

Duplications arise from an event termed unequal crossing-over that occurs during meiosis between misaligned homologous chromosomes. The chance of it happening is a function of the degree of sharing of repetitive elements between two chromosomes. The products of this recombination are a duplication at the site of the exchange and a reciprocal deletion. Ectopic recombination is typically mediated by sequence similarity at the duplicate breakpoints, which form direct repeats. Repetitive genetic elements such as transposable elements offer one source of repetitive DNA that can facilitate recombination, and they are often found at duplication breakpoints in plants and mammals.

Schematic of a region of a chromosome before and after a duplication event

Replication slippage

Replication slippage is an error in DNA replication that can produce duplications of short genetic sequences. During replication DNA polymerase begins to copy the DNA. At some point during the replication process, the polymerase dissociates from the DNA and replication stalls. When the polymerase reattaches to the DNA strand, it aligns the replicating strand to an incorrect position and incidentally copies the same section more than once. Replication slippage is also often facilitated by repetitive sequences, but requires only a few bases of similarity.

Retrotransposition

Retrotransposons, mainly L1, can occasionally act on cellular mRNA. Transcripts are reverse transcribed to DNA and inserted into random place in the genome, creating retrogenes. Resulting sequence usually lack introns and often contain poly, sequences that are also integrated into the genome. Many retrogenes display changes in gene regulation in comparison to their parental gene sequences, which sometimes results in novel functions. Retrogenes can move between different chromosomes to shape chromosomal evolution.

Aneuploidy

Aneuploidy occurs when nondisjunction at a single chromosome results in an abnormal number of chromosomes. Aneuploidy is often harmful and in mammals regularly leads to spontaneous abortions (miscarriages). Some aneuploid individuals are viable, for example trisomy 21 in humans, which leads to Down syndrome. Aneuploidy often alters gene dosage in ways that are detrimental to the organism; therefore, it is unlikely to spread through populations.

Polyploidy

Polyploidy, or whole genome duplication is a product of nondisjunction during meiosis which results in additional copies of the entire genome. Polyploidy is common in plants, but it has also occurred in animals, with two rounds of whole genome duplication (2R event) in the vertebrate lineage leading to humans. It has also occurred in the hemiascomycete yeasts ~100 mya.

After a whole genome duplication, there is a relatively short period of genome instability, extensive gene loss, elevated levels of nucleotide substitution and regulatory network rewiring. In addition, gene dosage effects play a significant role. Thus, most duplicates are lost within a short period, however, a considerable fraction of duplicates survive. Interestingly, genes involved in regulation are preferentially retained. Furthermore, retention of regulatory genes, most notably the Hox genes, has led to adaptive innovation.

Rapid evolution and functional divergence have been observed at the level of the transcription of duplicated genes, usually by point mutations in short transcription factor binding motifs. Furthermore, rapid evolution of protein phosphorylation motifs, usually embedded within rapidly evolving intrinsically disordered regions is another contributing factor for survival and rapid adaptation/neofunctionalization of duplicate genes. Thus, a link seems to exist between gene regulation (at least at the post-translational level) and genome evolution.

Polyploidy is also a well known source of speciation, as offspring, which have different numbers of chromosomes compared to parent species, are often unable to interbreed with non-polyploid organisms. Whole genome duplications are thought to be less detrimental than aneuploidy as the relative dosage of individual genes should be the same.

As an evolutionary event

Evolutionary fate of duplicate genes

Rate of gene duplication

Comparisons of genomes demonstrate that gene duplications are common in most species investigated. This is indicated by variable copy numbers (copy number variation) in the genome of humans or fruit flies. However, it has been difficult to measure the rate at which such duplications occur. Recent studies yielded a first direct estimate of the genome-wide rate of gene duplication in C. elegans, the first multicellular eukaryote for which such as estimate became available. The gene duplication rate in C. elegans is on the order of 10⁻⁷ duplications/gene/generation, that is, in a population of 10 million worms, one will have a gene duplication per generation. This rate is two orders of magnitude greater than the spontaneous rate of point mutation per nucleotide site in this species. Older (indirect) studies reported locus-specific duplication rates in bacteria, Drosophila, and humans ranging from 10⁻³ to 10⁻⁷/gene/generation.

Neofunctionalization

Main article: Neofunctionalization

Gene duplications are an essential source of genetic novelty that can lead to evolutionary innovation. Duplication creates genetic redundancy, where the second copy of the gene is often free from selective pressure—that is, mutations of it have no deleterious effects to its host organism. If one copy of a gene experiences a mutation that affects its original function, the second copy can serve as a 'spare part' and continue to function correctly. Thus, duplicate genes accumulate mutations faster than a functional single-copy gene, over generations of organisms, and it is possible for one of the two copies to develop a new and different function. Some examples of such neofunctionalization is the apparent mutation of a duplicated digestive gene in a family of ice fish into an antifreeze gene and duplication leading to a novel snake venom gene and the synthesis of 1 beta-hydroxytestosterone in pigs.

Gene duplication is believed to play a major role in evolution; this stance has been held by members of the scientific community for over 100 years. Susumu Ohno was one of the most famous developers of this theory in his classic book Evolution by gene duplication (1970). Ohno argued that gene duplication is the most important evolutionary force since the emergence of the universal common ancestor. Major genome duplication events can be quite common. It is believed that the entire yeast genome underwent duplication about 100 million years ago. Plants are the most prolific genome duplicators. For example, wheat is hexaploid (a kind of polyploid), meaning that it has six copies of its genome.

Subfunctionalization

Main article: Subfunctionalization

Another possible fate for duplicate genes is that both copies are equally free to accumulate degenerative mutations, so long as any defects are complemented by the other copy. This leads to a neutral "subfunctionalization" (a process of constructive neutral evolution) or DDC (duplication-degeneration-complementation) model, in which the functionality of the original gene is distributed among the two copies. Neither gene can be lost, as both now perform important non-redundant functions, but ultimately neither is able to achieve novel functionality.

Subfunctionalization can occur through neutral processes in which mutations accumulate with no detrimental or beneficial effects. However, in some cases subfunctionalization can occur with clear adaptive benefits. If an ancestral gene is pleiotropic and performs two functions, often neither one of these two functions can be changed without affecting the other function. In this way, partitioning the ancestral functions into two separate genes can allow for adaptive specialization of subfunctions, thereby providing an adaptive benefit.

Loss

Often the resulting genomic variation leads to gene dosage dependent neurological disorders such as Rett-like syndrome and Pelizaeus–Merzbacher disease. Such detrimental mutations are likely to be lost from the population and will not be preserved or develop novel functions. However, many duplications are, in fact, not detrimental or beneficial, and these neutral sequences may be lost or may spread through the population through random fluctuations via genetic drift.

Identifying duplications in sequenced genomes

Criteria and single genome scans

The two genes that exist after a gene duplication event are called paralogs and usually code for proteins with a similar function and/or structure. By contrast, orthologous genes present in different species which are each originally derived from the same ancestral sequence. (See Homology of sequences in genetics).

It is important (but often difficult) to differentiate between paralogs and orthologs in biological research. Experiments on human gene function can often be carried out on other species if a homolog to a human gene can be found in the genome of that species, but only if the homolog is orthologous. If they are paralogs and resulted from a gene duplication event, their functions are likely to be too different. One or more copies of duplicated genes that constitute a gene family may be affected by insertion of transposable elements that causes significant variation between them in their sequence and finally may become responsible for divergent evolution. This may also render the chances and the rate of gene conversion between the homologs of gene duplicates due to less or no similarity in their sequences.

Paralogs can be identified in single genomes through a sequence comparison of all annotated gene models to one another. Such a comparison can be performed on translated amino acid sequences (e.g. BLASTp, tBLASTx) to identify ancient duplications or on DNA nucleotide sequences (e.g. BLASTn, megablast) to identify more recent duplications. Most studies to identify gene duplications require reciprocal-best-hits or fuzzy reciprocal-best-hits, where each paralog must be the other's single best match in a sequence comparison.

Most gene duplications exist as low copy repeats (LCRs), rather highly repetitive sequences like transposable elements. They are mostly found in pericentronomic, subtelomeric and interstitial regions of a chromosome. Many LCRs, due to their size (>1Kb), similarity, and orientation, are highly susceptible to duplications and deletions.

Genomic microarrays detect duplications

Technologies such as genomic microarrays, also called array comparative genomic hybridization (array CGH), are used to detect chromosomal abnormalities, such as microduplications, in a high throughput fashion from genomic DNA samples. In particular, DNA microarray technology can simultaneously monitor the expression levels of thousands of genes across many treatments or experimental conditions, greatly facilitating the evolutionary studies of gene regulation after gene duplication or speciation.

Next generation sequencing

Gene duplications can also be identified through the use of next-generation sequencing platforms. The simplest means to identify duplications in genomic resequencing data is through the use of paired-end sequencing reads. Tandem duplications are indicated by sequencing read pairs which map in abnormal orientations. Through a combination of increased sequence coverage and abnormal mapping orientation, it is possible to identify duplications in genomic sequencing data.

Nomenclature

Human karyotype with annotated bands and sub-bands as used for the nomenclature of chromosome abnormalities. It shows dark and white regions as seen on G banding. Each row is vertically aligned at centromere level. It shows 22 homologous autosomal chromosome pairs, both the female (XX) and male (XY) versions of the two sex chromosomes, as well as the mitochondrial genome (at bottom left).

Further information: Karyotype

The International System for Human Cytogenomic Nomenclature (ISCN) is an international standard for human chromosome nomenclature, which includes band names, symbols and abbreviated terms used in the description of human chromosome and chromosome abnormalities. Abbreviations include dup for duplications of parts of a chromosome. For example, dup(17p12) causes Charcot–Marie–Tooth disease type 1A.

As amplification

Gene duplication does not necessarily constitute a lasting change in a species' genome. In fact, such changes often don't last past the initial host organism. From the perspective of molecular genetics, gene amplification is one of many ways in which a gene can be overexpressed. Genetic amplification can occur artificially, as with the use of the polymerase chain reaction technique to amplify short strands of DNA in vitro using enzymes, or it can occur naturally, as described above. If it's a natural duplication, it can still take place in a somatic cell, rather than a germline cell (which would be necessary for a lasting evolutionary change).

Role in cancer

Duplications of oncogenes are a common cause of many types of cancer. In such cases the genetic duplication occurs in a somatic cell and affects only the genome of the cancer cells themselves, not the entire organism, much less any subsequent offspring. Recent comprehensive patient-level classification and quantification of driver events in TCGA cohorts revealed that there are on average 12 driver events per tumor, of which 1.5 are amplifications of oncogenes.

Common oncogene amplifications in human cancers

Cancer type	Associated gene amplifications	Prevalence of amplification in cancer type (percent)
Breast cancer	MYC	20%
	ERBB2 (HER2)	20%
	CCND1 (Cyclin D1)	15–20%
	FGFR1	12%
	FGFR2	12%
Cervical cancer	MYC	25–50%
	ERBB2	20%
Colorectal cancer	HRAS	30%
	KRAS	20%
	MYB	15–20%
Esophageal cancer	MYC	40%
	CCND1	25%
	MDM2	13%
Gastric cancer	CCNE (Cyclin E)	15%
	KRAS	10%
	MET	10%
Glioblastoma	ERBB1 (EGFR)	33–50%
	CDK4	15%
Head and neck cancer	CCND1	50%
	ERBB1	10%
	MYC	7–10%
Hepatocellular cancer	CCND1	13%
Neuroblastoma	MYCN	20–25%
Ovarian cancer	MYC	20–30%
	ERBB2	15–30%
	AKT2	12%
Sarcoma	MDM2	10–30%
	CDK4	10%
Small cell lung cancer	MYC	15–20%

Whole-genome duplications are also frequent in cancers, detected in 30% to 36% of tumors from the most common cancer types. Their exact role in carcinogenesis is unclear, but they in some cases lead to loss of chromatin segregation leading to chromatin conformation changes that in turn lead to oncogenic epigenetic and transcriptional modifications.