View source for Gaussian process

In [[probability theory]] and [[statistics]], a '''Gaussian process''' is a [[stochastic process]] (a collection of random variables indexed by time or space), such that every finite collection of those random variables has a [[multivariate normal distribution]], i.e. every finite [[linear combination]] of them is normally distributed. The distribution of a Gaussian process is the [[joint distribution]] of all those (infinitely many) random variables, and as such, it is a distribution over functions with a continuous domain, e.g. time or space.

A machine-learning algorithm that involves a Gaussian process uses [[lazy learning]] and a measure of the similarity between points (the ''kernel function'') to predict the value for an unseen point from training data. The prediction is not just an estimate for that point, but also has uncertainty information—it is a one-dimensional Gaussian distribution (which is the [[marginal distribution]] at that point).<ref>{{cite web|url=http://platypusinnovation.blogspot.co.uk/2016/05/a-simple-intro-to-gaussian-processes.html|title=Platypus Innovation: A Simple Intro to Gaussian Processes (a great data modelling tool)|date=2016-05-10}}</ref>

For some kernel functions, matrix algebra can be used to calculate the predictions using the technique of [[kriging]]. When a parameterised kernel is used, optimisation software is typically used to fit a Gaussian process model.

The concept of Gaussian processes is named after [[Carl Friedrich Gauss]] because it is based on the notion of the Gaussian distribution ([[normal distribution]]). Gaussian processes can be seen as an infinite-dimensional generalization of multivariate normal distributions.

Gaussian processes are useful in [[statistical model]]ling, benefiting from properties inherited from the normal distribution. For example, if a [[random process]] is modelled as a Gaussian process, the distributions of various derived quantities can be obtained explicitly. Such quantities include the average value of the process over a range of times and the error in estimating the average using sample values at a small set of times.

==Definition==
A time continuous [[stochastic process]] <math>\left\{X_t ; t\in T\right\}</math> is Gaussian [[if and only if]] for every [[finite set]] of [[indexed family|indices]] <math>t_1,\ldots,t_k</math> in the index set <math>T</math>

: <math> \mathbf{X}_{t_1, \ldots, t_k} = (X_{t_1}, \ldots, X_{t_k}) </math>

is a [[multivariate normal distribution|multivariate Gaussian]] [[random variable]].<ref name=DrMacKayGPNN>{{cite book| last=MacKay | first=David, J.C.| authorlink=David J.C. MacKay| title=Information Theory, Inference, and Learning Algorithms |year=2003 | pages=540|publisher=[[Cambridge University Press]]| isbn=9780521642989|url=http://www.inference.phy.cam.ac.uk/itprnn/book.pdf | quote=The probability distribution of a function <math>y(\mathbf{x})</math> is a Gaussian processes if for any finite selection of points <math>\mathbf{x}^{(1)},\mathbf{x}^{(2)},\ldots,\mathbf{x}^{(N)}</math>, the density <math>P(y(\mathbf{x}^{(1)}),y(\mathbf{x}^{(2)}),\ldots,y(\mathbf{x}^{(N)}))</math>is a Gaussian}}</ref> That is the same as saying every linear combination of <math>
 (X_{t_1}, \ldots, X_{t_k}) </math> has a univariate normal (or Gaussian) distribution.

Using [[Characteristic function (probability theory)|characteristic functions]] of random variables, the Gaussian property can be formulated as follows: <math>\left\{X_t ; t\in T\right\}</math> is Gaussian if and only if, for every finite set of indices <math>t_1,\ldots,t_k</math>, there are real-valued <math>\sigma_{\ell j}</math>, <math>\mu_\ell</math> with <math>\sigma_{jj} > 0</math> such that the following equality holds for all <math>s_1,s_2,\ldots,s_k\in\mathbb{R}</math>

:<math> \operatorname{E}\left(\exp\left(i \ \sum_{\ell=1}^k s_\ell \ \mathbf{X}_{t_\ell}\right)\right) = \exp \left(-\frac{1}{2} \, \sum_{\ell, j} \sigma_{\ell j} s_\ell s_j + i \sum_\ell \mu_\ell s_\ell\right)</math>.

where <math>i</math> denotes the [[imaginary unit]] such that <math>i^2 =-1</math>.

The numbers <math>\sigma_{\ell j}</math> and <math>\mu_\ell</math> can be shown to be the [[covariance]]s and [[mean (mathematics)|means]] of the variables in the process.<ref>{{cite book |last=Dudley |first=R.M. |title=Real Analysis and Probability |year=1989 |publisher=Wadsworth and Brooks/Cole}}</ref>

==Variance==
The variance of a Gaussian process is finite at any time <math>t</math>, formally<ref name="Lapidoth2017">{{cite book|author=Amos Lapidoth|title=A Foundation in Digital Communication|url=https://books.google.com/books?id=6oTuDQAAQBAJ&printsec=frontcover#v=onepage&q=independence&f=false|date=8 February 2017|publisher=Cambridge University Press|isbn=978-1-107-17732-1}}</ref>{{rp|p. 515}}
:<math> \operatorname{var}[X(t)] = \operatorname{E}[|X(t)-\operatorname{E}[X(t)]|^2] < \infty \quad \text{for all } t \in T</math>.

==Stationarity==
For general stochastic processes [[Stationary process#strict-sense stationarity|strict-sense stationarity]] implies [[Stationary process#wide-sense stationarity|wide-sense stationarity]] but not every wide-sense stationary stochastic process is strict-sense stationary. However, for a Gaussian stochastic process the two concepts are equivalent.<ref name="Lapidoth2017"/>{{rp|p. 518}}

A Gaussian stochastic process is strict-sense stationary if, and only if, it is wide-sense stationary.

==Covariance functions==
A key fact of Gaussian processes is that they can be completely defined by their second-order statistics.<ref name="prml">{{cite book |last=Bishop |first=C.M. |title= Pattern Recognition and Machine Learning |year=2006 |publisher=[[Springer Science+Business Media|Springer]] |isbn=978-0-387-31073-2}}</ref> Thus, if a Gaussian process is assumed to have mean zero, defining the [[covariance function]] completely defines the process' behaviour. Importantly the non-negative definiteness of  this function enables its spectral decomposition using the [[Karhunen–Loève theorem|Karhunen–Loève expansion]]. Basic aspects that can be defined through the covariance function are the process' [[stationary process|stationarity]], [[isotropy]], [[smoothness]] and [[periodic function|periodicity]].<ref name="brml">{{cite book |last=Barber |first=David |title=Bayesian Reasoning and Machine Learning |url=http://web4.cs.ucl.ac.uk/staff/D.Barber/pmwiki/pmwiki.php?n=Brml.HomePage |year=2012 |publisher=[[Cambridge University Press]] |isbn=978-0-521-51814-7}}</ref><ref name="gpml">{{cite book |last=Rasmussen |first=C.E. |author2=Williams, C.K.I |title=Gaussian Processes for Machine Learning |url=http://www.gaussianprocess.org/gpml/ |year=2006 |publisher=[[MIT Press]] |isbn=978-0-262-18253-9}}</ref>

[[stationary process|Stationarity]] refers to the process' behaviour regarding the separation of any two points <math>x</math> and <math>x'</math>. If the process is stationary, it depends on their separation, <math>x-x'</math>, while if non-stationary it depends on the actual position of the points  <math>x</math> and <math>x'</math>. For example, the special case of an [[Ornstein–Uhlenbeck process]], a [[Brownian motion]] process, is stationary.

If the process depends only on <math>|x-x'|</math>, the Euclidean distance (not the direction) between <math>x</math> and <math>x'</math>,  then the process is considered isotropic. A process that is concurrently stationary and isotropic is considered to be [[homogeneous]];<ref name="PRP">{{cite book |last=Grimmett  |first=Geoffrey |author2=David Stirzaker|title= Probability and Random Processes| year=2001 |publisher=[[Oxford University Press]] |isbn=978-0198572220}}</ref> in practice these properties reflect the differences (or rather the lack of them) in the behaviour of the process given the location of the observer.

Ultimately Gaussian processes translate as taking priors on functions and the smoothness of these priors can be induced by the covariance function.<ref name ="brml"/> If we expect that for "near-by" input points <math>x</math> and <math>x'</math> their corresponding output points <math>y</math> and <math>y'</math> to be "near-by" also, then the assumption of continuity is present. If we wish to allow for significant displacement then we might choose a rougher covariance function. Extreme examples of the behaviour is the Ornstein&ndash;Uhlenbeck covariance function and the squared exponential where the former is never differentiable and the latter infinitely differentiable.

Periodicity refers to inducing periodic patterns within the behaviour of the process. Formally, this is achieved by mapping the input <math>x</math> to a two dimensional vector <math>u(x) = \left( \cos(x), \sin(x) \right)</math>.

===Usual covariance functions===
[[File:Gaussian process draws from prior distribution.png|thumbnail|right|The effect of choosing different kernels on the prior function distribution of the Gaussian process. Left is a squared exponential kernel. Middle is Brownian. Right is quadratic.]]
There are a number of common covariance functions:<ref name="gpml"/>
*Constant : <math> K_\operatorname{C}(x,x') = C </math>
*Linear: <math> K_\operatorname{L}(x,x') =  x^T x'</math>
*white Gaussian noise: <math> K_\operatorname{GN}(x,x') = \sigma^2 \delta_{x,x'}</math>
*Squared exponential: <math> K_\operatorname{SE}(x,x') = \exp \Big(-\frac{|d|^2}{2\ell^2} \Big)</math>
*Ornstein&ndash;Uhlenbeck: <math> K_\operatorname{OU}(x,x') = \exp \left(-\frac{|d|} \ell \right)</math>
*Matérn: <math> K_\operatorname{Matern}(x,x') = \frac{2^{1-\nu}}{\Gamma(\nu)} \Big(\frac{\sqrt{2\nu}|d|}{\ell} \Big)^\nu K_\nu \Big(\frac{\sqrt{2\nu}|d|}{\ell} \Big)</math>
*Periodic: <math> K_\operatorname{P}(x,x') = \exp\left(-\frac{ 2\sin^2\left(\frac d 2 \right)}{\ell^2} \right)</math>
*Rational quadratic: <math> K_\operatorname{RQ}(x,x') =  (1+|d|^2)^{-\alpha}, \quad \alpha \geq 0</math>

Here <math>d = x- x'</math>. The parameter ''ℓ'' is the characteristic length-scale of the process (practically, "how close" two points <math>x</math> and <math>x'</math> have to be to influence each other significantly), ''δ'' is the [[Kronecker delta]] and σ the [[standard deviation]] of the noise fluctuations. Moreover, <math>K_\nu</math> is the [[modified Bessel function]] of order <math>\nu</math> and <math>\Gamma(\nu)</math> is the [[gamma function]] evaluated at <math>\nu</math>. Importantly, a complicated covariance function can be defined as a linear combination of other simpler covariance functions in order to incorporate different insights about the data-set at hand.

Clearly, the inferential results are dependent on the values of the hyperparameters ''θ'' (e.g. ''ℓ'' and ''σ'') defining the model's behaviour. A popular choice for ''θ'' is to provide ''[[maximum a posteriori]]'' (MAP) estimates of it with some chosen prior. If the prior is very near uniform, this is the same as maximizing the [[marginal likelihood]] of the process; the  marginalization being done over the observed process values <math>y</math>.<ref name= "gpml"/> This approach is also known as ''maximum likelihood II'', ''evidence maximization'', or ''[[empirical Bayes]]''.<ref name="seegerGPML">{{cite journal |last1= Seeger| first1= Matthias |year= 2004 |title= Gaussian Processes for Machine Learning|journal= International Journal of Neural Systems|volume= 14|issue= 2|pages= 69–104 |doi=10.1142/s0129065704001899| pmid= 15112367 | citeseerx= 10.1.1.71.1079 }}</ref>

==Continuity==

For a Gaussian process, [[Continuous stochastic process#Continuity in probability|continuity in probability]] is equivalent to [[Continuous stochastic process#Mean-square continuity|mean-square continuity]],
<ref>{{cite book |chapterurl=https://www.mathunion.org/fileadmin/ICM/Proceedings/ICM1974.2/ICM1974.2.ocr.pdf |title=Proceedings of the International Congress of Mathematicians |editors= |first=R. M. |last=Dudley |authorlink=Richard M. Dudley |year=1975 |volume=2 |pages=143–146 |chapter=The Gaussian process and how to approach it}}</ref>{{rp|145}}
and [[Continuous stochastic process#Continuity with probability one|continuity with probability one]] is equivalent to [[Continuous stochastic process#Sample continuity|sample continuity]].<ref>{{cite journal |first=R. M. |last=Dudley |authorlink=Richard M. Dudley |title=Sample functions of the Gaussian process |journal=[[Annals of Probability]] |volume=1 |issue=1 |pages=66–103 |year=1973 |quote=|doi=10.1007/978-1-4419-5821-1_13 |isbn=978-1-4419-5820-4 }}</ref>{{rp|91 "Gaussian processes are discontinuous at fixed points."}}
The latter implies, but is not implied by, continuity in probability.
Continuity in probability holds if and only if the [[autocovariance|mean and autocovariance]] are continuous functions. In contrast, sample continuity was challenging even for [[Gaussian process#Stationarity|stationary Gaussian processes]] (as probably noted first by [[Andrey Kolmogorov]]), and more challenging for more general processes.<ref>{{cite book |last=Talagrand  |first=Michel |author-link=Michel Talagrand |title=Upper and lower bounds for stochastic processes: modern methods and classical problems |year=2014 |publisher=Springer, Heidelberg |isbn=978-3-642-54074-5 |url=https://www.springer.com/gp/book/9783642540745|series=Ergebnisse der Mathematik und ihrer Grenzgebiete. 3. Folge / A Series of Modern Surveys in Mathematics }}</ref>{{rp|Sect. 2.8}}
<ref>{{cite book |title=Lecture Notes in Mathematics |editors= |first=Michel |last=Ledoux |authorlink=Michel Ledoux |year=1994 |volume=1648 |publisher=Springer, Berlin |pages=165–294 |chapter=Isoperimetry and Gaussian analysis|doi=10.1007/BFb0095676 |title-link = Lecture Notes in Mathematics|isbn = 978-3-540-62055-6}}</ref>{{rp|69,81}}
<ref>{{cite journal |last=Adler |first=Robert J. |author-link=Robert Adler |title=An introduction to continuity, extrema, and related topics for general Gaussian processes |journal=Lecture Notes-Monograph Series |volume=12 |pages=i–155 |year=1990 |publisher=Institute of Mathematical Statistics |isbn= |jstor=4355563 }}</ref>{{rp|80}}
<ref>{{cite journal |title=Review of: Adler 1990 'An introduction to continuity...' |first=Simeon M. |last=Berman |date=1992 |journal=Mathematical Reviews|mr = 1088478}}</ref>
As usual, by a sample continuous process one means a process that admits a sample continuous [[Stochastic process#Modification|modification]].
<ref name=Dudley67>{{cite journal |first=R. M. |last=Dudley |authorlink=Richard M. Dudley |title=The sizes of compact subsets of Hilbert space and continuity of Gaussian processes |journal=Journal of Functional Analysis |volume=1 |issue= 3|pages=290–330 |year=1967|doi=10.1016/0022-1236(67)90017-1 }}</ref>{{rp|292}}
<ref name=MarcusShepp72>{{cite book |chapterurl=https://projecteuclid.org/euclid.bsmsp/1200514231 |title=Proceedings of the sixth Berkeley symposium on mathematical statistics and probability, vol. II: probability theory  |editors= |first1=M.B. |last1=Marcus |first2=Lawrence A. |last2=Shepp |authorlink2=Lawrence Shepp |year=1972 |volume= |publisher=Univ. California, Berkeley |pages=423–441 |chapter=Sample behavior of Gaussian processes}}</ref>{{rp|424}}

===Stationary case===

For a stationary Gaussian process <math>X=(X_t)_{t\in\mathbb R},</math> some conditions on its spectrum are sufficient for sample continuity, but fail to be necessary. A necessary and sufficient condition, sometimes called Dudley-Fernique theorem, involves the function <math>\sigma</math> defined by
:<math> \sigma(h) = \sqrt{ \mathbb E \big( X(t+h) - X(t) \big)^2 } </math>
(the right-hand side does not depend on <math>t</math> due to stationarity). Continuity of <math>X</math> in probability is equivalent to continuity of <math>\sigma</math> at <math>0.</math> When convergence of <math>\sigma(h)</math> to <math>0</math> (as <math>h\to 0</math>) is too slow, sample continuity of <math>X</math> may fail. Convergence of the following integrals matters:
:<math> I(\sigma) = \int_0^1 \frac{ \sigma(h) }{ h \sqrt{ \log(1/h) } } \, dh = \int_0^\infty 2\sigma(\mathbb e^{-x^2}) \, dx ,</math>
these two integrals being equal according to [[integration by substitution]] <math> h = \mathbb e^{-x^2}, </math> <math>\textstyle x = \sqrt{\log(1/h)} .</math> The first integrand need not be bounded as <math>h\to 0+,</math> thus the integral may converge (<math>I(\sigma)<\infty</math>) or diverge (<math>I(\sigma)=\infty</math>). Taking for example <math>\sigma(\mathbb e^{-x^2})=\tfrac1{x^a}</math> for large <math>x,</math> that is, <math>\sigma(h)=(\log(1/h))^{-a/2}</math> for small <math>h,</math> one obtains <math>I(\sigma)<\infty</math> when <math>a>1,</math> and <math>I(\sigma)=\infty</math> when <math>0<a\le 1.</math>
In these two cases the function <math>\sigma</math> is increasing on <math>[0,\infty),</math> but generally it is not. Moreover, the condition
: <math>(*)</math> &nbsp; there exists <math>\varepsilon>0</math> such that <math>\sigma</math> is monotone on <math>[0,\varepsilon]</math>
does not follow from continuity of <math>\sigma</math> and the evident relations <math>\sigma(h)\ge0</math> (for all <math>h</math>) and <math>\sigma(0)=0.</math>

'''Theorem 1.''' &nbsp; Let <math>\sigma</math> be continuous and satisfy <math>(*).</math> Then the condition <math>I(\sigma)<\infty</math> is necessary and sufficient for sample continuity of <math>X.</math>

Some history.<ref name=MarcusShepp72 />{{rp|424}}
Sufficiency was announced by [[Xavier Fernique]] in 1964, but the first proof was published by [[Richard M. Dudley]] in 1967.<ref name=Dudley67 />{{rp|Theorem 7.1}}
Necessity was proved by Michael B. Marcus and [[Lawrence Shepp]] in 1970.<ref name=MarcusShepp70>{{cite journal |first1=Michael B. |last1=Marcus |first2=Lawrence A. |last2=Shepp |authorlink2=Lawrence Shepp |title=Continuity of Gaussian processes |journal=[[Transactions of the American Mathematical Society]] |volume=151 |issue= 2|pages=377–391 |year=1970 |doi=10.2307/1995502|jstor=1995502 }}</ref>{{rp|380}}

There exist sample continuous processes <math>X</math> such that <math>I(\sigma)=\infty;</math> they violate condition <math>(*).</math> An example found by Marcus and Shepp <ref name=MarcusShepp70 />{{rp|387}} is a random [[Lacunary function#Lacunary trigonometric series|lacunary Fourier series]]
: <math> X_t = \sum_{n=1}^\infty c_n ( \xi_n \cos \lambda_n t + \eta_n \sin \lambda_n t ) ,</math>
where <math>\xi_1,\eta_1,\xi_2,\eta_2,\dots</math> are independent random variables with [[Normal distribution#Standard normal distribution|standard normal distribution]]; frequencies <math>0<\lambda_1<\lambda_2<\dots</math> are a fast growing sequence; and coefficients <math>c_n>0</math> satisfy <math>\textstyle \sum_n c_n < \infty.</math> The latter relation implies <math>\textstyle\mathbb E \sum_n c_n ( |\xi_n| + |\eta_n| ) = \sum_n c_n \mathbb E ( |\xi_n| + |\eta_n| ) = \text{const} \cdot \sum_n c_n < \infty,</math> whence <math>\sum_n c_n ( |\xi_n| + |\eta_n| ) < \infty</math> almost surely, which ensures uniform convergence of the Fourier series almost surely, and sample continuity of <math>X.</math>
[[File:Autocorrelation of a random lacunary Fourier series.svg|thumb|411px|Autocorrelation of a random lacunary Fourier series]]
Its autocovariation function
: <math> \mathbb E X_t X_{t+h} = \sum_{n=1}^\infty c_n^2 \cos \lambda_n h </math>
is nowhere monotone (see the picture), as well as the corresponding function <math>\sigma,</math>
: <math> \sigma(h) = \sqrt{ 2 \mathbb E X_t X_t - 2 \mathbb E X_t X_{t+h} } = 2 \sqrt{ \sum_{n=1}^\infty c_n^2 \sin^2 \frac{\lambda_n h}2 } .</math>

==Brownian motion as the integral of Gaussian processes==
A [[Wiener process]] (aka Brownian motion) is the integral of a [[White noise#Continuous-time white noise|white noise generalized Gaussian process]]. It is not [[stationary process|stationary]], but it has stationary increments.

The [[Ornstein–Uhlenbeck process]] is a [[stationary process|stationary]] Gaussian process.

The [[Brownian bridge]] is (like the Ornstein–Uhlenbeck process) an example of a Gaussian process whose increments are not [[statistical independence|independent]].

The [[fractional Brownian motion]] is a Gaussian process whose covariance function is a generalisation of that of the Wiener process.

==Applications==
[[Image:Regressions sine demo.svg|thumbnail|right|An example of Gaussian Process Regression (prediction) compared with other regression models.<ref>The documentation for [[scikit-learn]] also has similar [http://scikit-learn.org/stable/auto_examples/gaussian_process/plot_compare_gpr_krr.html examples].</ref>]]
A Gaussian process can be used as a [[prior probability distribution]] over [[Function (mathematics)|functions]] in [[Bayesian inference]].<ref name="gpml"/><ref>{{cite book |last=Liu |first=W. |author2=Principe, J.C. |author3=Haykin, S. |title=Kernel Adaptive Filtering: A Comprehensive Introduction |url=http://www.cnel.ufl.edu/~weifeng/publication.htm |year=2010 |publisher=[[John Wiley & Sons|John Wiley]] |isbn=978-0-470-44753-6}}</ref> Given any set of ''N'' points in the desired domain of your functions, take a [[multivariate Gaussian]] whose covariance [[matrix (mathematics)|matrix]] parameter is the [[Gram matrix]] of your ''N'' points with some desired [[stochastic kernel|kernel]], and [[sampling (mathematics)|sample]] from that Gaussian.

Inference of continuous values with a Gaussian process prior is known as Gaussian process regression, or [[kriging]]; extending Gaussian process regression to [[Kernel methods for vector output|multiple target variables]] is known as ''cokriging''.<ref>{{cite book |last=Stein |first=M.L. |title=Interpolation of Spatial Data: Some Theory for Kriging |year=1999 |publisher = [[Springer Science+Business Media|Springer]]}}</ref> Gaussian processes are thus useful as a powerful non-linear multivariate [[interpolation]] tool. Gaussian process regression can be further extended to address learning tasks in both [[Supervised learning|supervised]] (e.g. probabilistic classification<ref name="gpml"/>) and [[Unsupervised learning|unsupervised]] (e.g. [[manifold learning]]<ref name= "prml"/>) learning frameworks.

Gaussian processes can also be used in the context of mixture of experts models, for example.<ref>{{Cite journal |doi = 10.1109/TPAMI.2013.183|pmid = 26353224|title = Gaussian Process-Mixture Conditional Heteroscedasticity|journal = IEEE Transactions on Pattern Analysis and Machine Intelligence|volume = 36|issue = 5|pages = 888–900|year = 2014|last1 = Platanios|first1 = Emmanouil A.|last2 = Chatzis|first2 = Sotirios P.}}</ref><ref>{{Cite journal | doi=10.1016/j.neucom.2013.04.029| title=A latent variable Gaussian process model with Pitman–Yor process priors for multiclass classification| journal=Neurocomputing| volume=120| pages=482–489| year=2013| last1=Chatzis| first1=Sotirios P.}}</ref> The underlying rationale of such a learning framework consists in the assumption that a given mapping cannot be well captured by a single Gaussian process model. Instead, the observation space is divided into subsets, each of which is characterized by a different mapping function; each of these is learned via a different Gaussian process component in the postulated mixture.

===Gaussian process prediction, or Kriging===
[[File:Gaussian Process Regression.png|thumbnail|right|Gaussian Process Regression (prediction) with a squared exponential kernel. Left plot are draws from the prior function distribution. Middle are draws from the posterior. Right is mean prediction with one standard deviation shaded.]]
{{further|Kriging}}
When concerned with a general Gaussian process regression problem (Kriging), it is assumed that for a Gaussian process ''f'' observed at coordinates ''x'', the vector of values {{tmath|f(x)}} is just one sample from a multivariate Gaussian distribution of dimension equal to number of observed coordinates {{tmath|n}}. Therefore, under the assumption of a zero-mean distribution, {{tmath|f (x) \sim N (0, K(\theta,x,x'))}}, where {{tmath|K(\theta,x,x')}} is the covariance matrix between all possible pairs {{tmath|(x,x')}} for a given set of hyperparameters ''θ''.<ref name= "gpml"/>
As such the log marginal likelihood is:

: <math>\log p(f(x)\mid\theta,x) =  -\frac{1}{2}f(x)^T K(\theta,x,x')^{-1} f(x') -\frac 1 2 \log \det(K(\theta,x,x')) - \frac{n} 2 \log 2\pi </math>

and maximizing this marginal likelihood towards ''θ'' provides the complete specification of the Gaussian process ''f''. One can briefly note at this point that the first term corresponds to a penalty term for a model's failure to fit observed values and the second term to a penalty term that increases proportionally to a model's complexity. Having specified ''θ'' making predictions about unobserved values {{tmath|f(x^*)}} at coordinates ''x''* is then only a matter of drawing samples from the predictive distribution <math>p(y^*\mid x^*,f(x),x) = N(y^*\mid A,B)</math> where the posterior mean estimate ''A'' is defined as

:<math>A = K(\theta,x^*,x) K(\theta,x,x')^{-1} f(x)</math>

and the posterior variance estimate ''B'' is defined as:

:<math>B = K(\theta,x^*,x^*) - K(\theta,x^*,x)  K(\theta,x,x')^{-1}  K(\theta,x^*,x)^T </math>

where {{tmath|K(\theta,x^*,x)}} is the covariance between the new coordinate of estimation ''x''* and all other observed coordinates ''x'' for a given hyperparameter vector ''θ'', {{tmath|K(\theta,x,x')}} and {{tmath|f(x)}} are defined as before and {{tmath|K(\theta,x^*,x^*)}} is the variance at point ''x''* as dictated by ''θ''. It is important to note that practically the posterior mean estimate {{tmath|f(x^*)}} (the "point estimate") is just a linear combination of the observations {{tmath|f(x)}}; in a similar manner the variance of {{tmath|f(x^*)}} is actually independent of the observations {{tmath|f(x)}}. A known bottleneck in Gaussian process prediction is that the computational complexity of prediction is cubic in the number of points |''x''| and as such can become unfeasible for larger data sets.<ref name= "brml"/> Works on sparse Gaussian processes, that usually are based on the idea of building a ''representative set'' for the given process ''f'', try to circumvent this issue.<ref name="smolaSparse">{{cite journal |last1= Smola| first1= A.J.| last2=Schoellkopf | first2= B. |year= 2000 |title= Sparse greedy matrix approximation for machine learning |journal= Proceedings of the Seventeenth International Conference on Machine Learning| pages=911–918|url=http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.43.3153}}</ref><ref name="CsatoSparse">{{cite journal |last1= Csato| first1=L.| last2=Opper | first2= M. |year= 2002 |title= Sparse on-line Gaussian processes  |journal= Neural Computation |number=3| volume= 14 | pages=641–668 | doi=10.1162/089976602317250933| pmid=11860686| citeseerx=10.1.1.335.9713}}</ref>

==See also==
* [[Bayes linear statistics]]
* [[Bayesian interpretation of regularization]]
* [[Kriging]]
* [[Gaussian free field]]
* [[Gradient-enhanced kriging]] (GEK)

== References ==
{{Reflist}}

==External links==
* [http://www.GaussianProcess.org The Gaussian Processes Web Site, including the text of Rasmussen and Williams' Gaussian Processes for Machine Learning]
* [https://arxiv.org/abs/1505.02965 A gentle introduction to Gaussian processes]
* [http://publications.nr.no/917_Rapport.pdf A Review of Gaussian Random Fields and Correlation Functions]
* [https://pdfs.semanticscholar.org/c9f2/1b84149991f4d547b3f0f625f710750ad8d9.pdf Efficient Reinforcement Learning using Gaussian Processes]
===Software===
* [http://sourceforge.net/projects/kriging STK: a Small (Matlab/Octave) Toolbox for Kriging and GP modeling]
* [http://www.uqlab.com/ Kriging module in UQLab framework (Matlab)]
* [http://au.mathworks.com/matlabcentral/fileexchange/38880: Matlab/Octave function for stationary Gaussian fields]
* [https://github.com/Yelp/MOE Yelp MOE – A black box optimization engine using Gaussian process learning]
* [http://www.sumo.intec.ugent.be/ooDACE ooDACE] – A flexible object-oriented Kriging matlab toolbox.
* [http://becs.aalto.fi/en/research/bayes/gpstuff/ GPstuff – Gaussian process toolbox for Matlab and Octave]
* [https://github.com/SheffieldML/GPy GPy – A Gaussian processes framework in Python]
* [http://www.tmpl.fi/gp/ Interactive Gaussian process regression demo]
* [https://github.com/ChristophJud/GPR Basic Gaussian process library written in C++11]
* [http://scikit-learn.org scikit-learn] – A machine learning library for Python which includes Gaussian process regression and classification
* [https://github.com/modsim/KriKit] - The Kriging toolKit (KriKit) is developed at the Institute of Bio- and Geosciences 1 (IBG-1) of Forschungszentrum Jülich (FZJ)

===Video tutorials===
* [http://videolectures.net/gpip06_mackay_gpb Gaussian Process Basics by David MacKay]
* [http://videolectures.net/epsrcws08_rasmussen_lgp Learning with Gaussian Processes by Carl Edward Rasmussen]
* [http://videolectures.net/mlss07_rasmussen_bigp Bayesian inference and Gaussian processes by Carl Edward Rasmussen]

{{Stochastic processes}}

{{Authority control}}

{{DEFAULTSORT:Gaussian Process}}
[[Category:Stochastic processes]]
[[Category:Kernel methods for machine learning]]
[[Category:Nonparametric Bayesian statistics]]
[[Category:Normal distribution]]