View source for Kullback–Leibler divergence

{{Information theory}}
In [[mathematical statistics]], the '''Kullback–Leibler divergence''' (also called '''relative entropy''') is a measure of how one [[probability distribution]] is different from a second, reference probability distribution.<ref>{{cite journal
 |last1=Kullback |first1=S. |authorlink1=Solomon Kullback
 |last2=Leibler |first2=R.A. |authorlink2=Richard Leibler
 |year = 1951
 |title = On information and sufficiency
 |journal = [[Annals of Mathematical Statistics]]
 |volume = 22 |issue=1
 |pages=79–86
 |doi=10.1214/aoms/1177729694 |mr=39968
|url=https://www.jstor.org/stable/pdf/2236703.pdf}}</ref><ref name="Kullback">{{citation|last1=Kullback|first1=S.|title=Information Theory and Statistics |date=1959 |publisher=[[John Wiley & Sons]]}}. Republished by [[Dover Publications]] in 1968; reprinted in 1978: {{isbn|0-8446-5625-9}}.</ref> Applications include characterizing the relative [[Entropy (information theory)|(Shannon) entropy]] in information systems, randomness in continuous [[Time series|time-series]], and information gain when comparing statistical models of [[inference]]. In contrast to [[variation of information]], it is a distribution-wise ''asymmetric'' measure and thus does not qualify as a statistical ''metric'' of spread (it also does not satisfy the triangle inequality). In the simple case, a Kullback–Leibler divergence of 0 indicates that the two distributions in question are identical. In simplified terms, it is a measure of surprise, with diverse applications such as applied statistics, [[fluid mechanics]], [[neuroscience]] and [[machine learning]].

== Etymology ==
The Kullback–Leibler divergence was introduced by [[Solomon Kullback]] and [[Richard Leibler]] in 1951 as the '''directed divergence''' between two distributions; Kullback preferred the term '''discrimination information'''.<ref>{{cite journal
 |first=S. |last=Kullback |authorlink=Solomon Kullback
 |year=1987
 |title=Letter to the Editor: The Kullback–Leibler distance
 |journal=[[The American Statistician]]
 |volume=41 |issue=4 |pages=340–341
 |jstor=2684769
 |doi=10.1080/00031305.1987.10475510
}}</ref> The divergence is discussed in Kullback's 1959 book, ''Information Theory and Statistics''.<ref name="Kullback" />

==Definition==
For [[discrete probability distribution]]s <math>P</math> and <math>Q</math> defined on the same [[probability space]], the Kullback–Leibler divergence of <math>Q</math> from <math>P</math> is defined<ref>{{cite book| author= MacKay, David J.C.| year=2003|title=Information Theory, Inference, and Learning Algorithms|publisher=Cambridge University Press|edition=First|page=34|url=https://books.google.com/books?id=AKuMj4PN_EMC&printsec=frontcover#v=onepage&q=%22Kullback%E2%80%93Leibler%20divergence%22&f=false}}</ref> to be

:<math>D_\text{KL}(P \parallel Q) = \sum_{x\in\mathcal{X}} P(x) \log\left(\frac{P(x)}{Q(x)}\right).</math>
which is equivalent to

:<math>D_\text{KL}(P \parallel Q) = -\sum_{x\in\mathcal{X}} P(x) \log\left(\frac{Q(x)}{P(x)}\right)</math>

In other words, it is the [[expectation (statistics)|expectation]] of the logarithmic difference between the probabilities <math>P</math> and <math>Q</math>, where the expectation is taken using the probabilities <math>P</math>. The Kullback–Leibler divergence is defined only if for all <math>x</math>, <math>Q(x) = 0</math> implies <math>P(x) = 0</math> ([[absolute continuity]]). Whenever <math>P(x)</math> is zero the contribution of the corresponding term is interpreted as zero because

:<math>\lim_{x \to 0^{+}} x \log(x) = 0.</math>

For distributions <math>P</math> and <math>Q</math> of a [[continuous random variable]], the Kullback–Leibler divergence is defined to be the integral:<ref>Bishop C. (2006). ''Pattern Recognition and Machine Learning''</ref>{{rp|p. 55}}

:<math>D_\text{KL}(P \parallel Q) = \int_{-\infty}^\infty p(x) \log\left(\frac{p(x)}{q(x)}\right)\, dx</math>

where <math>p</math> and <math>q</math> denote the [[Probability density function|probability densities]] of <math>P</math> and <math>Q</math>.

More generally, if <math>P</math> and <math>Q</math> are probability [[measure (mathematics)|measure]]s over a set <math>{\mathcal{X}}</math>, and <math>P</math> is [[absolutely continuous measure|absolutely continuous]] with respect to <math>Q</math>, then the Kullback–Leibler divergence from <math>Q</math> to <math>P</math> is defined as

:<math>D_\text{KL}(P \parallel Q) = \int_{\mathcal{X}} \log\left(\frac{dP}{dQ}\right)\, dP,</math>

where <math>\frac{dP}{dQ}</math> is the [[Radon–Nikodym derivative]] of <math>P</math> with respect to <math>Q</math>, and provided the expression on the right-hand side exists. Equivalently (by the [[chain rule]]), this can be written as

:<math>D_\text{KL}(P \parallel Q) = \int_{\mathcal{X}} \log\left(\frac{dP}{dQ}\right) \frac{dP}{dQ}\, dQ,</math>

which is the [[Entropy (information theory)|entropy]] of <math>Q</math> relative to <math>P</math>. Continuing in this case, if <math>\mu</math> is any measure on <math>\mathcal{X}</math> for which <math>p = \frac{dP}{d\mu}</math> and <math>q = \frac{dQ}{d\mu}</math> exist (meaning that <math>p</math> and <math>q</math> are absolutely continuous with respect to <math>\mu</math>), then the Kullback–Leibler divergence from <math>Q</math> to <math>P</math> is given as

:<math>D_\text{KL}(P \parallel Q) = \int_{\mathcal{X}} p \log\left(\frac{p}{q}\right)\, d\mu.</math>

The logarithms in these formulae are taken to [[Base (exponentiation)|base]] 2 if information is measured in units of [[bit]]s, or to base <math>e</math> if information is measured in [[nat (unit)|nat]]s. Most formulas involving the Kullback–Leibler divergence hold regardless of the base of the logarithm.

Various conventions exist for referring to <math>D_\text{KL}(P \parallel Q)</math> in words. Often it is referred to as the divergence ''between'' <math>P</math> and <math>Q</math>, but this fails to convey the fundamental asymmetry in the relation. Sometimes, as in this article, it may be found described as the divergence of <math>P</math> ''from'' <math>Q</math> or as the divergence ''from'' <math>Q</math> ''to'' <math>P</math>. This reflects the [[asymmetry]] in [[Bayesian inference]], which starts ''from'' a [[Prior distribution|prior]] <math>Q</math> and updates ''to'' the [[Posterior distribution|posterior]] <math>P</math>. Another common way to refer to <math>D_\text{KL}(P \parallel Q)</math>is as the relative entropy of <math>P</math> ''with respect to'' <math>Q</math>.

==Basic example==

Kullback<ref name="Kullback" /> gives the following example (Table 2.1, Example 2.1). Let <math>P</math> and <math>Q</math> be the distributions shown in the table and figure. <math>P</math> is the distribution on the left side of the figure, a [[binomial distribution]] with <math>N = 2</math> and <math>p = 0.4</math>. <math>Q</math> is the distribution on the right side of the figure, a discrete uniform distribution with the three possible outcomes <math>x = 0</math>, <math>1</math>, or <math>2</math> (i.e. <math>\mathcal{X}=\{0,1,2\}</math>), each with probability <math>p = 1/3</math>.

[[File:Kullback–Leibler distributions example 1.svg|400px|Two distributions to illustrate Kullback–Leibler divergence]]
{| class="wikitable"
|-
! x !! 0 !! 1 !! 2 
|-
| Distribution ''P''(x) || 0.36 || 0.48 || 0.16
|-
| Distribution ''Q''(x) || 0.333 || 0.333 || 0.333
|}

The KL divergences <math>D_\text{KL}(P \parallel Q)</math> and <math>D_\text{KL}(Q \parallel P)</math> are calculated as follows. This example uses the [[Natural logarithm|natural log]] with base ''[[E (mathematical constant)|e]]'', designated <math>\operatorname{ln}</math> to get results in [[Nat (unit)|nats]] (see [[units of information]]).

:<math>\begin{align}
D_\text{KL}(P \parallel Q) &= \sum_{x\in\mathcal{X}} P(x) \ln\left(\frac{P(x)}{Q(x)}\right) \\
 &= 0.36 \ln\left(\frac{0.36}{0.333}\right) + 0.48 \ln\left(\frac{0.48}{0.333}\right) + 0.16 \ln\left(\frac{0.16}{0.333}\right) \\
 &= 0.0852996
\end{align}</math>

:<math>\begin{align}
D_\text{KL}(Q \parallel P) &= \sum_{x\in\mathcal{X}} Q(x) \ln\left(\frac{Q(x)}{P(x)}\right) \\
 &= 0.333 \ln\left(\frac{0.333}{0.36}\right) + 0.333 \ln\left(\frac{0.333}{0.48}\right) + 0.333 \ln\left(\frac{0.333}{0.16}\right) \\
 &= 0.097455
\end{align}</math>

==Interpretations==

The Kullback–Leibler divergence from <math>Q</math> to <math>P</math> is often denoted <math>D_\text{KL}(P \parallel Q)</math>.

In the context of [[machine learning]], <math>D_\text{KL}(P \parallel Q)</math> is often called the [[information gain in decision trees|information gain]] achieved if <math>Q</math> is used instead of <math>P</math>. By analogy with information theory, it is also called the '''relative entropy''' of <math>P</math> with respect to <math>Q</math>. In the context of [[coding theory]], <math>D_\text{KL}(P \parallel Q)</math> can be constructed by measuring the expected number of extra [[bit]]s required to [[Huffman coding|code]] samples from <math>P</math> using a code optimized for <math>Q</math> rather than the code optimized for <math>P</math>.

Expressed in the language of [[Bayesian inference]], <math>D_\text{KL}(P \parallel Q)</math> is a measure of the information gained when one revises one's beliefs from the [[prior probability|prior probability distribution]] <math>Q</math> to the [[posterior distribution|posterior probability distribution]] <math>P</math>. In other words, it is the amount of information lost when <math>Q</math> is used to approximate <math>P</math>.<ref>{{cite book |last=Burnham |first=K. P. |last2=Anderson |first2=D. R. |year=2002 |title=Model Selection and Multi-Model Inference |location= |publisher=Springer |edition=2nd |page=51 |url=https://books.google.com/books?id=fT1Iu-h6E-oC&pg=PA51 |isbn=9780387953649 }}</ref> In applications, <math>P</math> typically represents the "true" distribution of data, observations, or a precisely calculated theoretical distribution, while <math>Q</math> typically represents a theory, model, description, or [[approximation]] of <math>P</math>. In order to find a distribution <math>Q</math> that is closest to <math>P</math>, we can minimize KL divergence and compute an [[information projection]].

The Kullback–Leibler divergence is a special case of a broader class of [[statistical divergence]]s called [[f-divergence|''f''-divergences]] as well as the class of [[Bregman divergence]]s. It is the only such divergence over probabilities that is a member of both classes. Although it is often intuited as a way of measuring the distance between [[probability distributions]], the Kullback–Leibler divergence is not a true [[metric (mathematics)|metric]]. It does not obey the [[triangle inequality]], and in general <math>D_\text{KL}(P \parallel Q)</math> does not equal <math>D_\text{KL}(Q \parallel P)</math>. However, its [[Infinitesimal generator (stochastic processes)|infinitesimal]] form, specifically its [[Hessian matrix|Hessian]], gives a [[metric tensor]] known as the [[Fisher information metric]].

Arthur Hobson proved that the Kullback–Leibler divergence is the only measure of difference between probability distributions that satisfies some desired properties, which are the canonical extension to those appearing in a commonly used [[Entropy (information theory)#Characterization|characterization of entropy]].<ref>{{cite book|last1=Hobson|first1=Arthur|title=Concepts in statistical mechanics.| date=1971| publisher=Gordon and Breach|location=New York|isbn=978-0677032405}}</ref> Consequently, [[mutual information]] is the only measure of mutual dependence that obeys certain related conditions, since it can be defined [[Mutual information#Relation to other quantities|in terms of Kullback–Leibler divergence]].

==Motivation==

[[File:KL-Gauss-Example.png|thumb|right|320px|Illustration of the Kullback–Leibler (KL) divergence for two [[normal distribution]]s. The typical asymmetry for the Kullback–Leibler divergence is clearly visible.]]

In information theory, the [[Kraft–McMillan inequality|Kraft–McMillan theorem]] establishes that any directly decodable coding scheme for coding a message to identify one value <math>x_i</math> out of a set of possibilities <math>X</math> can be seen as representing an implicit probability distribution <math>q(x_i)=2^{-l_i}</math> over <math>X</math>, where <math>l_i</math> is the length of the code for <math>x_i</math> in bits. Therefore, the Kullback–Leibler divergence can be interpreted as the expected extra message-length per datum that must be communicated if a code that is optimal for a given (wrong) distribution <math>Q</math> is used, compared to using a code based on the true distribution <math>P</math>.

:<math>\begin{align}
D_\text{KL}(P\parallel Q) &= -\sum_{x\in\mathcal{X}} p(x) \log q(x) + \sum_{x\in\mathcal{X}} p(x) \log p(x) \\
 &= \Eta(P, Q) - \Eta(P)
\end{align}</math>

where <math>\Eta(P,Q)</math> is the [[cross entropy]] of <math>P</math> and <math>Q</math>, and <math>\Eta(P)</math> is the [[information entropy|entropy]] of <math>P</math> (which is the same as the cross-entropy of P with itself). 

The KL divergence <math>KL(P \mathop{\|} Q)</math> can be thought of as something like a measurement of how far the distribution Q is from the distribution P. The cross-entropy <math>H(P,Q)</math> is itself such a measurement, but it has the defect that <math>H(P,P)=:H(P)</math> isn't zero, so we subtract <math>H(P)</math> to make <math>KL(P \mathop{\|} Q)</math> agree more closely with our notion of distance. (Unfortunately it still isn't symmetric.)

Note also that there is a relation between the Kullback–Leibler divergence and the "[[rate function]]" in the theory of [[large deviations]].<ref name="Sanov">{{cite journal | last1 = Sanov | first1 = I.N. | year = 1957 | title = On the probability of large deviations of random magnitudes | url = | journal = Mat. Sbornik | volume = 42 | issue = 84| pages = 11–44 }}</ref><ref name="Novak">Novak S.Y. (2011), ''Extreme Value Methods with Applications to Finance'' ch. 14.5 ([[Chapman & Hall]]). {{isbn|978-1-4398-3574-6}}.</ref>

==Properties==

* The Kullback–Leibler divergence is always [[non-negative]],
::<math>D_\text{KL}(P\parallel Q) \geq 0,</math>
: a result known as [[Gibbs' inequality]], with <math>D_\text{KL}(P\parallel Q)</math> zero [[if and only if]] <math>P = Q</math> [[almost everywhere]]. The entropy <math>\Eta(P)</math> thus sets a minimum value for the cross-entropy <math>\Eta(P, Q)</math>, the [[Expected value|expected]] number of [[bit]]s required when using a code based on <math>Q</math> rather than <math>P</math>; and the Kullback–Leibler divergence therefore represents the expected number of extra bits that must be transmitted to identify a value <math>x</math> drawn from <math>X</math>, if a code is used corresponding to the probability distribution <math>Q</math>, rather than the "true" distribution <math>P</math>.

* The Kullback–Leibler divergence remains well-defined for continuous distributions, and furthermore is invariant under [[Change of variables|parameter transformations]]. For example, if a transformation is made from variable <math>x</math> to variable <math>y(x)</math>, then, since <math>P(x) dx = P(y) dy</math> and <math>Q(x) dx = Q(y) dy</math> the Kullback–Leibler divergence may be rewritten:
::<math>\begin{align}
 D_\text{KL}(P \parallel Q)
 &= \int_{x_a}^{x_b} P(x)\log\left(\frac{P(x)}{Q(x)}\right)\, dx \\[6pt]
 &= \int_{y_a}^{y_b} P(y)\log\left(\frac{P(y)\, \frac{dy}{dx}}{Q(y)\, \frac{dy}{dx}}\right)\, dy
 = \int_{y_a}^{y_b} P(y)\log\left(\frac{P(y)}{Q(y)}\right)\, dy
\end{align}</math>
:where <math>y_a = y(x_a)</math> and <math>y_b = y(x_b)</math>. Although it was assumed that the transformation was continuous, this need not be the case. This also shows that the Kullback–Leibler divergence produces a [[dimensional analysis|dimensionally consistent]] quantity, since if <math>x</math> is a dimensioned variable, <math>P(x)</math> and <math>Q(x)</math> are also dimensioned, since e.g. <math>P(x) dx</math> is dimensionless. The argument of the logarithmic term is and remains dimensionless, as it must. It can therefore be seen as in some ways a more fundamental quantity than some other properties in information theory<ref name="VerduLecture">See the section "differential entropy – 4" in [http://videolectures.net/nips09_verdu_re/ Relative Entropy] video lecture by [[Sergio Verdú]] [[Conference on Neural Information Processing Systems|NIPS]] 2009</ref> (such as [[self-information]] or [[Shannon entropy]]), which can become undefined or negative for non-discrete probabilities.

* The Kullback–Leibler divergence is [[Additive map|additive]] for [[Independent random variables|independent distributions]] in much the same way as Shannon entropy. If <math>P_1, P_2</math> are independent distributions, with the joint distribution <math>P(x, y) = P_1(x)P_2(y)</math>, and <math>Q, Q_1, Q_2</math> likewise, then
::<math> D_\text{KL}(P \parallel Q) = D_\text{KL}(P_1 \parallel Q_1) + D_\text{KL}(P_2 \parallel Q_2).</math>

* The Kullback–Leibler divergence <math> D_\text{KL}(P \parallel Q)</math> is [[Convex function|convex]] in the pair of [[probability mass function]]s <math>(p,q)</math>, i.e. if <math>(p_1,q_1)</math> and <math>(p_2,q_2)</math> are two pairs of probability mass functions, then

::<math>D_\text{KL}(\lambda p_1 + (1 - \lambda) p_2 \parallel \lambda q_1 + (1 - \lambda) q_2) \le 
\lambda D_\text{KL}(p_1 \parallel q_1) + (1 - \lambda)D_\text{KL}(p_2 \parallel q_2) \text{ for } 0 \le \lambda \le 1.</math>

== Examples ==

===Multivariate normal distributions===
Suppose that we have two [[multivariate normal distribution]]s, with means <math>\mu_0, \mu_1</math> and with (non-singular) [[covariance matrix|covariance matrices]] <math>\Sigma_0, \Sigma_1.</math> If the two distributions have the same dimension, <math>k</math>, then the Kullback–Leibler divergence between the distributions is as follows:<ref>Duchi J., "[http://web.stanford.edu/~jduchi/projects/general_notes.pdf Derivations for Linear Algebra and Optimization]".</ref>{{rp|p. 13}}

:<math>D_\text{KL} (\mathcal{N}_0 \parallel \mathcal{N}_1) = \frac{1}{2}\left( \operatorname{tr} \left(\Sigma_1^{-1}\Sigma_0\right) + (\mu_1 - \mu_0)^\mathsf{T} \Sigma_1^{-1}(\mu_1 - \mu_0) - k + \ln \left(\frac{\det\Sigma_1}{\det\Sigma_0}\right) \right).</math>

The [[logarithm]] in the last term must be taken to base ''[[e (mathematical constant)|e]]'' since all terms apart from the last are base-''e'' logarithms of expressions that are either factors of the density function or otherwise arise naturally. The equation therefore gives a result measured in [[nat (unit)|nat]]s. Dividing the entire expression above by <math>ln(2)</math> yields the divergence in [[bit]]s.

A special case, and a common quantity in [[variational inference]], is the KL-divergence between a diagonal multivariate normal, and a standard normal distribution (with zero mean and unit variance):

:<math>D_\text{KL}\left(\mathcal{N}\left((\mu_1, \ldots, \mu_k)^\mathsf{T}, \operatorname{diag} (\sigma_1^2, \ldots, \sigma_k^2)\right) \parallel \mathcal{N}\left(\mathbf{0}, \mathbf{I}\right)\right) = {1 \over 2} \sum_{i=1}^k (\sigma_i^2 + \mu_i^2 - \ln(\sigma_i^2) - 1).</math>

==Relation to metrics==

One might be tempted to call the Kullback–Leibler divergence a "[[metric space|distance metric]]" on the space of probability distributions, but this would not be correct as it is not [[symmetric]] – that is, <math>D_\text{KL}(P\parallel Q) \neq D_\text{KL}(Q\parallel P)</math> – nor does it satisfy the [[triangle inequality]]. Even so, being a [[premetric]], it generates a [[topology#Topologies on sets|topology]] on the space of [[probability distributions]]. More concretely, if <math>\{P_1,P_2,\ldots\}</math> is a sequence of distributions such that

: <math>\lim_{n \to \infty} D_\text{KL}(P_n\parallel Q) = 0</math>

then it is said that

: <math>P_n \xrightarrow{D} Q .</math>

[[Pinsker's inequality]] entails that

: <math>P_n \xrightarrow{D} P \Rightarrow P_n \xrightarrow{TV} P,</math>

where the latter stands for the usual convergence in [[total variation]].

Following [[Alfréd Rényi|Rényi]] (1970, 1961)<ref>{{cite book | author=Rényi A. |title =Probability Theory|year=1970|pages=Appendix, Sec.4|nopp = y| publisher=Elsevier|isbn = 978-0-486-45867-0}}</ref><ref>{{citation | first=A. | last= Rényi | contribution=On measures of entropy and information | title= Proceedings of the 4th Berkeley Symposium on Mathematics, Statistics and Probability 1960 | year=1961 | pages=547–561 | contribution-url = http://digitalassets.lib.berkeley.edu/math/ucb/text/math_s4_v1_article-27.pdf }}</ref>

===Fisher information metric===
The Kullback–Leibler divergence is directly related to the [[Fisher information metric]]. This can be made explicit as follows. Assume that the probability distributions <math>P</math> and <math>Q</math> are both parameterized by some (possibly multi-dimensional) parameter <math>\theta</math>. Consider then two close-by values of <math>P = P(\theta)</math> and <math>Q = P(\theta_0)</math> so that the parameter <math>\theta</math> differs by only a small amount from the parameter value <math>\theta_0</math>. Specifically, up to first order one has (using the [[Einstein summation convention]])
:<math>P(\theta) = P(\theta_0) + \Delta\theta_j P_j(\theta_0) + \cdots</math>

with <math>\Delta\theta_j = (\theta - \theta_0)_j</math> a small change of <math>\theta</math> in the <math>j</math> direction, and <math>P_j\left(\theta_0\right) = \frac{\partial P}{\partial \theta_j}(\theta_0)</math> the corresponding rate of change in the probability distribution. Since the Kullback–Leibler divergence has an absolute minimum 0 for <math>P = Q</math>, i.e. <math> \theta = \theta_0 </math>, it changes only to ''second'' order in the small parameters <math>\Delta\theta_j</math>. More formally, as for any minimum, the first derivatives of the divergence vanish

:<math>\left.\frac{\partial}{\partial \theta_j}\right|_{\theta = \theta_0} D_{KL}(P(\theta) \parallel P(\theta_0)) = 0,</math>

and by the [[Taylor series|Taylor expansion]] one has up to second order

:<math>D_\text{KL}(P(\theta) \parallel P(\theta_0)) = \frac{1}{2} \Delta\theta_j\Delta\theta^k g_{jk}(\theta_0) + \cdots</math>

where the [[Hessian matrix]] of the divergence

:<math>g_{jk}(\theta_0) = \left.\frac{\partial^2}{\partial\theta_j\, \partial\theta^k} \right|_{\theta = \theta_0} D_\text{KL}(P(\theta) \parallel P(\theta_0))</math>

must be [[Positive-definite matrix|positive semidefinite]]. Letting <math>\theta_0</math> vary (and dropping the subindex 0) the Hessian <math>g_{jk}(\theta)</math> defines a (possibly degenerate) [[Riemannian metric]] on the {{mvar|&theta;}} parameter space, called the Fisher information metric.

==== Fisher information metric theorem ====
When <math>p_{(x, \rho)}</math> satisfies the following regularity conditions:

:<math>\tfrac{\partial \log(p)}{\partial \rho}, \tfrac{\partial^2 \log(p)}{\partial \rho^2}, \tfrac{\partial^3 \log(p)}{\partial \rho^3}</math> exist,
:<math>\begin{align}
 \left|\frac{\partial p}{\partial \rho}\right| &< F(x): \int_{x=0}^\infty F(x)\,dx < \infty, \\ 
 \left|\frac{\partial^2 p}{\partial \rho^2}\right| &< G(x): \int_{x=0}^\infty G(x)\,dx < \infty \\ 
 \left|\frac{\partial^3 \log(p)}{\partial \rho^3}\right| &< H(x): \int_{x=0}^\infty p(x, 0)H(x)\,dx < \xi < \infty
\end{align}</math>

where {{mvar|&xi;}} is independent of {{mvar|&rho;}}
: <math>
 \left.\int_{x=0}^\infty \frac{\partial p(x, \rho)}{\partial \rho}\right|_{\rho=0}\, dx =
 \left.\int_{x=0}^\infty \frac{\partial^2 p(x, \rho)}{\partial \rho^2}\right|_{\rho=0}\, dx = 0
</math>
 
then:
: <math>\mathcal{D}(p(x, 0) \parallel p(x, \rho)) = \frac{c\rho^2}{2} + \mathcal{O}(\rho^3) \text{ as } \rho\to0.</math>

=== Variation of information ===
Another information-theoretic metric is [[Variation of information]], which is roughly a symmetrization of [[conditional entropy]]. It is a metric on the set of [[Partition of a set|partitions]] of a discrete [[probability space]].

==Relation to other quantities of information theory==
Many of the other quantities of information theory can be interpreted as applications of the Kullback–Leibler divergence to specific cases.

===Self-information===
{{Main|Information content}}
The [[self-information]], also known as the [[information content]] of a signal, random variable, or [[Event (probability theory)|event]] is defined as the negative logarithm of the [[probability]] of the given outcome occurring.

When applied to a [[discrete random variable]], the self-information can be represented as{{Citation needed|reason=Needs verification|date=August 2017}}

:<math>\operatorname \operatorname{I}(m) = D_\text{KL}(\delta_{im} \parallel \{ p_i \}),</math>

is the Kullback–Leibler divergence of the probability distribution <math>P(i)</math> from a [[Kronecker delta]] representing certainty that <math>i = m</math> — i.e. the number of extra bits that must be transmitted to identify <math>i</math> if only the probability distribution <math>P(i)</math> is available to the receiver, not the fact that <math>i = m</math>.

===Mutual information===
The [[mutual information]],{{Citation needed|reason=Needs verification|date=August 2017}}

:<math>\begin{align}
 \operatorname{I}(X; Y) & = D_\text{KL}(P(X, Y) \parallel P(X)P(Y) ) \\
 & = \operatorname{E}_X \{D_\text{KL}(P(Y\mid X) \parallel P(Y) ) \} \\
 & = \operatorname{E}_Y \{D_\text{KL}(P(X\mid Y) \parallel P(X) ) \}
\end{align}</math>

is the Kullback–Leibler divergence of the product <math>P(X)P(Y)</math> of the two [[marginal probability]] distributions from the [[joint probability distribution]] <math>P(X,Y)</math> — i.e. the expected number of extra bits that must be transmitted to identify <math>X</math> and <math>Y</math> if they are coded using only their marginal distributions instead of the joint distribution. Equivalently, if the joint probability <math>P(X,Y)</math> ''is'' known, it is the expected number of extra bits that must on average be sent to identify <math>Y</math> if the value of <math>X</math> is not already known to the receiver.

===Shannon entropy===
The [[Shannon entropy]],{{Citation needed|reason=Needs verification|date=August 2017}}

:<math>\begin{align}
 \Eta(X) & = \operatorname{E}[\operatorname{I}_X(x)] \\
 & = \log(N) - D_\text{KL}(p_X(x) \parallel P_U(X) )
\end{align}</math>

is the number of bits which would have to be transmitted to identify <math>X</math> from <math>N</math> equally likely possibilities, ''less'' the Kullback–Leibler divergence of the uniform distribution on the [[random variate]]s of <math>X</math>, <math>P_U(X)</math>, from the true distribution <math>P(X)</math> — i.e. ''less'' the expected number of bits saved, which would have had to be sent if the value of <math>X</math> were coded according to the uniform distribution <math>P_U(X)</math> rather than the true distribution <math>P(X)</math>.

===Conditional entropy===
The [[conditional entropy]],{{Citation needed|reason=Needs verification|date=August 2017}}

:<math>\begin{align}
 \Eta(X \mid Y) & = \log(N) - D_\text{KL}(P(X,Y) \parallel P_U(X) P(Y)) \\
 & = \log(N) - D_\text{KL}(P(X,Y) \parallel P(X) P(Y) ) - D_\text{KL}(P(X) \parallel P_U(X)) \\
 & = \Eta(X) - \operatorname{I}(X;Y) \\
 & = \log(N) - \operatorname{E}_Y \bigl[D_\text{KL}(P(X\mid Y) \parallel P_U(X)) \bigr]
\end{align}</math>

is the number of bits which would have to be transmitted to identify <math>X</math> from <math>N</math> equally likely possibilities, ''less'' the Kullback–Leibler divergence of the product distribution <math>P_U(X) P(Y)</math> from the true joint distribution <math>P(X,Y)</math> — i.e. ''less'' the expected number of bits saved which would have had to be sent if the value of <math>X</math> were coded according to the uniform distribution <math>P_U(X)</math> rather than the conditional distribution <math>P(X|Y)</math> of <math>X</math> given <math>Y</math>.

===Cross entropy===
The [[cross entropy]] between two [[probability distribution]]s measures the average number of [[bit]]s needed to identify an event from a set of possibilities, if a coding scheme is used based on a given probability distribution {{mvar|q}}, rather than the "true" distribution {{mvar|p}}. The cross entropy for two distributions {{mvar|p}} and {{mvar|q}} over the same [[probability space]] is thus defined as follows:{{Citation needed| reason=Needs verification|date=August 2017}}

:<math>\Eta(p, q) = \operatorname{E}_p[-\log(q)] = \Eta(p) + D_\text{KL}(p \parallel q).</math>

==Bayesian updating==
In [[Bayesian statistics]] the Kullback–Leibler divergence can be used as a measure of the information gain in moving from a [[prior distribution]] to a [[posterior distribution]]: <math>p(x) \to p(x\mid I)</math>. If some new fact <math>Y = y</math> is discovered, it can be used to update the posterior distribution for <math>X</math> from <math>p(x\mid I)</math> to a new posterior distribution <math>p(x\mid y,I)</math> using [[Bayes' theorem]]:

:<math>p(x \mid y, I) = \frac{p(y \mid x, I) p(x \mid I)}{p(y \mid I)}</math>

This distribution has a new [[entropy in thermodynamics and information theory|entropy]]:

:<math>\Eta\big(p(-\mid y, I)\big) = -\sum_x p(x \mid y, I) \log p(x \mid y, I),</math>

which may be less than or greater than the original entropy <math>\Eta(p(-\mid I))</math>. However, from the standpoint of the new probability distribution one can estimate that to have used the original code based on <math>p(x\mid I)</math> instead of a new code based on <math>p(x\mid y, I)</math> would have added an expected number of bits:

:<math> D_\text{KL}\big(p(- \mid y, I) \parallel p(- \mid I) \big) = \sum_x p(x \mid y, I) \log\left(\frac{p(x \mid y, I)}{p(x \mid I)}\right) </math>

to the message length. This therefore represents the amount of useful information, or information gain, about <math>X</math>, that we can estimate has been learned by discovering <math>Y = y</math>.

If a further piece of data, <math>Y_2 = y_2</math>, subsequently comes in, the probability distribution for <math>x</math> can be updated further, to give a new best guess <math>p(x \mid y_1,y_2,I)</math>. If one reinvestigates the information gain for using <math>p(x \mid y_1,I)</math> rather than <math>p(x \mid I)</math>, it turns out that it may be either greater or less than previously estimated:

:<math>\sum_x p(x \mid y_1, y_2, I) \log\left(\frac{p(x \mid y_1, y_2, I)}{p(x \mid I)}\right)</math> may be ≤ or > than <math>\displaystyle \sum_x p(x \mid y_1, I) \log\left(\frac{p(x \mid y_1, I)}{p(x \mid I)}\right)</math>

and so the combined information gain does ''not'' obey the triangle inequality:

:<math>D_\text{KL} \big( p(- \mid y_1, y_2, I) \parallel p(- \mid I) \big)</math> may be <, = or > than <math>D_\text{KL}\big( p(- \mid y_1, y_2, I) \parallel p(- \mid y_1, I)\big) + D_\text{KL}\big(p(- \mid y_1, I) \parallel p(- \mid I)\big)</math>

All one can say is that on ''average'', averaging using <math>p(y_2 \mid y_1, x, I)</math>, the two sides will average out.

===Bayesian experimental design===
A common goal in [[Bayesian experimental design]] is to maximise the expected Kullback–Leibler divergence between the prior and the posterior.<ref>{{cite journal | last1 = Chaloner | first1 = K. | last2 = Verdinelli | first2 = I. | year = 1995 | title = Bayesian experimental design: a review | url = | journal = [[Statistical Science]] | volume = 10 | issue = 3| pages = 273–304 | doi = 10.1214/ss/1177009939 }}</ref> When posteriors are approximated to be Gaussian distributions, a design maximising the expected Kullback–Leibler divergence is called [[D-optimal design|Bayes d-optimal]].

==Discrimination information==
The Kullback–Leibler divergence <math display="inline">D_\text{KL}\bigl(p(x \mid H_1) \parallel p(x \mid H_0)\bigr)</math> can also be interpreted as the expected '''discrimination information''' for <math>H_1</math> over <math>H_0</math>: the mean information per sample for discriminating in favor of a hypothesis <math>H_1</math> against a hypothesis <math>H_0</math>, when hypothesis <math>H_1</math> is true.<ref>{{Cite book | last1=Press | first1 = W.H. | last2=Teukolsky | first2=S.A. | last3=Vetterling | first3=W.T. | last4=Flannery | first4=B.P. | year=2007 | title=Numerical Recipes: The Art of Scientific Computing | edition=3rd | publisher=Cambridge University Press | isbn=978-0-521-88068-8 | chapter=Section 14.7.2. Kullback–Leibler Distance | chapter-url=http://apps.nrbook.com/empanel/index.html#pg=756 | title-link = Numerical Recipes }}</ref> Another name for this quantity, given to it by [[I. J. Good]], is the expected [[weight of evidence]] for <math>H_1</math> over <math>H_0</math> to be expected from each sample.

The expected weight of evidence for <math>H_1</math> over <math>H_0</math> is '''not''' the same as the information gain expected per sample about the probability distribution <math>p(H)</math> of the hypotheses,

:<math>D_\text{KL}(p(x \mid H_1) \parallel p(x \mid H_0)) \neq IG = D_\text{KL}(p(H \mid x) \parallel p(H \mid I)).</math>

Either of the two quantities can be used as a [[utility function]] in Bayesian experimental design, to choose an optimal next question to investigate: but they will in general lead to rather different experimental strategies.

On the entropy scale of ''information gain'' there is very little difference between near certainty and absolute certainty—coding according to a near certainty requires hardly any more bits than coding according to an absolute certainty. On the other hand, on the [[logit]] scale implied by weight of evidence, the difference between the two is enormous – infinite perhaps; this might reflect the difference between being almost sure (on a probabilistic level) that, say, the [[Riemann hypothesis]] is correct, compared to being certain that it is correct because one has a mathematical proof. These two different scales of [[loss function]] for uncertainty are ''both'' useful, according to how well each reflects the particular circumstances of the problem in question.

===<span id="Principle of Minimum Discrimination Information"></span> Principle of minimum discrimination information===
The idea of Kullback–Leibler divergence as discrimination information led Kullback to propose the Principle of '''{{visible anchor|Minimum Discrimination Information}}''' (MDI): given new facts, a new distribution <math>f</math> should be chosen which is as hard to discriminate from the original distribution <math>f_0</math> as possible; so that the new data produces as small an information gain <math>D_\text{KL}(f \parallel f_0)</math> as possible.

For example, if one had a prior distribution <math>p(x,a)</math> over <math>x</math> and <math>a</math>, and subsequently learnt the true distribution of <math>a</math> was <math>u(a)</math>, then the Kullback–Leibler divergence between the new joint distribution for <math>x</math> and <math>a</math>, <math>q(x\mid a)u(a)</math>, and the earlier prior distribution would be:

:<math>D_\text{KL}(q(x \mid a)u(a) \parallel p(x, a)) = \operatorname{E}_{u(a)}\left\{D_\text{KL}(q(x \mid a) \parallel p(x \mid a))\right\} + D_\text{KL}(u(a) \parallel p(a)),</math>

i.e. the sum of the Kullback–Leibler divergence of <math>p(a)</math> the prior distribution for <math>a</math> from the updated distribution <math>u(a)</math>, plus the expected value (using the probability distribution <math>u(a)</math>) of the Kullback–Leibler divergence of the prior conditional distribution <math>p(x\mid a)</math> from the new conditional distribution <math>q(x\mid a)</math>. (Note that often the later expected value is called the ''conditional Kullback–Leibler divergence'' (or ''conditional relative entropy'') and denoted by <math>D_\text{KL}(q(x\mid a) \parallel p(x\mid a))</math><ref>Thomas M. Cover, Joy A. Thomas (1991) ''Elements of Information Theory'' ([[John Wiley & Sons]])</ref>{{rp|p. 22}}) This is minimized if <math>q(x\mid a)=p(x\mid a)</math> over the whole support of <math>u(a)</math>; and we note that this result incorporates Bayes' theorem, if the new distribution <math>u(a)</math> is in fact a δ function representing certainty that <math>a</math> has one particular value.

MDI can be seen as an extension of [[Laplace]]'s [[Principle of Insufficient Reason]], and the [[Principle of Maximum Entropy]] of [[E.T. Jaynes]]. In particular, it is the natural extension of the principle of maximum entropy from discrete to continuous distributions, for which Shannon entropy ceases to be so useful (see ''[[differential entropy]]''), but the Kullback–Leibler divergence continues to be just as relevant.

In the engineering literature, MDI is sometimes called the '''Principle of Minimum Cross-Entropy''' (MCE) or '''Minxent''' for short. Minimising the Kullback–Leibler divergence from <math>m</math> to <math>p</math> with respect to <math>m</math> is equivalent to minimizing the cross-entropy of <math>p</math> and <math>m</math>, since

:<math>\Eta(p, m) = \Eta(p) + D_\text{KL}(p \parallel m),</math>

which is appropriate if one is trying to choose an adequate approximation to <math>p</math>. However, this is just as often ''not'' the task one is trying to achieve. Instead, just as often it is <math>m</math> that is some fixed prior reference measure, and <math>p</math> that one is attempting to optimise by minimising <math>D_\text{KL}(p \parallel m)</math> subject to some constraint. This has led to some ambiguity in the literature, with some authors attempting to resolve the inconsistency by redefining cross-entropy to be <math>D_\text{KL}(p \parallel m)</math>, rather than <math>\Eta(p,m)</math>.

==Relationship to available work==
[[File:ArgonKLdivergence.png|thumb|220px|right|Pressure versus volume plot of available work from a mole of argon gas relative to ambient, calculated as <math>T_o</math> times the Kullback–Leibler divergence.]]
[[Surprisal]]s<ref>Myron Tribus (1961), ''Thermodynamics and Thermostatics'' (D. Van Nostrand, New York)</ref> add where probabilities multiply. The surprisal for an event of probability <math>p</math> is defined as <math>s = k \ln(1 / p)</math>. If <math>k</math> is <math>\left\{1, 1/\ln 2, 1.38 \times 10^{-23}\right\}</math> then surprisal is in <math>\{</math>nats, bits, or <math>J/K\}</math> so that, for instance, there are <math>N</math> bits of surprisal for landing all "heads" on a toss of <math>N</math> coins.

Best-guess states (e.g. for atoms in a gas) are inferred by maximizing the ''average surprisal'' <math>S</math> ([[entropy]]) for a given set of control parameters (like pressure <math>P</math> or volume <math>V</math>). This constrained [[entropy maximization]], both classically<ref>{{cite journal| last1 = Jaynes | first1 = E. T. | year = 1957 | title = Information theory and statistical mechanics | url = http://bayes.wustl.edu/etj/articles/theory.1.pdf| journal = Physical Review | volume = 106 | issue = 4| pages = 620–630 | doi = 10.1103/physrev.106.620 |bibcode = 1957PhRv..106..620J }}</ref> and quantum mechanically,<ref>{{cite journal | last1 = Jaynes | first1 = E. T. | year = 1957 | title = Information theory and statistical mechanics II | url = http://bayes.wustl.edu/etj/articles/theory.2.pdf| journal = Physical Review | volume = 108 | issue = 2| pages = 171–190 | doi = 10.1103/physrev.108.171 |bibcode = 1957PhRv..108..171J }}</ref> minimizes [[Josiah Willard Gibbs|Gibbs]] availability in entropy units<ref>J.W. Gibbs (1873), "A method of geometrical representation of thermodynamic properties of substances by means of surfaces", reprinted in ''The Collected Works of J. W. Gibbs, Volume I Thermodynamics'', ed. W. R. Longley and R. G. Van Name (New York: Longmans, Green, 1931) footnote page 52.</ref> <math>A \equiv -k\ln(Z)</math> where <math>Z</math> is a constrained multiplicity or [[partition function (mathematics)|partition function]].

When temperature <math>T</math> is fixed, free energy (<math>T \times A</math>) is also minimized. Thus if <math>T, V</math> and number of molecules <math>N</math> are constant, the [[Helmholtz free energy]] <math>F \equiv U - TS</math> (where <math>U</math> is energy) is minimized as a system "equilibrates." If <math>T</math> and <math>P</math> are held constant (say during processes in your body), the [[Gibbs free energy]] <math>G = U + PV - TS</math> is minimized instead. The change in free energy under these conditions is a measure of available [[Work (thermodynamics)|work]] that might be done in the process. Thus available work for an ideal gas at constant temperature <math>T_o</math> and pressure <math>P_o</math> is <math>W = \Delta G = NkT_o\Theta(V/V_o)</math> where <math>V_o = NkT_o/P_o</math> and <math>\Theta(x) = x - 1 - \ln x \ge 0</math> (see also [[Gibbs inequality]]).

More generally<ref>{{cite journal | last1 = Tribus | first1 = M. | last2 = McIrvine | first2 = E. C. | year = 1971 | title = Energy and information | url = | journal = Scientific American | volume = 224 | issue = 3| pages = 179–186 | doi=10.1038/scientificamerican0971-179| bibcode = 1971SciAm.225c.179T }}</ref> the [[exergy|work available]] relative to some ambient is obtained by multiplying ambient temperature <math>T_o</math> by Kullback–Leibler divergence or ''net surprisal'' <math>\Delta I \ge 0,</math> defined as the average value of <math>k\ln(p/p_o)</math> where <math>p_o</math> is the probability of a given state under ambient conditions. For instance, the work available in equilibrating a monatomic ideal gas to ambient values of <math>V_o</math> and <math>T_o</math> is thus <math>W = T_o \Delta I</math>, where Kullback–Leibler divergence

:<math>\Delta I = Nk\left[\Theta\left(\frac{V}{V_o}\right) + \frac{3}{2}\Theta\left(\frac{T}{T_o}\right)\right].</math>

The resulting contours of constant Kullback–Leibler divergence, shown at right for a mole of Argon at standard temperature and pressure, for example put limits on the conversion of hot to cold as in flame-powered air-conditioning or in the unpowered device to convert boiling-water to ice-water discussed here.<ref>{{cite journal | last1 = Fraundorf | first1 = P. | year = 2007 | title = Thermal roots of correlation-based complexity | url = http://www3.interscience.wiley.com/cgi-bin/abstract/117861985/ABSTRACT| journal = Complexity | volume = 13 | issue = 3| pages = 18–26 | doi= 10.1002/cplx.20195| bibcode = 2008Cmplx..13c..18F| arxiv = 1103.2481}}{{dead link|date=February 2019|bot=medic}}{{cbignore|bot=medic}}</ref> Thus Kullback–Leibler divergence measures thermodynamic availability in bits.

==Quantum information theory==
For [[density matrix|density matrices]] <math>P</math> and <math>Q</math> on a [[Hilbert space]], the K–L divergence (or [[quantum relative entropy]] as it is often called in this case) from <math>Q</math> to <math>P</math> is defined to be

:<math>D_\text{KL}(P \parallel Q) = \operatorname{Tr}(P(\log(P) - \log(Q))).</math>

In [[quantum information science]] the minimum of <math> D_\text{KL}(P\parallel Q)</math> over all separable states <math>Q</math> can also be used as a measure of [[quantum entanglement|entanglement]] in the state <math>P</math>.

==Relationship between models and reality==
Just as Kullback–Leibler divergence of "actual from ambient" measures thermodynamic availability, Kullback–Leibler divergence of "reality from a model" is also useful even if the only clues we have about reality are some experimental measurements. In the former case Kullback–Leibler divergence describes ''distance to equilibrium'' or (when multiplied by ambient temperature) the amount of ''available work'', while in the latter case it tells you about surprises that reality has up its sleeve or, in other words, ''how much the model has yet to learn''.

Although this tool for evaluating models against systems that are accessible experimentally may be applied in any field, its application to selecting a [[statistical model]] via [[Akaike information criterion]] are particularly well described in papers<ref>{{cite journal| last1 = Burnham | first1 = K.P. | last2 = Anderson | first2 = D.R. | year = 2001 | title = Kullback–Leibler information as a basis for strong inference in ecological studies | url = http://www.publish.csiro.au/paper/WR99107.htm| journal = Wildlife Research | volume = 28 | issue = 2| pages = 111–119 | doi = 10.1071/WR99107 }}</ref> and a book<ref>Burnham, K. P. and Anderson D. R. (2002), ''Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, Second Edition'' (Springer Science) {{isbn|978-0-387-95364-9}}.</ref> by Burnham and Anderson. In a nutshell the Kullback–Leibler divergence of reality from a model may be estimated, to within a constant additive term, by a function (like the squares summed) of the deviations observed between data and the model's predictions. Estimates of such divergence for models that share the same additive term can in turn be used to select among models.

When trying to fit parametrized models to data there are various estimators which attempt to minimize Kullback–Leibler divergence, such as [[maximum likelihood]] and [[maximum spacing estimation|maximum spacing]] estimators.

==Symmetrised divergence==
Kullback and Leibler themselves actually defined the divergence as:

:<math>D_\text{KL}(P \parallel Q) + D_\text{KL}(Q \parallel P)</math>

which is symmetric and nonnegative. This quantity has sometimes been used for [[feature selection]] in [[statistical classification|classification]] problems, where <math>P</math> and <math>Q</math> are the conditional [[Probability density function|pdfs]] of a feature under two different classes. In the Banking and Finance industries, this quantity is referred to as Population Stability Index, and is used to assess distributional shifts in model features through time.

An alternative is given via the <math>\lambda</math> divergence,

:<math>D_\lambda(P \parallel Q) = \lambda D_\text{KL}(P \parallel \lambda P + (1 - \lambda)Q) + (1 - \lambda) D_\text{KL}(Q \parallel \lambda P + (1 - \lambda)Q),</math>

which can be interpreted as the expected information gain about <math>X</math> from discovering which probability distribution <math>X</math> is drawn from, <math>P</math> or <math>Q</math>, if they currently have probabilities <math>\lambda</math> and <math>1-\lambda</math> respectively.{{Clarify|date=May 2018}} {{Citation needed|date=May 2018}}

The value <math>\lambda = 0.5</math> gives the [[Jensen–Shannon divergence]], defined by

:<math>D_\text{JS} = \frac{1}{2} D_\text{KL} (P \parallel M) + \frac{1}{2} D_\text{KL}(Q \parallel M)</math>

where <math>M</math> is the average of the two distributions,

:<math>M = \frac{1}{2}(P + Q).</math>

<math>D_{JS}</math> can also be interpreted as the capacity of a noisy information channel with two inputs giving the output distributions <math>P</math> and <math>Q</math>. The Jensen–Shannon divergence, like all ''f''-divergences, is ''locally'' proportional to the [[Fisher information metric]]. It is similar to the [[Hellinger metric]] (in the sense that induces the same affine connection on a [[statistical manifold]]).

==Relationship to other probability-distance measures==

There are many other important measures of [[Statistical distance|probability distance]]. Some of these are particularly connected with the Kullback–Leibler divergence. For example:
* The [[total variation distance of probability measures|total variation distance]], <math>\delta(p,q)</math>. This is connected to the divergence through [[Pinsker's inequality]]: <math>\delta(P, Q) \le \sqrt{\frac{1}{2} D_\text{KL}(P \parallel Q)}</math>
* The family of [[Rényi divergence]]s provide generalizations of the Kullback–Leibler divergence. Depending on the value of a certain parameter, <math>\alpha</math>, various inequalities may be deduced.

Other notable measures of distance include the [[Hellinger distance]], ''histogram intersection'', ''[[chi-squared test|Chi-squared statistic]]'', ''quadratic form distance'', ''[[matching distance|match distance]]'', ''[[Kolmogorov–Smirnov test|Kolmogorov–Smirnov distance]]'', and ''[[earth mover's distance]]''.<ref name="earth">{{cite journal | last1 = Rubner | first1 = Y. | authorlink3 = Leonidas J. Guibas | last2 = Tomasi | first2 = C. | last3 = Guibas | first3 = L. J. | year = 2000 | title = The earth mover's distance as a metric for image retrieval | url = | journal = [[International Journal of Computer Vision]] | volume = 40 | issue = 2| pages = 99–121 | doi = 10.1023/A:1026543900054 }}</ref>

==Data differencing==
{{Main|Data differencing}}
Just as ''absolute'' entropy serves as theoretical background for [[data compression|data ''compression'']], ''relative'' entropy serves as theoretical background for [[data differencing|data ''differencing'']] – the absolute entropy of a set of data in this sense being the data required to reconstruct it (minimum compressed size), while the relative entropy of a target set of data, given a source set of data, is the data required to reconstruct the target ''given'' the source (minimum size of a [[patch (computing)|patch]]).

==See also==
{{Div col|colwidth=20em}}
*[[Akaike information criterion]]
*[[Bayesian information criterion]]
*[[Bregman divergence]]
*[[Cross-entropy]]
*[[Deviance information criterion]]
*[[Entropic value at risk]]
*[[Entropy power inequality]]
*[[Information gain in decision trees]]
*[[Information gain ratio]]
*[[Information theory and measure theory]]
*[[Jensen–Shannon divergence]]
*[[Quantum relative entropy]]
*[[Solomon Kullback]] and [[Richard Leibler]]
{{div col end}}

==References==
{{Reflist|30em}}

==External links==
* [https://bitbucket.org/szzoli/ite/ Information Theoretical Estimators Toolbox]
* [https://github.com/evansenter/diverge Ruby gem for calculating Kullback–Leibler divergence]
* [https://arxiv.org/abs/1404.2000 Jon Shlens' tutorial on Kullback–Leibler divergence and likelihood theory]
* [http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=13089&objectType=file Matlab code for calculating Kullback–Leibler divergence for discrete distributions]
* [[Sergio Verdú]], [http://videolectures.net/nips09_verdu_re/ Relative Entropy], [[Conference on Neural Information Processing Systems|NIPS]] 2009. One-hour video lecture.
* [https://arxiv.org/abs/math/0604246 A modern summary of info-theoretic divergence measures]

{{DEFAULTSORT:Kullback-Leibler Divergence}}
[[Category:Entropy and information]]
[[Category:F-divergences]]
[[Category:Information geometry]]
[[Category:Thermodynamics]]