View source for Empirical distribution function

{{See also|Frequency distribution}}

{{CSS image crop|Image = ECDF-100.png|bSize = 2500|cWidth = 250|cHeight = 250|oTop = {{#expr:{{#invoke:Random|number|9}}*250}}|oLeft = {{#expr:{{#invoke:Random|number|9}}*250}}|Location = right|Description = The green curve, which asymptotically approaches heights of 0 and 1 without reaching them, is the true cumulative distribution function of the [[standard normal distribution]]. The grey hash marks represent the observations in a particular [[sample (statistics)|sample]] drawn from that distribution, and the horizontal steps of the blue step function (including the leftmost point in each step but not including the rightmost point) form the empirical distribution function of that sample. ({{purge|Click here to load a new graph.}})}}

In [[statistics]], an '''empirical distribution function''' is the distribution function associated with the [[empirical measure]] of a [[sample (statistics)|sample]]. This [[cumulative distribution function]] is a [[step function]] that jumps up by {{math|1/''n''}} at each of the {{math|''n''}} data points. Its value at any specified value of the measured variable is the fraction of observations of the measured variable that are less than or equal to the specified value.

The empirical distribution function is an estimate of the cumulative distribution function that generated the points in the sample. It converges with probability 1 to that underlying distribution, according to the [[Glivenko–Cantelli theorem]].  A number of results exist to quantify the rate of convergence of the empirical distribution function to the underlying cumulative distribution function.

== Definition ==
Let {{math|(''X''<sub>1</sub>, …, ''X''<sub>''n''</sub>)}} be [[iid|independent, identically distributed]] real random variables with the common [[cumulative distribution function]] {{math|''F''(''t'')}}. Then the '''empirical distribution function''' is defined as<ref name="vdv265">
  {{cite book
  | last = van der Vaart | first = A.W.
  | title = Asymptotic statistics
  | year = 1998
  | publisher = Cambridge University Press
  | isbn = 0-521-78450-6
  | page = 265
  }}
</ref><ref>[http://planetmath.org/empiricaldistributionfunction PlanetMath] {{webarchive |url=https://web.archive.org/web/20130509235041/http://planetmath.org/empiricaldistributionfunction |date=May 9, 2013 }}</ref>
:<math>\widehat F_n(t) = \frac{\mbox{number of elements in the sample} \leq t}{n} = \frac{1}{n} \sum_{i=1}^n \mathbf{1}_{X_i \le t},</math>
where <math style="vertical-align:-.3em">\mathbf{1}_{A}</math> is the [[indicator function|indicator]] of [[event (probability theory)|event]] {{math|''A''}}. For a fixed {{math|''t''}}, the indicator <math style="vertical-align:-.3em">\mathbf{1}_{X_i \le t}</math> is a [[Bernoulli distribution|Bernoulli random variable]] with parameter {{math|''p'' {{=}} ''F''(''t'')}}; hence <math style="vertical-align:-.3em">n \widehat F_n(t)</math> is a [[binomial distribution|binomial random variable]] with [[mean]] {{math|''nF''(''t'')}} and [[variance]] {{math|''nF''(''t'')(1 − ''F''(''t''))}}. This implies that <math style="vertical-align:-.3em">\widehat F_n(t)</math> is an [[bias of an estimator|unbiased]] estimator for {{math|''F''(''t'')}}.

However, in some textbooks, the definition is given as
<math>\widehat F_n(t) = \frac{1}{n+1} \sum_{i=1}^n \mathbf{1}_{X_i \le t}</math><ref>Coles, S. (2001) ''An Introduction to Statistical Modeling of Extreme Values''. Springer, p. 36, Definition 2.4.  {{ISBN|978-1-4471-3675-0}}.</ref><ref>Madsen, H.O., Krenk, S., Lind, S.C. (2006) ''Methods of Structural Safety''. Dover Publications. p. 148-149. {{ISBN|0486445976}}</ref>

==Small-sample properties==

The [[mean]] of the empirical distribution is an [[bias of an estimator|unbiased estimator]] of the mean of the population distribution.

The [[variance]] of the empirical distribution times <math>\tfrac{n}{n-1}</math> is an unbiased estimator of the variance of the population distribution.

== Asymptotic properties ==
Since the ratio {{math|(''n''&nbsp;+&nbsp;1)/''n''}} approaches 1 as {{math|''n''}} goes to infinity, the asymptotic properties of the two definitions that are given above are the same.

By the [[strong law of large numbers]], the estimator <math style="vertical-align:-.3em">\scriptstyle\widehat{F}_n(t)</math> converges to {{math|''F''(''t'')}} as {{math|''n'' → ∞}} [[almost sure convergence|almost surely]], for every value of {{math|''t''}}:<ref name="vdv265" />
: <math>
    \widehat F_n(t)\ \xrightarrow{\text{a.s.}}\ F(t);
  </math>
thus the estimator <math style="vertical-align:-.3em">\scriptstyle\widehat{F}_n(t)</math> is [[consistent estimator|consistent]]. This expression asserts the pointwise convergence of the empirical distribution function to the true cumulative distribution function. There is a stronger result, called the [[Glivenko–Cantelli theorem]], which states that the convergence in fact happens uniformly over {{math|''t''}}:<ref name="vdv266">{{cite book
  | last = van der Vaart | first = A.W.
  | title = Asymptotic statistics
  | year = 1998
  | publisher = Cambridge University Press
  | isbn = 0-521-78450-6
  | page = 266
  }}</ref>
: <math>
    \|\widehat F_n-F\|_\infty \equiv
    \sup_{t\in\mathbb{R}} \big|\widehat F_n(t)-F(t)\big|\ \xrightarrow{\text{a.s.}}\ 0.
  </math>
The sup-norm in this expression is called the [[Kolmogorov–Smirnov test|Kolmogorov–Smirnov statistic]] for testing the goodness-of-fit between the empirical distribution <math style="vertical-align:-.3em">\scriptstyle\widehat{F}_n(t)</math> and the assumed true cumulative distribution function {{math|''F''}}. Other [[norm (mathematics)|norm function]]s may be reasonably used here instead of the sup-norm. For example, the [[Lp norm|L<sup>2</sup>-norm]] gives rise to the [[Cramér–von Mises criterion|Cramér–von Mises statistic]].

The asymptotic distribution can be further characterized in several different ways. First, the  [[central limit theorem]] states that ''pointwise'', <math style="vertical-align:-.3em">\scriptstyle\widehat{F}_n(t)</math> has asymptotically normal distribution with the standard <math style="vertical-align:-.3em">\sqrt{n}</math> rate of convergence:<ref name="vdv265"/>
: <math>
    \sqrt{n}\big(\widehat F_n(t) - F(t)\big)\ \ \xrightarrow{d}\ \ \mathcal{N}\Big( 0, F(t)\big(1-F(t)\big) \Big).
  </math>
This result is extended by the [[Donsker’s theorem]], which asserts that the ''[[empirical process]]'' <math style="vertical-align:-.3em">\scriptstyle\sqrt{n}(\widehat{F}_n - F)</math>, viewed as a function indexed by <math style="vertical-align:-.3em">\scriptstyle t\in\mathbb{R}</math>, [[convergence in distribution|converges in distribution]] in the [[Skorokhod space]] <math style="vertical-align:-.3em">\scriptstyle D[-\infty, +\infty]</math> to the mean-zero [[Gaussian process]] <math style="vertical-align:-.3em">\scriptstyle G_F = B \circ F</math>, where {{math|''B''}} is the standard [[Brownian bridge]].<ref name="vdv266"/> The covariance structure of this Gaussian process is
: <math>
    \operatorname{E}[\,G_F(t_1)G_F(t_2)\,] = F(t_1\wedge t_2) - F(t_1)F(t_2).
  </math>
The uniform rate of convergence in Donsker’s theorem can be quantified by the result known as the [[Hungarian embedding]]:<ref name="vdv268">{{cite book
  | last = van der Vaart | first = A.W.
  | title = Asymptotic statistics
  | year = 1998
  | publisher = Cambridge University Press
  | isbn = 0-521-78450-6
  | page = 268
  }}</ref>
: <math>
    \limsup_{n\to\infty} \frac{\sqrt{n}}{\ln^2 n} \big\| \sqrt{n}(\widehat F_n-F) - G_{F,n}\big\|_\infty < \infty, \quad \text{a.s.}
  </math>

Alternatively, the rate of convergence of <math style="vertical-align:-.3em">\scriptstyle\sqrt{n}(\widehat{F}_n-F)</math> can also be quantified in terms of the asymptotic behavior of the sup-norm of this expression. Number of results exist in this venue, for example the [[Dvoretzky–Kiefer–Wolfowitz inequality]] provides bound on the tail probabilities of <math style="vertical-align:-.3em">\scriptstyle\sqrt{n}\|\widehat{F}_n-F\|_\infty</math>:<ref name="vdv268"/>
: <math>
    \Pr\!\Big( \sqrt{n}\|\widehat{F}_n-F\|_\infty > z \Big) \leq 2e^{-2z^2}.
  </math>
In fact, Kolmogorov has shown that if the cumulative distribution function {{math|''F''}} is continuous, then the expression <math style="vertical-align:-.3em">\scriptstyle\sqrt{n}\|\widehat{F}_n-F\|_\infty</math> converges in distribution to <math style="vertical-align:-.3em">\scriptstyle\|B\|_\infty</math>, which has the [[Kolmogorov distribution]] that does not depend on the form of {{math|''F''}}.

Another result, which follows from the [[law of the iterated logarithm]], is that <ref name="vdv268"/>
: <math>
    \limsup_{n\to\infty} \frac{\sqrt{n}\|\widehat{F}_n-F\|_\infty}{\sqrt{2\ln\ln n}} \leq \frac12, \quad \text{a.s.}
  </math>
and
: <math>
    \liminf_{n\to\infty} \sqrt{2n\ln\ln n} \|\widehat{F}_n-F\|_\infty = \frac{\pi}{2}, \quad \text{a.s.}
  </math>

== Confidence Intervals and Confidence Bands ==
{{cleanup|reason = The text is missing some mathematical notations, has extra spaces, and various grammatical errors.|date=November 2019}}

The definition of a confidence interval can be extended to a function by considering x fixed and calculating a 1 - α '''pointwise confidence interval''' about x. Formally, C(x) defines the region defined by the pointwise confidence interval, given by:

<math>P({F(x) \in C(x)}) \geq 1 - \alpha</math>

The issue with this pointwise definition is that each of the x values are computed independently. So if we computed the pointwise confidence interval for each x, any individual x would obey the confidence interval. However, it would not guarantee that for each x, C(x) simultaneously would be a  confidence region because the guarantee is only for each point in isolation.

Due to this issue, we can instead define a 1 - α '''confidence band or confidence envelope''' that is applied to all x (denoted by B(x)):

<math>P({F(x) \in B(x), \forall x}) \geq 1 - \alpha</math>

Using a result called the [[Dvoretzky–Kiefer–Wolfowitz inequality|Dvoretzky-Kiefer-Wolfowitz (DKW) inequality]]<ref>{{Citation|title=Dvoretzky–Kiefer–Wolfowitz inequality|date=2018-12-15|url=https://en.wikipedia.org/w/index.php?title=Dvoretzky%E2%80%93Kiefer%E2%80%93Wolfowitz_inequality&oldid=873880410|work=Wikipedia|language=en|access-date=2019-11-20}}</ref>, a two sided confidence band is given by the following:

<math>P(sup_{x \in \Re}|\hat{F_n(x)-F(x)| > \epsilon})\leq 2e^{-2n\epsilon^2}</math>

Notice that this defines the probability of being ''outside'' a particular band around F(x). So setting the RHS to α, we can find an  to create a confidence band:

<math>\alpha = 2e^{-2n\epsilon^2}</math>

<math>\log\left ( \frac{\alpha}{2} \right ) = -2n\epsilon^2</math>

<math>\epsilon = \sqrt{\frac{1}{2n}\log\left ( \frac{2}{\alpha} \right )}</math>

Thus, we can define our confidence bands as such:

<math>L(x) = max\{\hat{F(x)}- \epsilon, 0\}</math>

<math>U(x) = min\{\hat{F(x)}+ \epsilon, 1\}</math>

== Simulation ==
<syntaxhighlight lang="python3">
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
from statsmodels.tools.tools import ECDF

def epsilon(n, alpha=0.05):
   return np.sqrt(1. / (2. * n) * np.log(2. / alpha))

def plot_dist(rv, n, name, low, high, pltnum=111):
    samples = rv.rvs(n)
    ecdf = ECDF(samples)

    x = np.linspace(low, high, 10000)
    eps = epsilon(n=len(samples))
    df = pd.DataFrame(ecdf(x), index=x)
    df['ecdf'] = ecdf(x)
    df['cdf'] = rv.cdf(x)
    
    plt.subplot(pltnum)
    df['ecdf'].plot(label='ECDF')
    df['cdf'].plot(label='CDF', style='r--')
    df['upper'] = pd.Series(ecdf(x), index=x).apply(lambda x: min(x + eps, 1.))
    df['lower'] = pd.Series(ecdf(x), index=x).apply(lambda x: max(x - eps, 0.))
    plt.fill_between(x, df['upper'], df['lower'], 
                     alpha=0.1, label='Confidence Band')
    plt.legend(loc='best')
    plt.title('%s ECDF (n=%d)' % (name, len(samples)))
</syntaxhighlight>Now that we have some convenient helper functions, let's plot several different distributions to visually see how the confidence bands work.<syntaxhighlight lang="python3">
# Plot a normal ECDF and confidence bands
plt.figure(1, figsize=(12, 10))
norm = stats.norm(0., 1.)
plot_dist(norm, 50, name='Normal Distribution', low=-3, high=3, pltnum=221)
plot_dist(norm, 100, name='Normal Distribution', low=-3, high=3, pltnum=222)
plot_dist(norm, 500, name='Normal Distribution', low=-3, high=3, pltnum=223)
plot_dist(norm, 1000, name='Normal Distribution', low=-3, high=3, pltnum=224)
plt.show()
</syntaxhighlight><br />

== See also ==
* [[Càdlàg]] functions
* [[Count data]]
* [[Distribution fitting]]
* [[Dvoretzky–Kiefer–Wolfowitz inequality]]
* [[Empirical probability]]
* [[Empirical process]]
* [[Quantile#Estimating quantiles from a sample|Estimating quantiles from a sample]]
* [[Frequency (statistics)]]
* [[Kaplan–Meier estimator]] for censored processes
* [[Survival function]]

== References ==
{{reflist}}

== Further reading ==
* {{cite book
  | last1 = Shorack | first1 = G.R.
  | last2 = Wellner | first2 = J.A.
  | title = Empirical Processes with Applications to Statistics
  | year = 1986
  | publisher = Wiley
  | location = New York
  | isbn = 0-471-86725-X
  | ref = harv
  }}

== External links ==
* {{Commons category-inline|Cumulative frequency distribution|Empirical distribution functions}}

{{-}}
{{Statistics|inference}}

{{DEFAULTSORT:Empirical Distribution Function}}
[[Category:Nonparametric statistics]]
[[Category:Empirical process]]