Changes
Undid revision 865463830 by 24.12.193.145 (talk)
{{short description|A method of estimating the parameters of a statistical model, given observations}}
{{About|the statistical techniques|computer data storage|Partial response maximum likelihood}}
{{merge from|Partial likelihood methods for panel data|date=October 2017}}
{{More footnotes|date=September 2009}}
In statistics, '''maximum likelihood estimation''' ('''MLE''') is a method of [[estimator|estimating]] the [[Statistical parameter|parameters]] of a [[statistical model]], given observations. MLE attempts to find the parameter values that maximize the [[likelihood function]], given the observations. The resulting estimate is called a '''maximum likelihood estimate''', which is also abbreviated as MLE.
The method of maximum likelihood is used with a wide range of statistical analyses. As an example, suppose that we are interested in the heights of adult female penguins, but are unable to measure the height of every penguin in a population (due to cost or time constraints). Assuming that the heights are [[Normal distribution|normally distributed]] with some unknown [[mean]] and [[variance]], the mean and variance can be estimated with MLE while only knowing the heights of some sample of the overall population. MLE would accomplish that by taking the mean and variance as parameters and finding particular parametric values that make the observed results the most probable given the normal model.
From the point of view of [[Bayesian inference]], MLE is a special case of [[maximum a posteriori estimation]] (MAP) that assumes a [[uniform distribution (continuous)|uniform]] [[Prior probability|prior distribution]] of the parameters. In [[frequentist inference]], MLE is one of several methods to get estimates of parameters without using prior distributions. Priors are avoided by not making probability statements about the parameters, but only about their estimates, whose properties are fully defined by the observations and the statistical model.
== Principles ==
The method of maximum likelihood is based on the [[likelihood function]], <math>\mathcal L(\theta\,;x)</math>. We are given a [[statistical model]], i.e. a family of distributions <math>\{ f(\cdot\,;\theta) \mid \theta \in \Theta \}</math>, where <math>\theta</math> denotes the (possibly multi-dimensional) parameter for the model. The method of maximum likelihood finds the values of the model parameter, <math>\theta</math>, that maximize the likelihood function, <math>\mathcal L(\theta\,;x)</math>. Intuitively, this selects the parameter values that make the data most probable.
The method defines a maximum likelihood estimate:
: <math>
\hat\theta \in \{ \underset{\theta\in\Theta}{\operatorname{arg\,max}}\ \mathcal L(\theta\,;x) \},
</math>
if a maximum exists.
In practice, it is often convenient to work with the [[natural logarithm]] of the likelihood function, called the log-likelihood:
: <math>
\ell(\theta\,;x) = \ln\mathcal{L}(\theta\,;x),
</math>
or the average log-likelihood:
: <math>
\hat\ell(\theta\,;x) = \frac1n \ln\mathcal{L(\theta\,;x)}.
</math>
The [[circumflex|hat]] over <math>\ell</math> indicates that it is akin to an [[estimator]]. Indeed, <math>\hat\ell</math> estimates the expected log-likelihood of a single observation in the model.
An MLE is the same regardless of whether we maximize the likelihood or the log-likelihood, because log is a [[Monotonic function|strictly increasing]] function.
For many models, a maximum likelihood estimator can be found as an explicit function of the observed data <math>x</math>. For many other models, however, no closed-form solution to the maximization problem is known or available, and an MLE can only be found via numerical [[global optimization]]. For some problems, there may be multiple values that maximize the likelihood. For other problems, no maximum likelihood estimate exists: either the log-likelihood function increases without ever reaching a [[supremum]] value, or the supremum does exist but is outside the bounds of <math>\Theta</math>, the set of acceptable parameter values.
== Properties ==
A maximum likelihood estimator is an [[extremum estimator]] obtained by maximizing, as a function of ''θ'', the ''objective function'' (cf. [[loss function]]) <math>\hat\ell(\theta\,;x)</math>. If the data are [[independent and identically distributed]], then we have
: <math>
\hat\ell(\theta\,;x)=\frac1n \sum_{i=1}^n \ln f(x_i\mid\theta),
</math>
this being the sample analogue of the expected log-likelihood <math>\ell(\theta) = \operatorname{E}[\, \ln f(x_i\mid\theta) \,]</math>, where this expectation is taken with respect to the true density.
Maximum-likelihood estimators have no optimum properties for finite samples, in the sense that (when evaluated on finite samples) other estimators may have greater concentration around the true parameter-value.<ref>{{harvtxt |Pfanzagl |1994 |p=206 }}</ref> However, like other estimation methods, maximum likelihood estimation possesses a number of attractive [[Asymptotic theory (statistics)|limiting properties]]: As the sample size increases to infinity, sequences of maximum likelihood estimators have these properties:
* [[consistency of an estimator|Consistency]]: the sequence of MLEs converges in probability to the value being estimated.
* [[efficient estimator|Efficiency]], i.e. it achieves the [[Cramér–Rao lower bound]] when the sample size tends to infinity. This means that no consistent estimator has lower asymptotic [[mean squared error]] than the MLE (or other estimators attaining this bound).
* Second-order efficiency after correction for bias.
=== Consistency ===
Under the conditions outlined below, the maximum likelihood estimator is '''[[consistent estimator|consistent]]'''. The consistency means that if the data were generated by <math>f(\cdot\,;\theta_0)</math> and we have a sufficiently large number of observations ''n'', then it is possible to find the value of ''θ''<sub>0</sub> with arbitrary precision. In mathematical terms this means that as ''n'' goes to infinity the estimator <math>\hat\theta</math> [[convergence in probability|converges in probability]] to its true value:
: <math>
\hat\theta_\mathrm{mle}\ \xrightarrow{\text{p}}\ \theta_0.
</math>
Under slightly stronger conditions, the estimator converges [[almost sure convergence|almost surely]] (or ''strongly''):
: <math>
\hat\theta_\mathrm{mle}\ \xrightarrow{\text{a.s.}}\ \theta_0.
</math>
Note that, in practical applications, data is never generated by <math>f(\cdot\,;\theta_0)</math>. Rather, <math>f(\cdot\,;\theta_0)</math> is a model, often in idealized form, of the process that generated the data. It is a common aphorism in statistics that ''[[all models are wrong]]''. Thus, true consistency does not occur in practical applications. Nevertheless, consistency is often considered to be a desirable property for an estimator to have.
To establish consistency, the following conditions are sufficient.<ref>{{harvtxt |Newey |McFadden |1994 |loc=Theorem 2.5. }}</ref>
{{ordered list
|1= '''[[Identifiability|Identification]]''' of the model:
: <math>
\theta \neq \theta_0 \quad \Leftrightarrow \quad f(\cdot\mid\theta)\neq f(\cdot\mid\theta_0).
</math>
In other words, different parameter values ''θ'' correspond to different distributions within the model. If this condition did not hold, there would be some value ''θ''<sub>1</sub> such that ''θ''<sub>0</sub> and ''θ''<sub>1</sub> generate an identical distribution of the observable data. Then we would not be able to distinguish between these two parameters even with an infinite amount of data—these parameters would have been [[observational equivalence|observationally equivalent]]. <br />
The identification condition is absolutely necessary for the ML estimator to be consistent. When this condition holds, the limiting likelihood function ''ℓ''(''θ''{{!}}·) has unique global maximum at ''θ''<sub>0</sub>.
|2= '''Compactness''': the parameter space Θ of the model is [[compact set|compact]].
[[File:Ee noncompactness.svg|240px|right]]
The identification condition establishes that the log-likelihood has a unique global maximum. Compactness implies that the likelihood cannot approach the maximum value arbitrarily close at some other point (as demonstrated for example in the picture on the right).
Compactness is only a sufficient condition and not a necessary condition. Compactness can be replaced by some other conditions, such as:
{{unordered list
| both [[Concave function|concavity]] of the log-likelihood function and compactness of some (nonempty) upper [[level set]]s of the log-likelihood function, or
| existence of a compact [[Neighbourhood (mathematics)|neighborhood]] ''N'' of ''θ''<sub>0</sub> such that outside of ''N'' the log-likelihood function is less than the maximum by at least some {{nowrap|''ε'' > 0}}.
}}
|3= '''Continuity''': the function ln ''f''(''x''{{!}}''θ'') is continuous in ''θ'' for almost all values of ''x'':
: <math>
\operatorname{P}\!\big[\; \ln f(x\mid\theta) \;\in\; \mathbb{C}^0(\Theta) \;\big] = 1.
</math>
The continuity here can be replaced with a slightly weaker condition of [[upper semi-continuous|upper semi-continuity]].
|4= '''Dominance''': there exists ''D''(''x'') integrable with respect to the distribution ''f''(''x''{{!}}''θ''<sub>0</sub>) such that
: <math>
\big|\ln f(x\mid\theta)\big| < D(x) \quad \text{ for all } \theta\in\Theta.
</math>
By the [[uniform law of large numbers]], the dominance condition together with continuity establish the '''uniform convergence in probability''' of the log-likelihood:
: <math>
\sup_{\theta\in\Theta} \big|\hat\ell(\theta\mid x) - \ell(\theta)\,\big|\ \xrightarrow{\text{p}}\ 0.
</math>
}}
The dominance condition can be employed in the case of [[i.i.d.]] observations. In the non-i.i.d. case, the uniform convergence in probability can be checked by showing that the sequence <math>\hat\ell(\theta\mid x)</math> is [[stochastic equicontinuity|stochastically equicontinuous]].
If one wants to demonstrate that the ML estimator <math>\hat\theta</math> converges to ''θ''<sub>0</sub> [[almost sure convergence|almost surely]], then a stronger condition of '''uniform convergence almost surely''' has to be imposed:
: <math>
\sup_{\theta\in\Theta} \big\|\;\hat\ell(x\mid\theta) - \ell(\theta)\;\big\| \ \xrightarrow{\text{a.s.}}\ 0.
</math>
Additionally, if (as assumed above) the data were generated by <math>f(\cdot\,;\theta_0)</math>, then under certain conditions, it can also be shown that the maximum likelihood estimator [[Convergence in distribution|converges in distribution]] to a normal distribution. Specifically,<ref>{{harvtxt |Newey |McFadden |1994 |loc=Theorem 3.3. }}</ref>
: <math>
\sqrt{n}\big(\hat\theta_\mathrm{mle} - \theta_0\big)\ \xrightarrow{d}\ \mathcal{N}(0,\,I^{-1})
</math>
where {{math|''I''}} is the [[Fisher information|Fisher information matrix]].
=== Functional invariance ===
The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, if <math>\widehat{\theta}</math> is the MLE for <math>\theta</math>, and if <math>g(\theta)</math> is any transformation of <math>\theta</math>, then the MLE for <math>\alpha=g(\theta)</math> is by definition
:<math>\widehat{\alpha} = g(\,\widehat{\theta}\,). \,</math>
It maximizes the so-called [[Likelihood function#Profile likelihood|profile likelihood]]:
:<math>\bar{L}(\alpha) = \sup_{\theta: \alpha = g(\theta)} L(\theta). \, </math>
The MLE is also invariant with respect to certain transformations of the data. If <math>y=g(x)</math> where <math>g</math> is one to one and does not depend on the parameters to be estimated, then the density functions satisfy
:<math>f_Y(y) = \frac{f_X(x)}{|g'(x)|} </math>
and hence the likelihood functions for <math>X</math> and <math>Y</math> differ only by a factor that does not depend on the model parameters.
For example, the MLE parameters of the log-normal distribution are the same as those of the normal distribution fitted to the logarithm of the data.
===Higher-order properties===
As noted above, the maximum likelihood estimator is {{sqrt|''n'' }}-consistent and asymptotically efficient, meaning that it reaches the [[Cramér–Rao bound]]:
: <math>
\sqrt{n}(\hat\theta_\text{mle} - \theta_0)\ \ \xrightarrow{d}\ \ \mathcal{N}(0,\ I^{-1}),
</math>
where <math>I</math> is the [[Fisher information matrix]]:
: <math>
I_{jk} = \operatorname{E}_X\bigg[\;{-\frac{\partial^2\ln f_{\theta_0}(X_t)}{\partial\theta_j\,\partial\theta_k}}
\;\bigg].
</math>
In particular, it means that the [[bias of an estimator|bias]] of the maximum likelihood estimator is equal to zero up to the order {{frac|1|{{sqrt|''n'' }}}}. However, when we consider the higher-order terms in the [[Edgeworth expansion|expansion]] of the distribution of this estimator, it turns out that {{math|''θ''<sub>mle</sub>}} has bias of order {{frac|1|''n''}}. This bias is equal to (componentwise)<ref>{{harvtxt |Cox |Snell |1968 | loc=formula (20) }}</ref>
: <math>
b_h \equiv \operatorname{E}\bigg[\;(\hat\theta_\mathrm{mle} - \theta_0)_h\;\bigg]
= \frac{1}{n} \sum_{i, j, k = 1}^n I^{h i}I^{j k} \big( \frac{1}{2} K_{i j k} + J_{j,i k} \big)
</math>
where <math>I^{j k}</math> denotes the (''j,k'')-th component of the ''inverse'' Fisher information matrix <math>I^{-1}</math>, and
: <math>
\tfrac12 K_{ijk} + J_{j,ik} = \operatorname{E}_X \bigg[\;
\frac12 \frac{\partial^3 \ln f_{\theta_0}(X_t)}{\partial\theta_i\,\partial\theta_j\,\partial\theta_k} +
\frac{\partial\ln f_{\theta_0}(X_t)}{\partial\theta_j} \frac{\partial^2\ln f_{\theta_0}(X_t)}{\partial\theta_i\,\partial\theta_k}
\; \bigg] .
</math>
Using these formulae it is possible to estimate the second-order bias of the maximum likelihood estimator, and ''correct'' for that bias by subtracting it:
: <math>
\hat\theta^*_\text{mle} = \hat\theta_\text{mle} - \hat{b}.
</math>
This estimator is unbiased up to the terms of order {{frac|1|''n''}}, and is called the '''bias-corrected maximum likelihood estimator'''.
This bias-corrected estimator is ''second-order efficient'' (at least within the curved exponential family), meaning that it has minimal mean squared error among all second-order bias-corrected estimators, up to the terms of the order {{frac|1|''n'' ² }}. It is possible to continue this process, that is to derive the third-order bias-correction term, and so on. However, as was shown by {{harvtxt| Kano |1996 }}, the maximum likelihood estimator is '''not''' third-order efficient.
==Relation to Bayesian inference==
A maximum likelihood estimator coincides with the [[Maximum a posteriori|most probable]] [[Bayesian estimator]] given a [[Uniform distribution (continuous)|uniform]] [[prior probability|prior distribution]] on the [[parameter space|parameters]]. Indeed, the [[maximum a posteriori estimate]] is the parameter {{mvar|θ}} that maximizes the probability of {{mvar|θ}} given the data, given by Bayes' theorem:
: <math>
\operatorname{P}(\theta\mid x_1,x_2,\ldots,x_n) = \frac{f(x_1,x_2,\ldots,x_n\mid\theta)\operatorname{P}(\theta)}{\operatorname{P}(x_1,x_2,\ldots,x_n)}
</math>
where <math>P(\theta)</math> is the prior distribution for the parameter {{mvar|θ}} and where <math>\operatorname{P}(x_1,x_2,\ldots,x_n)</math> is the probability of the data averaged over all parameters. Since the denominator is independent of {{mvar|θ}}, the Bayesian estimator is obtained by maximizing <math>f(x_1,x_2,\ldots,x_n\mid\theta)\operatorname{P}(\theta)</math> with respect to {{mvar|θ}}. If we further assume that the prior <math>P(\theta)</math> is a uniform distribution, the Bayesian estimator is obtained by maximizing the likelihood function <math>f(x_1,x_2,\ldots,x_n\mid\theta)</math>. Thus the Bayesian estimator coincides with the maximum likelihood estimator for a uniform prior distribution <math>\operatorname{P}(\theta)</math>.
== Examples ==
{{merge from|Maximum likelihood estimation with flow data|date=October 2017}}
=== Discrete uniform distribution ===
{{Main|German tank problem}}
Consider a case where ''n'' tickets numbered from 1 to ''n'' are placed in a box and one is selected at random (''see [[Uniform distribution (discrete)|uniform distribution]]''); thus, the sample size is 1. If ''n'' is unknown, then the maximum likelihood estimator <math>\hat{n}</math> of ''n'' is the number ''m'' on the drawn ticket. (The likelihood is 0 for ''n'' < ''m'', {{frac|1|''n''}} for ''n'' ≥ ''m'', and this is greatest when ''n'' = ''m''. Note that the maximum likelihood estimate of ''n'' occurs at the lower extreme of possible values {''m'', ''m'' + 1, ...}, rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) The [[expected value]] of the number ''m'' on the drawn ticket, and therefore the expected value of <math>\hat{n}</math>, is (''n'' + 1)/2. As a result, with a sample size of 1, the maximum likelihood estimator for ''n'' will systematically underestimate ''n'' by (''n'' − 1)/2.
=== Discrete distribution, finite parameter space ===
Suppose one wishes to determine just how biased an [[unfair coin]] is. Call the probability of tossing a ‘[[Obverse and reverse|head]]’ ''p''. The goal then becomes to determine ''p''.
Suppose the coin is tossed 80 times: i.e. the sample might be something like ''x''<sub>1</sub> = H, ''x''<sub>2</sub> = T, ..., ''x''<sub>80</sub> = T, and the count of the number of [[Obverse and reverse|heads]] "H" is observed.
The probability of tossing [[Obverse and reverse|tails]] is 1 − ''p'' (so here ''p'' is ''θ'' above). Suppose the outcome is 49 heads and 31 [[Obverse and reverse|tails]], and suppose the coin was taken from a box containing three coins: one which gives heads with probability ''p'' = {{frac|1|3}}, one which gives heads with probability ''p'' = {{frac|1|2}} and another which gives heads with probability ''p'' = {{frac|2|3}}. The coins have lost their labels, so which one it was is unknown. Using '''maximum likelihood estimation''' the coin that has the largest likelihood can be found, given the data that were observed. By using the [[probability mass function]] of the [[binomial distribution]] with sample size equal to 80, number successes equal to 49 but for different values of ''p'' (the "probability of success"), the likelihood function (defined below) takes one of three values:
:<math>
\begin{align}
\operatorname{P}\big[\;\mathrm{H} = 49 \mid p=\tfrac{1}{3}\;\big] & = \binom{80}{49}(\tfrac{1}{3})^{49}(1-\tfrac{1}{3})^{31} \approx 0.000, \\[6pt]
\operatorname{P}\big[\;\mathrm{H} = 49 \mid p=\tfrac{1}{2}\;\big] & = \binom{80}{49}(\tfrac{1}{2})^{49}(1-\tfrac{1}{2})^{31} \approx 0.012, \\[6pt]
\operatorname{P}\big[\;\mathrm{H} = 49 \mid p=\tfrac{2}{3}\;\big] & = \binom{80}{49}(\tfrac{2}{3})^{49}(1-\tfrac{2}{3})^{31} \approx 0.054.
\end{align}
</math>
The likelihood is maximized when ''p'' = {{frac|2|3}}, and so this is the ''maximum likelihood estimate'' for ''p''.
=== Discrete distribution, continuous parameter space ===
Now suppose that there was only one coin but its ''p'' could have been any value 0 ≤ ''p'' ≤ 1. The likelihood function to be maximised is
:<math>
L(p) = f_D(\mathrm{H} = 49 \mid p) = \binom{80}{49} p^{49}(1 - p)^{31},
</math>
and the maximisation is over all possible values 0 ≤ ''p'' ≤ 1.
[[File:MLfunctionbinomial-en.svg|thumb|200px|likelihood function for proportion value of a binomial process (''n'' = 10)]]
One way to maximize this function is by [[derivative|differentiating]] with respect to ''p'' and setting to zero:
:<math>
\begin{align}
{0}&{} = \frac{\partial}{\partial p} \left( \binom{80}{49} p^{49}(1-p)^{31} \right), \\[8pt]
{0}&{}= 49 p^{48}(1-p)^{31} - 31 p^{49}(1-p)^{30} \\[8pt]
& {}= p^{48}(1-p)^{30}\left[ 49 (1-p) - 31 p \right] \\[8pt]
& {}= p^{48}(1-p)^{30}\left[ 49 - 80 p \right]
\end{align}
</math>
which has solutions ''p'' = 0, ''p'' = 1, and ''p'' = {{frac|49|80}}. The solution which maximizes the likelihood is clearly ''p'' = {{frac|49|80}} (since ''p'' = 0 and ''p'' = 1 result in a likelihood of zero). Thus the ''maximum likelihood estimator'' for ''p'' is {{frac|49|80}}.
This result is easily generalized by substituting a letter such as ''s'' in the place of 49 to represent the observed number of 'successes' of our [[Bernoulli trial]]s, and a letter such as ''n'' in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields {{frac|''s''|''n''}} which is the maximum likelihood estimator for any sequence of ''n'' Bernoulli trials resulting in ''s'' 'successes'.
=== Continuous distribution, continuous parameter space ===
For the [[normal distribution]] <math>\mathcal{N}(\mu, \sigma^2)</math> which has [[probability density function]]
:<math>f(x\mid \mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}\ }
\exp{\left(-\frac {(x-\mu)^2}{2\sigma^2} \right)}, </math>
the corresponding [[probability density function]] for a sample of {{mvar|n}} [[independent identically distributed]] normal random variables (the likelihood) is
:<math>f(x_1,\ldots,x_n \mid \mu,\sigma^2) = \prod_{i=1}^{n} f( x_{i}\mid \mu, \sigma^2) = \left( \frac{1}{2\pi\sigma^2} \right)^{n/2} \exp\left( -\frac{ \sum_{i=1}^{n}(x_i-\mu)^2}{2\sigma^2}\right),</math>
or more conveniently,
:<math>f(x_1,\ldots,x_n \mid \mu,\sigma^2) = \left( \frac{1}{2\pi\sigma^2} \right)^{n/2} \exp\left(-\frac{ \sum_{i=1}^{n}(x_i-\bar{x})^2+n(\bar{x}-\mu)^2}{2\sigma^2}\right),</math>
where <math> \bar{x} </math> is the [[sample mean]].
This family of distributions has two parameters: {{math|''θ'' {{=}} (''μ'', ''σ'')}}; so we maximize the likelihood, <math>\mathcal{L} (\mu,\sigma) = f(x_1,\ldots,x_n \mid \mu, \sigma)</math>, over both parameters simultaneously, or if possible, individually.
Since the [[natural logarithm|logarithm]] function itself is a [[continuous function|continuous]] [[strictly increasing]] function over the [[range (mathematics)|range]] of the likelihood, the values which maximize the likelihood will also maximize its logarithm (the log-likelihood itself is not necessarily strictly increasing). The log-likelihood can be written as follows:
:<math>
\log\Big( \mathcal{L} (\mu,\sigma)\Big) = -\frac{\,n\,}{2} \log(2\pi\sigma^2)
- \frac{1}{2\sigma^2} \sum_{i=1}^{n} (\,x_i-\mu\,)^2
</math>
(Note: the log-likelihood is closely related to [[information entropy]] and [[Fisher information]].)
We now compute the derivatives of this log-likelihood as follows.
:<math>
\begin{align}
0 & = \frac{\partial}{\partial \mu} \log\Big( \mathcal{L} (\mu,\sigma)\Big) =
0 - \frac{\;-2\!n(\bar{x}-\mu)\;}{2\sigma^2}.
\end{align}
</math>
This is solved by
:<math>\hat\mu = \bar{x} = \sum^n_{i=1} \frac{\,x_i\,}{n}. </math>
This is indeed the maximum of the function, since it is the only turning point in {{mvar|μ}} and the second derivative is strictly less than zero. Its [[expected value]] is equal to the parameter {{mvar|μ}} of the given distribution,
:<math>\operatorname{E}\big[\;\widehat\mu\;\big] = \mu, \, </math>
which means that the maximum likelihood estimator <math>\widehat\mu</math> is unbiased.
Similarly we differentiate the log-likelihood with respect to {{mvar|σ}} and equate to zero:
:<math>
\begin{align}
0 & = \frac{\partial}{\partial \sigma} \log \Bigg[ \left( \frac{1}{2\pi\sigma^2} \right)^{n/2} \exp\left(-\frac{ \sum_{i=1}^{n}(x_i-\bar{x})^2+n(\bar{x}-\mu)^2}{2\sigma^2}\right) \Bigg] \\[6pt]
& = \frac{\partial}{\partial \sigma} \left[ \frac{n}{2}\log\left( \frac{1}{2\pi\sigma^2} \right) - \frac{ \sum_{i=1}^{n}(x_i-\bar{x})^2+n(\bar{x}-\mu)^2}{2\sigma^2} \right] \\[6pt]
& = -\frac{\,n\,}{\sigma} + \frac{ \sum_{i=1}^{n}(x_i-\bar{x})^2+n(\bar{x}-\mu)^2}{\sigma^3}
\end{align}
</math>
which is solved by
:<math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n(x_i-\mu)^2.</math>
Inserting the estimate <math>\mu = \widehat\mu</math> we obtain
:<math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2 = \frac{1}{n}\sum_{i=1}^n x_i^2 -\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n x_i x_j.</math>
To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables ([[statistical error]]) <math>\delta_i \equiv \mu - x_i</math>. Expressing the estimate in these variables yields
: <math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (\mu - \delta_i)^2 -\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n (\mu - \delta_i)(\mu - \delta_j).</math>
Simplifying the expression above, utilizing the facts that <math>\operatorname{E}\big[\;\delta_i\;\big] = 0 </math> and <math>\operatorname{E}\big[\;\delta_i^2\;\big] = \sigma^2 </math>, allows us to obtain
:<math>\operatorname{E}\big[\;\widehat\sigma^2\;\big]= \frac{\,n-1\,}{n}\sigma^2.</math>
This means that the estimator <math>\widehat\sigma</math> is biased. However, <math>\widehat\sigma</math> is consistent.
Formally we say that the ''maximum likelihood estimator'' for <math>\theta=(\mu,\sigma^2)</math> is
:<math>\widehat{\theta} = \left(\widehat{\mu},\widehat{\sigma}^2\right).</math>
In this case the MLEs could be obtained individually. In general this may not be the case, and the MLEs would have to be obtained simultaneously.
The normal log-likelihood at its maximum takes a particularly simple form:
:<math>
\log\Big( \mathcal{L}(\hat\mu,\hat\sigma)\Big) = \frac{\,-n\;\;}{2} \big(\,\log(2\pi\hat\sigma^2) +1\,\big)
</math>
This maximum log-likelihood can be shown to be the same for more general [[least squares]], even for [[non-linear least squares]]. This is often used in determining likelihood-based approximate [[confidence interval]]s and [[confidence region]]s, which are generally more accurate than those using the asymptotic normality discussed above.
== Non-independent variables ==
It may be the case that variables are correlated, that is, not independent. Two random variables ''X'' and ''Y'' are independent only if their joint probability density function is the product of the individual probability density functions, i.e.
:<math>f(x,y)=f(x)f(y)\,</math>
Suppose one constructs an order-''n'' Gaussian vector out of random variables <math>(x_1,\ldots,x_n)</math>, where each variable has means given by <math>(\mu_1, \ldots, \mu_n)</math>. Furthermore, let the [[covariance matrix]] be denoted by <math>\mathit\Sigma</math>.
The joint probability density function of these ''n'' random variables is then given by:
:<math>f(x_1,\ldots,x_n)=\frac{1}{(2\pi)^{n/2}\sqrt{\text{det}(\mathit\Sigma)}} \exp\left( -\frac{1}{2} \left[x_1-\mu_1,\ldots,x_n-\mu_n\right]\mathit\Sigma^{-1} \left[x_1-\mu_1,\ldots,x_n-\mu_n\right]^\mathrm{T} \right)</math>
In the two variable case, the joint probability density function is given by:
:<math> f(x,y) = \frac{1}{2\pi \sigma_x \sigma_y \sqrt{1-\rho^2}} \exp\left[ -\frac{1}{2(1-\rho^2)} \left(\frac{(x-\mu_x)^2}{\sigma_x^2} - \frac{2\rho(x-\mu_x)(y-\mu_y)}{\sigma_x\sigma_y} + \frac{(y-\mu_y)^2}{\sigma_y^2}\right) \right] </math>
In this and other cases where a joint density function exists, the likelihood function is defined as above, in the section [[Maximum likelihood#Principles|Principles]], using this density.
== Iterative procedures ==
Consider problems where both states <math>x_i</math> and parameters such as <math>\sigma^2 </math> require to be estimated. Iterative procedures such as [[Expectation-maximization algorithm]]s may be used to solve joint state-parameter estimation problems.
For example, suppose that {{mvar|n}} samples of state estimates <math>\hat{x}_{i}</math> together with a sample mean <math>\bar{x}</math> have been calculated by either a minimum-variance [[Kalman filter]] or a minimum-variance smoother using a previous variance estimate <math>\widehat\sigma^2 </math>. Then the next variance iterate may be obtained from the maximum likelihood estimate calculation
:<math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^{n} (\hat{x}_i - \bar{x})^2. </math>
The convergence of MLEs within filtering and smoothing EM algorithms has been studied in the literature.<ref>{{cite journal
|last = Einicke
|first = G. A.
|last2 = Falco
|first2 = G.
|last3 = Malos
|first3 = J. T.
|title = EM Algorithm State Matrix Estimation for Navigation
|journal = IEEE Signal Processing Letters
|volume = 17
|issue = 5
|pages = 437–440
|date=May 2010
|postscript = <!--None-->
|doi=10.1109/LSP.2010.2043151
}}</ref><ref>{{cite journal
|last = Einicke
|first = G. A.
|last2 = Falco
|first2 = G.
|last3 = Dunn
|first3 = M. T.
|last4 = Reid
|first4 = D. C.
|title = Iterative Smoother-Based Variance Estimation
|journal = IEEE Signal Processing Letters
|volume = 19
|issue = 5
|pages = 275–278
|date=May 2012
|postscript = <!--None-->
|doi=10.1109/LSP.2012.2190278
}}</ref>
== History ==
Early users of maximum likelihood were [[Carl Friedrich Gauss]], [[Pierre-Simon Laplace]], [[Thorvald N. Thiele]], and [[Francis Ysidro Edgeworth]].<ref>{{harvtxt |Edgeworth |September 1908 }} and {{harvtxt |Edgeworth |December 1908 }}</ref>
[[File:Youngronaldfisher2.JPG|thumb|right|220px|Ronald Fisher in 1913]]
However its widespread use arose between 1912 and 1922 when [[Ronald Fisher]] recommended, widely popularized, and carefully analyzed maximum-likelihood estimation (with fruitless attempts at [[mathematical proof|proofs]]).<ref name="Pfanzagl">{{cite book|title=Parametric Statistical Theory|author=Pfanzagl, Johann, with the assistance of R. Hamböker|year=1994|publisher=[[Walter de Gruyter]]|isbn=3-11-013863-8|pages=207–208}}</ref>
Maximum-likelihood estimation finally transcended heuristic justification in a proof published by [[Samuel S. Wilks]] in 1938, now called "[[Likelihood-ratio test#Wilks' theorem|Wilks' theorem]]".<ref>{{harvtxt|Wilks|1938}}</ref> The theorem shows that the error in the logarithm of likelihood values for estimates from multiple independent samples is asymptotically [[chi-squared distribution|''χ'' ² distributed]], which enables convenient determination of a [[confidence region]] around any one estimate of the parameters. The only difficult part of [[Samuel S. Wilks|Wilks]]’ proof depends on the expected value of the [[Fisher information]] matrix, which ironically is provided by a theorem proven by [[Ronald Fisher|Fisher]].<ref>Owen, Art B. (2001). ''Empirical Likelihood''. London: Chapman & Hall/Boca Raton, FL: CRC Press. {{isbn|978-1584880714}}.</ref> Wilks continued to improve on the generality of the theorem throughout his life, with his most general proof published in 1962.<ref>Wilks, Samuel S. (1962), ''Mathematical Statistics'', New York: John Wiley & Sons. {{isbn|978-0471946502}}.</ref>
Some of the theory behind maximum likelihood estimation was developed for [[Bayesian statistics]].<ref name="Pfanzagl" />
Reviews of the development of maximum likelihood estimation have been provided by a number of authors.<ref>{{harvtxt |Savage |1976 }}, {{harvtxt |Pratt |1976 }}, {{harvs |txt=yes |last=Stigler |year=1978 |year2=1986 |year3=1999 }}, {{harvs |txt=yes |last=Hald |year=1998 |year2=1999 }}, and {{harvtxt |Aldrich |1997 }}</ref>
== See also ==
{{Portal|Statistics}}
* '''Other estimation methods'''
** [[Generalized method of moments]] are methods related to the likelihood equation in maximum likelihood estimation
** [[M-estimator]], an approach used in robust statistics
** [[Maximum a posteriori]] (MAP) estimator, for a contrast in the way to calculate estimators when prior knowledge is postulated
** [[Maximum spacing estimation]], a related method that is more robust in many situations
** [[Principle of maximum entropy|Maximum entropy estimation]]
** [[Method of moments (statistics)]], another popular method for finding parameters of distributions
** [[Method of support]], a variation of the maximum likelihood technique
** [[Minimum distance estimation]]
** [[Quasi-maximum likelihood]] estimator, an MLE estimator that is misspecified, but still consistent
** [[Restricted maximum likelihood]], a variation using a likelihood function calculated from a transformed set of data
* '''Related concepts''':
** [[Akaike information criterion]], a criterion to compare statistical models, based on MLE
** [[Extremum estimator]], a more general class of estimators to which MLE belongs
** [[Fisher information]], information matrix, its relationship to covariance matrix of ML estimates
** [[Mean squared error]], a measure of how 'good' an estimator of a distributional parameter is (be it the maximum likelihood estimator or some other estimator)
** [[RANSAC]], a method to estimate parameters of a mathematical model given data that contains [[outliers]]
** [[Rao–Blackwell theorem]], which yields a process for finding the best possible unbiased estimator (in the sense of having minimal [[mean squared error]]); the MLE is often a good starting place for the process
** [[Likelihood-ratio test#Wilks' theorem|Wilks’ Theorem]] provides a means of estimating the size and shape of the region of roughly equally-probable estimates for the population's parameter values, using the information from a single sample, using a [[chi-squared distribution]]
==Notes==
{{Reflist|30em}}
==References: non-historical==
{{Refbegin}}
* {{cite journal
| last1 = Cox | first1 = David R. | authorlink1=David R. Cox
| last2 = Snell | first2 = E. Joyce
| title = A general definition of residuals
| year = 1968
| journal = [[Journal of the Royal Statistical Society, Series B]]
| pages = 248–275
| jstor = 2984505
| ref = CITEREFCoxSnell1968
| volume=30
}}
* {{cite journal
|last = Kano
|first = Yutaka
|title = Third-order efficiency implies fourth-order efficiency
|year = 1996
|journal = Journal of the Japan Statistical Society
|volume = 26
|pages = 101–117
|url = http://www.journalarchive.jst.go.jp/english/jnlabstract_en.php?cdjournal=jjss1995&cdvol=26&noissue=1&startpage=101
|ref = harv
|doi = 10.14490/jjss1995.26.101
}}{{Dead link|date=September 2018 |bot=InternetArchiveBot |fix-attempted=yes }}
* {{cite book
| last1 = Newey | first1 = Whitney K.
| last2 = McFadden | first2 = Daniel | authorlink2 = Daniel McFadden
| chapter = Chapter 35: Large sample estimation and hypothesis testing
| editor1-first= Robert | editor1-last=Engle | editor2-first=Dan | editor2-last=McFadden
| title = Handbook of Econometrics, Vol.4
| year = 1994
| publisher = Elsevier Science
| pages = 2111–2245
| isbn=0-444-88766-0
| ref = CITEREFNeweyMcFadden1994
}}
{{Refend}}
==References: historical==
{{refbegin}}
* {{cite journal
| last = Aldrich | first = John
| title = R. A. Fisher and the making of maximum likelihood 1912–1922
| year = 1997
| journal = Statistical Science
| volume = 12 | issue = 3
| pages = 162–176
| doi = 10.1214/ss/1030037906
| mr = 1617519
| ref = CITEREFAldrich1997
}}
* {{cite journal
| doi = 10.2307/2339293
| last = Edgeworth | first = Francis Y. | authorlink = Francis Ysidro Edgeworth
| title = On the probable errors of frequency-constants
|date=Sep 1908
| ref= CITEREFEdgeworthSeptember_1908
| journal = Journal of the Royal Statistical Society
| volume = 71 | issue = 3
| pages = 499–512
| jstor = 2339293
}}
* {{cite journal
| doi = 10.2307/2339378
| last = Edgeworth | first = Francis Y.
| title = On the probable errors of frequency-constants
|date=Dec 1908
| ref= CITEREFEdgeworthDecember_1908
| journal = Journal of the Royal Statistical Society
| volume = 71 | issue = 4
| pages = 651–678
| jstor = 2339378
}}
* {{cite book
| last = Hald | first = Anders | authorlink=Anders Hald
| title = A history of mathematical statistics from 1750 to 1930
| year = 1998
| publisher = Wiley
| location = New York, NY
| isbn = 0-471-17912-4
| ref = CITEREFHald1998
}}
* {{cite journal
| last = Hald | first = Anders
| title = On the history of maximum likelihood in relation to inverse probability and least squares
| year = 1999
| journal = Statistical Science
| volume = 14 | issue = 2
| pages =214–222
| jstor = 2676741
| ref = CITEREFHald1999
| doi=10.1214/ss/1009212248
| url = http://projecteuclid.org/download/pdf_1/euclid.ss/1009212248
}}
* {{cite journal
| doi = 10.1214/aos/1176343457
| last = Pratt | first = John W.
| title = F. Y. Edgeworth and R. A. Fisher on the efficiency of maximum likelihood estimation
| year = 1976
| journal = The Annals of Statistics
| volume = 4 | issue = 3
| pages = 501–514
| jstor = 2958222
| ref = CITEREFPratt1976
}}
* {{cite journal
| doi = 10.1214/aos/1176343456
| last = Savage | first = Leonard J. | authorlink = Leonard J. Savage
| title = On rereading R. A. Fisher
| year = 1976
| journal = The Annals of Statistics
| volume = 4 | issue = 3
| pages = 441–500
| jstor = 2958221
| ref = CITEREFSavage1976
}}
* {{cite journal
| doi = 10.2307/2344804
| last = Stigler | first = Stephen M. | authorlink = Stephen M. Stigler
| title = Francis Ysidro Edgeworth, statistician
| year = 1978
| journal = Journal of the Royal Statistical Society, Series A
| volume = 141 | issue = 3
| pages = 287–322
| jstor = 2344804
| ref = CITEREFStigler1978
}}
* {{cite book
| last = Stigler | first = Stephen M.
| title = The history of statistics: the measurement of uncertainty before 1900
| year = 1986
| publisher = Harvard University Press
| isbn = 0-674-40340-1
| ref = CITEREFStigler1986
}}
* {{cite book
| last = Stigler | first = Stephen M.
| title = Statistics on the table: the history of statistical concepts and methods
| year = 1999
| publisher = Harvard University Press
| isbn = 0-674-83601-4
| ref = CITEREFStigler1999
}}
* {{citation|last=Wilks | first= S. S. |year= 1938 | title= The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses | journal= [[Annals of Mathematical Statistics]] | volume=9| pages= 60–62| doi= 10.1214/aoms/1177732360}}.
{{Refend}}
== Further reading ==
* Andersen, Erling B. (1970); "Asymptotic Properties of Conditional Maximum Likelihood Estimators", ''[[Journal of the Royal Statistical Society]]'' '''B''' 32, 283–301
* Andersen, Erling B. (1980); ''Discrete Statistical Models with Social Science Applications'', North Holland, 1980
* [[Debabrata Basu|Basu, Debabrata]] (1988); ''Statistical Information and Likelihood : A Collection of Critical Essays by Dr. D. Basu''; in Ghosh, Jayanta K., editor; ''Lecture Notes in Statistics'', Volume 45, Springer-Verlag, 1988
* {{cite book
| author = Einicke, G.A.
| year = 2012
| title = Smoothing, Filtering and Prediction: Estimating the Past, Present and Future
| publisher = Intech
| location = Rijeka, Croatia
| isbn = 978-953-307-752-9
| url = http://www.intechopen.com/books/smoothing-filtering-and-prediction-estimating-the-past-present-and-future}}
* {{cite journal |last=Ferguson |first=Thomas S. |title=An inconsistent maximum likelihood estimate |journal=Journal of the American Statistical Association |volume=77 |issue=380 |year=1982 |pages=831–834 |jstor=2287314 |doi=10.1080/01621459.1982.10477894}}
* {{cite book
| last = Ferguson | first = Thomas S.
| title = A Course in Large Sample Theory
| publisher = Chapman & Hall
| year = 1996
| isbn = 0-412-04371-8
}}
* {{cite journal
| last = Le Cam | first = Lucien | authorlink = Lucien Le Cam
| title = Maximum likelihood — an introduction
| journal = ISI Review
| volume = 58 | issue = 2
| year = 1990
| pages = 153–171
| doi=10.2307/1403464
}}
* {{cite book
| last1 = Le Cam | first1 = Lucien
| last2 = Yang | first2 = Grace Lo
| title = Asymptotics in Statistics: some basic concepts
| year = 2000
| edition = Second
| publisher = Springer
| isbn = 0-387-95036-2
| ref = CITEREFLeCam2000
}}
* {{cite book
| last1 = Lehmann | first1 = Erich L. | authorlink = Erich Leo Lehmann
| last2 = Casella | first2 = George
| title = Theory of Point Estimation, 2nd ed
| year = 1998
| publisher = Springer
| isbn = 0-387-98502-6
| ref = CITEREFLehamnnCasella1998
}}
* {{cite book | title=Maximum Likelihood Estimation and Inference | last= Millar | first= R. B. | year= 2011 | publisher= Wiley}}
* {{cite book |author=Ruppert, David |title=Statistics and Data Analysis for Financial Engineering |publisher=Springer |year=2010 |isbn=978-1-4419-7786-1 |page=98 |url=https://books.google.com/books?id=i2bD50PbIikC&pg=PA98 |ref=CITEREFRuppert2010 }}
* {{cite book
| last = van der Vaart | first = Aad W. |author-link= Aad van der Vaart
| title = Asymptotic Statistics
| year = 1998
| isbn = 0-521-78450-6
}}
== External links ==
* {{springer|title=Maximum-likelihood method|id=p/m063100}}
* Lemke, D. (2016). Maximum likelihood estimation and EM fixed point ideals for binary tensors. ([[San Francisco State University]]: Masters Theses Collection – Degree in Mathematics). OCLC# 980058195 URL: <nowiki>http://hdl.handle.net/10211.3/173201</nowiki>
* [http://statgen.iop.kcl.ac.uk/bgim/mle/sslike_1.html Maximum Likelihood Estimation Primer (an excellent tutorial)]
* [http://www.mayin.org/ajayshah/KB/R/documents/mle/mle.html Implementing MLE for your own likelihood function using R]
* [http://www.netstorm.be/home/mle A selection of likelihood functions in R]
* {{cite journal | title = Tutorial on maximum likelihood estimation | citeseerx = 10.1.1.74.671 | journal = [[Journal of Mathematical Psychology]] | last= Myung | first= I. J. | year= 2003 }}
{{Statistics}}
[[Category:Maximum likelihood estimation]]
[[Category:M-estimators]]
[[Category:Probability distribution fitting]]
{{About|the statistical techniques|computer data storage|Partial response maximum likelihood}}
{{merge from|Partial likelihood methods for panel data|date=October 2017}}
{{More footnotes|date=September 2009}}
In statistics, '''maximum likelihood estimation''' ('''MLE''') is a method of [[estimator|estimating]] the [[Statistical parameter|parameters]] of a [[statistical model]], given observations. MLE attempts to find the parameter values that maximize the [[likelihood function]], given the observations. The resulting estimate is called a '''maximum likelihood estimate''', which is also abbreviated as MLE.
The method of maximum likelihood is used with a wide range of statistical analyses. As an example, suppose that we are interested in the heights of adult female penguins, but are unable to measure the height of every penguin in a population (due to cost or time constraints). Assuming that the heights are [[Normal distribution|normally distributed]] with some unknown [[mean]] and [[variance]], the mean and variance can be estimated with MLE while only knowing the heights of some sample of the overall population. MLE would accomplish that by taking the mean and variance as parameters and finding particular parametric values that make the observed results the most probable given the normal model.
From the point of view of [[Bayesian inference]], MLE is a special case of [[maximum a posteriori estimation]] (MAP) that assumes a [[uniform distribution (continuous)|uniform]] [[Prior probability|prior distribution]] of the parameters. In [[frequentist inference]], MLE is one of several methods to get estimates of parameters without using prior distributions. Priors are avoided by not making probability statements about the parameters, but only about their estimates, whose properties are fully defined by the observations and the statistical model.
== Principles ==
The method of maximum likelihood is based on the [[likelihood function]], <math>\mathcal L(\theta\,;x)</math>. We are given a [[statistical model]], i.e. a family of distributions <math>\{ f(\cdot\,;\theta) \mid \theta \in \Theta \}</math>, where <math>\theta</math> denotes the (possibly multi-dimensional) parameter for the model. The method of maximum likelihood finds the values of the model parameter, <math>\theta</math>, that maximize the likelihood function, <math>\mathcal L(\theta\,;x)</math>. Intuitively, this selects the parameter values that make the data most probable.
The method defines a maximum likelihood estimate:
: <math>
\hat\theta \in \{ \underset{\theta\in\Theta}{\operatorname{arg\,max}}\ \mathcal L(\theta\,;x) \},
</math>
if a maximum exists.
In practice, it is often convenient to work with the [[natural logarithm]] of the likelihood function, called the log-likelihood:
: <math>
\ell(\theta\,;x) = \ln\mathcal{L}(\theta\,;x),
</math>
or the average log-likelihood:
: <math>
\hat\ell(\theta\,;x) = \frac1n \ln\mathcal{L(\theta\,;x)}.
</math>
The [[circumflex|hat]] over <math>\ell</math> indicates that it is akin to an [[estimator]]. Indeed, <math>\hat\ell</math> estimates the expected log-likelihood of a single observation in the model.
An MLE is the same regardless of whether we maximize the likelihood or the log-likelihood, because log is a [[Monotonic function|strictly increasing]] function.
For many models, a maximum likelihood estimator can be found as an explicit function of the observed data <math>x</math>. For many other models, however, no closed-form solution to the maximization problem is known or available, and an MLE can only be found via numerical [[global optimization]]. For some problems, there may be multiple values that maximize the likelihood. For other problems, no maximum likelihood estimate exists: either the log-likelihood function increases without ever reaching a [[supremum]] value, or the supremum does exist but is outside the bounds of <math>\Theta</math>, the set of acceptable parameter values.
== Properties ==
A maximum likelihood estimator is an [[extremum estimator]] obtained by maximizing, as a function of ''θ'', the ''objective function'' (cf. [[loss function]]) <math>\hat\ell(\theta\,;x)</math>. If the data are [[independent and identically distributed]], then we have
: <math>
\hat\ell(\theta\,;x)=\frac1n \sum_{i=1}^n \ln f(x_i\mid\theta),
</math>
this being the sample analogue of the expected log-likelihood <math>\ell(\theta) = \operatorname{E}[\, \ln f(x_i\mid\theta) \,]</math>, where this expectation is taken with respect to the true density.
Maximum-likelihood estimators have no optimum properties for finite samples, in the sense that (when evaluated on finite samples) other estimators may have greater concentration around the true parameter-value.<ref>{{harvtxt |Pfanzagl |1994 |p=206 }}</ref> However, like other estimation methods, maximum likelihood estimation possesses a number of attractive [[Asymptotic theory (statistics)|limiting properties]]: As the sample size increases to infinity, sequences of maximum likelihood estimators have these properties:
* [[consistency of an estimator|Consistency]]: the sequence of MLEs converges in probability to the value being estimated.
* [[efficient estimator|Efficiency]], i.e. it achieves the [[Cramér–Rao lower bound]] when the sample size tends to infinity. This means that no consistent estimator has lower asymptotic [[mean squared error]] than the MLE (or other estimators attaining this bound).
* Second-order efficiency after correction for bias.
=== Consistency ===
Under the conditions outlined below, the maximum likelihood estimator is '''[[consistent estimator|consistent]]'''. The consistency means that if the data were generated by <math>f(\cdot\,;\theta_0)</math> and we have a sufficiently large number of observations ''n'', then it is possible to find the value of ''θ''<sub>0</sub> with arbitrary precision. In mathematical terms this means that as ''n'' goes to infinity the estimator <math>\hat\theta</math> [[convergence in probability|converges in probability]] to its true value:
: <math>
\hat\theta_\mathrm{mle}\ \xrightarrow{\text{p}}\ \theta_0.
</math>
Under slightly stronger conditions, the estimator converges [[almost sure convergence|almost surely]] (or ''strongly''):
: <math>
\hat\theta_\mathrm{mle}\ \xrightarrow{\text{a.s.}}\ \theta_0.
</math>
Note that, in practical applications, data is never generated by <math>f(\cdot\,;\theta_0)</math>. Rather, <math>f(\cdot\,;\theta_0)</math> is a model, often in idealized form, of the process that generated the data. It is a common aphorism in statistics that ''[[all models are wrong]]''. Thus, true consistency does not occur in practical applications. Nevertheless, consistency is often considered to be a desirable property for an estimator to have.
To establish consistency, the following conditions are sufficient.<ref>{{harvtxt |Newey |McFadden |1994 |loc=Theorem 2.5. }}</ref>
{{ordered list
|1= '''[[Identifiability|Identification]]''' of the model:
: <math>
\theta \neq \theta_0 \quad \Leftrightarrow \quad f(\cdot\mid\theta)\neq f(\cdot\mid\theta_0).
</math>
In other words, different parameter values ''θ'' correspond to different distributions within the model. If this condition did not hold, there would be some value ''θ''<sub>1</sub> such that ''θ''<sub>0</sub> and ''θ''<sub>1</sub> generate an identical distribution of the observable data. Then we would not be able to distinguish between these two parameters even with an infinite amount of data—these parameters would have been [[observational equivalence|observationally equivalent]]. <br />
The identification condition is absolutely necessary for the ML estimator to be consistent. When this condition holds, the limiting likelihood function ''ℓ''(''θ''{{!}}·) has unique global maximum at ''θ''<sub>0</sub>.
|2= '''Compactness''': the parameter space Θ of the model is [[compact set|compact]].
[[File:Ee noncompactness.svg|240px|right]]
The identification condition establishes that the log-likelihood has a unique global maximum. Compactness implies that the likelihood cannot approach the maximum value arbitrarily close at some other point (as demonstrated for example in the picture on the right).
Compactness is only a sufficient condition and not a necessary condition. Compactness can be replaced by some other conditions, such as:
{{unordered list
| both [[Concave function|concavity]] of the log-likelihood function and compactness of some (nonempty) upper [[level set]]s of the log-likelihood function, or
| existence of a compact [[Neighbourhood (mathematics)|neighborhood]] ''N'' of ''θ''<sub>0</sub> such that outside of ''N'' the log-likelihood function is less than the maximum by at least some {{nowrap|''ε'' > 0}}.
}}
|3= '''Continuity''': the function ln ''f''(''x''{{!}}''θ'') is continuous in ''θ'' for almost all values of ''x'':
: <math>
\operatorname{P}\!\big[\; \ln f(x\mid\theta) \;\in\; \mathbb{C}^0(\Theta) \;\big] = 1.
</math>
The continuity here can be replaced with a slightly weaker condition of [[upper semi-continuous|upper semi-continuity]].
|4= '''Dominance''': there exists ''D''(''x'') integrable with respect to the distribution ''f''(''x''{{!}}''θ''<sub>0</sub>) such that
: <math>
\big|\ln f(x\mid\theta)\big| < D(x) \quad \text{ for all } \theta\in\Theta.
</math>
By the [[uniform law of large numbers]], the dominance condition together with continuity establish the '''uniform convergence in probability''' of the log-likelihood:
: <math>
\sup_{\theta\in\Theta} \big|\hat\ell(\theta\mid x) - \ell(\theta)\,\big|\ \xrightarrow{\text{p}}\ 0.
</math>
}}
The dominance condition can be employed in the case of [[i.i.d.]] observations. In the non-i.i.d. case, the uniform convergence in probability can be checked by showing that the sequence <math>\hat\ell(\theta\mid x)</math> is [[stochastic equicontinuity|stochastically equicontinuous]].
If one wants to demonstrate that the ML estimator <math>\hat\theta</math> converges to ''θ''<sub>0</sub> [[almost sure convergence|almost surely]], then a stronger condition of '''uniform convergence almost surely''' has to be imposed:
: <math>
\sup_{\theta\in\Theta} \big\|\;\hat\ell(x\mid\theta) - \ell(\theta)\;\big\| \ \xrightarrow{\text{a.s.}}\ 0.
</math>
Additionally, if (as assumed above) the data were generated by <math>f(\cdot\,;\theta_0)</math>, then under certain conditions, it can also be shown that the maximum likelihood estimator [[Convergence in distribution|converges in distribution]] to a normal distribution. Specifically,<ref>{{harvtxt |Newey |McFadden |1994 |loc=Theorem 3.3. }}</ref>
: <math>
\sqrt{n}\big(\hat\theta_\mathrm{mle} - \theta_0\big)\ \xrightarrow{d}\ \mathcal{N}(0,\,I^{-1})
</math>
where {{math|''I''}} is the [[Fisher information|Fisher information matrix]].
=== Functional invariance ===
The maximum likelihood estimator selects the parameter value which gives the observed data the largest possible probability (or probability density, in the continuous case). If the parameter consists of a number of components, then we define their separate maximum likelihood estimators, as the corresponding component of the MLE of the complete parameter. Consistent with this, if <math>\widehat{\theta}</math> is the MLE for <math>\theta</math>, and if <math>g(\theta)</math> is any transformation of <math>\theta</math>, then the MLE for <math>\alpha=g(\theta)</math> is by definition
:<math>\widehat{\alpha} = g(\,\widehat{\theta}\,). \,</math>
It maximizes the so-called [[Likelihood function#Profile likelihood|profile likelihood]]:
:<math>\bar{L}(\alpha) = \sup_{\theta: \alpha = g(\theta)} L(\theta). \, </math>
The MLE is also invariant with respect to certain transformations of the data. If <math>y=g(x)</math> where <math>g</math> is one to one and does not depend on the parameters to be estimated, then the density functions satisfy
:<math>f_Y(y) = \frac{f_X(x)}{|g'(x)|} </math>
and hence the likelihood functions for <math>X</math> and <math>Y</math> differ only by a factor that does not depend on the model parameters.
For example, the MLE parameters of the log-normal distribution are the same as those of the normal distribution fitted to the logarithm of the data.
===Higher-order properties===
As noted above, the maximum likelihood estimator is {{sqrt|''n'' }}-consistent and asymptotically efficient, meaning that it reaches the [[Cramér–Rao bound]]:
: <math>
\sqrt{n}(\hat\theta_\text{mle} - \theta_0)\ \ \xrightarrow{d}\ \ \mathcal{N}(0,\ I^{-1}),
</math>
where <math>I</math> is the [[Fisher information matrix]]:
: <math>
I_{jk} = \operatorname{E}_X\bigg[\;{-\frac{\partial^2\ln f_{\theta_0}(X_t)}{\partial\theta_j\,\partial\theta_k}}
\;\bigg].
</math>
In particular, it means that the [[bias of an estimator|bias]] of the maximum likelihood estimator is equal to zero up to the order {{frac|1|{{sqrt|''n'' }}}}. However, when we consider the higher-order terms in the [[Edgeworth expansion|expansion]] of the distribution of this estimator, it turns out that {{math|''θ''<sub>mle</sub>}} has bias of order {{frac|1|''n''}}. This bias is equal to (componentwise)<ref>{{harvtxt |Cox |Snell |1968 | loc=formula (20) }}</ref>
: <math>
b_h \equiv \operatorname{E}\bigg[\;(\hat\theta_\mathrm{mle} - \theta_0)_h\;\bigg]
= \frac{1}{n} \sum_{i, j, k = 1}^n I^{h i}I^{j k} \big( \frac{1}{2} K_{i j k} + J_{j,i k} \big)
</math>
where <math>I^{j k}</math> denotes the (''j,k'')-th component of the ''inverse'' Fisher information matrix <math>I^{-1}</math>, and
: <math>
\tfrac12 K_{ijk} + J_{j,ik} = \operatorname{E}_X \bigg[\;
\frac12 \frac{\partial^3 \ln f_{\theta_0}(X_t)}{\partial\theta_i\,\partial\theta_j\,\partial\theta_k} +
\frac{\partial\ln f_{\theta_0}(X_t)}{\partial\theta_j} \frac{\partial^2\ln f_{\theta_0}(X_t)}{\partial\theta_i\,\partial\theta_k}
\; \bigg] .
</math>
Using these formulae it is possible to estimate the second-order bias of the maximum likelihood estimator, and ''correct'' for that bias by subtracting it:
: <math>
\hat\theta^*_\text{mle} = \hat\theta_\text{mle} - \hat{b}.
</math>
This estimator is unbiased up to the terms of order {{frac|1|''n''}}, and is called the '''bias-corrected maximum likelihood estimator'''.
This bias-corrected estimator is ''second-order efficient'' (at least within the curved exponential family), meaning that it has minimal mean squared error among all second-order bias-corrected estimators, up to the terms of the order {{frac|1|''n'' ² }}. It is possible to continue this process, that is to derive the third-order bias-correction term, and so on. However, as was shown by {{harvtxt| Kano |1996 }}, the maximum likelihood estimator is '''not''' third-order efficient.
==Relation to Bayesian inference==
A maximum likelihood estimator coincides with the [[Maximum a posteriori|most probable]] [[Bayesian estimator]] given a [[Uniform distribution (continuous)|uniform]] [[prior probability|prior distribution]] on the [[parameter space|parameters]]. Indeed, the [[maximum a posteriori estimate]] is the parameter {{mvar|θ}} that maximizes the probability of {{mvar|θ}} given the data, given by Bayes' theorem:
: <math>
\operatorname{P}(\theta\mid x_1,x_2,\ldots,x_n) = \frac{f(x_1,x_2,\ldots,x_n\mid\theta)\operatorname{P}(\theta)}{\operatorname{P}(x_1,x_2,\ldots,x_n)}
</math>
where <math>P(\theta)</math> is the prior distribution for the parameter {{mvar|θ}} and where <math>\operatorname{P}(x_1,x_2,\ldots,x_n)</math> is the probability of the data averaged over all parameters. Since the denominator is independent of {{mvar|θ}}, the Bayesian estimator is obtained by maximizing <math>f(x_1,x_2,\ldots,x_n\mid\theta)\operatorname{P}(\theta)</math> with respect to {{mvar|θ}}. If we further assume that the prior <math>P(\theta)</math> is a uniform distribution, the Bayesian estimator is obtained by maximizing the likelihood function <math>f(x_1,x_2,\ldots,x_n\mid\theta)</math>. Thus the Bayesian estimator coincides with the maximum likelihood estimator for a uniform prior distribution <math>\operatorname{P}(\theta)</math>.
== Examples ==
{{merge from|Maximum likelihood estimation with flow data|date=October 2017}}
=== Discrete uniform distribution ===
{{Main|German tank problem}}
Consider a case where ''n'' tickets numbered from 1 to ''n'' are placed in a box and one is selected at random (''see [[Uniform distribution (discrete)|uniform distribution]]''); thus, the sample size is 1. If ''n'' is unknown, then the maximum likelihood estimator <math>\hat{n}</math> of ''n'' is the number ''m'' on the drawn ticket. (The likelihood is 0 for ''n'' < ''m'', {{frac|1|''n''}} for ''n'' ≥ ''m'', and this is greatest when ''n'' = ''m''. Note that the maximum likelihood estimate of ''n'' occurs at the lower extreme of possible values {''m'', ''m'' + 1, ...}, rather than somewhere in the "middle" of the range of possible values, which would result in less bias.) The [[expected value]] of the number ''m'' on the drawn ticket, and therefore the expected value of <math>\hat{n}</math>, is (''n'' + 1)/2. As a result, with a sample size of 1, the maximum likelihood estimator for ''n'' will systematically underestimate ''n'' by (''n'' − 1)/2.
=== Discrete distribution, finite parameter space ===
Suppose one wishes to determine just how biased an [[unfair coin]] is. Call the probability of tossing a ‘[[Obverse and reverse|head]]’ ''p''. The goal then becomes to determine ''p''.
Suppose the coin is tossed 80 times: i.e. the sample might be something like ''x''<sub>1</sub> = H, ''x''<sub>2</sub> = T, ..., ''x''<sub>80</sub> = T, and the count of the number of [[Obverse and reverse|heads]] "H" is observed.
The probability of tossing [[Obverse and reverse|tails]] is 1 − ''p'' (so here ''p'' is ''θ'' above). Suppose the outcome is 49 heads and 31 [[Obverse and reverse|tails]], and suppose the coin was taken from a box containing three coins: one which gives heads with probability ''p'' = {{frac|1|3}}, one which gives heads with probability ''p'' = {{frac|1|2}} and another which gives heads with probability ''p'' = {{frac|2|3}}. The coins have lost their labels, so which one it was is unknown. Using '''maximum likelihood estimation''' the coin that has the largest likelihood can be found, given the data that were observed. By using the [[probability mass function]] of the [[binomial distribution]] with sample size equal to 80, number successes equal to 49 but for different values of ''p'' (the "probability of success"), the likelihood function (defined below) takes one of three values:
:<math>
\begin{align}
\operatorname{P}\big[\;\mathrm{H} = 49 \mid p=\tfrac{1}{3}\;\big] & = \binom{80}{49}(\tfrac{1}{3})^{49}(1-\tfrac{1}{3})^{31} \approx 0.000, \\[6pt]
\operatorname{P}\big[\;\mathrm{H} = 49 \mid p=\tfrac{1}{2}\;\big] & = \binom{80}{49}(\tfrac{1}{2})^{49}(1-\tfrac{1}{2})^{31} \approx 0.012, \\[6pt]
\operatorname{P}\big[\;\mathrm{H} = 49 \mid p=\tfrac{2}{3}\;\big] & = \binom{80}{49}(\tfrac{2}{3})^{49}(1-\tfrac{2}{3})^{31} \approx 0.054.
\end{align}
</math>
The likelihood is maximized when ''p'' = {{frac|2|3}}, and so this is the ''maximum likelihood estimate'' for ''p''.
=== Discrete distribution, continuous parameter space ===
Now suppose that there was only one coin but its ''p'' could have been any value 0 ≤ ''p'' ≤ 1. The likelihood function to be maximised is
:<math>
L(p) = f_D(\mathrm{H} = 49 \mid p) = \binom{80}{49} p^{49}(1 - p)^{31},
</math>
and the maximisation is over all possible values 0 ≤ ''p'' ≤ 1.
[[File:MLfunctionbinomial-en.svg|thumb|200px|likelihood function for proportion value of a binomial process (''n'' = 10)]]
One way to maximize this function is by [[derivative|differentiating]] with respect to ''p'' and setting to zero:
:<math>
\begin{align}
{0}&{} = \frac{\partial}{\partial p} \left( \binom{80}{49} p^{49}(1-p)^{31} \right), \\[8pt]
{0}&{}= 49 p^{48}(1-p)^{31} - 31 p^{49}(1-p)^{30} \\[8pt]
& {}= p^{48}(1-p)^{30}\left[ 49 (1-p) - 31 p \right] \\[8pt]
& {}= p^{48}(1-p)^{30}\left[ 49 - 80 p \right]
\end{align}
</math>
which has solutions ''p'' = 0, ''p'' = 1, and ''p'' = {{frac|49|80}}. The solution which maximizes the likelihood is clearly ''p'' = {{frac|49|80}} (since ''p'' = 0 and ''p'' = 1 result in a likelihood of zero). Thus the ''maximum likelihood estimator'' for ''p'' is {{frac|49|80}}.
This result is easily generalized by substituting a letter such as ''s'' in the place of 49 to represent the observed number of 'successes' of our [[Bernoulli trial]]s, and a letter such as ''n'' in the place of 80 to represent the number of Bernoulli trials. Exactly the same calculation yields {{frac|''s''|''n''}} which is the maximum likelihood estimator for any sequence of ''n'' Bernoulli trials resulting in ''s'' 'successes'.
=== Continuous distribution, continuous parameter space ===
For the [[normal distribution]] <math>\mathcal{N}(\mu, \sigma^2)</math> which has [[probability density function]]
:<math>f(x\mid \mu,\sigma^2) = \frac{1}{\sqrt{2\pi\sigma^2}\ }
\exp{\left(-\frac {(x-\mu)^2}{2\sigma^2} \right)}, </math>
the corresponding [[probability density function]] for a sample of {{mvar|n}} [[independent identically distributed]] normal random variables (the likelihood) is
:<math>f(x_1,\ldots,x_n \mid \mu,\sigma^2) = \prod_{i=1}^{n} f( x_{i}\mid \mu, \sigma^2) = \left( \frac{1}{2\pi\sigma^2} \right)^{n/2} \exp\left( -\frac{ \sum_{i=1}^{n}(x_i-\mu)^2}{2\sigma^2}\right),</math>
or more conveniently,
:<math>f(x_1,\ldots,x_n \mid \mu,\sigma^2) = \left( \frac{1}{2\pi\sigma^2} \right)^{n/2} \exp\left(-\frac{ \sum_{i=1}^{n}(x_i-\bar{x})^2+n(\bar{x}-\mu)^2}{2\sigma^2}\right),</math>
where <math> \bar{x} </math> is the [[sample mean]].
This family of distributions has two parameters: {{math|''θ'' {{=}} (''μ'', ''σ'')}}; so we maximize the likelihood, <math>\mathcal{L} (\mu,\sigma) = f(x_1,\ldots,x_n \mid \mu, \sigma)</math>, over both parameters simultaneously, or if possible, individually.
Since the [[natural logarithm|logarithm]] function itself is a [[continuous function|continuous]] [[strictly increasing]] function over the [[range (mathematics)|range]] of the likelihood, the values which maximize the likelihood will also maximize its logarithm (the log-likelihood itself is not necessarily strictly increasing). The log-likelihood can be written as follows:
:<math>
\log\Big( \mathcal{L} (\mu,\sigma)\Big) = -\frac{\,n\,}{2} \log(2\pi\sigma^2)
- \frac{1}{2\sigma^2} \sum_{i=1}^{n} (\,x_i-\mu\,)^2
</math>
(Note: the log-likelihood is closely related to [[information entropy]] and [[Fisher information]].)
We now compute the derivatives of this log-likelihood as follows.
:<math>
\begin{align}
0 & = \frac{\partial}{\partial \mu} \log\Big( \mathcal{L} (\mu,\sigma)\Big) =
0 - \frac{\;-2\!n(\bar{x}-\mu)\;}{2\sigma^2}.
\end{align}
</math>
This is solved by
:<math>\hat\mu = \bar{x} = \sum^n_{i=1} \frac{\,x_i\,}{n}. </math>
This is indeed the maximum of the function, since it is the only turning point in {{mvar|μ}} and the second derivative is strictly less than zero. Its [[expected value]] is equal to the parameter {{mvar|μ}} of the given distribution,
:<math>\operatorname{E}\big[\;\widehat\mu\;\big] = \mu, \, </math>
which means that the maximum likelihood estimator <math>\widehat\mu</math> is unbiased.
Similarly we differentiate the log-likelihood with respect to {{mvar|σ}} and equate to zero:
:<math>
\begin{align}
0 & = \frac{\partial}{\partial \sigma} \log \Bigg[ \left( \frac{1}{2\pi\sigma^2} \right)^{n/2} \exp\left(-\frac{ \sum_{i=1}^{n}(x_i-\bar{x})^2+n(\bar{x}-\mu)^2}{2\sigma^2}\right) \Bigg] \\[6pt]
& = \frac{\partial}{\partial \sigma} \left[ \frac{n}{2}\log\left( \frac{1}{2\pi\sigma^2} \right) - \frac{ \sum_{i=1}^{n}(x_i-\bar{x})^2+n(\bar{x}-\mu)^2}{2\sigma^2} \right] \\[6pt]
& = -\frac{\,n\,}{\sigma} + \frac{ \sum_{i=1}^{n}(x_i-\bar{x})^2+n(\bar{x}-\mu)^2}{\sigma^3}
\end{align}
</math>
which is solved by
:<math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n(x_i-\mu)^2.</math>
Inserting the estimate <math>\mu = \widehat\mu</math> we obtain
:<math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (x_i - \bar{x})^2 = \frac{1}{n}\sum_{i=1}^n x_i^2 -\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n x_i x_j.</math>
To calculate its expected value, it is convenient to rewrite the expression in terms of zero-mean random variables ([[statistical error]]) <math>\delta_i \equiv \mu - x_i</math>. Expressing the estimate in these variables yields
: <math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^n (\mu - \delta_i)^2 -\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n (\mu - \delta_i)(\mu - \delta_j).</math>
Simplifying the expression above, utilizing the facts that <math>\operatorname{E}\big[\;\delta_i\;\big] = 0 </math> and <math>\operatorname{E}\big[\;\delta_i^2\;\big] = \sigma^2 </math>, allows us to obtain
:<math>\operatorname{E}\big[\;\widehat\sigma^2\;\big]= \frac{\,n-1\,}{n}\sigma^2.</math>
This means that the estimator <math>\widehat\sigma</math> is biased. However, <math>\widehat\sigma</math> is consistent.
Formally we say that the ''maximum likelihood estimator'' for <math>\theta=(\mu,\sigma^2)</math> is
:<math>\widehat{\theta} = \left(\widehat{\mu},\widehat{\sigma}^2\right).</math>
In this case the MLEs could be obtained individually. In general this may not be the case, and the MLEs would have to be obtained simultaneously.
The normal log-likelihood at its maximum takes a particularly simple form:
:<math>
\log\Big( \mathcal{L}(\hat\mu,\hat\sigma)\Big) = \frac{\,-n\;\;}{2} \big(\,\log(2\pi\hat\sigma^2) +1\,\big)
</math>
This maximum log-likelihood can be shown to be the same for more general [[least squares]], even for [[non-linear least squares]]. This is often used in determining likelihood-based approximate [[confidence interval]]s and [[confidence region]]s, which are generally more accurate than those using the asymptotic normality discussed above.
== Non-independent variables ==
It may be the case that variables are correlated, that is, not independent. Two random variables ''X'' and ''Y'' are independent only if their joint probability density function is the product of the individual probability density functions, i.e.
:<math>f(x,y)=f(x)f(y)\,</math>
Suppose one constructs an order-''n'' Gaussian vector out of random variables <math>(x_1,\ldots,x_n)</math>, where each variable has means given by <math>(\mu_1, \ldots, \mu_n)</math>. Furthermore, let the [[covariance matrix]] be denoted by <math>\mathit\Sigma</math>.
The joint probability density function of these ''n'' random variables is then given by:
:<math>f(x_1,\ldots,x_n)=\frac{1}{(2\pi)^{n/2}\sqrt{\text{det}(\mathit\Sigma)}} \exp\left( -\frac{1}{2} \left[x_1-\mu_1,\ldots,x_n-\mu_n\right]\mathit\Sigma^{-1} \left[x_1-\mu_1,\ldots,x_n-\mu_n\right]^\mathrm{T} \right)</math>
In the two variable case, the joint probability density function is given by:
:<math> f(x,y) = \frac{1}{2\pi \sigma_x \sigma_y \sqrt{1-\rho^2}} \exp\left[ -\frac{1}{2(1-\rho^2)} \left(\frac{(x-\mu_x)^2}{\sigma_x^2} - \frac{2\rho(x-\mu_x)(y-\mu_y)}{\sigma_x\sigma_y} + \frac{(y-\mu_y)^2}{\sigma_y^2}\right) \right] </math>
In this and other cases where a joint density function exists, the likelihood function is defined as above, in the section [[Maximum likelihood#Principles|Principles]], using this density.
== Iterative procedures ==
Consider problems where both states <math>x_i</math> and parameters such as <math>\sigma^2 </math> require to be estimated. Iterative procedures such as [[Expectation-maximization algorithm]]s may be used to solve joint state-parameter estimation problems.
For example, suppose that {{mvar|n}} samples of state estimates <math>\hat{x}_{i}</math> together with a sample mean <math>\bar{x}</math> have been calculated by either a minimum-variance [[Kalman filter]] or a minimum-variance smoother using a previous variance estimate <math>\widehat\sigma^2 </math>. Then the next variance iterate may be obtained from the maximum likelihood estimate calculation
:<math>\widehat\sigma^2 = \frac{1}{n} \sum_{i=1}^{n} (\hat{x}_i - \bar{x})^2. </math>
The convergence of MLEs within filtering and smoothing EM algorithms has been studied in the literature.<ref>{{cite journal
|last = Einicke
|first = G. A.
|last2 = Falco
|first2 = G.
|last3 = Malos
|first3 = J. T.
|title = EM Algorithm State Matrix Estimation for Navigation
|journal = IEEE Signal Processing Letters
|volume = 17
|issue = 5
|pages = 437–440
|date=May 2010
|postscript = <!--None-->
|doi=10.1109/LSP.2010.2043151
}}</ref><ref>{{cite journal
|last = Einicke
|first = G. A.
|last2 = Falco
|first2 = G.
|last3 = Dunn
|first3 = M. T.
|last4 = Reid
|first4 = D. C.
|title = Iterative Smoother-Based Variance Estimation
|journal = IEEE Signal Processing Letters
|volume = 19
|issue = 5
|pages = 275–278
|date=May 2012
|postscript = <!--None-->
|doi=10.1109/LSP.2012.2190278
}}</ref>
== History ==
Early users of maximum likelihood were [[Carl Friedrich Gauss]], [[Pierre-Simon Laplace]], [[Thorvald N. Thiele]], and [[Francis Ysidro Edgeworth]].<ref>{{harvtxt |Edgeworth |September 1908 }} and {{harvtxt |Edgeworth |December 1908 }}</ref>
[[File:Youngronaldfisher2.JPG|thumb|right|220px|Ronald Fisher in 1913]]
However its widespread use arose between 1912 and 1922 when [[Ronald Fisher]] recommended, widely popularized, and carefully analyzed maximum-likelihood estimation (with fruitless attempts at [[mathematical proof|proofs]]).<ref name="Pfanzagl">{{cite book|title=Parametric Statistical Theory|author=Pfanzagl, Johann, with the assistance of R. Hamböker|year=1994|publisher=[[Walter de Gruyter]]|isbn=3-11-013863-8|pages=207–208}}</ref>
Maximum-likelihood estimation finally transcended heuristic justification in a proof published by [[Samuel S. Wilks]] in 1938, now called "[[Likelihood-ratio test#Wilks' theorem|Wilks' theorem]]".<ref>{{harvtxt|Wilks|1938}}</ref> The theorem shows that the error in the logarithm of likelihood values for estimates from multiple independent samples is asymptotically [[chi-squared distribution|''χ'' ² distributed]], which enables convenient determination of a [[confidence region]] around any one estimate of the parameters. The only difficult part of [[Samuel S. Wilks|Wilks]]’ proof depends on the expected value of the [[Fisher information]] matrix, which ironically is provided by a theorem proven by [[Ronald Fisher|Fisher]].<ref>Owen, Art B. (2001). ''Empirical Likelihood''. London: Chapman & Hall/Boca Raton, FL: CRC Press. {{isbn|978-1584880714}}.</ref> Wilks continued to improve on the generality of the theorem throughout his life, with his most general proof published in 1962.<ref>Wilks, Samuel S. (1962), ''Mathematical Statistics'', New York: John Wiley & Sons. {{isbn|978-0471946502}}.</ref>
Some of the theory behind maximum likelihood estimation was developed for [[Bayesian statistics]].<ref name="Pfanzagl" />
Reviews of the development of maximum likelihood estimation have been provided by a number of authors.<ref>{{harvtxt |Savage |1976 }}, {{harvtxt |Pratt |1976 }}, {{harvs |txt=yes |last=Stigler |year=1978 |year2=1986 |year3=1999 }}, {{harvs |txt=yes |last=Hald |year=1998 |year2=1999 }}, and {{harvtxt |Aldrich |1997 }}</ref>
== See also ==
{{Portal|Statistics}}
* '''Other estimation methods'''
** [[Generalized method of moments]] are methods related to the likelihood equation in maximum likelihood estimation
** [[M-estimator]], an approach used in robust statistics
** [[Maximum a posteriori]] (MAP) estimator, for a contrast in the way to calculate estimators when prior knowledge is postulated
** [[Maximum spacing estimation]], a related method that is more robust in many situations
** [[Principle of maximum entropy|Maximum entropy estimation]]
** [[Method of moments (statistics)]], another popular method for finding parameters of distributions
** [[Method of support]], a variation of the maximum likelihood technique
** [[Minimum distance estimation]]
** [[Quasi-maximum likelihood]] estimator, an MLE estimator that is misspecified, but still consistent
** [[Restricted maximum likelihood]], a variation using a likelihood function calculated from a transformed set of data
* '''Related concepts''':
** [[Akaike information criterion]], a criterion to compare statistical models, based on MLE
** [[Extremum estimator]], a more general class of estimators to which MLE belongs
** [[Fisher information]], information matrix, its relationship to covariance matrix of ML estimates
** [[Mean squared error]], a measure of how 'good' an estimator of a distributional parameter is (be it the maximum likelihood estimator or some other estimator)
** [[RANSAC]], a method to estimate parameters of a mathematical model given data that contains [[outliers]]
** [[Rao–Blackwell theorem]], which yields a process for finding the best possible unbiased estimator (in the sense of having minimal [[mean squared error]]); the MLE is often a good starting place for the process
** [[Likelihood-ratio test#Wilks' theorem|Wilks’ Theorem]] provides a means of estimating the size and shape of the region of roughly equally-probable estimates for the population's parameter values, using the information from a single sample, using a [[chi-squared distribution]]
==Notes==
{{Reflist|30em}}
==References: non-historical==
{{Refbegin}}
* {{cite journal
| last1 = Cox | first1 = David R. | authorlink1=David R. Cox
| last2 = Snell | first2 = E. Joyce
| title = A general definition of residuals
| year = 1968
| journal = [[Journal of the Royal Statistical Society, Series B]]
| pages = 248–275
| jstor = 2984505
| ref = CITEREFCoxSnell1968
| volume=30
}}
* {{cite journal
|last = Kano
|first = Yutaka
|title = Third-order efficiency implies fourth-order efficiency
|year = 1996
|journal = Journal of the Japan Statistical Society
|volume = 26
|pages = 101–117
|url = http://www.journalarchive.jst.go.jp/english/jnlabstract_en.php?cdjournal=jjss1995&cdvol=26&noissue=1&startpage=101
|ref = harv
|doi = 10.14490/jjss1995.26.101
}}{{Dead link|date=September 2018 |bot=InternetArchiveBot |fix-attempted=yes }}
* {{cite book
| last1 = Newey | first1 = Whitney K.
| last2 = McFadden | first2 = Daniel | authorlink2 = Daniel McFadden
| chapter = Chapter 35: Large sample estimation and hypothesis testing
| editor1-first= Robert | editor1-last=Engle | editor2-first=Dan | editor2-last=McFadden
| title = Handbook of Econometrics, Vol.4
| year = 1994
| publisher = Elsevier Science
| pages = 2111–2245
| isbn=0-444-88766-0
| ref = CITEREFNeweyMcFadden1994
}}
{{Refend}}
==References: historical==
{{refbegin}}
* {{cite journal
| last = Aldrich | first = John
| title = R. A. Fisher and the making of maximum likelihood 1912–1922
| year = 1997
| journal = Statistical Science
| volume = 12 | issue = 3
| pages = 162–176
| doi = 10.1214/ss/1030037906
| mr = 1617519
| ref = CITEREFAldrich1997
}}
* {{cite journal
| doi = 10.2307/2339293
| last = Edgeworth | first = Francis Y. | authorlink = Francis Ysidro Edgeworth
| title = On the probable errors of frequency-constants
|date=Sep 1908
| ref= CITEREFEdgeworthSeptember_1908
| journal = Journal of the Royal Statistical Society
| volume = 71 | issue = 3
| pages = 499–512
| jstor = 2339293
}}
* {{cite journal
| doi = 10.2307/2339378
| last = Edgeworth | first = Francis Y.
| title = On the probable errors of frequency-constants
|date=Dec 1908
| ref= CITEREFEdgeworthDecember_1908
| journal = Journal of the Royal Statistical Society
| volume = 71 | issue = 4
| pages = 651–678
| jstor = 2339378
}}
* {{cite book
| last = Hald | first = Anders | authorlink=Anders Hald
| title = A history of mathematical statistics from 1750 to 1930
| year = 1998
| publisher = Wiley
| location = New York, NY
| isbn = 0-471-17912-4
| ref = CITEREFHald1998
}}
* {{cite journal
| last = Hald | first = Anders
| title = On the history of maximum likelihood in relation to inverse probability and least squares
| year = 1999
| journal = Statistical Science
| volume = 14 | issue = 2
| pages =214–222
| jstor = 2676741
| ref = CITEREFHald1999
| doi=10.1214/ss/1009212248
| url = http://projecteuclid.org/download/pdf_1/euclid.ss/1009212248
}}
* {{cite journal
| doi = 10.1214/aos/1176343457
| last = Pratt | first = John W.
| title = F. Y. Edgeworth and R. A. Fisher on the efficiency of maximum likelihood estimation
| year = 1976
| journal = The Annals of Statistics
| volume = 4 | issue = 3
| pages = 501–514
| jstor = 2958222
| ref = CITEREFPratt1976
}}
* {{cite journal
| doi = 10.1214/aos/1176343456
| last = Savage | first = Leonard J. | authorlink = Leonard J. Savage
| title = On rereading R. A. Fisher
| year = 1976
| journal = The Annals of Statistics
| volume = 4 | issue = 3
| pages = 441–500
| jstor = 2958221
| ref = CITEREFSavage1976
}}
* {{cite journal
| doi = 10.2307/2344804
| last = Stigler | first = Stephen M. | authorlink = Stephen M. Stigler
| title = Francis Ysidro Edgeworth, statistician
| year = 1978
| journal = Journal of the Royal Statistical Society, Series A
| volume = 141 | issue = 3
| pages = 287–322
| jstor = 2344804
| ref = CITEREFStigler1978
}}
* {{cite book
| last = Stigler | first = Stephen M.
| title = The history of statistics: the measurement of uncertainty before 1900
| year = 1986
| publisher = Harvard University Press
| isbn = 0-674-40340-1
| ref = CITEREFStigler1986
}}
* {{cite book
| last = Stigler | first = Stephen M.
| title = Statistics on the table: the history of statistical concepts and methods
| year = 1999
| publisher = Harvard University Press
| isbn = 0-674-83601-4
| ref = CITEREFStigler1999
}}
* {{citation|last=Wilks | first= S. S. |year= 1938 | title= The Large-Sample Distribution of the Likelihood Ratio for Testing Composite Hypotheses | journal= [[Annals of Mathematical Statistics]] | volume=9| pages= 60–62| doi= 10.1214/aoms/1177732360}}.
{{Refend}}
== Further reading ==
* Andersen, Erling B. (1970); "Asymptotic Properties of Conditional Maximum Likelihood Estimators", ''[[Journal of the Royal Statistical Society]]'' '''B''' 32, 283–301
* Andersen, Erling B. (1980); ''Discrete Statistical Models with Social Science Applications'', North Holland, 1980
* [[Debabrata Basu|Basu, Debabrata]] (1988); ''Statistical Information and Likelihood : A Collection of Critical Essays by Dr. D. Basu''; in Ghosh, Jayanta K., editor; ''Lecture Notes in Statistics'', Volume 45, Springer-Verlag, 1988
* {{cite book
| author = Einicke, G.A.
| year = 2012
| title = Smoothing, Filtering and Prediction: Estimating the Past, Present and Future
| publisher = Intech
| location = Rijeka, Croatia
| isbn = 978-953-307-752-9
| url = http://www.intechopen.com/books/smoothing-filtering-and-prediction-estimating-the-past-present-and-future}}
* {{cite journal |last=Ferguson |first=Thomas S. |title=An inconsistent maximum likelihood estimate |journal=Journal of the American Statistical Association |volume=77 |issue=380 |year=1982 |pages=831–834 |jstor=2287314 |doi=10.1080/01621459.1982.10477894}}
* {{cite book
| last = Ferguson | first = Thomas S.
| title = A Course in Large Sample Theory
| publisher = Chapman & Hall
| year = 1996
| isbn = 0-412-04371-8
}}
* {{cite journal
| last = Le Cam | first = Lucien | authorlink = Lucien Le Cam
| title = Maximum likelihood — an introduction
| journal = ISI Review
| volume = 58 | issue = 2
| year = 1990
| pages = 153–171
| doi=10.2307/1403464
}}
* {{cite book
| last1 = Le Cam | first1 = Lucien
| last2 = Yang | first2 = Grace Lo
| title = Asymptotics in Statistics: some basic concepts
| year = 2000
| edition = Second
| publisher = Springer
| isbn = 0-387-95036-2
| ref = CITEREFLeCam2000
}}
* {{cite book
| last1 = Lehmann | first1 = Erich L. | authorlink = Erich Leo Lehmann
| last2 = Casella | first2 = George
| title = Theory of Point Estimation, 2nd ed
| year = 1998
| publisher = Springer
| isbn = 0-387-98502-6
| ref = CITEREFLehamnnCasella1998
}}
* {{cite book | title=Maximum Likelihood Estimation and Inference | last= Millar | first= R. B. | year= 2011 | publisher= Wiley}}
* {{cite book |author=Ruppert, David |title=Statistics and Data Analysis for Financial Engineering |publisher=Springer |year=2010 |isbn=978-1-4419-7786-1 |page=98 |url=https://books.google.com/books?id=i2bD50PbIikC&pg=PA98 |ref=CITEREFRuppert2010 }}
* {{cite book
| last = van der Vaart | first = Aad W. |author-link= Aad van der Vaart
| title = Asymptotic Statistics
| year = 1998
| isbn = 0-521-78450-6
}}
== External links ==
* {{springer|title=Maximum-likelihood method|id=p/m063100}}
* Lemke, D. (2016). Maximum likelihood estimation and EM fixed point ideals for binary tensors. ([[San Francisco State University]]: Masters Theses Collection – Degree in Mathematics). OCLC# 980058195 URL: <nowiki>http://hdl.handle.net/10211.3/173201</nowiki>
* [http://statgen.iop.kcl.ac.uk/bgim/mle/sslike_1.html Maximum Likelihood Estimation Primer (an excellent tutorial)]
* [http://www.mayin.org/ajayshah/KB/R/documents/mle/mle.html Implementing MLE for your own likelihood function using R]
* [http://www.netstorm.be/home/mle A selection of likelihood functions in R]
* {{cite journal | title = Tutorial on maximum likelihood estimation | citeseerx = 10.1.1.74.671 | journal = [[Journal of Mathematical Psychology]] | last= Myung | first= I. J. | year= 2003 }}
{{Statistics}}
[[Category:Maximum likelihood estimation]]
[[Category:M-estimators]]
[[Category:Probability distribution fitting]]
