Estimation¶
Estimation means using a sample to determine the value of an unknown population parameter \(\theta\), such as a mean \(\mu\), a proportion \(p\) or a variance \(\sigma^2\).
Point estimation¶
A point estimator \(\hat\theta = T(X_1, \dots, X_n)\) is a rule that turns the data into a single number. Familiar examples are the sample mean \(\bar X\) for \(\mu\), the sample proportion \(\hat p\) for \(p\), and the sample variance \(S^2\) for \(\sigma^2\). An estimator is a random variable; the number obtained from one particular sample is called an estimate.
Properties of estimators¶
- Unbiasedness: \(E(\hat\theta) = \theta\). On average over repeated samples, the estimator hits the true value. Dividing by \(n-1\) rather than \(n\) makes \(S^2\) unbiased for \(\sigma^2\).
- Efficiency: among unbiased estimators, the one with the smallest variance is preferred.
- Consistency: \(\hat\theta\) converges in probability to \(\theta\) as \(n \to \infty\).
The standard deviation of an estimator is called its standard error. For the sample mean it is \(\sigma/\sqrt{n}\), estimated by \(s/\sqrt{n}\).
Mean squared error¶
The mean squared error combines both sources of inaccuracy:
Connection to machine learning
The same decomposition reappears as the bias-variance trade-off. An estimator with a small bias but much lower variance can have a smaller MSE than an unbiased one, which is the rationale behind regularization.
Confidence intervals¶
A point estimate says nothing about its precision. A confidence interval at level \(1-\alpha\) is a random interval constructed so that, over repeated samples, it contains the true parameter with probability \(1-\alpha\).
For the mean of a normally distributed population with unknown variance, the interval is
where \(t_{n-1,\,1-\alpha/2}\) is the \(1-\alpha/2\) quantile of the \(t\) distribution with \(n-1\) degrees of freedom. Thanks to the central limit theorem, the same interval is approximately valid for non-normal data when \(n\) is large; the \(t\) quantile then approaches the normal quantile, which is about 1.96 for a 95 % interval.
Interpreting a confidence interval
The 95 % describes the long-run behavior of the procedure: 95 % of intervals constructed this way contain the true value. It is not the probability that the true value lies in one particular, already computed interval; in the frequentist framework the parameter is fixed, so that interval either contains it or does not. Statements of the form "the parameter lies in this interval with probability 95 %" belong to Bayesian credible intervals.
Maximum likelihood estimation¶
Maximum likelihood (ML) is the most widely used general method for constructing estimators. Its idea is to choose the parameter value under which the observed data are most probable. For independent observations with density or mass function \(f(x \mid \theta)\), the likelihood and the log-likelihood are
and the ML estimator is \(\hat\theta_{\text{ML}} = \arg\max_\theta \ell(\theta)\). Maximizing \(\ell\) instead of \(L\) gives the same result and is easier, because the product becomes a sum.
Example: estimating a proportion
Suppose \(k\) of \(n\) independent Bernoulli trials are successes. Then \(\ell(p) = k \log p + (n-k)\log(1-p)\). Setting the derivative \(k/p - (n-k)/(1-p)\) to zero gives \(\hat p = k/n\), the observed proportion.
Under standard regularity conditions, ML estimators are consistent and asymptotically normal, and their variance approaches the smallest achievable value (the Cramér–Rao bound) as \(n\) grows. Maximum likelihood is used again for logistic regression, and training a neural network by minimizing cross-entropy is a form of maximum likelihood estimation.