Fundamentals¶
Random variables¶
A random variable \(X\) is a numerical quantity whose value is determined by the outcome of a random process. It is discrete if it takes countably many values (the number of defective items in a batch) and continuous if it can take any value in an interval (a person's height).
A discrete random variable is described by its probability mass function \(P(X = x)\). A continuous random variable is described by a probability density function \(f(x)\); probabilities are areas under the density, \(P(a \le X \le b) = \int_a^b f(x)\,dx\), and the probability of any single exact value is zero.
Expectation and variance¶
The expected value is the probability-weighted average of the possible values, and the variance is the expected squared deviation from it:
The standard deviation \(\sigma = \sqrt{\operatorname{Var}(X)}\) is measured in the same units as \(X\).
Common distributions¶
| Distribution | Type | Typical use | Mean | Variance |
|---|---|---|---|---|
| Bernoulli(\(p\)) | discrete | a single yes/no outcome | \(p\) | \(p(1-p)\) |
| Binomial(\(n, p\)) | discrete | number of successes in \(n\) independent trials | \(np\) | \(np(1-p)\) |
| Poisson(\(\lambda\)) | discrete | counts of events in a fixed interval | \(\lambda\) | \(\lambda\) |
| Normal(\(\mu, \sigma^2\)) | continuous | measurement errors, sums of many small effects | \(\mu\) | \(\sigma^2\) |
| Student's \(t_\nu\) | continuous | inference on means when \(\sigma\) is estimated | \(0\) (for \(\nu > 1\)) | \(\nu/(\nu-2)\) (for \(\nu > 2\)) |
| \(\chi^2_\nu\) | continuous | variances, goodness-of-fit, contingency tables | \(\nu\) | \(2\nu\) |
Here \(\nu\) denotes the degrees of freedom.
Populations and samples¶
The population is the complete set of units we want to learn about, for example all items produced on a manufacturing line in one year. A sample is the subset we actually observe. Valid inference requires that the sample be drawn by a known random mechanism. In a simple random sample, every unit has the same chance of selection, and the observations \(X_1, \dots, X_n\) can be treated as independent and identically distributed (i.i.d.).
A quantity computed from the sample is called a statistic. The most important ones are the sample mean and the sample variance:
Because a statistic depends on which units happen to be sampled, it is itself a random variable. Its distribution over all possible samples is called the sampling distribution. For i.i.d. data, \(E(\bar X) = \mu\) and \(\operatorname{Var}(\bar X) = \sigma^2/n\).
Law of large numbers and central limit theorem¶
The law of large numbers states that the sample mean \(\bar X\) converges to the population mean \(\mu\) as the sample size grows.
The central limit theorem (CLT) describes the shape of the remaining uncertainty. If the \(X_i\) are i.i.d. with finite variance \(\sigma^2\), then the standardized mean approaches a standard normal distribution:
In practice this means that \(\bar X\) is approximately \(N(\mu, \sigma^2/n)\) for large \(n\), whatever the distribution of the individual observations.
Why the CLT matters
Most confidence intervals and tests in the following pages rely on the CLT. It also explains why precision improves only with the square root of the sample size: halving the standard error requires four times as many observations.
How large is large?
The popular rule "\(n \ge 30\) is enough" is only a rough guide. For heavily skewed or heavy-tailed distributions, the normal approximation can remain poor for much larger samples.