Hypothesis Testing¶
A hypothesis test is a procedure for deciding, on the basis of data, whether there is enough evidence against a default assumption, while controlling the probability of a false alarm.
Null and alternative hypotheses¶
- The null hypothesis \(H_0\) is the default position, usually "no effect" or "no difference", for example \(H_0: \mu_1 = \mu_2\).
- The alternative hypothesis \(H_1\) is what we look for evidence of, either two-sided (\(\mu_1 \ne \mu_2\)) or one-sided (\(\mu_1 > \mu_2\)).
A test statistic \(T\) is computed from the data. Its distribution under \(H_0\) is known, so we can judge whether the observed value would be unusual if \(H_0\) were true.
The p-value¶
The p-value is the probability, computed under the assumption that \(H_0\) is true, of obtaining a test statistic at least as extreme as the one observed. For a two-sided test,
If \(p \le \alpha\), where the significance level \(\alpha\) is fixed in advance (commonly 0.05), \(H_0\) is rejected and the result is called statistically significant.
Common misinterpretations
- The p-value is not the probability that \(H_0\) is true.
- A small p-value does not mean that the effect is large or practically important.
- A large p-value does not show that \(H_0\) is true; the study may simply lack power.
The American Statistical Association addressed these issues in a 2016 statement on p-values (Wasserstein & Lazar, 2016). Reporting effect sizes with confidence intervals alongside p-values is good practice.
Type I and type II errors¶
| \(H_0\) true | \(H_0\) false | |
|---|---|---|
| reject \(H_0\) | type I error (probability \(\alpha\)) | correct (probability \(1-\beta\), the power) |
| do not reject \(H_0\) | correct | type II error (probability \(\beta\)) |
The significance level \(\alpha\) is the probability of a type I error that we are willing to accept. The power \(1-\beta\) is the probability of detecting an effect that really exists; it increases with the effect size, the sample size and \(\alpha\), and decreases with the variability of the data. Studies are usually planned to achieve a power of at least 80 % for the smallest effect of practical interest.
Connection to classification
A type I error is a false positive and a type II error a false negative, exactly the off-diagonal cells of a confusion matrix.
Common tests¶
| Test | Question | Main assumptions |
|---|---|---|
| One-sample \(t\) test | Does a mean differ from a given value \(\mu_0\)? | approximately normal data or large \(n\) |
| Welch's two-sample \(t\) test | Do the means of two independent groups differ? | independent groups; equal variances not required |
| Paired \(t\) test | Do paired measurements differ on average? | differences approximately normal |
| \(\chi^2\) test of independence | Are two categorical variables associated? | expected cell counts not too small (often \(\ge 5\)) |
| Mann–Whitney \(U\) test | Do two independent groups tend to differ in level? | ordinal or continuous data; no normality needed |
| One-way ANOVA (\(F\) test) | Do the means of three or more groups differ? | normality, equal variances, independence |
The one-sample \(t\) statistic is
which follows a \(t\) distribution with \(n-1\) degrees of freedom under \(H_0\) when the data are normal.
Multiple testing¶
When many hypotheses are tested, some will appear significant by chance. With 20 independent tests at \(\alpha = 0.05\) and all null hypotheses true, the probability of at least one false positive is \(1 - 0.95^{20} \approx 0.64\). The Bonferroni correction tests each of \(m\) hypotheses at level \(\alpha/m\), which controls the probability of any false positive. The Benjamini–Hochberg procedure controls the false discovery rate, the expected proportion of false positives among the rejections, and is less conservative.