Linear Regression

Linear regression describes how the mean of a continuous response variable \(y\) depends on one or more predictors \(x_1, \dots, x_p\).

The model

For observations \(i = 1, \dots, n\),

\[ y_i = \beta_0 + \beta_1 x_{i1} + \dots + \beta_p x_{ip} + \varepsilon_i , \]

or, in matrix form, \(\mathbf{y} = \mathbf{X}\boldsymbol\beta + \boldsymbol\varepsilon\), where \(\mathbf{X}\) is the \(n \times (p+1)\) design matrix whose first column consists of ones. The coefficient \(\beta_0\) is the intercept, \(\beta_1, \dots, \beta_p\) are the slopes, and \(\varepsilon_i\) are unobserved random errors.

The model is called linear because it is linear in the coefficients. The predictors themselves may be transformed: \(y = \beta_0 + \beta_1 x + \beta_2 x^2 + \varepsilon\) is still a linear model.

Ordinary least squares

Ordinary least squares (OLS) chooses the coefficients that minimize the sum of squared residuals:

\[ \hat{\boldsymbol\beta} = \arg\min_{\boldsymbol\beta} \sum_{i=1}^n \big(y_i - \mathbf{x}_i^\top \boldsymbol\beta\big)^2 = (\mathbf{X}^\top\mathbf{X})^{-1}\mathbf{X}^\top\mathbf{y}, \]

where \(\mathbf{x}_i^\top\) is the \(i\)-th row of \(\mathbf{X}\). The closed-form solution exists whenever \(\mathbf{X}\) has full column rank. With a single predictor it reduces to

\[ \hat\beta_1 = \frac{\sum_{i}(x_i - \bar x)(y_i - \bar y)}{\sum_{i}(x_i - \bar x)^2}, \qquad \hat\beta_0 = \bar y - \hat\beta_1 \bar x . \]

Least squares and maximum likelihood

If the errors are independent and normally distributed with constant variance, the OLS estimator is also the maximum likelihood estimator of \(\boldsymbol\beta\).

Assumptions

The usual standard errors, tests and confidence intervals rely on the following assumptions:

1. Linearity: \(E(y_i \mid \mathbf{x}_i) = \mathbf{x}_i^\top\boldsymbol\beta\).
2. Exogeneity: \(E(\varepsilon_i \mid \mathbf{X}) = 0\); in particular, no omitted variable is correlated with the included predictors.
3. Homoscedasticity: the errors have constant variance \(\sigma^2\).
4. No autocorrelation: errors of different observations are uncorrelated, which is often violated in time series or clustered data.
5. No perfect multicollinearity: no predictor is an exact linear combination of the others.
6. Normality of the errors, needed for exact small-sample \(t\) and \(F\) tests; for large samples the CLT makes it less important.

Under assumptions 1–5, the Gauss–Markov theorem states that OLS is the best linear unbiased estimator (BLUE): no other linear unbiased estimator has smaller variance. Residual plots are the main diagnostic tool, and heteroscedasticity-robust standard errors offer protection against violations of assumption 3.

Interpretation

• \(\hat\beta_j\) is the estimated change in the mean of \(y\) associated with a one-unit increase in \(x_j\), holding all other predictors fixed.
• The \(t\) test of \(H_0: \beta_j = 0\) (see Hypothesis Testing) asks whether \(x_j\) is associated with \(y\) once the other predictors are accounted for.
• The coefficient of determination

\[ R^2 = 1 - \frac{\sum_i (y_i - \hat y_i)^2}{\sum_i (y_i - \bar y)^2} \]

is the proportion of the variance of \(y\) explained by the model. Because \(R^2\) never decreases when predictors are added, the adjusted \(R^2\) is preferred for comparing models of different size.

Example

Suppose a fitted model for apartment prices is \(\widehat{\text{price}} = 50\,000 + 3\,000 \cdot \text{area}\), with price in euros and area in square meters. Each additional square meter is associated with an expected price that is 3,000 euros higher.

Association is not causation

A regression coefficient measures an association. Interpreting it causally requires a design that rules out confounding, such as a randomized experiment, or dedicated methods of causal inference.

Outlook

If the response takes only the values 0 and 1, a straight line can predict values below 0 or above 1, and the constant-variance assumption fails. Logistic regression solves both problems.

Next: Logistic Regression