Logistic Regression

Logistic regression models the probability of a binary outcome \(y \in \{0, 1\}\), such as disease or no disease, purchase or no purchase, as a function of predictors.

The bridge to machine learning

Logistic regression appears in all three parts of this wiki. In statistics it is used to estimate and interpret effects on the odds of an outcome. In machine learning it serves as a classifier. In neural networks it is identical to a single neuron with a sigmoid activation.

The model

As in linear regression, we start from a linear predictor \(\eta = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p = \mathbf{x}^\top\boldsymbol\beta\), which can take any real value. The logistic function, also called the sigmoid, maps it to a probability between 0 and 1:

\[ \pi(\mathbf{x}) = P(y = 1 \mid \mathbf{x}) = \sigma(\eta) = \frac{1}{1 + e^{-\eta}} . \]

Odds and the logit

The odds of an event with probability \(\pi\) are \(\pi/(1-\pi)\); a probability of 0.8 corresponds to odds of 4 to 1. The inverse of the logistic function is the logit, the logarithm of the odds:

\[ \operatorname{logit}(\pi) = \log\frac{\pi}{1-\pi} = \beta_0 + \beta_1 x_1 + \dots + \beta_p x_p . \]

Logistic regression is therefore a linear model for the log-odds. It is a member of the family of generalized linear models (GLMs), with a binomial response distribution and the logit as link function.

Interpreting coefficients: the odds ratio

A one-unit increase in \(x_j\), with all other predictors held fixed, changes the log-odds by \(\beta_j\) and therefore multiplies the odds by

\[ \text{OR}_j = e^{\beta_j}, \]

the odds ratio. An odds ratio above 1 indicates higher odds of \(y = 1\), a value of 1 indicates no association, and a value below 1 indicates lower odds.

Example

In a model for heart disease, suppose the estimated coefficient of a smoking indicator is \(\hat\beta = 0.69\). Then \(e^{0.69} \approx 2.0\): holding the other predictors fixed, smokers have about twice the odds of heart disease compared with non-smokers.

Odds ratios are not risk ratios

Doubling the odds is not the same as doubling the probability. The odds ratio approximates the relative risk only when the outcome is rare.

Maximum likelihood estimation

Unlike OLS, logistic regression has no closed-form estimator. The coefficients are estimated by maximum likelihood. Each \(y_i\) is a Bernoulli variable with success probability \(\pi_i = \sigma(\mathbf{x}_i^\top\boldsymbol\beta)\), so the log-likelihood is

\[ \ell(\boldsymbol\beta) = \sum_{i=1}^n \Big[ y_i \log \pi_i + (1 - y_i)\log(1 - \pi_i) \Big]. \]

This function is concave, so any maximum is global. It is maximized numerically, traditionally by Newton–Raphson, which for this model is equivalent to iteratively reweighted least squares (IRLS). The negative log-likelihood is the binary cross-entropy, the loss function used to train classifiers and neural networks, typically with gradient-based methods.

Individual coefficients are tested with Wald tests; nested models are compared with likelihood ratio tests.

Separation

If a predictor or combination of predictors separates the two outcome classes perfectly, the likelihood keeps increasing as the coefficients grow without bound, and no finite ML estimate exists. Penalized methods such as Firth's bias-reduced logistic regression or regularization resolve this.

Extensions

• Multinomial logistic regression handles more than two unordered categories; in machine learning it is known as softmax regression.
• Ordinal logistic regression handles ordered categories.
• Penalized logistic regression (ridge, lasso) handles many predictors.

What comes next

This completes the statistical inference part. Machine learning asks a different question: not whether an effect is significant, but how accurately a model predicts outcomes for new cases.

Continue to Machine Learning