Model Evaluation¶

A model is only as good as its performance on data it has not seen. This page covers how to estimate that performance reliably, which metrics to use, and how regularization helps models generalize.

Cross-validation¶

In \(k\)-fold cross-validation, the data are split into \(k\) folds of roughly equal size. Each fold is used once as the validation set while the model is trained on the other \(k-1\) folds. The cross-validation estimate is the average of the \(k\) validation errors:

\[ \text{CV}_{(k)} = \frac{1}{k}\sum_{j=1}^k \text{Err}_j . \]

Common choices are \(k = 5\) or \(k = 10\); the special case \(k = n\) is leave-one-out cross-validation.

Important variants:

Stratified cross-validation keeps the class proportions the same in every fold, which matters for imbalanced data.
Grouped cross-validation keeps all observations from the same group (for example, the same patient) in the same fold.
For time series, models are trained on the past and validated on the future, for example with a rolling origin.
Nested cross-validation uses an inner loop for tuning hyperparameters and an outer loop for an unbiased performance estimate.

Metrics¶

Regression¶

Mean squared error (MSE) and its square root, the RMSE, which is in the units of \(y\).
Mean absolute error (MAE), which is less sensitive to large errors.
\(R^2\), the proportion of variance explained (see Linear Regression).

Classification¶

Binary classification results are summarized in a confusion matrix:

	actual positive	actual negative
predicted positive	true positive (TP)	false positive (FP)
predicted negative	false negative (FN)	true negative (TN)

From it we derive

\[ \text{accuracy} = \frac{TP + TN}{TP + TN + FP + FN}, \qquad \text{precision} = \frac{TP}{TP + FP}, \qquad \text{recall} = \frac{TP}{TP + FN}, \]

\[ F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} . \]

Recall is also called sensitivity or the true positive rate. Specificity is \(TN/(TN + FP)\). False positives and false negatives correspond to type I and type II errors.

Imbalanced classes

If only 1 % of cases are positive, as in fraud detection, a model that always predicts "negative" achieves 99 % accuracy and is useless. Precision, recall, the \(F_1\) score or the area under the precision-recall curve are more informative in such cases.

When a model outputs probabilities, two further aspects matter. The log loss (cross-entropy) rewards confident correct predictions and heavily penalizes confident wrong ones. Calibration asks whether predicted probabilities match observed frequencies: among cases with a predicted probability of 0.3, about 30 % should be positive.

ROC curve and AUC¶

A probabilistic classifier such as logistic regression produces scores, and the predicted class depends on the threshold. The receiver operating characteristic (ROC) curve plots the true positive rate against the false positive rate (\(1 -\) specificity) as the threshold varies over all possible values.

The area under the curve (AUC) summarizes the ROC curve in one number. An AUC of 0.5 corresponds to random guessing and an AUC of 1 to perfect separation. The AUC equals the probability that a randomly chosen positive case receives a higher score than a randomly chosen negative case (ties counted as one half). It does not depend on a threshold, but it says nothing about calibration.

Regularization¶

Regularization reduces overfitting by adding a penalty on the size of the coefficients to the loss. It deliberately introduces some bias in exchange for a larger reduction in variance (see the bias-variance trade-off):

\[ \hat{\boldsymbol\beta} = \arg\min_{\boldsymbol\beta} \; \sum_{i=1}^n L\big(y_i, \mathbf{x}_i^\top\boldsymbol\beta\big) + \lambda\, P(\boldsymbol\beta), \qquad \lambda \ge 0 . \]

Ridge regression (L2 penalty, Hoerl & Kennard, 1970): \(P(\boldsymbol\beta) = \sum_{j=1}^p \beta_j^2\). All coefficients are shrunk towards zero.
Lasso (L1 penalty, Tibshirani, 1996): \(P(\boldsymbol\beta) = \sum_{j=1}^p |\beta_j|\). Some coefficients become exactly zero, so the lasso also performs variable selection.
Elastic net: a weighted combination of both penalties.

The intercept is usually not penalized. The penalty strength \(\lambda\) is a hyperparameter chosen by cross-validation. Other forms of regularization include early stopping, limiting tree depth and, for neural networks, dropout and weight decay.

Standardize features before penalizing

Because the penalty depends on the magnitude of the coefficients, and that magnitude depends on the units of each feature, features should be standardized before fitting ridge or lasso models.

What comes next¶

This completes the classical machine learning toolkit. The next part turns to neural networks, the foundation of modern AI.

Continue to AI