Supervised Learning¶
In supervised learning we observe pairs \((\mathbf{x}_i, y_i)\) and learn a function that predicts \(y\) for new inputs \(\mathbf{x}\). This page introduces the most widely used classical methods.
Logistic regression as a classifier¶
Logistic regression estimates a probability \(\hat\pi(\mathbf{x}) = P(y = 1 \mid \mathbf{x})\). To obtain a class label, the probability is compared with a threshold \(c\), by default 0.5:
The resulting decision boundary is linear (a hyperplane in the feature space). The threshold can be moved to trade false positives against false negatives. In the machine learning setting, attention shifts from p-values to predictive measures such as the AUC, and the model is usually regularized when there are many features.
Start with a baseline
Logistic regression is fast, stable and interpretable. It is a sensible first model for any binary classification problem, and more complex models should be judged by how much they improve on it.
Decision trees¶
A decision tree partitions the feature space through a sequence of binary questions such as "Is age below 40?". Each leaf predicts the majority class (classification) or the mean response (regression) of the training points it contains. Splits are chosen greedily to make the resulting nodes as pure as possible, for example by minimizing the Gini impurity
where \(p_k\) is the proportion of class \(k\) in the node. CART (Breiman et al., 1984) is the best-known algorithm.
Trees are easy to explain, need no feature scaling and capture interactions automatically. Their weaknesses are a strong tendency to overfit and high instability: a small change in the data can produce a very different tree. Limiting the depth or pruning helps, but combining many trees helps more.
Random forests¶
A random forest (Breiman, 2001) averages the predictions of many decision trees, each made different by two sources of randomness:
- Bagging: each tree is grown on a bootstrap sample of the training data.
- Random feature subsets: at each split, only a random subset of the features is considered.
Averaging many trees whose errors are only weakly correlated greatly reduces variance. Random forests perform well with little tuning and provide measures of variable importance.
Boosting¶
Boosting builds an ensemble sequentially: each new model, typically a small tree, focuses on what the current ensemble still gets wrong. Gradient boosting (Friedman, 2001) fits each new tree \(h_m\) to the negative gradient of the loss function, which for squared-error loss is simply the residuals, and updates
with a learning rate \(0 < \nu \le 1\). Implementations such as XGBoost, LightGBM and CatBoost are among the most accurate methods for tabular data. Because boosting keeps reducing training error, it needs to be controlled through the learning rate, tree depth and early stopping.
Support vector machines¶
A support vector machine (SVM) for two classes looks for the separating hyperplane with the largest margin, the distance to the closest training points of either class. These closest points are the support vectors. The soft-margin version tolerates some misclassified points, with a penalty parameter \(C\) controlling the trade-off.
Using the kernel trick, for example with a radial basis function (RBF) kernel, an SVM can learn nonlinear decision boundaries by implicitly working in a high-dimensional feature space. SVMs were among the leading classifiers before the rise of deep learning.
k-nearest neighbors¶
k-nearest neighbors (kNN) predicts for a new point by finding the \(k\) closest training points, usually in Euclidean distance, and taking their majority class or average response. There is no training phase beyond storing the data.
- A small \(k\) gives flexible, high-variance predictions; a large \(k\) gives smoother, more biased ones.
- Features must be on comparable scales, so standardization is essential.
- In high dimensions, distances become less informative (the curse of dimensionality), and kNN performs poorly.
Overview¶
| Method | Interpretability | Typical strength |
|---|---|---|
| Logistic regression | high | baseline, roughly linear effects |
| Decision tree | high | transparent rules |
| Random forest | medium | robust, little tuning |
| Gradient boosting | medium | top accuracy on tabular data |
| SVM | low to medium | moderate-sized, high-dimensional data |
| kNN | medium | simple, low-dimensional problems |