Machine Learning¶
Machine learning (ML) studies algorithms that improve at a task by learning from data rather than following explicitly programmed rules. Many of its methods come from statistics; the main difference is one of emphasis. Machine learning cares above all about generalization, meaning accurate predictions for new, unseen data, while classical statistics focuses more on interpreting parameters and quantifying uncertainty.
Two cultures
In his essay Statistical Modeling: The Two Cultures (2001), Leo Breiman contrasted a "data modeling" culture, which assumes a stochastic model for the data, with an "algorithmic modeling" culture, which treats the data-generating mechanism as unknown and judges models by predictive accuracy. Today the two cultures overlap considerably.
Contents¶
- Fundamentals: types of learning, loss functions, training and test data, overfitting, the bias-variance trade-off.
- Supervised Learning: logistic regression as a classifier, decision trees, random forests, boosting, support vector machines, nearest neighbors.
- Unsupervised Learning: k-means, hierarchical clustering, principal component analysis.
- Model Evaluation: cross-validation, metrics, ROC curves and AUC, regularization.
Statistics and machine learning compared¶
| Classical statistics | Machine learning | |
|---|---|---|
| Primary goal | inference and explanation | prediction |
| Typical question | Is this effect different from zero? | How accurate are predictions on new data? |
| Model choice | guided by theory and assumptions | guided by validation performance |
| Typical data size | small to moderate | moderate to very large |
| Typical models | simple, interpretable | from simple to highly flexible |
What comes next¶
The most flexible models in machine learning are neural networks, which lead into the AI part.