Glossary

Brief definitions of key terms, in alphabetical order. Each entry links to the page where the term is explained in detail.

Attention

A mechanism that computes each output as a weighted average of values, with weights determined by the similarity between queries and keys; the core of the transformer. → Transformers

AUC

Area under the ROC curve; the probability that a random positive case is scored higher than a random negative case. 0.5 means chance, 1 means perfect separation. → Model Evaluation

Backpropagation

An efficient method for computing the gradient of a loss with respect to all weights of a neural network by applying the chain rule backwards through the layers. → Neural Networks

Bias-variance trade-off

The tension between the systematic error of overly simple models and the sensitivity to the training data of overly flexible ones. → ML Fundamentals

Boosting

An ensemble method that adds models sequentially, each one correcting the errors of the current ensemble. → Supervised Learning

Central limit theorem

The result that the standardized mean of many i.i.d. random variables with finite variance is approximately standard normal. → Fundamentals

Confidence interval

An interval computed from the data by a procedure that, over repeated samples, contains the true parameter with a stated probability. → Estimation

Cross-entropy

A loss function for probabilistic classifiers; equal to the negative log-likelihood of the observed labels. → Logistic Regression

Cross-validation

Repeatedly splitting the data into training and validation parts to estimate how well a model generalizes. → Model Evaluation

Decision tree

A model that predicts by applying a hierarchy of yes/no splits on the features. → Supervised Learning

Diffusion model

A generative model that learns to reverse a gradual noising process, generating data by step-by-step denoising. → Recent Developments

Embedding

A learned vector representation of an object, such as a token, in which similar objects lie close together. → Transformers

Fine-tuning

Further training of a pretrained model on a smaller, task-specific data set. → Language Models

Gradient descent

An iterative optimization method that updates parameters in the direction of the negative gradient of the loss. → Neural Networks

Hallucination

Fluent but false or unsupported output from a language model. → Language Models

k-means

A clustering method that partitions data into \(k\) groups by minimizing the squared distances to the group means. → Unsupervised Learning

Logit

The logarithm of the odds, \(\log\frac{\pi}{1-\pi}\); the inverse of the sigmoid function. → Logistic Regression

Maximum likelihood

An estimation principle that chooses the parameter values under which the observed data are most probable. → Estimation

Odds ratio

The factor by which the odds of an outcome are multiplied when a predictor increases by one unit, \(e^{\beta}\) in logistic regression. → Logistic Regression

Ordinary least squares (OLS)

Estimating regression coefficients by minimizing the sum of squared residuals. → Linear Regression

Overfitting

Fitting the training data so closely that noise is learned as if it were signal, leading to poor performance on new data. → ML Fundamentals

p-value

The probability, assuming the null hypothesis is true, of a test statistic at least as extreme as the one observed. → Hypothesis Testing

Perceptron

An early model of an artificial neuron that outputs 1 when a weighted sum of its inputs exceeds a threshold. → Neural Networks

Principal component analysis (PCA)

A linear method that finds uncorrelated directions of maximal variance, used for dimensionality reduction. → Unsupervised Learning

Random forest

An ensemble of decision trees grown on bootstrap samples with random feature subsets, whose predictions are averaged. → Supervised Learning

Regularization

Adding a penalty on model complexity, such as the size of the coefficients, to reduce overfitting. → Model Evaluation

Retrieval-augmented generation (RAG)

Supplying a language model with passages retrieved from a document collection to ground its answers. → Recent Developments

RLHF

Reinforcement learning from human feedback; optimizing a model against a reward model trained on human preference judgments. → Language Models

Sigmoid

The logistic function \(\sigma(z) = 1/(1+e^{-z})\), which maps any real number to the interval \((0,1)\). → Logistic Regression

Transformer

A neural network architecture based on attention, introduced in 2017; the basis of modern language models. → Transformers

Type I and type II errors

Rejecting a true null hypothesis (type I, false positive) or failing to reject a false one (type II, false negative). → Hypothesis Testing