Neural Networks & Deep Learning¶
An artificial neural network is a function built from many simple units, called neurons, arranged in layers. Each unit computes a weighted sum of its inputs and applies a nonlinear function. The design was loosely inspired by biological neurons, but in practice a neural network is best understood as a flexible mathematical model whose parameters are learned from data.
The perceptron¶
The perceptron, published by Frank Rosenblatt in 1958, outputs 1 if a weighted sum of its inputs exceeds a threshold and 0 otherwise:
with weight vector \(\mathbf{w}\) and bias \(b\). Rosenblatt's learning rule finds a separating hyperplane whenever the two classes are linearly separable, but a single perceptron cannot solve problems that are not, such as the XOR function. Minsky and Papert analyzed these limitations in their 1969 book Perceptrons, which contributed to a decline in neural network research in the following years.
Logistic regression as a neuron¶
Replacing the hard threshold with the smooth sigmoid function gives
This is exactly the logistic regression model: the weights \(\mathbf{w}\) correspond to the slopes \(\beta_1, \dots, \beta_p\) and the bias \(b\) to the intercept \(\beta_0\). Training this neuron by minimizing binary cross-entropy is the same as maximum likelihood estimation of the logistic regression coefficients.
The common thread
A neural network for binary classification with a sigmoid output unit is a logistic regression whose inputs are not the raw features but features learned by the preceding layers.
Multilayer networks¶
A multilayer perceptron (MLP), also called a feedforward network, stacks layers of neurons:
The activation function \(g\) must be nonlinear; otherwise the composition of layers would collapse into a single linear map. The rectified linear unit, ReLU, \(g(z) = \max(0, z)\), is a common default for hidden layers. For classification with more than two classes, the output layer applies the softmax function, which turns a vector of scores into probabilities that sum to one.
The universal approximation theorem (Cybenko, 1989; Hornik, 1991) states that a network with a single hidden layer and a suitable nonlinear activation can approximate any continuous function on a compact domain to any desired accuracy, given enough hidden units. The theorem says nothing about how many units are needed or how to find the weights. In practice, deep networks with many layers learn useful representations far more efficiently, hence the term deep learning.
Training: gradient descent and backpropagation¶
Training means adjusting all parameters \(\boldsymbol\theta\) to minimize a loss function \(L\), such as the cross-entropy for classification. Gradient descent repeatedly takes a small step in the direction of steepest descent:
where \(\eta > 0\) is the learning rate. The gradient is computed by backpropagation, an efficient application of the chain rule that passes error signals backwards from the output layer through the network. The underlying technique, reverse-mode automatic differentiation, was described by Seppo Linnainmaa in 1970; Paul Werbos proposed applying it to neural networks, and Rumelhart, Hinton and Williams (1986) demonstrated and popularized its use for training multilayer networks. In practice, gradients are estimated on small random subsets of the data (mini-batch stochastic gradient descent), often with adaptive methods such as Adam (Kingma & Ba, 2014).
Why deep learning took off around 2012
The core ideas are decades old. The breakthrough came from the combination of large labeled data sets such as ImageNet, fast parallel computation on graphics processing units (GPUs), and practical improvements such as ReLU activations, better weight initialization and dropout.
Important architectures¶
- Convolutional neural networks (CNNs) apply small learned filters across an image, sharing weights between positions. They dominated computer vision for most of the 2010s.
- Recurrent neural networks (RNNs), especially long short-term memory networks (LSTMs; Hochreiter & Schmidhuber, 1997), process sequences one element at a time while maintaining an internal state. They were the standard for language tasks before transformers.
- Autoencoders learn to compress inputs into a low-dimensional code and reconstruct them, a nonlinear relative of PCA.
- Transformers replace recurrence with attention and are covered on the next page.
Overfitting
Deep networks often have millions or billions of parameters. Regularization techniques such as dropout, weight decay, data augmentation and early stopping, together with careful model evaluation, are essential.