Transformers¶
The transformer is a neural network architecture introduced by Vaswani et al. in the 2017 paper Attention Is All You Need. It was designed for machine translation, but it has since become the foundation of virtually all large language models and is widely used for images, audio and other kinds of data.
Why not recurrence?¶
Recurrent networks read a sequence one element at a time. This causes two problems:
- Limited parallelism: step \(t\) cannot be computed before step \(t-1\), which makes training on modern parallel hardware slow.
- Long-range dependencies: information from early in a sequence must be carried through many intermediate steps and tends to fade.
The transformer removes recurrence altogether. Every position in a sequence can attend directly to every other position, and all positions are processed in parallel.
Attention¶
The central operation is self-attention. Each token (a word or part of a word) is represented by a vector. Three learned linear projections turn each of these vectors into a query (what this position is looking for), a key (what it offers to others) and a value (the information it passes on). Stacking these vectors for all positions gives matrices \(Q\), \(K\) and \(V\). Scaled dot-product attention is then
The dot products in \(QK^\top\) measure how well each query matches each key. The softmax, applied row by row, turns these scores into weights that sum to one, and each output is the correspondingly weighted average of the values. Dividing by \(\sqrt{d_k}\), where \(d_k\) is the dimension of the keys, keeps the scores from growing with the dimension, which would otherwise push the softmax into regions with very small gradients.
Intuition
In the sentence "The cat sat on the mat because it was tired", self-attention allows the representation of "it" to draw heavily on the representation of "cat", resolving the reference.
Multi-head attention¶
Rather than computing a single attention function, the transformer runs several in parallel, each with its own projections. Each of these heads can specialize in a different kind of relationship, for example syntactic or semantic. Their outputs are concatenated and projected back to the model dimension.
The architecture¶
Each transformer layer consists of two sub-layers:
- multi-head self-attention, and
- a position-wise feed-forward network, a small MLP applied to each position separately.
Each sub-layer is wrapped in a residual connection and layer normalization. Many such layers are stacked; the base model in the original paper used six layers each in the encoder and decoder.
Attention by itself ignores the order of the tokens, so information about position must be added. The original paper added fixed sinusoidal positional encodings to the input embeddings; later models often use learned or relative position schemes, such as rotary position embeddings (RoPE).
The original transformer has two parts:
| Part | Role | Well-known descendants |
|---|---|---|
| Encoder | reads the whole input and builds a contextual representation of every token | BERT |
| Decoder | generates the output one token at a time; causal masking prevents each position from attending to later positions | the GPT family |
In translation, the decoder also attends to the encoder's output through cross-attention. Most modern generative language models are decoder-only.
Tokens and embeddings¶
Before text enters the model, it is split into tokens by a tokenizer, usually into subword units produced by algorithms such as byte-pair encoding. Each token is mapped to a learned vector, its embedding. At the output, a linear layer followed by a softmax over the vocabulary gives a probability distribution for the next token. This final step is a multinomial logistic regression, the many-class extension of logistic regression.
Quadratic cost
Self-attention compares every token with every other token, so computation and memory grow quadratically with sequence length. Handling long contexts efficiently is an active research area.
Why transformers succeeded
Transformers parallelize well on GPUs and scale smoothly: with more data, parameters and compute, their performance improves predictably across many tasks. This scalability is a key reason for the rise of large language models.