Language Models¶
A language model assigns probabilities to sequences of tokens. Most modern models are autoregressive: using the chain rule of probability, they factor the probability of a sequence into a product of next-token predictions,
Large language models (LLMs) implement this with transformers that have billions of parameters and are trained on very large text corpora.
BERT and GPT¶
Two model families introduced in 2018 shaped the field (GPT: OpenAI technical report, June 2018; BERT: arXiv preprint, October 2018, published at NAACL 2019):
| BERT (Devlin et al., Google) | GPT (Radford et al., OpenAI) | |
|---|---|---|
| Architecture | transformer encoder | transformer decoder |
| Pretraining task | predict randomly masked tokens (masked language modeling) | predict the next token |
| Context | bidirectional | left-to-right (causal) |
| Typical use | understanding tasks: classification, search, entity recognition | text generation, dialogue |
BERT became the standard starting point for language understanding tasks. The GPT line grew through GPT-2 (2019) and GPT-3 (2020). GPT-3, with 175 billion parameters, showed that sufficiently large models can perform new tasks from a few examples given in the prompt, without any change to their weights (Brown et al., 2020). This ability is called in-context learning or few-shot prompting.
Pretraining¶
During pretraining, a model learns from large amounts of unlabeled text such as web pages, books and source code. This is self-supervised learning: the target at each position is simply the token that actually comes next. The model minimizes the cross-entropy
which is the negative log-likelihood, so pretraining is a form of maximum likelihood estimation. To predict text well, the model has to pick up grammar, facts about the world and patterns of reasoning. Empirically, the loss decreases smoothly and predictably as model size, data and compute increase; these regularities are known as scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022).
The result is a base model. It continues text fluently but does not reliably follow instructions or behave like an assistant.
Fine-tuning¶
Fine-tuning continues training a pretrained model on a smaller, task-specific data set:
- Task-specific fine-tuning, for example adapting BERT to sentiment classification, is classical transfer learning.
- Instruction tuning, also called supervised fine-tuning (SFT), trains a model on examples of instructions paired with good responses, so that it behaves as a helpful assistant.
- Parameter-efficient fine-tuning methods such as LoRA (Hu et al., 2021) train only a small number of additional parameters, greatly reducing memory and compute requirements.
Reinforcement learning from human feedback¶
Reinforcement learning from human feedback (RLHF) aligns a model's behavior with human preferences. Christiano et al. (2017) showed that deep reinforcement learning agents can be trained from human preference comparisons; Ouyang et al. (2022) applied the method to large language models with InstructGPT. ChatGPT (November 2022) was trained in the same way and was described by OpenAI as a sibling model of InstructGPT. A typical pipeline is:
- The model generates several responses to each prompt.
- Human annotators rank the responses by quality.
- A reward model is trained to predict these rankings.
- The language model is optimized with reinforcement learning, for example with proximal policy optimization (PPO), to achieve high reward while a penalty keeps it close to the original model.
Direct preference optimization (DPO; Rafailov et al., 2023) is a simpler alternative that learns directly from preference pairs without training a separate reward model.
Logistic regression again
Reward models are usually trained with the Bradley–Terry model: the probability that response A is preferred over response B is \(\sigma(r_A - r_B)\), where \(r_A\) and \(r_B\) are the predicted rewards. This is a logistic regression on the difference of rewards.
Limitations¶
Hallucinations
Language models can produce fluent, confident text that is false, including invented facts, sources and quotations. They model which text is probable, not which text is true. Important claims should be checked; techniques such as retrieval-augmented generation can reduce the problem.
Other limitations include a fixed knowledge cutoff, sensitivity to the exact wording of the prompt, and systematic errors inherited from patterns in the training data.