From Predictor to Assistant

A model that has only been pretrained continues text. It does not follow instructions reliably. Assistants are usually built in several stages:

1. 1. Pretraining: predicting the next token on a very large collection of text. This gives the model broad knowledge of language and the world, and also its mistakes and biases.
2. 2. Supervised fine-tuning: further training on examples of good responses to instructions or questions, usually written or curated by people.
3. 3. Preference training: people (or models guided by rules) compare alternative answers, and the model is trained to produce the preferred kind. One approach is reinforcement learning from human feedback (RLHF): a reward model is learned from the comparisons and then used to improve the assistant (Ouyang et al., 2022). Direct preference optimization (DPO) is a simpler alternative (Rafailov et al., 2023).
4. 4. Deployment: the application adds instructions (system prompts), safety measures and sometimes tools such as search.

The exact methods vary between companies and are not always published. These stages change behavior, such as tone, helpfulness and refusals, but they do not turn the model into a database of verified facts (section 21).