Training
But where did all these weights come from?
Weights are learned by comparing the model's predictions with the real target and gradually adjusting.
In more detail
The main goal in pretraining a language model is to predict the next token. The model sees a large amount of text. For each position, it receives the text before it and outputs a probability for the true next token. The loss (cross-entropy) is −log p(true token): small when the model gave the right token a high probability, large when it gave it a low one.
The loss is averaged over many positions. Backpropagation computes how the loss would change if each parameter changed slightly (the gradient), and an optimizer changes every parameter a little in the direction that lowers the loss. This is repeated for an enormous number of steps.
Notice that nothing in this objective says "be truthful" or "be helpful". A pretrained base model is a text continuer. Assistants such as ChatGPT go through further stages (section 20).
Training is the expensive part. Inference, the use of a trained model, only runs the forward pass with fixed weights. The model does not learn from your chat by changing its weights.
Simplification: the visualization updates a single weight to make the idea clear. Real training updates billions of parameters at once.