Training
Training changes the model’s numbers so it makes better predictions. A model can improve on its practice data and still do poorly on new data.
Training helps a model do well on examples it has seen. Test it on new examples too. A good score on practice data does not guarantee a good score on new data.
What training does
Scroll sideways to see the full diagram.
Many language models learn by guessing the next token in text. For example, after “The sky is,” the next token might be “blue.” The model compares its guess with the actual next token. A number called loss measures how far off the prediction was. A program called an optimizer changes the model’s learned numbers, or weights, to reduce that loss.
The loss for next-token prediction is usually cross-entropy. It is low when the model gives high probability to the correct token. It is high when the model is confident and wrong. The average loss over a dataset is a measure of fit, not of truth. For example, if the correct next token is 'rain', assigning it 80% probability gives a lower loss than assigning it 10%.
Training uses some examples for practice and keeps others separate for testing. A training run has these parts:
- Data. Text, code, or other sequences, split into training and validation sets.
- Batches. Small groups of sequences processed together.
- Forward pass. The model computes predictions and loss.
- Backward pass. The model works backward to find how each weight affected the error. This step is called backpropagation.
- Optimizer step. The optimizer updates the weights, usually with Adam or a variant.
Pretraining uses a very large corpus. Post-training then changes the model to follow instructions or to prefer certain answers. Both stages use gradients. The scale and the data differ.
Gradients and step size
Imagine walking downhill to find a low point. The gradient tells the optimizer which way is uphill. The optimizer usually moves the weights the other way, toward lower loss. The learning rate controls the size of that move.
η is the learning rate; ∇L is the gradient.
A high learning rate can jump over a good region, or make the loss grow without bound. A low learning rate can take too long, or stop in a poor region. Real runs use schedules: a warmup, then a decay. They also clip gradients, because one large gradient can damage the weights.
Momentum and Adam keep state from earlier steps. Momentum carries speed through flat regions. Adam scales each update by the recent gradient size. That helps in high dimensions, but it does not remove the need to tune the rate.
Watch three things. First, a high rate can overshoot the valley and climb the other side. Second, momentum can carry the point past a minimum. Third, noise changes the path, but it does not fix a rate that is far too high.
Overfitting and data
If a model sees the same data many times, it can memorize the training set. Training loss falls, but validation loss stops falling or rises. That gap is overfitting. Regularization, dropout, weight decay, and early stopping reduce it. More or better data can also help; the result depends on the task and training run.
Clean examples are important when training a model further. Duplicate examples, label errors, and leaked test data all distort the result. A benchmark score can rise because the model saw the benchmark during training. That is contamination, not skill.
For language models, generalization is hard to measure. Perplexity on held-out text measures fit. Task accuracy measures behavior. Neither one covers safety, factuality, or robustness. Use several evaluations, and keep a held-out set that you never tune on.
What training does not give you
Training does not store a database of facts. It adjusts weights so that the model predicts text that looks like its data. A fact can be approximate, outdated, or missing. A fluent answer can still be wrong.
Training also does not guarantee that the model follows instructions. Instruction tuning and preference tuning are separate steps. They are covered on the fine-tuning page.