Fine-tuning

Full fine-tuning

A model has many learned numbers called weights. Full fine-tuning changes all of them. This gives the model more freedom to adapt. It also costs the most memory.

A training step needs the weights, the gradients, and the optimizer state. A common mixed-precision Adam estimate is about 16 bytes per parameter for weights, gradients, master weights, and optimizer state. The actual layout varies; activations and runtime overhead are extra.

A 27B model therefore needs about 432 GB for a full fine-tune. That is more than the weights alone. Most single-GPU work uses a smaller model, or a parameter-efficient method.

Full fine-tuning can also cause catastrophic forgetting. The model can lose skills that were not in the new data. A small learning rate and a mix of old and new data reduce the risk.

Parameter-efficient tuning

LoRA is a way to change a model while keeping its original weights fixed. It trains two small sets of numbers, called matrices, instead. The product of those matrices is added to a chosen weight. For a weight of shape d by k, the adapter has about r(d + k) parameters, where r is the rank. Rank controls capacity, not quality.

QLoRA keeps the base model in 4 bits and trains the adapter in higher precision. That cuts the weight payload to about a quarter of its 16-bit size, before extra storage costs. The adapter still needs optimizer state during training. At serving time, you can merge the adapter into the weights, or load it as a separate module.

Other methods exist: adapters, prefix tuning, and low-rank updates to other modules. They share the same idea. Train few parameters, keep the base model fixed, and evaluate the result on held-out data.

Look at the memory bars. Full fine-tuning needs the weights, gradients, and optimizer state. LoRA needs the frozen base plus a small adapter. That gap can make LoRA feasible on a device that cannot hold a full fine-tune.

Supervised fine-tuning

Supervised fine-tuning (SFT) trains on examples of an input and the desired answer. For a chat model, the pair is a conversation with a target reply. The loss usually applies only to the assistant tokens, so the model learns to produce the reply, not to repeat the prompt.

Data quality dominates. A few thousand clean, consistent examples can beat a large noisy set. Cover the formats you need: refusal, tool calls, long answers, short answers. Include hard cases, not only easy ones.

Watch for overfitting. Validation loss can fall while the model gets worse at everything else. Test on tasks that are not in the training set. Keep an untouched evaluation set.

Preference tuning

Preference tuning changes which of two answers the model prefers. One method, RLHF, uses a separate model to score answers and then trains the language model to improve that score. Another, DPO, learns directly from pairs of preferred and rejected answers. Both need good preference data and a clear definition of "better".

Preference tuning is not a substitute for SFT. It works best after the model already follows the format. It can also make the model reward-hack: it learns the labeler's biases, not the task.

What fine-tuning does not solve by itself

• Keep facts current. A model can memorize facts from fine-tuning data, but retrieval is cheaper and easier to update.
• Replace the tokenizer freely. Changing the vocabulary requires compatible embeddings and additional training; an ordinary fine-tune keeps the tokenizer.
• Make long context free. Extending a window can require positional changes and more training. Cache memory and attention work still grow.
• Guarantee safety. A model can still be wrong or rude after tuning. Keep output checks.

When you serve an adapter, check the engine support. Some engines merge the adapter into the weights. Others load it at request time and can serve several adapters. Measure the effect on throughput and on time to first token.