Multi-token prediction

A model can try to write faster by guessing several tokens at once, then checking the guesses.

A glass and brass machine produces multiple tiles along parallel tracks.

Guessing ahead only helps if enough guesses are accepted to make up for the time spent guessing. Measure both the number accepted and the overall speed.

What MTP is

Multi-token prediction (MTP) trains extra parts of the model to guess tokens beyond the next one. These parts are called heads. They use information from the main model and can help it make a draft.

At serving time, the heads can draft a short chain of tokens. The main model then checks the chain in one forward pass. This is speculative decoding. The draft can come from MTP heads, a smaller draft model, or a prompt lookup that copies a suffix from the context.

The training benefit and the serving benefit are different. MTP can affect the representations learned during training; any quality benefit depends on the model and training setup. Speculative decoding uses the heads to cut the number of target passes. A model can have one without the other.

How verification works

Scroll sideways to see the full diagram.

The model keeps the accepted beginning, replaces the rejected token, and discards the rest of the draft.

The main model, called the target, checks the proposed tokens together. When sampling, it compares the draft’s probabilities with its own and applies an acceptance test. If a token fails, the target picks a replacement using a corrected probability rule. The later draft tokens are thrown away.

Done correctly, this keeps the output distribution identical to the target model. A naive rule such as "accept if the tokens match" does not. With greedy decoding, the target always chooses its most likely next token. It checks each guess against that choice and stops at the first mismatch.

For example, a draft proposes A, B, C. The verifier may accept A and B, reject C, and sample a replacement at that position. Later draft tokens are discarded. The actual acceptance test depends on the decoding method.

One target pass can check several drafts. The cache for the drafts must be kept or recomputed, and the verifier needs the draft tokens as input. The cost per round is therefore one target forward pass plus the draft cost, not one pass per token.

The curve shows the break-even point. Below it, the draft costs more than it saves. Above it, the speedup rises with acceptance. The gain also depends on the draft cost. A cheap MTP head can pay off at a lower acceptance rate than a separate draft model.

What acceptance depends on

Acceptance is the fraction of draft tokens that the target accepts. It depends on the task, the temperature, the draft depth, and the quality of the draft. Code completion and predictable text accept more often than open-ended reasoning. Temperature changes both distributions and can change acceptance in either direction; measure it for the chosen draft and target.

Acceptance often falls with depth, but the pattern depends on the draft method and workload. A confidence threshold can stop a chain early, before a weak draft wastes a verification slot. Serving stacks expose counters for drafted and accepted tokens, so you can measure the real rate.

Low acceptance is not neutral. The target still runs a full pass, and the draft work is wasted. A speculative path with poor acceptance can be slower than plain decode. Test it on your own traffic, not on a benchmark prompt.

Costs and limits

Draft compute. MTP heads add a small forward pass per draft. A separate draft model has its own compute and memory cost.
Verification memory. The draft tokens need cache slots, and the verifier reads them.
Batching. Speculative decoding helps a single stream most. In a large batch, the target is already busy, so the relative gain falls.
Distribution. The rejection rule must preserve the target distribution. A wrong rule changes the output.
Long context. The draft still needs the context, and the cache traffic grows with it.

Speculative decoding does not increase the hardware’s memory bandwidth. It increases the number of output tokens per target pass. If the draft is cheap and accurate, the effective tokens per byte rises. If not, the extra draft work can make generation slower.

MTP in a serving stack

Some checkpoints ship with MTP heads. The serving engine loads the heads beside the main weights and can cap the draft depth or stop a chain at a confidence threshold. The engine also counts drafted and accepted tokens, which gives the acceptance rate directly.

Prompt lookup is a cheaper draft source. It copies a suffix from the prompt or the cache and verifies it. It works well for edits, repeated text, and code. It does not need a trained head, but it only helps when the reply repeats the context.

How to measure it

Measure plain decode on a fixed prompt set, with a fixed sampler and output length.
Enable the draft path with the same settings.
Record accepted tokens, drafted tokens, effective tokens per second, and p95 latency.
Check that the output matches the baseline under greedy decoding, using a token hash.
Repeat at the batch sizes and context lengths you serve.