Inference

The request path

When you send a message, the server turns it into tokens, adds the model’s expected message format, and runs the model. It then turns the output tokens back into text. The model does its work in two phases:

1. Prefill. The model processes all prompt tokens in parallel. It builds the KV cache for those tokens.
2. Decode. The model produces one token, appends its key and value to the cache, and repeats. Each step reads the weights and the cache.

A stop condition ends the loop. It can be a special end token, a maximum token count, a stop string, or a client cancellation. The server also applies sampling settings, which change how it picks each token.

One request follows this timeline: arrival → queue → prefill → first token → decode steps → final token. TTFT includes queueing and prefill; inter-token latency describes the gaps after the first token.

Prefill

During prefill, the model reads the prompt and saves information it will need for the answer. Longer prompts usually take more work. The wait from sending the request to receiving the first token is called time to first token (TTFT). It includes any time spent waiting in a queue.

A long prompt can dominate the request. In a local GB10 run, a 32K-token prompt took 13.26 seconds to prefill, against 0.57 seconds for a 1K prompt. These are measured examples, not predictions for another workload. About the measurements.

Two methods reduce prefill cost. Prefix caching reuses cache for a shared prompt prefix. Chunked prefill splits a long prompt into pieces, so decode work can interleave and other requests can start sooner.

Decode

Decode produces one token per step. Each step reads the model weights and the KV cache, then writes a small amount of new state. The arithmetic per byte is low, so decode is usually limited by memory bandwidth, not by compute.

Decode speed depends on model size, precision, cache size, batch size, and the kernels. A dense model reads all of its weights for every token. A sparse model reads fewer. The memory wall page shows how to estimate the ceiling.

The timeline shows why users see a pause before the first token, then a stream. The second chart shows the trade. A larger batch raises aggregate throughput, but each request waits longer per token. At some batch size, the cache no longer fits, and the server must queue or reject work.

Batching and latency

Continuous batching lets the server add new requests to a running batch. It raises utilization, because one decode step can serve many requests. It also raises the time per step, because each step reads more cache and does more work.

Throughput and latency pull in opposite directions. Measure both. Report p50 and p95 latency, not only the mean. A server with high throughput can still miss a latency target for the slowest requests.

Admission control protects the target. When the cache is full, the server can queue, reject, or evict a request. A policy that ignores cache pressure will slow every request and can stall the queue.

Sampling and stop conditions

Sampling settings change the reply and its length. Sampling work can also affect per-token cost, especially for large vocabularies or complex constraints. Temperature flattens or sharpens the distribution. Top-p and top-k cut the tail. Greedy decoding picks the most likely token. Repetition penalties can discourage repeated tokens, but can also suppress useful repetition.

Set a maximum token count for every request. Set stop strings when the format has a clear end. Support cancellation, so a client can stop a long reply and free the cache slot.

Metrics that match the question