Serving systems

What a serving engine does

A serving engine accepts messages, runs the model, and sends answers back. It decides which requests get GPU time and where to keep their saved information. It also tracks speed and limits how much work can enter the queue.

Choose an engine that supports your exact checkpoint, weight format, hardware, and required features. Then test it with your prompt lengths and request rate. A support label does not tell you whether that path is fast.

The engine is not a neutral wrapper. Its scheduler and cache policy change the latency and the quality of service. Two engines with the same model can behave very differently under load.

Continuous batching

Continuous batching adds and removes requests at each decode step. It keeps the batch full instead of waiting for a fixed group to finish. That raises throughput, because one weight read serves more tokens.

The cost is latency. A larger batch takes longer per step, and a new request can wait for a prefill slot. If the scheduler admits too much work, the cache fills and every request slows down.

Chunked prefill splits a long prompt into pieces. It lets decode steps interleave, so a long prompt does not block short chats. It also makes the time to first token less predictable, because the prompt competes with other work.

The curve shows one simulation of a queueing cliff. Its percentiles cover only requests completed within the 120-second window; rejected and still-pending requests are excluded. Below it, the server has spare capacity and p95 stays close to the service time. As arrivals approach service capacity, waiting time can rise sharply. If arrivals exceed capacity, a finite queue fills and requests are rejected or time out. The exact curve depends on traffic and scheduling.

KV cache management

A paged cache splits saved information into fixed-size blocks called pages. The server can give pages to requests, share them, or free them when they are no longer needed. This can reduce wasted memory.

Prefix caching reuses cache for a shared prefix. A system prompt or a long document can be cached once and shared by many requests. That cuts prefill time and memory traffic.

Sharing has a security cost. If two tenants share a prefix, the cache must not leak one tenant's data to the other. Use a salt or a separate cache namespace per tenant. Check that the engine supports isolation before you enable sharing across users.

Context copies are a related trick. The engine copies a suffix from the prompt or the cache and verifies it as a draft. That helps edits and repeated text. It is not a substitute for a trained draft head.

Parallelism

Tensor parallelism splits a layer across devices. Each device computes part of a matrix, and the devices exchange partial results. The link between devices becomes part of every layer.

Pipeline parallelism splits layers into stages. It reduces the link traffic per stage, but it adds a pipeline bubble and needs enough requests to fill the stages. Expert parallelism splits experts and uses an all-to-all exchange for routing.

Data parallelism runs full replicas behind a router. It can scale throughput across replicas. Shared caches or prefix-aware routing can improve reuse, but are not required for scaling. It does not reduce the memory per replica.

Admission and service targets

Set a maximum batch size, a maximum context length, and a queue limit. Reject early when the queue is full. A fast rejection is better than a slow timeout.

Track goodput: the number of requests that meet the latency target. Goodput is a better capacity measure than raw throughput. A server can report high tokens per second and still miss the target for most users.

Use priorities if some work matters more. A background summarizer should not block an interactive chat. Reserve capacity for the priority class, and cap the low-priority queue.

Choosing a stack

Benchmark on your own prompts, your own output lengths, and your own batch sizes. A vendor benchmark with a short prompt and a large batch does not predict a chat workload.