How-to recipes

Size a model for a device

Use this when you must decide whether a model fits, and how fast it can answer.

You need the model’s total and active parameter counts, its weight format, its cache details, and the length of text you want it to handle. You also need the device’s memory capacity and bandwidth. See the linked pages below if any of these terms are new.

1. Estimate space for the weights. Multiply the number of parameters by the bytes per weight. Better still, use the actual model file size. If you only have a rough estimate, add 10% for extra information and check the real size later.
2. Estimate space for the KV cache. Use 2 × layers × KV heads × head dimension × bytes per element × context tokens. The cache explanation shows an example.
3. Add space for the software and its temporary data. Start with 2 to 4 GiB as a rough guess. Actual use can be much larger, so measure the peak.
4. Add 20% extra space as a starting cushion for the operating system and other overhead. Adjust it after measuring.
5. Compare the total with the device memory. If it is over, use a smaller model, shorter context, or another device.
6. For a single request, estimate cache traffic per decode step for the full current context, not cache capacity for one stored token. Add it to active weight bytes. Divide usable bandwidth by that total for an optimistic rate; actual traffic and kernel costs vary.
7. Write the budget down: weights, cache, runtime, headroom, ceiling, and the target rate.

Illustrative example. A 27B dense model at 4 bits has 13.5 GB (about 12.6 GiB) of weight payload. Assume 10% overhead: about 13.8 GiB. At one byte per element, a 32,768-token cache takes 2 GiB for this model shape. Add 3 GiB for runtime and 20% headroom: about 22.6 GiB. On a 32 GiB device this estimate fits.

Actual peak memory and concurrency must be measured.

If bandwidth is 273 GB/s, each step reads 14.85 GB of weights plus 2.15 GB of cache. The optimistic batch-one estimate is about 16 tokens/s before other overhead. Use the actual achieved bandwidth for a more realistic estimate.

Check. The sum fits with headroom. The ceiling is above the target rate. The KV estimate uses the real model shape.

If it does not fit. Lower the bits, shorten the context, choose a model with fewer KV heads, or split the model across devices. Do not ignore the cache.

If it fits but is slow. Lower the active parameter count or the bits. Add batching or speculative decoding. Check the ceiling first.

Why: the memory wall, the KV cache, mixture of experts.

Choose a quantization

Use this when you want fewer bytes per weight, and you cannot accept a quality surprise.

You need a baseline checkpoint, a task check, your production prompt set, and the kernel support for each candidate format.

1. Write the task check first. Start with 20 to 50 representative prompts from your traffic as a screen, then use a larger held-out set sized for the decision. Record the baseline outputs and scores.
2. Pick two or three checkpoints that the serving engine supports. Prefer checkpoints that the model author qualified.
3. Run the task check at the production context length and sampler. Compare task scores and a predeclared quality threshold. Different quantizations can produce different tokens without a quality failure.
4. Measure time to first token and tokens per second at the batch sizes you serve.
5. Compare memory use and the rollback path. Keep the exact baseline image.
6. Keep a checkpoint that meets the quality, speed, and memory targets; lower bit width alone does not guarantee a better result. Record the result and the date.

Example decision. For an illustrative extraction task, require at least 95% field accuracy on a held-out set. Allow no more than a 1 percentage point drop from the baseline. Require a measured rate of at least 15 tokens/s. Choose thresholds and a sample size that suit the real risk before looking at candidate results.

Check. The candidate meets a predeclared task-quality threshold and the speed and memory targets. Test the rollback.

If quality fails. Try QAD or QAT, a higher bit width, mixed precision, or a higher-precision KV cache. Test each change on the same prompts.

If speed fails. Check kernel support for the format. A 4-bit format with a slow kernel can lose to an 8-bit format.

Why: quantization, kernels.

Benchmark a change

Use this when you need to decide whether a change is real.

You need a fixed prompt set, a fixed output length, a sampler, a warmup, and a way to record every run.

1. Write the question and decide what result would count as a win. For example: “Does the new version cut the 95th-percentile wait by at least 5%?”
2. Fix the model, tokenizer, prompt set, output length, sampler, and stop conditions.
3. Warm up the model. Match the cache state to the test: measure cold prompts and reused prefixes separately if both matter.
4. Alternate between the original and the changed version. This helps both versions experience similar conditions.
5. Start with at least five repeats per condition; use more when the uncertainty is too large. Record every run, including the slow ones.
6. Check correctness before speed. Use a token hash for greedy decoding.
7. Report the median, the full range, sample count, and an interval for the difference in the chosen metric. For p95, collect enough requests per condition to estimate the tail; five runs alone are not enough. State what you did not test.

Check. Both versions used the same settings. The tests alternated. The answers still meet your needs, and the measured difference supports your decision.

If the result varies a lot. Run more tests. Reduce other work on the machine. Run the two versions close together in time, and report all results.

Why: measurement.

Set serving limits

Use this when you put a model behind an API and need a latency target.

You need a load test, a latency target, the cache size per request, and a policy for rejects.

1. Measure time to first token, inter-token latency, and p95 at the target load. Use the real prompt and output lengths.
2. Set a maximum context length. Reserve room for the reply and the system prompt.
3. Set a maximum batch size that keeps p95 inside the target.
4. Set a queue limit. Reject early when the queue is full.
5. Isolate the prefix cache between tenants. Use a salt or a separate namespace.
6. Load test below, at, and above the target rate. Include bursts and long requests. Watch p95 and the reject rate.
7. Track goodput and cache use. Review the limits when the traffic changes.

Example target. For an illustrative chat service, require 95% of accepted requests at 10 requests per minute to reach their first token within 3 seconds. Keep rejects below 1%. State separately how rejected and timed-out requests count toward goodput.

Check. At the target arrival rate, the chosen p95 latency stays below the stated limit with enough requests to estimate it. Rejects are explicit. The cache is isolated.

If the server overloads. Reduce the admitted request rate or context limit. Tune the batch cap against both throughput and latency; lowering it can reduce capacity. Reject earlier or add capacity. Do not let the queue grow without bound.

Why: serving systems, inference.