Measurement

A benchmark is a test of speed or quality. Repeating the test helps you tell a real improvement from a lucky run.

A brass balance and glass apparatus compare patterned tiles.

One fast run does not prove a change is faster. Repeat the test, record all the results, and check that the output still meets your needs.

What to measure

Before testing, decide what you want to learn. For example: “Does this change cut the wait for the first token by at least 5%?” Decide what to compare it with. Decide what result counts as a success.

Metric	Definition	Watch out for
TTFT	Request to first token	Prefill queueing, cache misses
ITL	Gap between output tokens	Batch changes, streaming buffering
Tokens/s	Tokens after the first divided by decode time	State whether the first token and prefill are included
p95 latency	A time that 95% of measured requests did not exceed	Sample size, bursty arrivals
Goodput	Requests that meet the target	Target definition, reject policy
Memory	Weights, cache, peak use	Allocator overhead, fragmentation

Measure correctness at the same time. For greedy decoding, compare token hashes. For sampled decoding, a fixed seed helps diagnose changes but does not prove distributional equivalence. Compare distributions or task outcomes when exact output is not expected. Decide which differences are acceptable before measuring speed.

Sources of noise

Run-to-run variance comes from clocks, temperature, background processes, cache warmth, memory placement, and the order of the runs. A first run can be slower because the model is not loaded or the cache is cold.

The workload adds more variance. Prompt length, output length, batch size, sampler, and stop conditions all change the result. A benchmark with a short prompt and a fixed output cap does not predict a chat workload.

Software versions matter. A kernel change, a driver update, or a different quantization can move the result by more than the feature you are testing. Record the image digest, the model hash, and the engine version.

Design the comparison

Write the question and the decision rule.
Fix the model, the tokenizer, the prompt set, the output length, and the sampler.
Warm up the model and the cache.
Run the baseline and the candidate in an interleaved order, not all of one then all of the other.
Start with at least five repeats per condition; add runs when the interval is too wide for the decision. For p95, collect enough request observations per condition to estimate the tail.
Record every run, not only the best one.
Check correctness before you compare speed.

Interleaving reduces drift. If the machine warms up during the session, both conditions see the same drift. Randomize the order if you can. Do not run the candidate only after the baseline.

The bars show a 95% interval for each mean. Their overlap alone does not decide whether a difference is resolved. The verdict uses the interval for the difference. Check whether it includes zero and whether the plausible differences matter for your decision.

Statistics without ceremony

Report the median and the full range. The mean is sensitive to one slow run. If you report a mean, report the standard deviation and the number of runs.

A confidence interval is a range calculated from the observations. It shows how uncertain an estimate is under the chosen method and assumptions. With few runs, the range is often wide, so a small difference may be hard to distinguish from noise.

Use a two-sample test only when the design supports it. When each baseline run is deliberately matched to a nearby candidate run under comparable conditions, analyze the differences within those pairs; this can reduce variance. Independent runs need a two-sample method. Either way, the effect size matters more than the p value.

Do not test many metrics and report only the one that passed. Pick the primary metric in advance. Treat the rest as exploration, and say so.

Correctness and quality

For a speed change, check that the output is the same under greedy decoding. Use a token hash over the full output, not a string comparison after detokenization. Detokenization can hide a token difference.

For a quality change, use a held-out set and a task metric. Perplexity is a fit measure, not a task measure. An LLM judge can help, but it has biases and needs its own validation.

Run the quality check at the same context length and sampler as production. A quantized model can pass a short test and fail a long extraction. Report the sample size and the confidence interval.

Reported local example. Evidence and limits

A kernel sweep compared a control with three variants. The control took 13.2589 s at 32K prefill; one variant took 13.2587 s, a difference of 0.0002 s. A third took 13.8188 s, about 4% slower. All runs produced the same token hash, and the team kept the control. The near-zero difference is not evidence of a useful win without repeated measurements.

How to report

Write down the command, the image digest, the model hash, the prompt set, the output length, the sampler, and the number of runs. Put the raw runs in a file next to the report. State what you did not test.

Example report. Question: does the candidate reduce median TTFT by at least 5%? Workload: a fixed prompt set and output cap. Record the model and engine versions, hardware, cache state, paired run values, request count, and the interval for the difference. Decision: keep the candidate only if the quality check passes and the evidence supports the target. State any untested workloads.

Separate the result from the interpretation. "The candidate ran at 13.26 s" is a result. "The candidate is faster" is an interpretation. Keep the two apart, especially when the difference is small.