Mixed review
Twenty questions across the guide. Answers are hidden.
Mixed questions
1. A prompt has 10,000 English words. Is it 10,000 tokens?
No. English text is usually about 1.3 to 1.6 tokens per word, so expect roughly 13,000 to 16,000 tokens. Code, numbers, and non-Latin scripts use more. Count with the model's own tokenizer before you budget.
2. A model has 32 layers, 8 KV heads, a head dim of 128, and a 1-byte cache. How big is the cache at 32K tokens?
2 × 32 × 8 × 128 × 1 × 32,768 = 2 GiB. The formula is 2 × layers × KV heads × head dim × bytes × tokens. At 1M tokens the same cache is 64 GiB.
3. Why does a language model need a causal mask?
Training predicts the next token. Without a mask, a position could read the token it must predict, which makes the task trivial. The mask allows only the current position and earlier ones.
4. Why can a high learning rate make the loss worse?
The step can overshoot the valley and climb the other side. A very high rate can make the loss grow without bound. Warmup and decay reduce the risk.
5. Rank 16 LoRA on the query and value matrices of a 27B model: is the trainable fraction above or below 1%?
Below 1%. For 64 layers and a hidden size of 5,120, the adapter is about 21M parameters, which is roughly 0.08% of 27B. The exact count depends on the model shapes.
6. What is the difference between time to first token and tokens per second?
Time to first token measures prefill and queueing. Tokens per second measures decode. A server can have a fast decode rate and a slow first token, or the reverse.
7. A 27B dense model at 8 bits runs on a device with 273 GB/s. What is the decode ceiling?
About 10 tokens per second. The weights are 27 GB, and 273 GB/s ÷ 27 GB ≈ 10.1 tokens/s. This weight-only estimate excludes cache traffic and other overhead.
8. Why does a 4-bit model not always run twice as fast as an 8-bit model?
Scale overhead, cache traffic, kernel support, and fixed per-step costs limit the gain. A format without a fast kernel can even be slower. Measure the exact checkpoint.
9. Where does kernel fusion save the most traffic: batch 1 or batch 4,096?
At batch 4,096. At batch 1, the weights dominate and the intermediates are small. At a large batch, the intermediates are large, so removing their global writes and reads saves most of the traffic.
10. Why does a strided read waste bandwidth?
In the guide’s NVIDIA-style example, the GPU fetches 32-byte sectors. With a large stride, each thread touches a different sector, so most of each sector is unused. Coalesced reads put neighboring threads on neighboring addresses.
11. A sparse model has 120B total parameters and 30B active. What must be available to the server?
If all weights are kept resident, all 120B parameters need space. Offload or sharding can allow a model to run when they do not fit on one device, with extra costs. Active parameters help estimate compute and traffic, but do not determine speed alone.
12. Draft depth 4, acceptance 0.5, draft cost 0.1. What is the approximate speedup?
About 1.4×. Expected accepted tokens are 0.5 + 0.25 + 0.125 + 0.0625 = 0.9375. One round yields about 1.94 tokens for 1.4 target-pass times, so the speedup is about 1.38×. It is not 2×.
13. Why is 95% utilization a risky target?
Arrivals are uneven. Near full utilization, a burst fills the queue and the queue drains slowly. p95 latency rises sharply, even when the mean looks fine.
14. A candidate is 3% faster in three runs. Is it faster?
Not proven. The conclusion depends on the spread of the measurements. Three runs can leave substantial uncertainty. Interleave the runs, repeat, and report the spread. Check the token hash before you compare speed.
15. Why can a higher video resolution cost more than the pixel increase suggests?
More pixels add tokens. Full attention can grow faster than token count, but actual scaling depends on the model and processing path. Measure detail at the delivery size.
16. What does a prefix cache reuse, and what must it protect?
It reuses KV cache for a shared prompt prefix, which cuts prefill time. It must isolate tenants, or one user's cached tokens can affect another user's request.
17. What does quantization-aware distillation add?
It trains a low-bit student to match a high-precision teacher. That can recover quality at 4 bits, but it costs training compute and still needs a final quality check.
18. Why must a speculative verifier use a rejection rule?
A naive rule changes the output distribution. A correct rejection rule accepts a draft only when it passes a test against the target distribution, and samples a replacement when it fails. That keeps the output equivalent to the target model.
19. What is the difference between distillation and step caching?
Distillation trains a student to reach a good result in fewer steps. Step caching reuses features across steps without retraining. Both cut time, and both can change the output. Test them on your own prompts.
20. A model fits in memory but answers too slowly. What are the first levers?
Lower the bytes per token: fewer active parameters, lower precision, or a smaller cache. Then use batching or speculative decoding to get more tokens per weight read. Check the bandwidth ceiling before you tune kernels.
See About the numbers for the limits of the measured examples.