How a transformer works
A transformer is a kind of model. It reads tokens and uses earlier tokens to help predict what comes next.
When the model writes an answer, it saves information about earlier tokens. This saved information is called the KV cache. It saves work, but it uses memory.
The shape of the computation
Show the labeled diagram
Scroll sideways to see the full diagram.
A transformer has a stack of layers. Each layer changes a list of numbers for each token. This list is called a vector. An attention block lets a token use information from earlier tokens. Another block transforms the result. After the last layer, the model gives each possible next token a score.
For one position, a layer first mixes information from allowed earlier positions, then transforms the resulting vector. The next layer repeats that process with the updated vector.
The forward pass has these stages:
- Map each token id to an embedding vector.
- Run the vector through many layers. Each layer has an attention block and a feed-forward block.
- Project the final vector to a score for every vocabulary token.
- Turn the scores into a probability distribution, then sample one token.
The weights hold the learned numbers. A dense model uses all of its weights for every token. A mixture-of-experts model uses only some of them, which changes the cost. This page describes the dense case first.
Attention in plain words
Attention helps a token use information from other tokens. For example, the word “its” in “The cat ate its food” can use information about “cat”. The model makes three lists of numbers for each token: a query, a key, and a value. It compares queries with keys to make scores. A function called softmax turns the scores into proportions that add up to one. The result combines the values using those proportions.
With a causal mask, a position can read only itself and earlier positions.
Here Q, K, and V are the query, key, and value arrays; d is the size of a query or key vector. The mask removes scores for future positions before softmax.
A causal mask stops a token from reading future tokens. Without it, the model could see the answer during training. The mask makes the next-token prediction task honest.
Real models run many attention heads in parallel. Grouped-query attention (GQA) shares key and value heads across groups of query heads. That cuts the KV cache. Multi-head latent attention (MLA) compresses the cache in a different way.
The grid is small on purpose. A real model has thousands of positions and many heads. The rule is the same: a query at position t can read keys at positions 0 to t, and nothing later.
Why the KV cache exists
Scroll sideways to see the full diagram.
When the model writes an answer, it can reuse the keys and values it already worked out. The KV cache holds them. For each new token, the model reads the saved information and adds new keys and values. It does not need to work out the whole prompt again each time.
For standard full-attention layers with the same GQA shape and cache precision, the cache has a fixed size per stored token. Sliding windows, compressed caches, mixed layers, and allocation overhead change the total:
For a model with 32 layers, 8 KV heads, and a head dim of 128, the cache uses 64 KiB per token. That assumes one byte per element. At 32K tokens, that is 2 GiB. At 128K, it is 8 GiB. At 1M, it is 64 GiB. A 27B model at 4 bits needs about 12.6 GiB for its weights, so the cache can dominate.
Three levers reduce the cache: fewer KV heads (GQA), a smaller head dim, or fewer bytes per element (FP8 or INT8 cache). A fourth lever is to share cache between requests that have the same prefix. That is prefix caching.
What attention does not do
Attention is not a database. It mixes values with learned weights. A model can fail to recall a fact that is present in the context. This is more likely when the fact sits in the middle of a long prompt.
The attention matrix grows with the square of the sequence length. Full-attention work therefore grows faster than linear as the prompt grows; total prefill time also includes other work. Decode avoids that square cost for past tokens by using the cache, but it still reads the whole cache for every new token.
Position encoding, training data, and the instruction template all change how well a model uses context. A large window is a capacity limit, not a guarantee of recall.
Where MoE and MTP fit
A mixture-of-experts model replaces the feed-forward block with many expert blocks, and a router picks a few for each token. The attention block and the KV cache stay. The next page in the serving path is mixture of experts.
Multi-token prediction adds small heads that guess several future tokens. The main model then checks those guesses in one pass. That is speculative decoding, and it uses the same cache and kernels.