Tokens and context

What a token is

A token is a piece of text. A tool called a tokenizer splits text into pieces and gives each piece a number. A common word might be one token. A longer or less common word might be several. The exact split depends on the tokenizer.

Most modern tokenizers use byte-pair encoding (BPE) or a close variant. Byte-level BPE starts with bytes and learns frequent pairs to merge; other tokenizers use different starting units or rules. The result is a vocabulary of tens of thousands of pieces. The model learns a list of numbers, called an embedding, for each token ID.

Token counts drive cost. Providers bill per token. A context window counts tokens. A prompt of 1,000 English words is about 1,300 to 1,600 tokens. Code, numbers, and non-Latin scripts usually use more tokens per character.

Use the tokenizer that ships with your model when you estimate cost or fit a prompt into a window.

The context window

The context window is the amount of text a model can use at once, measured in tokens. Think of it as a limited workspace. Your message, earlier messages, instructions, and the reply all need room. If the text is too long, the server may cut some of it off or reject the request.

Long context has two costs. The KV cache grows in a straight line with the number of tokens, so memory use grows. The attention step grows faster than that during prefill, so the time to the first token grows.

A large window does not guarantee good recall. Models can miss facts in the middle of a long prompt. Position, formatting, and the question all change the result. Test recall on your own documents before you trust a large window.

Why token counts matter

• Billing. Providers charge for input tokens and output tokens, often at different rates.
• Latency. A long prompt raises the time to first token. A long reply raises the total time.
• Memory. A long conversation can use more memory for its cache than for the model’s weights.
• Truncation. A request that exceeds the window loses the oldest tokens, or fails.
• Evaluation. A benchmark that gives models different token budgets is not a fair comparison.

When you plan a workload, count tokens with the real tokenizer. Then add the system prompt and the tool schemas. Then reserve room for the reply. The sum must fit inside the window with a margin.

Chat templates and special tokens

Instruction-tuned models expect a chat template. The template adds special tokens that mark roles, such as system, user, and assistant. It may also add tokens for tools or for the end of a turn.

Use the template that came with the model. A mismatched template can turn a strong model into a weak one. The template may render special token IDs as visible strings for inspection. Other role text can be ordinary tokens. Wrong markers change the pattern the model learned during training.

For example, a template might represent a message as [user] What is a token? [end], followed by an assistant marker. These labels are illustrative; use the actual template and token IDs for the model.

Tool calls use their own tokens and their own format. Read the model card before you build a parser. A program that expects one format can fail if the model uses another.

Common mistakes

• Estimating tokens from word count. English averages about 1.3 to 1.6 tokens per word, but the range is wide.
• Treating the context window as free memory. Every token adds KV cache and prefill work.
• Forgetting the system prompt and tool schemas in the budget.
• Using a long context when a short, focused prompt works better.
• Assuming that a model with a large window will find every fact in it.