Glossary

Type to filter by term, definition, or tag. The list works offline.

Acceptance rate Serving

The share of proposed draft tokens that the main model accepts. It affects how much speculative decoding speeds things up. Read more

Activation Foundations

A set of numbers produced by a model layer for a particular input. Read more

Admission control Serving

A way for a server to limit incoming work. It may make requests wait or turn them away when it is full. Read more

Attention Foundations

A way for a model to use information from other tokens when it processes a token. Read more

Autoregressive Foundations

Describes a model that generates an output one token at a time, using earlier tokens to choose the next one. Read more

Batch Serving

A group of requests or examples processed together. A bigger batch can improve total throughput but increase the wait for each request. Read more

BF16 Serving

A 16-bit floating-point format with a wide exponent. It is common for training and for unquantized serving. Read more

Bits per weight Compression

How much space one model weight uses, including its share of any scales and metadata. Read more

Byte-pair encoding (BPE) Foundations

A tokenizer method that repeatedly merges frequent pairs of pieces. Byte-level versions start from bytes. The learned pieces form a fixed vocabulary. Read more

Calibration Compression

Using sample data to choose settings for quantization. The sample should resemble the work the model will do. Read more

Capacity factor Serving

A setting used by some MoE implementations to limit assignments to an expert. Overflow handling varies. Read more

Catastrophic forgetting Training

The loss of earlier skills after training on a narrow dataset. Read more

Causal mask Foundations

A rule that stops a token from using information from later tokens when predicting the next token. Read more

Checkpoint Foundations

A saved copy of a model’s learned weights. It may also include the settings and other files needed to load it. Read more

Chunked prefill Serving

Splitting a long prompt into pieces so a server can work on other requests between pieces. Read more

Classifier-free guidance (CFG) Media

A diffusion technique that pushes the result toward the prompt. A high scale can oversaturate the output. Read more

Coalescing Serving

Arranging threads so neighboring threads read neighboring addresses. Poor coalescing wastes memory sectors. Read more

Contamination Measurement

Test data that leaked into training. It can make a benchmark score look better than the model's real skill. Read more

Context copies Serving

A draft method that copies a suffix from the prompt or cache and verifies it. It helps edits and repeated text. Read more

Context window Foundations

The amount of text a model can use at once, measured in tokens. The prompt and the reply both need space. Read more

Continuous batching Serving

Adding and removing requests as the model runs, instead of waiting for a whole fixed group to finish. Read more

CUDA Serving

NVIDIA's programming model for GPU kernels. Kernels run as grids of thread blocks. Read more

Decode Serving

The part of a request where a model writes its answer, usually one token at a time. Read more

Diffusion Media

A way to make images or video by starting with noise and gradually changing it into a result. Read more

Distillation Training

Training a smaller or faster model using another model as a teacher. Read more

DPO Training

Direct preference optimization. It trains on pairs of preferred and rejected answers with a simpler loss than RLHF. Read more

Draft token Serving

A guessed next token that the main model checks before using it. Read more

Embedding Foundations

A list of numbers the model uses to represent a token. Read more

EXL3 Compression

A trellis-coded weight format with a configurable bit budget. It needs compatible ExLlamaV3 kernels. Read more

Expert Serving

A learned part of a mixture-of-experts model. A router chooses which experts work on each token. Read more

Expert parallelism Serving

Splitting experts across devices. Tokens move over an all-to-all link to reach their expert. Read more

Fine-tuning Training

Extra training that changes an already trained model for a particular job or behavior. Read more

FP16 Serving

A 16-bit floating-point format. It is common for weights, activations, and KV caches. Read more

FP8 Compression

A family of 8-bit floating-point formats. The value payload is half the size of FP16; scales and other metadata add storage. Read more

GGUF Compression

A model file format used by llama.cpp. It stores quantized tensors and metadata for local serving. Read more

Goodput Measurement

The number of requests that finish within the chosen service target. Define how rejected requests count too. Read more

Gradient Training

Numbers that tell the optimizer how the loss changes when it changes each weight. They help it choose which way to move. Read more

Grouped-query attention (GQA) Foundations

An attention design in which several query heads share key and value heads. This can make the KV cache smaller. Read more

Hallucination Foundations

An answer that sounds convincing but is wrong or not supported by the available information. Read more

Held-out set Measurement

Examples kept separate from training and tuning, used to test how well a model handles new inputs. Read more

Inference Serving

Using a trained model to process an input and produce an output. Read more

Inter-token latency (ITL) Serving

The time between output tokens in a streamed answer. Read more

Kernel Serving

A small program that runs on a GPU. Kernels perform tasks such as matrix multiplication or attention. Read more

Kernel fusion Serving

Combining several operations into one kernel so intermediate tensors stay on chip instead of going to global memory. Read more

KV cache Serving

Saved keys and values from earlier tokens. The model reuses them when it writes new tokens, which saves work but uses memory. Read more

KV paging Serving

Storing the cache in fixed-size pages so the scheduler can allocate, share, and evict it without copying a whole sequence. Read more

Latency Serving

The time an operation or request takes, measured from a defined start to a defined end. Read more

Load-balancing loss Training

A training term that discourages the router from sending most tokens to a few experts. Read more

Logits Foundations

Raw scores for each vocabulary token before softmax. Sampling turns them into a choice. Read more

LoRA Training

A fine-tuning method that keeps the original model weights fixed and trains a small set of additional weights. Read more

Loss Training

A number that measures how far a model’s prediction is from the desired result. Training tries to reduce it. Read more

Memory bandwidth Serving

The bytes a device can move per second. It can limit decode speed when memory traffic is the bottleneck. Read more

Memory wall Serving

A speed limit caused by how quickly a device can move data to and from memory. Read more

Mixture of experts (MoE) Serving

A model with many learned parts called experts. A router chooses a few experts to process each token. Read more

Multi-head latent attention (MLA) Foundations

A compressed attention cache design. It reduces the bytes per token compared with a standard KV cache. Read more

Multi-query attention (MQA) Foundations

An attention layout with one key and value head shared by all query heads. It uses less cache than otherwise identical multi-head attention. Read more

Multi-token prediction (MTP) Serving

A training approach that adds predictions for tokens beyond the next one. Some models use those predictions to draft tokens. Read more

MXFP4 / MXFP8 Compression

Block-scaled float formats with a shared scale for 32 values. They are an open standard with new kernel support. Read more

NVFP4 Compression

A 4-bit float format with FP8 block scales, used on Blackwell hardware. It stores about 4.5 bits per weight. Read more

Occupancy Serving

The share of a multiprocessor’s maximum warp capacity that is active. Registers and shared memory can limit it. Read more

Overfitting Training

When a model fits its training examples so closely that it does worse on new examples. Read more

Parameter Foundations

A learned number in a model. Parameters are often called weights. Read more

Perplexity Measurement

The exponent of cross-entropy on a text set. It measures fit, not task quality. Read more

Pipeline parallelism Serving

Splitting layers into stages across devices. It adds a pipeline bubble and needs enough requests to fill the stages. Read more

Prefill Serving

The stage where a model reads the prompt and saves information needed to write an answer. Read more

Prefix cache Serving

Saved information for the beginning of a prompt that can be reused when another request starts with the same text. Read more

Q4_K_M Compression

A GGUF 4-bit k-quant mix at about 4.5 bits per weight. Some tensors stay wider than others. Read more

QLoRA Training

LoRA on a 4-bit base model. It cuts the base memory by about four times. Read more

Quantization Compression

A way to store numbers using fewer bits. It can save memory, but the stored numbers may be less exact. Read more

Quantization-aware distillation (QAD) Compression

Training a low-bit student to match a high-precision teacher. It can recover quality at a training cost. Read more

Quantization-aware training (QAT) Compression

Simulating low precision during training so the model learns to tolerate the error. Read more

RLHF Training

Reinforcement learning from human feedback. It trains a policy against a reward model built from preference data. Read more

Roofline Serving

A model that compares compute and bandwidth limits against arithmetic intensity. Read more

Router Serving

The part of a mixture-of-experts model that chooses which experts work on each token. Read more

Sampling Foundations

Choosing the next token from the model’s probability distribution. Settings such as temperature affect the choice. Read more

Softmax Foundations

A function that turns a list of scores into probabilities that add up to one. Read more

Speculative decoding Serving

Guessing several tokens and having the main model check them together. It can speed up generation when enough guesses are useful. Read more

Step cache Media

Reusing features across diffusion steps to cut run time. It changes the output and needs a quality check. Read more

Supervised fine-tuning (SFT) Training

Training on input-output pairs, usually with the loss on the assistant tokens. Read more

Tensor core Serving

A GPU unit for fast matrix work. Kernels must feed it the right tile shapes and data types. Read more

Tensor parallelism Serving

Splitting a matrix across devices. It adds an all-reduce exchange to every layer. Read more

Throughput Serving

How much work a system completes in a given time, such as tokens or requests per second. Read more

Time to first token (TTFT) Serving

Time to first token: the wait from sending a request until the first piece of the answer arrives. Read more

Token Foundations

A piece of text that a tokenizer maps to a number. One word can be one token or several. Read more

Tokenizer Foundations

A tool that splits text into tokens and converts those tokens to numbers and back. Read more

Top-k sampling Foundations

Sampling from the k most likely tokens. It cuts the tail of the distribution. Read more

Top-p sampling Foundations

Sampling from the smallest set of tokens whose probability adds up to p. Read more

Transformer Foundations

A stack of attention and feed-forward layers. It is the base architecture for most language models. Read more

VAE Media

An encoder and decoder for latent media models. It compresses images or video before diffusion. Read more

Verification Serving

The target model's check of draft tokens. A correct rule keeps the output distribution unchanged. Read more

Warmup Measurement

An initial run that loads weights and warms caches before measurement. Read more

Weight Foundations

A learned number in a model. The model uses its weights to turn an input into a prediction. Read more

No term matches. Try a shorter search.