KV Cache (Transformer Inference)
An in-memory store of key and value tensors from prior tokens used during transformer decoding to avoid recomputing attention over the full sequence at every step.
The KV cache is a memory structure used during autoregressive decoding in transformer language models. When generating text one token at a time, each new token requires attention against every prior token. Without caching, the model would recompute key and value projections for the entire prefix at every step, making generation quadratic in sequence length. Because the key and value projections of a token do not change once that token is fixed, they can be computed once and stored. The KV cache holds these tensors so that generating token N only requires fresh computation for the new query vector, while the stored keys and values are reused. This converts the per-step cost from O(N) attention computation to O(1) for the cached portion plus the new token's contribution, at the cost of additional GPU memory. The KV cache is local to a single inference request and lives only as long as that request's GPU allocation. It is distinct from prompt caching, which reuses prior internal state across separate API requests with shared prefixes, and from semantic cache layers, which match whole responses by query similarity. Memory pressure from large KV caches has motivated techniques such as paged attention, multi-query attention, grouped-query attention, and various compression schemes for long-context serving.