KV Cache: How LLMs Remember Context Efficiently
The key-value cache is the mechanism that lets LLMs process long conversations without recomputing everything from scratch on every token.
Key Takeaways
| Takeaway | Details |
|---|---|
| KV Cache Purpose | Stores key and value attention matrices to avoid recomputing attention over all previous tokens during generation. |
| Performance Improvement | Reduces generation complexity from O(n²) to O(n) work per token by reusing cached vectors. |
| Memory Requirements | Can require tens of gigabytes per request for large models, often exceeding model weight memory usage. |
| Compression Techniques | Quantized caches, sliding window attention, and grouped-query attention help reduce memory consumption. |
| Prefix Caching | Reuses cached KV representations across requests sharing common prefixes like system prompts or documents. |
| Cost Benefits | Cached prefix tokens are billed at up to 90% discount from regular input token prices. |
The Problem the KV Cache Solves
During autoregressive text generation, every new token must attend to all previous tokens. Without caching, generating the nth token would require recomputing attention over all n-1 previous tokens, O(n²) work for an n-token response. For long documents or multi-turn conversations, this would make generation prohibitively slow and expensive.
The key-value (KV) cache solves this by storing the key and value attention matrices for every token as they are computed. When generating a new token, the model reuses cached K and V vectors from all previous tokens instead of recomputing them. Only the new token's K/V vectors need to be computed, reducing generation to O(n) work per token.
Memory Tradeoffs
The KV cache dramatically speeds up generation but consumes substantial GPU memory. For a 70B parameter model with a 128K context window, the KV cache can require tens of gigabytes of memory per concurrent request, often more than the model weights themselves. This memory pressure is a primary constraint on how many simultaneous requests an inference server can handle.
Techniques like quantized KV caches (storing K/V in 4-bit or 8-bit instead of 16-bit), sliding window attention (only caching the most recent N tokens), and grouped-query attention (GQA, which reduces the number of K/V heads) all help compress KV cache memory. These tradeoffs directly affect how much context length is practical at a given cost.
Prefix Caching
Prefix caching (also called prompt caching) extends the KV cache idea across requests: if many queries share the same prefix (a system prompt, document, or code file), the provider computes and caches the KV representations for that prefix once, reusing them across all subsequent requests. This dramatically reduces cost and latency for high-prefix applications.
Anthropic, OpenAI, and Google all offer prompt caching as a feature. Tokens in the cached prefix are billed at a significant discount (up to 90% off input token prices). For applications with long, stable system prompts or shared context, enabling prompt caching is one of the highest-ROI optimizations available.
Read next
LLM Inference: How Models Generate Text
Inference is what happens when an LLM produces a response. Understanding it helps you optimize for speed, cost, and quality.
Context Windows Explained: The Working Memory of LLMs
What context windows are, why they matter for building AI applications, how they've grown from 4K to 10M tokens, and how to manage them effectively.
The Attention Mechanism: How LLMs Understand Context
A clear explanation of self-attention, the mathematical operation at the heart of every transformer that allows language models to understand relationships between words.