Tech AI News - 테카이

KV cache: why generating tokens one by one doesn't get slower each time

· 7 min read

This post was translated from the Korean original by AI.한국어 원문 읽기 →

It's easy to assume that "generating tokens one at a time means it gets slower as you go." In reality, attention computes the key and value matrices for past tokens once, stores them in a cache, and reuses them — so the compute cost of producing a single token stays roughly constant. The catch is that this cache isn't free. As context grows and concurrent users increase, it's this cache — not the model weights — that eats up GPU memory faster.


Background — why this term now

As more models push context windows to hundreds of thousands or even a million tokens, discussion of GPU memory is shifting from "how many GB is the model" to "how much more piles up during inference." The official Hugging Face Transformers documentation explains the background of this shift with a simple analogy: just as it would be inefficient if, every time you spoke, the other person forgot everything said before and had to hear it all again from scratch, an autoregressive model (one that feeds its previous output back in as input, one token at a time) also slows down if it recomputes the entire context at every step.

The KV cache (key-value cache) is the mechanism that eliminates this inefficiency: the attention layer stores the key-value pairs of tokens it has already processed, then pulls them back out when generating the next token. The NVIDIA developer blog (December 2025) explains that for servers handling long contexts and large numbers of concurrent requests, the size of this cache itself becomes the key variable in GPU memory design.


Key data & current state

MetricValueSource · date
LLaMA2-7B model weight memory (fp16)~14GB"Efficient LLM Inference with Kcache" paper, April 2024
Same model, KV cache memory at batch 8, sequence length 32K (32,768 tokens)~128GBSame paper, April 2024
KV cache per token for Qwen3-8B (36 layers, 8 KV heads, head dim 128, bf16)~144KiBSebastian Raschka FAQ
Cache for one 32,768-token sequence under the above conditions~4.5GiBSame FAQ
Total cache for 8 concurrent requests under the same conditions~36GiBSame FAQ
GPU memory waste from pre-allocation (reserving for max sequence length)~60%–80%PagedAttention paper (SOSP 2023, arXiv 2309.06180)
Memory waste after applying PagedAttentionUnder 4%Same paper
KV cache savings from NVFP4 quantization (vs. FP8)Up to ~50%NVIDIA developer blog, December 2025

Interpretation: Model weights are fixed — the server loads them into memory once, and every request shares them. The KV cache, by contrast, accumulates separately for each in-flight request, so it grows directly with context length and the number of concurrent requests. In the LLaMA2-7B example, the cache (128GB) being over 9x larger than the weights (14GB) is simply a result of the batch size and sequence length used.

Deep dive

1) What exactly is the KV cache

Attention compares each token's query (Q) against the keys (K) of every token so far to get weights, then uses those weights to sum up values (V) and build the next token's representation. Under causal masking (the constraint that blocks a token from seeing tokens that come after it), no token's K and V ever change because of a token that appears later. This is the key point the official Hugging Face documentation makes: once a past token's K and V are computed, they stay that value forever, so there's no reason to recompute them.

So when generating a new token, only that token's K and V are computed and appended to the cache, and attention uses "the past K and V already in the cache, plus the K and V just computed." Thanks to this structure, the per-token attention cost grows only linearly as the context lengthens. Without the cache, you'd have to recompute the entire sequence at every step, which would make the cost grow quadratically.

2) A common misunderstanding — why it reads this way

The most common misunderstanding is thinking that "since the cache keeps things from slowing down, it must also save memory." In reality, it's closer to the opposite. The KV cache is a technique that trades computation for storage, so the longer the sequence, the more K and V values pile up. It preserves speed, but keeps consuming more memory.

The second misunderstanding is "shouldn't the query (Q) be cached too?" Borrowing an explanation from the Hugging Face forums, the query is a value used once — "for that token to look at the rest" — and then discarded. Generating the next token only needs the query of that new token; the queries of previous tokens were already fully used in their own attention computation and have no further use. Keys and values, on the other hand, need to be kept around because every later token's query has to keep comparing against them. The required shape differs too — at every step, what's needed is the keys and values of all past tokens, but for the query, just the most recent token's is enough.

3) What actually matters when judging it

To judge the claim that "the KV cache becomes a bigger constraint than the weights," you need to look at how differently the two grow. Weights have a fixed size per model and are done once loaded. The KV cache grows by multiplying a "per-token cost" — number of layers × number of KV heads × head dimension × precision (bytes) — by the sequence length and the number of concurrent requests. As Raschka explains, even if a single token costs only about 144KiB under Qwen3-8B's configuration, the cache alone reaches 36GiB once eight 30,000-token conversations come in at the same time.

Because of this multiplicative structure, past a certain point the cache overtakes the weights. The LLaMA2-7B example (14GB of weights, 128GB of cache at batch 8 and 32K tokens) illustrates that crossover. Analysis: exactly when this flip happens depends on the model's architecture (layers, heads) and serving conditions (batch size, context length), so there's no general rule for "after how many tokens" — it has to be recalculated every time.

4) What approaches to easing memory pressure actually target

Several techniques exist, each aimed at shrinking one term of that multiplication. GQA (grouped-query attention, which keeps the query heads but reduces and shares the number of key/value heads) reduces the "number of KV heads" term. Low-precision quantization methods like NVFP4 target the "precision (bytes)" term — NVIDIA explains that compared to FP8, it cuts the cache by up to about 50%, letting roughly 2x more context fit in the same memory.

PagedAttention tackles a different axis. Rather than shrinking the multiplied values themselves, it eliminates the waste (60%–80%) that came from pre-reserving memory for the maximum possible sequence length — since you don't know in advance how long a sequence will end up — and instead allocates only what's needed, in fixed-size blocks. As a result, waste drops to under 4%, and requests sharing the same prefix can even share blocks. All three techniques attack the same problem — "the KV cache is expensive" — from different angles: head count, cost per byte, and allocation efficiency.

In practice — 5 principles for reading the KV cache correctly

  1. "No need to recompute every time" applies only to keys and values. The query at each step is used once, for the most recent token, and then discarded, so there's no reason to cache it.
  2. The KV cache isn't a memory-saving technique — it's a technique that trades computation for memory. Per-token compute gets lighter, but the longer the context, the more has to be stored.
  3. Weights are loaded once per model and shared by every request, but the KV cache is created separately for each in-flight request. So as concurrent users grow or context lengthens, the cache eats GPU memory faster than the weights do.
  4. Per-token cache size is set by the number of layers, number of KV heads, head dimension, and precision. Techniques that shrink the cache, like GQA or quantization, ultimately target one of these terms.
  5. When you see an announcement about "supporting up to X tokens of context," also ask "how many users can use that length at once." The number of concurrent requests the cache memory allows is what actually determines perceived performance.

References

  • #how it works
  • #kv cache
  • #attention
  • #llm inference
  • #gpu memory
  • #pagedattention
How It Works

Attention — How AI Picks Out 'What Matters' in a Sentence

Attention is the mechanism by which AI understands long text not by remembering every word equally, but by concentrating on the few words it needs right now. This article explains why cost grows as context gets longer, and how optimizations like GQA and MLA affect real-world response speed and pricing.

· 6 min