The KV Cache Evolution

The Transformer Moment: Where This Starts

June 2017. Eight researchers at Google published a paper with a bold title: "Attention Is All You Need." Inside was the Transformer. At the time, it was a sequence-modeling architecture. Looking back, it also started the memory problem this article is about.

To see why it mattered, remember what came before. The dominant approach was the Recurrent Neural Network (RNN). Imagine passing a sticky note down a line of 500 people. Each person reads the previous note, compresses the whole story into their own short message, adds something new, and passes it along. By person 300, the first message is blurry. Worse, person 10 cannot even start until person 9 is finished. That was the shape of recurrence: useful, but painfully sequential.

The Transformer changed the setup. Instead of passing one note forward, it let every token look at every other token directly and decide what mattered. That is attention. There is no need to wait for the previous step to finish, and no single compressed hidden state has to carry the whole past. The cost is a large matrix operation, but that operation maps well to GPUs.

The benchmark numbers made the case. On WMT 2014 English-to-German, the Transformer reached 28.4 BLEU, more than 2 points above the previous best. That may sound small from the outside, but in machine translation those points were hard to win. Training also became much faster. The architecture improved accuracy and fit the hardware better.

The next decade built on that foundation: BERT, GPT-2, GPT-3, GPT-4, LLaMA, Gemma, Qwen, DeepSeek, and Mistral. More layers, more parameters, more data. For a while, the recipe kept working. Then users started asking models to read long codebases, research archives, legal documents, and multi-hour conversations. A quieter cost began to dominate. It was not the cost of the weights. It was the cost of remembering the context. That cost is the KV cache.

Queries, Keys, and Values: The Small Mechanism Behind the Big Cost

Before the cache makes sense, the objects inside it have to make sense. The Transformer's central operation, scaled dot-product attention, turns each token into three vectors: Query (Q), Key (K), and Value (V). The names come from database language, but a food court is an easier way to picture it.

Picture a food court with 200 stalls.

Inside the model, every token makes the same move. It sends out a Query, compares that Query against other tokens' Keys, and then blends the Values according to those scores. A token that looks relevant contributes more. A token that looks irrelevant fades into the background. The final representation is not copied from one place. It is mixed, weighted, and rebuilt from the full context. The operation is compact on paper. At scale, across every token and every layer, it becomes expensive.

The softmax turns raw scores into weights that sum to 100%. The √d_k term, the square root of the key dimension, keeps the scores from becoming too large. Under the notation, the mechanism is plain: ask what matters, then blend the content that receives weight. It is simple, but it becomes memory-heavy once the context grows.

How Big Are These Vectors?

Each Q, K, and V vector has a specific size called the head dimension (d_h), typically 128 numbers. In a model like LLaMA-2 7B, there are 32 attention heads, each with dimension 128, giving a total hidden size of 4,096.

What Is KV Cache and Why Does It Exist?

The first time I looked closely at LLM inference, the slow part was more ordinary than I expected. The model does not write a paragraph in one breath. It writes one token, looks back, writes the next token, and looks back again. Every step repeats part of the same work.

Say it has written "The cat sat on the" and now needs token 6. It attends over tokens 1–5. Then token 7 attends over 1–6. Token 8 over 1–7. Without caching, the model keeps rebuilding Key and Value vectors for old tokens it has already processed. Nothing is mathematically broken. It is simply repetitive, and the repetition compounds.

The KV cache is the notebook the model keeps open beside it. Compute each token's Key and Value once. Store them. When the next token arrives, read the old entries instead of rebuilding them. It is a plain idea, which is why the memory cost is easy to underestimate.

The Speed Benefit

Without the cache, the model keeps paying for old work. With the cache, each step computes K and V only for the new token, then attends over the stored history. This does not make attention constant-time. Each new token still scans a growing context, so decoding still gets more expensive as the sequence length grows. The gain is concrete: fewer repeated projections and fewer wasted bytes moving around.

The Memory Cost

Then the tradeoff arrives. The cached K and V vectors live in GPU memory, and they grow linearly with sequence length. In FP16, the memory formula is:

The first "2" is for Keys and Values. The final "2 bytes" is FP16 precision. Put real model configs into that formula and the numbers get large quickly:

The Problem: When Memory Becomes the Bottleneck

The cache saves time, then slowly becomes the thing that controls the system. At 2K tokens, it is easy to ignore. At 32K, 128K, or a million tokens, it decides which model fits, how many users can be batched, and whether the request can finish. Three pressure points show up again and again:

Why Memory Bandwidth Matters

An A100 can perform about 312 trillion floating-point operations per second, but it moves data at roughly 2 TB/s. During decode, the arithmetic is not the hard part: one query vector is compared against a long row of cached keys. The expensive part is hauling those cached vectors out of memory every step. Engineers call this memory-bandwidth bound. The GPU is not short on math. It is waiting for bytes.

It is like a very fast cook working from a pantry down the hallway. The cooking takes seconds. The walking takes most of the shift. A faster stove will not fix that; you need fewer trips, a closer pantry, or lighter ingredients.

The Memory Fragmentation Problem

There is also waste that comes from bookkeeping. In serving, requests have different lengths, but older systems often reserved fixed-size memory blocks anyway. A short prompt could occupy space reserved for a much longer one. The PagedAttention paper measured this kind of fragmentation at 60–80% of KV cache memory in existing systems.

Most KV cache work answers one of those pressures. The techniques fall into five groups:

Multi-Head Attention (MHA): The Expensive Baseline

The original Transformer used what we now call Multi-Head Attention (MHA). It is the reference point for almost every optimization that follows. The design is still elegant: instead of one large attention pass, the model runs many smaller heads in parallel. One head may become sensitive to grammar. Another may track coreference. Another may notice negation. Nobody writes those roles by hand; they emerge during training.

The price is memory. In MHA, every head keeps its own separate Key and Value matrices. For a 32-head, 32-layer model, the cache grows like this:

For LLaMA-2 7B: 2 × 32 layers × 32 heads × 128 dim × 2 bytes = 524,288 bytes = 512 KB per token.

This is why MHA remains a good baseline. It gives every head its own memory, so each head can specialize freely. But the cost is blunt: more heads mean more cached vectors. With 32 heads and 128-dimensional heads, the model stores 64 vectors per token per layer. That is a lot of memory just to remember what has already been read.

Multi-Query Attention (MQA): Sharing One Memory Shelf

November 2019. Noam Shazeer, one of the original Transformer authors, published a short paper with a very sharp idea: what if all attention heads shared one set of Keys and Values?

That is Multi-Query Attention (MQA). You keep many Query heads, so the model can still ask many different questions. But instead of giving each head its own Key-Value archive, every head reads from the same archive. Imagine 32 researchers working from one filing cabinet. They disagree, they ask different questions, they write different notes, but the cabinet itself is shared.

The idea is simple. Queries need variety because each head asks a different question. Keys and Values are the context being questioned. Maybe the model does not need 32 full copies of that context.

For a model with 32 query heads but only 1 KV head, the KV cache shrinks by 32× compared with MHA.

The Results

The speed gain follows the memory story. Decode is usually bandwidth-bound, so a 32× smaller KV cache means far fewer bytes read every step. Tokens per second improves. The quality cost exists, but in 2019 it looked small enough to be worth the trade.

The Trade-off

Over time, that phrase "modest quality loss" became less comforting. MQA holds up well on short answers and many factual tasks. On long code, multi-step reasoning, and dense document analysis, the gap becomes easier to feel. By 2023, "slightly worse" was no longer good enough for many top models. That is where GQA enters.

Models Using MQA

MQA did make it into real models, including PaLM, Falcon-7B, and StarCoder. One detail worth keeping straight: Falcon-40B and Falcon-180B use GQA with 8 KV heads, not single-head MQA. The distinction matters because the field eventually moved toward that middle ground.

Grouped-Query Attention (GQA): The Middle Path

By mid-2023, MQA had a reputation problem. The memory savings were excellent. The quality tradeoff was harder to ignore. Forcing 32 query heads to share one KV head was efficient, but too severe for many larger models and harder tasks.

Joshua Ainslie and the Google team asked the practical question: if MHA is too expensive and MQA is too compressed, what sits between them? Their answer was Grouped-Query Attention (GQA). Divide the query heads into groups. Each group shares one KV head. With 32 query heads and 8 KV groups, the cache drops by 4×. Not as dramatic as MQA, but often much easier to live with.

The conversion recipe may be the most valuable part of the paper. A trained MHA model could be uptrained into GQA using about 5% of the original pretraining compute. No full restart. No huge retraining bill. GQA-8 stayed close enough to MHA on benchmarks that the quality gap became a deployment tradeoff rather than a blocker.

What the Paper Actually Shows

GQA fit the way model teams actually work. It saved enough memory to matter, and it could be adopted without throwing away an already-trained model. That combination is why it spread so quickly through open model families.

GQA's Impact on the Industry

After that, GQA became ordinary in the best possible way. LLaMA-3, Qwen2, Gemma 2, Phi-4, Mistral: the pattern shows up everywhere. Many 7–8B models settled around 8 KV heads. Larger models sometimes go lower. The exact number varies, but the direction is consistent.

Multi-Head Latent Attention (MLA): DeepSeek's Different Question

May 2024. GQA had become the sensible default, and it would have been easy for the field to keep refining that line of work. Then DeepSeek-V2 introduced MLA, and the question changed.

MHA, MQA, and GQA all ask some version of the same question: how many copies of Keys and Values should we store? DeepSeek asked a more unsettling one: do we need to store full Keys and Values at all? MLA stores a compact latent vector per token, then reconstructs the needed Key and Value information through learned projections during inference.

How MLA Works (The Intuition)

The music analogy helps. MHA stores full uncompressed recordings of every instrument. GQA stores one recording per section. MLA stores something closer to a compact score: enough structure to reconstruct what is needed later. The latent vector is the score; the model weights are the synthesizer. Reconstruction costs compute, but GPUs are good at matrix multiplication. The cache stays small.

The important trick is placement. Reconstruction matrices live in the model weights, not in the per-token cache. You pay for them once when the model is loaded, while the memory that grows with context remains compact.

In formula form, MLA stores a compressed vector c_t rather than the full Key and Value matrices for each token. The dimension d_c is much smaller than the full KV representation.

The Numbers Make the Point

DeepSeek-V2 MLA Configuration

In DeepSeek-V2, the specific parameters are:

• Number of attention heads (n_h): 128
• Per-head dimension (d_h): 128
• KV compression dimension (d_c): 512
• Decoupled RoPE dimension (d_h^R): 64

The cache stores 512 + 64 = 576 elements per token per layer. Standard MHA with 128 heads would store 2 × 128 × 128 = 32,768 elements. That is about a 56× reduction in the V2 configuration. It is roughly like having 2.25 KV groups, a configuration GQA cannot naturally express.

DeepSeek-V3: Scaling MLA Further

V2 proved the idea. V3 showed it could scale. DeepSeek kept MLA while moving to a 671B-parameter MoE model with 61 transformer layers. The cache cost landed around 70 KB per token. That is not tiny in isolation, but it is low for a model of that scale.

Live: How KV Cache Works in MHA vs MQA vs GQA vs MLA

The animation below runs all four attention methods side by side. Watch tokens arrive one at a time and see how much cached attention state each method stores. This is a normalized 32-layer FP16 comparison so the methods are compared on the same footing.

Cross-Layer Attention (CLA): Sharing Across Layers

Every approach so far has attacked the "head" axis of the KV cache, fewer heads, smaller heads, compressed heads. But a transformer has two axes that matter for cache size: heads and layers. If you have 32 layers, you're storing 32 separate sets of KV vectors per token. Nobody questioned this for years. Then a paper from Microsoft did.

The May 2024 CLA paper asks: do adjacent transformer layers really need their own independent KV cache? The observation is that nearby layers tend to attend to similar positions, they're looking at roughly the same context. So what if layers within a group just shared the KV cache from the first layer in that group?

That's Cross-Layer Attention (CLA). Group your 32 layers into pairs. Each pair shares one KV cache instead of two. Immediately, your cache is half the size. The accuracy loss is small enough that on most benchmarks you can't confidently attribute degradation to CLA specifically. And because CLA acts on the layer axis and GQA/MQA act on the head axis, you can combine them, the reductions multiply.

Results

CLA alone gives roughly 2× KV cache reduction. Stack it with GQA (8 groups) and you're looking at 8–16× over vanilla MHA. It's not widely deployed yet as a primary technique, most labs are still on GQA, but CLA sits in the toolbox as a clean, low-risk complement to anything else you're doing.

Sliding Window & Local-Global Attention

There's one more lever that hasn't come up yet: you can limit how far back the model looks at all. Every approach so far caches everything and then figures out how to compress it. This one asks: what if most of that history isn't needed in the first place?

Sliding Window Attention (Mistral 7B)

The Mistral 7B paper from October 2023 introduced Sliding Window Attention (SWA), each layer only attends to the 4,096 most recent tokens, and older tokens fall off the window. The KV cache per layer is capped at a fixed size no matter how long the conversation gets. That's a hard memory ceiling, which is very appealing from a serving perspective.

The natural worry is that important early context gets lost. The Mistral answer to this is stacking: with a window of 4,096 and 32 layers, information can propagate across a theoretical span of roughly 131,000 tokens as signals pass between layers. Whether that theoretical span translates to practical recall on hard tasks is a separate question, and I think it's fair to be somewhat skeptical about very long-range dependencies in SWA-only models. But for most real-world queries, where the relevant context is recent, it works well.

Interleaved Local-Global Attention (Gemma 2 and Gemma 3)

Google's Gemma team took a more careful approach. Rather than making every layer local, they interleaved two types:

Local layers use sliding window attention with a short, fixed window, tiny KV cache, fast, cheap. Global layers use full attention across all tokens, accurate, but expensive. The trick is the ratio. Gemma 3 runs 5 local layers for every 1 global. So only one out of every six layers actually needs to see the full context. The other five cap out at 1,024 tokens. At 128K context, that's the difference between paying for 128K-length KV cache at every layer versus paying for it at 1 in 6 layers. The memory curve flattens dramatically.

PagedAttention: Virtual Memory for KV Blocks

Everything up to this point has been about making the KV cache smaller in theory. MLA stores latents instead of full KV. GQA cuts the head count. Quantization shrinks each number. But there is a separate systems problem: the software managing GPU memory for KV cache can waste a lot of space.

Before vLLM, most serving systems pre-allocated a contiguous block of memory for each request, sized for the maximum possible sequence length. If you pre-allocate for 4,096 tokens and the user sends 200, you've wasted 3,896 tokens worth of memory. Multiply by thousands of concurrent users and you're throwing away 60–80% of your GPU memory before a single token is generated. The SOSP 2023 paper that measured this was not subtle about how bad the situation was.

Woosuk Kwon and the Berkeley team borrowed from a place software engineers already knew well: operating systems. Linux virtual memory does not require physically contiguous RAM. It breaks memory into pages, maps them through a page table, and lets the OS place pages wherever space exists. PagedAttention applies that pattern to GPU KV cache.

The Problem: Memory Fragmentation

The way pre-vLLM systems handled memory is almost comically wasteful once you see it. Each incoming request got a contiguous block of GPU memory sized for the maximum possible response. Ask for a 4,096-token limit, get 4,096 tokens of memory, whether you use 50 tokens or 4,000. Three kinds of waste compound on each other:

The paper found that existing systems waste 60–80% of KV cache memory!

The Solution: Paging

PagedAttention splits the KV cache into small fixed-size blocks, often 16 or 32 tokens per block. Blocks are allocated as needed, not upfront. They also do not need to be physically adjacent in memory. A page table maps logical sequence positions to wherever the physical blocks actually sit. Request A's block 3 can sit next to Request B's block 7 in physical memory. The attention kernel does not care; it uses the table.

Results

The system built on this, vLLM, quickly became one of the most used LLM servers. The throughput numbers are strong, but the bigger change is KV cache sharing across requests. Multiple users hitting the same system prompt do not each need their own copy of its KV blocks. That sharing follows naturally from the paging design.

TensorRT-LLM and SGLang both adopted similar paging approaches. For production LLM serving, paged KV cache management is now one of the first memory tools to check.

KV Cache Eviction: Keeping Only What Matters

One uncomfortable finding keeps showing up in KV eviction papers: much of the cache is barely used. Attention is highly skewed. A small set of tokens attracts most of the attention weight, while many others contribute almost nothing during generation. Once you see that pattern, the next question is natural: why keep every token at full cost?

H₂O: Heavy-Hitter Oracle (NeurIPS 2023)

Zhang et al. studied how attention scores are distributed across tokens during generation and found that they are not spread evenly. A small number of tokens, the "heavy hitters", absorb most cumulative attention across heads and decode steps. Many other tokens receive almost none. H₂O tracks which tokens collect the most attention weight and keeps those, plus a small window of recent tokens for continuity. Everything else gets evicted.

With only 20% heavy hitters retained, H₂O improved throughput by up to 29× over DeepSpeed Zero-Inference and Hugging Face Accelerate on OPT-6.7B and OPT-30B models.

Scissorhands (NeurIPS 2023)

Scissorhands builds on an observation that complements H2O. Where H2O measures importance dynamically as generation proceeds, Scissorhands asks: is importance stable over time? Their answer, the "Persistence of Importance" hypothesis, is that once a token is heavily attended to, it tends to stay heavily attended to. You don't need to keep re-evaluating which tokens matter. Identify them early, keep them, discard the rest. It's a simpler runtime than H2O but relies on this stability assumption holding.

Scissorhands achieves up to 5× KV cache reduction without quality loss, and when combined with 4-bit quantization, reaches 20× compression.

StreamingLLM: Infinite Context with Fixed Cache (ICLR 2024)

Xiao et al. at MIT studied why removing the first few tokens from the KV cache causes such severe quality loss, more than their content alone would suggest. They found that those initial tokens work as "attention sinks." No matter what the tokens say, the model routes a large share of attention weight to them. The beginning of the sequence seems to act like a scratch pad for attention that does not belong anywhere else. Remove the sinks, and attention becomes unstable. Keep the sinks plus a sliding window of recent tokens, and the model can process very long sequences with a fixed cache budget.

StreamingLLM enabled models like Llama-2, MPT, and Falcon to process up to 4 million tokens with stable performance, achieving 22.2× speedup over sliding window recomputation baselines.

SnapKV (NeurIPS 2024)

SnapKV shifts the problem earlier. Instead of deciding what to keep during generation, it decides during prefill, before a single output token is written. The observation is that attention patterns in the last few tokens of a prompt are already highly predictive of which earlier prompt tokens will matter during generation. So SnapKV reads those final prompt attention weights, pre-selects the high-importance prefix tokens, and throws away the rest before generation even begins. You're compressing the cache at the moment it's cheapest to do so.

PyramidInfer and PyramidKV

Both PyramidInfer and PyramidKV reached the same insight from different angles: token importance is not uniform across layers. Lower layers, which build basic syntactic structure, need to attend to more tokens. Higher layers, which handle abstraction and semantics, can often get by with fewer. Instead of giving every layer the same cache budget, early layers get more and late layers get less. That gives the cache a pyramid shape. PyramidKV matched full-cache performance while keeping only 12% of the total cache in the reported benchmarks.

PyramidInfer achieved 2.2× throughput improvement with over 54% GPU memory reduction. PyramidKV matched full KV cache performance while retaining only 12% of the cache.

KV Cache Quantization: Shrinking Each Number

All the approaches so far have tried to store fewer things in the KV cache. Quantization takes a different angle: store the same things, but use fewer bits per number. Think of it like image compression. You are not removing pixels; you are encoding each one more compactly. Standard KV cache uses 16-bit floats (FP16). That is two bytes per number. The quantization question is: how far can you push this before the model starts getting confused by its own rough memories? 8 bits? 4 bits? 2 bits?

This is harder than it sounds. Keys and Values are not arbitrary numbers. They carry structures that attention depends on. Distort the Key representations and the model starts attending to the wrong tokens. Distort the Values and the output gets garbled. KV quantization methods have to respect those structures instead of applying generic compression blindly.

KIVI: 2-Bit Quantization (ICML 2024)

KIVI is the paper I would point people to for why KV quantization is harder than it looks. Keys and Values have different outlier problems. Key matrices have specific channels (column indices) that produce very large values across all tokens. If you quantize naively, those channels dominate the scale and precision is lost everywhere else. Value matrices have the opposite issue: certain tokens produce spiking activations across all channels. KIVI handles this by using per-channel quantization for Keys and per-token quantization for Values, matching the compression axis to the actual outlier axis. The result is stable 2-bit quantization with near-lossless quality. Moving from 16 bits to 2 bits is an 8× compression, and KIVI does it without fine-tuning.

The results held across LLaMA, Falcon, and Mistral families with negligible quality degradation. No retraining, no fine-tuning. Just better quantization choices. The throughput improvements come largely from the reduced memory bandwidth, smaller cache means fewer bytes to move per decode step.

KVQuant: Toward 10 Million Context (NeurIPS 2024)

Hooper et al. pushed harder. Where KIVI focuses on the quantization axis and granularity, KVQuant layers in several additional tricks: applying quantization before RoPE (positional encoding) is added to keys, using non-uniform quantization bins, and handling outliers with a dense-and-sparse decomposition. Each addition is well-motivated in the paper, and the cumulative effect is significant.

The results enabled:

• Less than 0.1 perplexity degradation at 3-bit quantization
• LLaMA-7B with 1 million context on a single A100-80GB GPU
• Up to 10 million context on an 8-GPU system
• 1.7× speedup with custom CUDA kernels vs FP16 baseline

TurboQuant: Near-Optimal Compression (2025)

TurboQuant is the most theoretically grounded method in this space, and the data-oblivious claim is its most striking property. No calibration data, no model-specific tuning, no setup cost, apply it to any model at inference time. This works because TurboQuant doesn't try to learn from the model's activation statistics. Instead it applies a random rotation to spread information uniformly across vector coordinates, then runs near-optimal scalar quantizers on each coordinate independently. The theoretical backing for random rotation is solid (it's a randomized dimensionality technique with good guarantees), and the empirical results track the theory closely, which is rarer than you'd think in this area.

The method works in three steps:

1. Randomly rotating input vectors to distribute information evenly across coordinates
2. Applying optimal scalar quantizers to each coordinate independently
3. Using a 1-bit Quantized JL transform on residuals for unbiased inner products

Bonsai 8B + Turbo1Bit: 1-Bit Weights Meet KV Cache Compression

All the quantization methods above shrink the KV cache. But what happens when you also shrink the model weights to the extreme? On March 31, 2026, Caltech spinoff PrismML announced Bonsai 8B, a language model where every weight is represented as just one bit: either +1 or −1. An 8.2 billion parameter model that fits in 1.15 GB (compared to ~16 GB for a standard FP16 8B model), a 14× size reduction for the weights alone.

There is a catch: 1-bit weights do NOT mean 1-bit KV cache. The weights (W_K, W_V, W_Q, etc.) are 1-bit, but when they multiply against input activations, the resulting Key and Value vectors are still full-precision floating point numbers. Bonsai's weights are tiny, but the KV cache during inference remains FP16-sized and grows linearly with context length. The memory wall is still there.

This is where Turbo1Bit comes in: an open-source project that stacks TurboQuant-style KV cache quantization on top of Bonsai's 1-bit weights.

Turbo1Bit's KV cache compression findings for 1-bit models specifically:

Quantization Comparison Overview

FlashAttention: Using Hardware Better

FlashAttention is a slightly different kind of topic for this survey. It does not shrink the KV cache. It does not change how many KV heads you have or what precision you use. I include it because it explains why memory movement matters so much in long-context inference.

The core observation from Tri Dao's 2022 paper is that standard attention implementations move data between GPU HBM (large, slower) and SRAM (small, fast) more often than needed. FlashAttention restructures the computation into SRAM-sized tiles, reducing that movement. The algorithm is mathematically identical. The output is the same. The implementation is 2–4× faster and uses memory proportional to sequence length rather than its square.

FlashAttention (NeurIPS 2022)

Standard attention computes the full n×n score matrix and holds it in HBM while doing later operations. For a 32K-token sequence with 128 heads, that matrix is enormous. Every operation has to load data from HBM into the GPU cores, do work, and write results back. FlashAttention breaks the attention matrix into chunks small enough to fit in on-chip SRAM, finishes the operations for each chunk while it is in fast memory, and never materializes the full matrix in HBM. The output is the same, but the memory access pattern is different.

Key result: 2–4× speedup over optimized baselines, with linear memory usage instead of quadratic. This means the n×n attention matrix is never fully materialized in GPU memory.

FlashAttention-2 (2023)

FlashAttention-2 is mostly an engineering improvement. Dao revisited work partitioning between GPU thread blocks, reduced non-matmul operations, and improved handling of varying sequence lengths. None of these changes affect the algorithm. They push GPU utilization from around 35% to 72%, which matters a lot in deployment.

FlashAttention is now common in vLLM, Hugging Face Transformers, PyTorch, and many other stacks. Its link to KV cache is indirect but real: when attention is computed more efficiently, the per-step cost of attending over a long KV cache is lower. That makes the other memory reductions pay off more.

Model-by-Model Comparison: How the Pieces Fit

After the methods, the fastest way to build intuition is to look at real configs. The tables below use published configs and model cards where available. When a row needs an assumption because of mixed head dimensions, shared KV, or sliding-window layers, I flag it rather than hiding the uncertainty.

A pattern appears quickly. LLaMA-2 7B, with pure MHA, sits at 512 KB per token. Mistral 7B, with GQA plus a sliding window, drops to 128 KB before any KV quantization. MLA cuts far deeper. Qwen3.5-9B gets to 32 KB per token by mixing attention with DeltaNet-style recurrent layers. The table is dense, but the pattern is simple: modern models are not relying on one trick. They are stacking several.

Part A: Legacy & Foundational Models (2023–2024)

Part B: Frontier Models (2025–2026)

Exact KV Cache Memory Requirements, Calculated from config.json

Below are KV cache memory calculations derived from published config.json files where available. Rows marked with * are estimates due to hybrid architectures (mixed head dimensions, KV sharing, or sliding windows) where an exact single number is not straightforward.

Patterns in the Table

Visual Summary: How Each Technique Reduces KV Cache

DFlash: Drafting More Than One Token at a Time

So far, most of the story has been about making the cache smaller. DFlash comes from a neighboring problem: even with a smaller cache, generation still happens one token at a time unless the serving system finds a way around it. Speculative decoding is one such workaround. In February 2026, Z Lab published DFlash, which changes the drafter itself: instead of drafting autoregressively, it drafts a block with diffusion.

The Speculative Decoding Bottleneck

Standard autoregressive decoding is narrow by design: one forward pass, one new token. Even after the KV cache is optimized, each decode step still has to read the relevant cache from HBM, compute attention, and produce a single output token. The GPU often has plenty of compute left unused; the step is waiting on memory.

Speculative decoding uses a smaller draft model to propose several candidate tokens, then asks the larger target model to verify them in one pass. If k candidates are accepted, one expensive target-model step yields k output tokens. The catch is that most draft models are also autoregressive. They still draft token by token. The bottleneck moves to a cheaper model, but it remains serial.

DFlash replaces that serial drafter with a block diffusion model. The draft block is generated together, not one token at a time. That single change makes the speculative path much wider.

How DFlash Works

Step 1: Context Feature Extraction

After prefill or verification, DFlash reads hidden-state features from the target model. It does not rely only on the final layer. It samples across depth, so early syntactic signals and later semantic signals both reach the draft model. A lightweight projection fuses those features.

Step 2: KV Cache Injection into the Draft Model

This is the heart of the method. The fused target-model features are injected into the Key/Value projections of every draft-model layer and stored in the draft model's own KV cache. The drafter is not handed a one-time summary and left to drift. It carries target-model context throughout the diffusion process.

Step 3: Parallel Verification

The target model then verifies the proposed block in one pass. Matching tokens are accepted. At the first mismatch, the target model corrects the output and the remaining draft tokens are discarded. The output distribution stays the same as the target model alone; only the route to that output changes.

Performance Results

The paper reports over 6× lossless acceleration across a range of models and benchmarks. "Lossless" has a precise meaning here: with correct rejection sampling, speculative decoding preserves the target model's output distribution. The model does not say something different; it reaches the same distribution with fewer expensive target-model passes.

Relationship to KV Cache Efficiency

DFlash sits beside KV-cache architecture work rather than replacing it:

When the two are combined, the gains can compound. A smaller KV cache makes target-model verification cheaper; DFlash reduces how often the target model has to run. One shrinks the memory bill per step. The other reduces the number of expensive steps.

Kimi K2.5: How Agent Workloads Change Memory

Everything up to this point assumes the model gets one prompt and writes one response. Agents work differently. An agent may get a task like "research competitors and write a market analysis" and then run for hours. It searches the web, reads documents, calls code tools, checks outputs, revises its plan, and searches again. Each tool call can return 5,000–20,000 tokens. After 100 tool calls, the context can contain millions of tokens, many of them stale. At that point, KV cache is no longer only an efficiency issue. It becomes a session-management issue.

Moonshot AI's Kimi K2.5 paper (February 2026) is worth reading because it describes both the system and the failures they hit while running long agent sessions. Those failure modes explain why the design looks the way it does.

The Problem: Why Agents Break Traditional KV Cache

In a standard chatbot conversation, the context grows slowly: one user message, one model response, and so on. The KV cache grows predictably and stays within the context window.

An agent is nothing like this. Consider an agent researching a topic:

After 200 tool calls, the context might contain millions of tokens, most of which are tool outputs the model only needed briefly. The KV cache grows out of control, latency spikes, and eventually the model hits its context limit.

Kimi K2.5's Three-Part Solution

1. Agent Swarm: Parallel Decomposition

Instead of one agent working through a problem step by step, which creates one large and growing KV cache, K2.5 introduces Agent Swarm. A central orchestrator breaks a complex task into independent sub-problems, then starts multiple sub-agents to work on them at the same time.

The result: 4.5× latency reduction over single-agent baselines on complex web search tasks, and a 17.8% accuracy improvement on BrowseComp (60.6% → 78.4%).

2. PARL: Teaching the Model When to Parallelize

The orchestrator doesn't use hand-coded rules to decide when to spawn sub-agents. Instead, Moonshot trained it with a custom reinforcement learning method called Parallel-Agent RL (PARL). The reward function has three components:

A key design choice: sub-agents are frozen, so their weights do not update during PARL training. Only the orchestrator learns. This avoids a hard credit-assignment problem in multi-agent RL, where it is unclear which agent caused a good outcome.

3. Toggle: Budget-Aware Reasoning

Kimi K2.5 uses a training technique called Toggle that alternates between two modes:

For KV cache, reasoning tokens are expensive. A thinking model like K2.5 might generate 36,000 internal reasoning tokens before producing a 200-token answer. Those 36K tokens all live in the KV cache. Toggle teaches the model to spend fewer tokens on easy sub-tasks and save the larger budget for harder ones. At ~70 KB per token with MLA, that is the difference between 490 MB and 2.5 GB of KV cache just for the thinking phase.

Discard-All Context Management

For very long agent sessions, K2.5 uses a strategy called "Discard-all". After each major reasoning phase, the model discards the full context, including tool outputs, and starts fresh with only the accumulated conclusions. This resets the KV cache periodically instead of letting it grow to the context limit.

On the BrowseComp benchmark, using Discard-all boosted accuracy from 60.6% to 74.9%, beating GPT-5.2's reported 65.8%, Claude Opus 4.5 (37.0%), and Gemini 3 Pro (37.8%). The useful lesson is that throwing away context can help when old tool outputs add more noise than value.

The KV Cache Implications

Linear Attention & SSMs: The O(1) KV Cache Alternative

Every chapter so far has assumed the cache exists and asked how to make it smaller. This chapter is different. What if you got rid of the cache altogether?

That's the premise behind State Space Models and linear attention. Instead of storing a growing list of Key and Value vectors for every previous token, these architectures compress the entire history into a single fixed-size state, and update it as each new token arrives. No cache that grows with context length. Constant memory at inference time, regardless of how long the sequence gets.

How Standard Attention Scales

For context: standard self-attention is O(n²) in compute and O(n) in memory per layer. At 1 million tokens, the KV cache is enormous, and every new decode step requires loading the entire thing from HBM. SSMs break that scaling entirely, but they break it by accepting lossy compression of the past, which Transformers never do.

State Space Models (Mamba, RWKV, RetNet)

Mamba (Selective State Spaces)

Mamba (Gu & Dao, 2023) is probably the most-discussed SSM in the LLM space. The key word in its full name, selective state space model, is "selective." Earlier SSMs like S4 had fixed transition matrices. Mamba learns to select which information to retain and which to discard based on the current input, which makes it much more expressive. The hidden state is fixed-size regardless of sequence length; what changes is how that state gets updated. Hardware-efficient scan operations make this fast on GPU without needing custom attention kernels.

RWKV (Receptance Weighted Key Value)

RWKV is interesting for a different reason. It's been developed almost entirely as an open, community-driven effort, unlike most of the models in this survey. Technically it reformulates attention as a linear recurrence, the running "KV cache" collapses to a single exponentially-weighted state. O(1) memory at inference. The cost is that old information decays; you can't retrieve something from 10,000 steps ago with the same fidelity as something from 100 steps ago. For many practical tasks that's fine. For precise document lookup, it's not.

RetNet (Retention Networks, Microsoft)

RetNet is clever in a specific way: it offers parallel training (O(n²), like a Transformer) and recurrent inference (O(1) memory, O(1) per step) from the same underlying formulation. Training is efficient on GPU because you can parallelize. Inference is efficient because it's recurrent. Microsoft scaled this to 65B parameters with results they describe as competitive, though independent evaluation at that scale is limited enough that I'd hold that claim loosely.

Liquid Foundation Models (LFM)

Liquid AI's LFMs are worth mentioning as a different flavor of the same idea. Where Mamba and RWKV use discrete recurrence, LFMs are built on Liquid Time-Constant networks, continuous-time dynamical systems where each neuron has a learnable time constant that controls how fast it forgets. The key difference from standard SSMs is that these time constants adapt per input, rather than being fixed structure. This makes LFMs more expressive for data with irregular timing, audio, sensor streams, time series, though less clearly better for straightforward text generation.

Head-to-Head: Transformer KV Cache vs SSM Hidden State

Hybrid Models: Getting Practical

The pure-SSM vs pure-Transformer debate has been largely settled by practitioners: you want both. Attention for precise recall. SSM layers for cheap context compression. Several architectures now alternate between them:

• Jamba (AI21 Labs): mixes Mamba blocks with full attention layers, adds MoE on top. First large-scale hybrid to reach competitive quality
• Zamba: a single shared attention layer surrounded by many Mamba layers, the attention acts as a periodic "recalibration" of the SSM state
• DeepSeek V4's CSA/HCA: arguably a hybrid within the attention family, compresses long-range context aggressively (HCA) while keeping some layers precise (CSA)

Prefix Caching & RadixAttention: Reuse Across Requests

All the optimization techniques so far are about a single request, one user, one conversation, one growing KV cache. But production LLM services don't look like that. You're serving thousands of users at once, and many of them start with the exact same system prompt, the exact same few-shot examples, maybe even the same retrieved documents. Without prefix caching, you're recomputing those KV vectors from scratch for every request. You're literally doing the same work thousands of times over.

I find this one of the most under-discussed efficiency wins in the whole space, maybe because it doesn't require any model changes. It's purely a serving-layer optimization. But the math is stark. A 2,000-token system prompt served to 1,000 users per second means 2 billion redundant KV computations per second. Cache that prefix once and you've eliminated all of them.

What Counts as a Prefix?

Any text that appears at the start of multiple requests. The obvious cases: system prompts, few-shot examples, long documents sent with every question, RAG chunks that keep appearing in similar queries. The less obvious case: multi-turn conversation history. Once a user has had 10 turns with your assistant, the first 9 turns are a shared prefix for every subsequent request. Cache those and you're not re-reading the whole conversation history on every new message.

Automatic Prefix Caching (APC)

APC in vLLM hashes incoming token sequences, checks for prefix matches in a stored KV block pool, and if a match exists, reattaches those cached blocks to the new request without recomputing them. The prefill phase for those matched tokens is skipped entirely. It's completely transparent to the model, the output is identical to a full recompute.

RadixAttention (SGLang)

APC handles exact prefix matches. RadixAttention, from the SGLang paper, goes further. It organizes the KV cache as a radix tree, basically a trie over token sequences. This lets you cache and reuse shared subtrees along with common leading prefixes. If Request A and Request B share the same first 500 tokens but diverge after that, they still share those 500 tokens of KV cache in the tree, even if neither request is an exact prefix of the other. Multi-turn conversations fit this structure well because they branch from shared history.

LRU eviction happens at the leaves. The least recently used branches get dropped first when memory is needed, preserving the most commonly used paths. Multiple concurrent requests can read from the same node without lock contention. It is a simple data structure doing useful work.

Real-World Impact

Mixture of Experts & KV Cache: Sparse Activation

Some recent KV cache efficiency gains come from a different architectural choice: Mixture of Experts (MoE). DeepSeek V3, Qwen3-MoE, Mixtral, and DeepSeek V4 all use MoE. The KV cache benefits are indirect, but they matter.

MoE is often left out of KV cache surveys because it is not a KV cache technique by itself. But when choosing a model architecture, MoE versus dense affects how much memory headroom is left for the cache. That makes it relevant here.

How MoE Works

Standard Transformer FFN layers run every token through the same network. MoE replaces each FFN with N parallel "expert" FFNs and a learned router that picks the top-k experts for each token. In DeepSeek V3, that's 256 experts with top-8 routing. For any given token, 248 of the experts do nothing. Their weights sit in memory but don't consume compute or bandwidth for that token.

MoE's Effect on KV Cache

1. Attention stays dense, MoE doesn't directly help the cache

The important nuance: MoE only applies to FFN sublayers. The attention layers are fully dense, every token attends to every other token, and full KV is kept for every layer. MoE does not reduce KV cache size on its own. DeepSeek V3 at 671B total parameters has the same KV cache structure as a dense model with 37B active parameters. The 93% KV reduction in V3 comes from MLA, not MoE. The two techniques are separate and can be stacked.

2. More parameters, same cache, the memory split matters

MoE does affect total GPU memory, but indirectly. Expert weights all need to live somewhere; you cannot load 64 expert FFNs from disk on every decode step. The GPU memory budget gets split between KV cache, which scales with context, and expert weights, which scale with model size. Serving a large MoE model can leave less memory headroom for KV cache, which may force more aggressive quantization or eviction even when the cache architecture is efficient.

DeepSeek's MoE + MLA Combination

DeepSeek combines MoE for compute efficiency with MLA for KV cache efficiency. Together, they make a 671B-parameter model practical to serve on a small GPU cluster. MoE reduces active compute; MLA reduces memory bandwidth. Neither solves the other's problem directly, but together they address two major bottlenecks in serving large models.

KV Cache Offloading: CPU & NVMe as Overflow

Sometimes GQA, quantization, and prefix caching are still not enough. The context is too long for GPU memory. You may be generating a 300,000-token legal analysis, and the cache simply does not fit. Now what?

You spill it. Move the blocks that are not needed right now to CPU RAM, or to NVMe if RAM is full. Load them back when the attention head reaches those positions. It is slow, but it can be the difference between a failed request and a completed one. For batch workloads where latency is flexible, the throughput can be acceptable.

FlexGen

FlexGen (Sheng et al., 2023) treats offloading as a scheduling problem, rather than only an emergency fallback. GPU, CPU DRAM, and NVMe are resources with different bandwidth and capacity. A linear programming solver chooses where to place model weights, activations, and KV cache across all three, given the throughput target and memory constraints.

Running a 30B model on a single 16 GB GPU is normally unrealistic. FlexGen makes it possible by accepting high latency and optimizing throughput instead. For research workloads and offline batch jobs, this can be a valid tradeoff. For interactive use, it usually is not.

InfiniGen

InfiniGen is selective about what it offloads. Rather than moving everything to CPU and fetching it back blindly, it runs a lightweight predictor that scores which KV blocks the current decode step will likely need. It prefetches those blocks and leaves low-scoring blocks on disk. Attention bandwidth is spent only on blocks that are likely to matter.

When to Use Offloading

Disaggregated Prefill & Decode: Splitting the Serving Job

In a standard LLM server, the same GPU handles prefill and decode. Prefill processes the prompt. Decode generates output one token at a time. They look similar from the outside, but they stress the hardware in very different ways.

The Problem: Prefill-Decode Interference

When a new request arrives during ongoing decode, the GPU has to make a bad choice. Handle the compute-heavy prefill and delay existing users, or keep decoding and make the new user wait for the first token. That collision is prefill-decode interference.

Disaggregated Serving (Splitwise, DistServe, Mooncake)

The fix is easy to describe and harder to operate: separate the two phases. A prefill cluster builds the prompt's KV cache. The KV blocks are transferred over NVLink or InfiniBand. A decode cluster handles generation. Two fleets with different resource profiles create much less interference.

Ring Attention: Sequence Parallelism for 1M+ Tokens

Even with disaggregated serving and aggressive quantization, there are contexts long enough that no single GPU can hold the full KV cache. A million-token context at even 70 KB/token is 70 GB, bigger than most GPU memory budgets. Ring Attention handles this by distributing the KV cache across a ring of GPUs. Each GPU holds a slice of the sequence. The attention computation rotates KV blocks around the ring in a round-robin while queries stay local. Every GPU participates; the total capacity scales with the ring size.

Production long-context systems rarely disclose whether they use Ring Attention specifically. Most model cards say "supports 1M token context" without describing the serving stack. Ring Attention is the main published technique for this problem, but that does not mean it is used everywhere.

Where Things Stand in 2026

This field moves quickly, so any summary has a short shelf life. By early 2026, one pattern is clear: KV cache efficiency is a stack, not a single technique.

What's Working Today

What Is Still Unsettled

1. Trillion-scale MLA: Kimi K2 from Moonshot AI is the main data point. It takes DeepSeek-style MLA to roughly 1 trillion parameters with 384 routed experts. The public config shows the latent cache approach survives at that scale. Quality tradeoffs still need broader evaluation.

2. Ten-million-token context: Meta's LLaMA 4 Scout model card reports a 10 million token context window, roughly thousands of pages in one prompt. The card does not expose enough attention internals to map every technique, but the target shows how far long-context systems are being pushed.

3. On-device long context: Gemma 4 E2B and E4B aim at mobile and edge hardware while keeping 128K context. Their KV budget comes from a blend of low KV-head counts, local-global attention, cross-layer sharing, and per-layer embeddings. Gemma 4 26B-A4B is the MoE member; E2B and E4B are not.

4. Aggressive KV-head reduction: Qwen3's MoE variant uses 4 KV heads with 64 query heads, a 16:1 ratio. The field keeps testing how few KV heads are enough.

5. Data-oblivious quantization: TurboQuant does not require calibration per model. That makes it attractive for teams serving many architectures. The exact compression factor still depends on model, workload, and context length.

The Journey in Numbers