The Evolution of KV Cache

The Transformer Revolution — Where It All Began

June 2017. Eight researchers at Google drop a paper with one of the boldest titles in computer science history: "Attention Is All You Need." Inside was a new architecture they called the Transformer — and it would go on to power virtually every major AI system built in the decade that followed.

To appreciate why the Transformer mattered, you need to understand what came before it. The reigning champion of text processing was the Recurrent Neural Network (RNN). Imagine a relay race where each runner must memorize and recite every message carried by all previous runners before passing the baton. Runner #50 has to recite 49 accumulated messages, then add their own. Inevitably, early messages get garbled or lost. Worse, the race is strictly sequential: runner #10 can't start until runner #9 finishes, no matter how many spare runners you have standing around. That's the RNN — slow, forgetful, and impossible to speed up by adding more hardware.

The Transformer discarded that relay race entirely. Its breakthrough? Attention. Instead of passing messages down a chain one at a time, it spreads the entire text on a giant table and lets every word look at every other word simultaneously. Think of a room of 500 people at a networking event where everyone can hear every conversation happening at once and instantly decide which ones are relevant to them. No chain. No waiting. Massively parallel.

The numbers spoke for themselves. On the WMT 2014 English-to-German translation benchmark, the Transformer hit a BLEU score of 28.4 — blowing past the previous best by over 2 points. In machine translation, that's not an incremental gain; it's a big jump. On top of that, training was much faster because the architecture could chew through all tokens in parallel rather than one at a time.

What followed was an explosion. BERT. GPT-2. GPT-3. GPT-4. LLaMA. Gemma. Qwen. DeepSeek. Mistral. Hundreds of models, all built on the same Transformer backbone. But as these models ballooned from millions to hundreds of billions of parameters — and as users demanded they handle longer and longer documents — a hidden cost quietly grew in the background. A cost that would eventually consume more GPU memory than the model weights themselves. That cost has a name: the KV Cache.

Understanding Keys, Values, and Queries

To understand the KV Cache, we first need to understand how attention works. The Transformer uses a mechanism called Scaled Dot-Product Attention, and it relies on three things: Queries (Q), Keys (K), and Values (V).

Here is an analogy that makes it concrete. Imagine you are ordering at a massive food court with 200 stalls:

In the Transformer, each word (or "token") generates three things: a Query, a Key, and a Value — all represented as lists of numbers (vectors). The process works like the food court: your craving (Query) gets compared against every stall's menu board (Keys) to produce a relevance score. High-scoring stalls contribute more of their dish (Value) to your final plate. Low-scoring ones contribute almost nothing. The result is a weighted blend of Values, customized to what this particular token was "looking for."

In plain English: "Scan your craving (Q) against every stall's menu board (K), calculate a match percentage for each (softmax turns raw scores into percentages that add up to 100%), then blend the dishes (V) in proportion to how well they matched."

The √d_k part is just a scaling factor to keep numbers from getting too large. It's the square root of the dimension of the key vectors.

How Big Are These Vectors?

Each Q, K, and V vector has a specific size called the head dimension (d_h), typically 128 numbers. In a model like LLaMA-2 7B, there are 32 attention heads, each with dimension 128, giving a total hidden size of 4,096.

What Is KV Cache and Why Does It Exist?

Here's how a language model actually generates text. It doesn't write a whole sentence at once — it produces one token (roughly one word) at a time. Suppose the model has already written "The cat sat on the" and now needs to pick the next word. To make that decision, it has to run the attention mechanism, which means comparing the new token against every single previous token.

Now here's the wasteful part. Without caching, the model recomputes the Key and Value vectors for "The," "cat," "sat," "on," and "the" from scratch every single time it generates a new word. At token 1,000? It recomputes K and V for all 999 previous tokens. At token 1,001? All 1,000. You can see how this snowballs into an absurd amount of redundant work.

The KV Cache is the fix. It simply stores the Key and Value vectors once they're computed and reuses them for all future tokens. Each new token only needs to compute its own Q, K, V, then look up everything else from the cache. No redundant recomputation. Simple and effective.

The Speed Benefit

Without KV Cache, generating a sequence of length n requires O(n²) total computation (because each new token must attend to all previous tokens, and this happens n times). With KV Cache, it drops to O(n) per new token, because we only compute attention for one new query against cached keys and values.

The Memory Cost

Here's the catch: those cached K and V vectors consume GPU memory, and they grow linearly with sequence length. The formula for KV Cache memory in FP16 (16-bit floating point, 2 bytes per number) is:

The first "2" is for Keys and Values. The final "2 bytes" is for FP16 precision. Let's see what this means for real models:

The Problem — When Memory Becomes the Bottleneck

The KV Cache creates three critical problems that limit how we can deploy and use large language models:

Why Memory Bandwidth Matters

An NVIDIA A100 GPU can perform 312 trillion floating-point operations per second (312 TFLOPS). That's a lot of raw math power. But it can only move data from its memory at about 2 terabytes per second. When a model generates text one token at a time, the actual math is trivial — multiply one query vector against a bunch of cached keys. The bottleneck isn't doing the multiplication; it's loading all those keys and values from memory fast enough. Engineers call this being memory-bandwidth bound.

Here's an analogy that makes this click. Imagine a pizza oven that can bake 1,000 pizzas per minute (the GPU's compute). Sounds amazing, right? Except the kitchen only has one narrow doorway for ingredients to come through (memory bandwidth). The oven sits idle most of the time, waiting for dough and toppings to arrive. Making the oven bigger doesn't help. You need either a wider doorway — or smaller pizzas.

The Memory Fragmentation Problem

There's another subtle issue. When serving multiple users simultaneously, each request may need a different amount of KV cache (because they have different conversation lengths). Traditional systems pre-allocate a fixed-size memory block for each request, leading to massive waste when the actual conversation is shorter than the maximum. The PagedAttention paper (Section 11) showed that existing systems waste 60-80% of KV cache memory due to fragmentation.

These problems inspired a decade of research that we'll explore in the following chapters. The solutions fall into several categories:

Multi-Head Attention (MHA) — The Baseline

The original Transformer uses Multi-Head Attention (MHA). In MHA, the model runs multiple parallel "attention heads" — each one learns to focus on different types of word relationships. One head might specialize in tracking who did what to whom (subject-object). Another might track temporal cues ("before," "after," "meanwhile"). A third might notice negation patterns. They each develop their own specialty without being explicitly told what to learn.

In MHA, every attention head has its own separate set of Keys, Values, and Queries. For a model with 32 attention heads:

For LLaMA-2 7B: 2 × 32 layers × 32 heads × 128 dim × 2 bytes = 524,288 bytes = 512 KB per token.

MHA provides the best quality because each head maintains its own representation of Keys and Values. However, the KV cache grows proportionally to the number of heads, making it the most memory-expensive approach.

Multi-Query Attention (MQA) — The First Big Fix

November 2019. Noam Shazeer — one of the eight original Transformer authors — publishes a short, almost casual paper with an idea so simple it seems like it shouldn't work: make all the attention heads share a single set of Keys and Values.

That's Multi-Query Attention (MQA) in one sentence. You still have 32 query heads asking 32 different questions about the text. But instead of each head maintaining its own personal reference sheet (its own Keys and Values), they all consult the exact same sheet. Imagine 32 food critics at a restaurant — each one evaluates the meal from a different angle (texture, aroma, plating, spice level), but they're all tasting from the same set of plates. One kitchen. Thirty-two opinions.

For a model with 32 query heads but only 1 KV head, the KV cache shrinks by 32× compared to MHA!

The Results

Shazeer demonstrated that since autoregressive decoding is bottlenecked by memory bandwidth (not compute), shrinking the KV cache directly speeds up token generation. In practice, models using MQA generated tokens significantly faster, while benchmark scores dropped only slightly compared to the full MHA setup.

The Trade-off

While MQA provides the maximum memory savings, forcing all heads to share a single KV representation can hurt model quality, especially for complex reasoning tasks. This quality concern led to the development of Grouped-Query Attention (next chapter).

Models Using MQA

MQA was adopted by several notable models including PaLM (Google, 2022), Falcon-7B (TII, 2023), and StarCoder (BigCode, 2023). Note: Falcon-40B and Falcon-180B use GQA (8 KV heads), not MQA — only the 7B variant uses true single-head MQA. Concerns about quality degradation led most subsequent models to prefer GQA instead.

Grouped-Query Attention (GQA) — The Best of Both Worlds

By mid-2023, the AI community faced a dilemma: MHA gave the best quality but was memory-hungry, while MQA saved memory but sacrificed accuracy. Joshua Ainslie and his team at Google asked the natural question: what if we do something in between?

Their answer was Grouped-Query Attention (GQA). Rather than giving each query head its own private KV pair (MHA) or forcing all query heads to share one KV pair (MQA), GQA organizes query heads into small groups. Each group gets its own dedicated KV pair. For instance, with 32 query heads split into 8 groups, every 4 query heads share one KV head.

Key Findings from the Paper

The paper made two important contributions:

1. Uptraining Recipe: Existing MHA models can be converted to GQA using only 5% of original pre-training compute. This means you don't need to retrain from scratch.

2. Quality-Speed Trade-off: GQA with 8 KV heads achieves quality close to MHA while maintaining speed comparable to MQA. It's the Goldilocks solution.

GQA's Impact on the Industry

GQA quickly became the de facto standard for modern large language models. The following table shows its widespread adoption:

Multi-Head Latent Attention (MLA) — DeepSeek's Breakthrough

May 2024. While the rest of the industry was settling into GQA as "good enough," a Chinese AI lab called DeepSeek dropped a paper that made everyone do a double take. Their approach didn't just tweak the number of KV heads — it threw out the idea of storing Keys and Values at all. Instead, DeepSeek compressed the full K and V information for each token into a tiny "latent" vector, then reconstructed the actual Keys and Values on-the-fly during inference from this compressed seed.

How MLA Works (The Intuition)

Think of it in terms of music. MHA is like storing full uncompressed WAV recordings of every instrument in an orchestra — pristine audio, but terabytes of disk space. GQA is like grouping instruments into sections (strings, brass, woodwinds) and keeping one recording per section. MLA goes further still: it doesn't store any audio at all. Instead, it stores a tiny MIDI-like encoding — just the notes, velocities, and timing — from which a synthesizer can reconstruct a faithful rendition of the full orchestra on demand. That MIDI encoding is the "latent vector," and the synthesizer is a learned matrix multiplication built into the model weights.

Mathematically, instead of storing the full Key matrix K (size: n_heads × d_head) and Value matrix V (size: n_heads × d_head) for each token, MLA stores a single compressed vector c_t of dimension d_c, where d_c is much smaller than the full KV size.

The Numbers Tell the Story

DeepSeek-V2 MLA Configuration

In DeepSeek-V2, the specific parameters are:

• Number of attention heads (n_h): 128
• Per-head dimension (d_h): 128
• KV compression dimension (d_c): 512
• Decoupled RoPE dimension (d_h^R): 64

The KV cache per token requires only (d_c + d_h^R) = 576 elements, which is equivalent to GQA with only 2.25 KV groups. For comparison, standard MHA with 128 heads would need 2 × 128 × 128 = 32,768 elements per layer.

DeepSeek-V3: Scaling MLA Further

In December 2024, DeepSeek released V3, which continued using MLA and achieved even more impressive results:

Live: How KV Cache Works in MHA vs MQA vs GQA vs MLA

The animation below runs all four attention methods side by side. Watch tokens arrive one at a time and see how much cached attention state each method stores. This is a normalized 32-layer FP16 comparison so the methods are compared on the same footing.

Cross-Layer Attention (CLA) — Sharing Across Layers

All the methods discussed so far (MHA, MQA, GQA, MLA) reduce KV cache along the "head" dimension — fewer or compressed heads. But what about another dimension: layers?

In May 2024, researchers from Microsoft proposed Cross-Layer Attention (CLA): instead of each transformer layer maintaining its own KV cache, adjacent layers can share their Key-Value pairs.

The idea is simple: divide the transformer layers into groups (e.g., pairs of adjacent layers), and within each group, all layers use the Keys and Values from the bottom layer of the group. This means you only need to store KV cache for half the layers.

Results

CLA can reduce the KV cache size by 2× while maintaining nearly the same accuracy as standard MQA. Importantly, CLA is orthogonal to head-level compression — it can be combined with GQA or MQA for even greater savings. For example, GQA + CLA could achieve a combined 8–16× reduction over MHA.

Sliding Window & Local-Global Attention

Another approach to reducing KV cache is limiting how far back the model looks. Instead of attending to every previous token (potentially hundreds of thousands), the model only looks at a fixed window of recent tokens.

Sliding Window Attention (Mistral 7B)

In October 2023, Mistral AI published the Mistral 7B model, which introduced Sliding Window Attention (SWA) with a window size of 4,096 tokens. Each layer only attends to the previous 4,096 tokens, meaning the KV cache per layer is capped at a fixed size regardless of the total sequence length.

The key insight is that information can still flow across the entire sequence through stacked layers. With a window of W = 4,096 and k layers stacked, the effective attention span is W × k. For Mistral 7B with 32 layers, this gives a theoretical span of approximately 131,000 tokens even though each layer only looks at 4,096.

Interleaved Local-Global Attention (Gemma 2 and Gemma 3)

Google's Gemma models took a more nuanced approach by interleaving two types of attention layers:

Local layers use sliding window attention: they only look at nearby tokens, so their KV cache is small and fixed. Global layers use full attention: they look at all tokens, requiring a larger KV cache. By making most layers local (5:1 ratio in Gemma 3), the total KV cache is drastically smaller.

PagedAttention — Virtual Memory for AI

Everything we've discussed so far makes the KV cache theoretically smaller. But there's a different, more mundane problem: even when the cache should fit in GPU memory, the software managing that memory does an awful job. Chunks get allocated, freed, re-allocated in different sizes — and before long, the GPU's memory looks like a parking lot full of oddly-spaced cars with no room for a new one.

In September 2023, Woosuk Kwon and colleagues at UC Berkeley published a fix that was hiding in plain sight for decades — in your computer's operating system, of all places. The idea: treat GPU memory for KV cache the same way Linux treats RAM. Break it into fixed-size pages. Don't require them to be contiguous. Map them with a lightweight table. They called it PagedAttention.

The Problem: Memory Fragmentation

In traditional LLM serving, the system pre-allocates a contiguous block of memory for each request's KV cache based on the maximum possible sequence length. This leads to three types of waste:

The paper found that existing systems waste 60–80% of KV cache memory!

The Solution: Paging

PagedAttention divides the KV cache into fixed-size blocks (like pages in virtual memory). These blocks don't need to be contiguous in physical GPU memory — they can be scattered anywhere, connected by a simple lookup table. This eliminates fragmentation almost entirely.

Results

The resulting system, called vLLM, posted clear gains:

vLLM became one of the most widely adopted open-source LLM serving frameworks. PagedAttention's approach to KV cache memory management has been adopted by several other serving systems including TensorRT-LLM and SGLang.

KV Cache Eviction — Keeping Only What Matters

Another powerful strategy: instead of caching Keys and Values for every token, only keep the ones that actually matter. Research has shown that attention is highly concentrated — a small fraction of tokens receive the vast majority of attention.

H₂O: Heavy-Hitter Oracle (NeurIPS 2023)

Zhang et al. analyzed how attention is distributed across tokens and found a clear pattern: the vast majority of the total attention score is concentrated on a tiny fraction of tokens — what they call "heavy hitters." Most tokens contribute almost nothing to the final output. Based on this observation, their H₂O system retains only these high-impact tokens alongside a window of recent tokens in the KV cache, discarding everything else.

With only 20% heavy hitters retained, H₂O improved throughput by up to 29× over DeepSpeed Zero-Inference and Hugging Face Accelerate on OPT-6.7B and OPT-30B models.

Scissorhands (NeurIPS 2023)

Liu et al. proposed the "Persistence of Importance" hypothesis: tokens that were important at one generation step will continue to be important in future steps. This insight allows the system to identify important tokens early and keep only those.

Scissorhands achieves up to 5× KV cache reduction without quality loss, and when combined with 4-bit quantization, reaches 20× compression.

StreamingLLM: Infinite Context with Fixed Cache (ICLR 2024)

Xiao et al. at MIT uncovered a curious behavior in transformer models: no matter what text you feed them, the very first few tokens in the sequence always attract unusually high attention scores — even when those tokens carry no meaningful content (like a period or a padding symbol). The researchers named these positions "attention sinks." By preserving just these initial sink tokens alongside a sliding window of the most recent tokens, the model can process infinitely long sequences with a fixed memory budget.

StreamingLLM enabled models like Llama-2, MPT, and Falcon to process up to 4 million tokens with stable performance, achieving 22.2× speedup over sliding window recomputation baselines.

SnapKV (NeurIPS 2024)

Li et al. noticed something surprising: even before the model starts generating new tokens, the pattern of which prompt tokens each attention head focuses on is already stable and predictable. By examining attention weights in a small window near the end of the prompt, SnapKV can anticipate which earlier tokens will matter most and pre-select them for caching, discarding the rest.

PyramidInfer and PyramidKV

Two complementary papers discovered that the number of important tokens decreases across layers. Lower layers need more cached tokens, while higher layers can function with very few. This pyramidal pattern allows allocating more cache budget to lower layers and less to higher layers.

PyramidInfer achieved 2.2× throughput improvement with over 54% GPU memory reduction. PyramidKV matched full KV cache performance while retaining only 12% of the cache.

KV Cache Quantization — Shrinking Each Number

Every approach so far has tried to cache fewer things. Quantization takes a completely different angle: cache the same things, but represent each number using fewer bits. It's the difference between storing fewer photos versus compressing each photo. In standard deployments, each number in the KV cache occupies 16 bits (FP16). The question quantization researchers ask: can we get away with 8, 4, or even 2 bits?

KIVI: 2-Bit Quantization (ICML 2024)

Liu et al. stumbled onto something that previous quantization work had missed. When you look at the raw numbers inside Keys and Values, they have different shapes of outliers. The Key matrix has certain channels (columns) that consistently contain huge values across all tokens — so you need to calibrate each channel separately. The Value matrix is the opposite: certain tokens (rows) spike across all channels. KIVI exploits this asymmetry by using per-channel quantization for Keys and per-token quantization for Values, squeezing both down to just 2 bits per number — a raw 8× compression from FP16.

Across LLaMA, Falcon, and Mistral model families, KIVI preserved benchmark accuracy with negligible degradation — all without any retraining or fine-tuning step.

KVQuant: Toward 10 Million Context (NeurIPS 2024)

Hooper et al. pushed quantization further with several innovations: per-channel key quantization, pre-RoPE key quantization, non-uniform quantization, and per-vector dense-and-sparse quantization.

The results enabled:

• Less than 0.1 perplexity degradation at 3-bit quantization
• LLaMA-7B with 1 million context on a single A100-80GB GPU
• Up to 10 million context on an 8-GPU system
• 1.7× speedup with custom CUDA kernels vs FP16 baseline

TurboQuant: Near-Optimal Compression (2025)

Google's TurboQuant represents the cutting edge of KV cache quantization. Unlike previous methods that required careful calibration data or model-specific tuning, TurboQuant is completely data-oblivious — it works on any model without seeing any data.

The method works by:

1. Randomly rotating input vectors to distribute information evenly across coordinates
2. Applying optimal scalar quantizers to each coordinate independently
3. Using a 1-bit Quantized JL transform on residuals for unbiased inner products

Bonsai 8B + Turbo1Bit: 1-Bit Weights Meet KV Cache Compression

All the quantization methods above shrink the KV cache. But what happens when you also shrink the model weights to the extreme? In mid-2025, Caltech spinoff PrismML released Bonsai 8B — a language model where every single weight is represented as just one bit: either +1 or −1. An 8.2 billion parameter model that fits in 1.15 GB (compared to ~16 GB for a standard FP16 8B model). That's a 14× size reduction for the weights alone.

But here's the catch that relates to our KV cache story: 1-bit weights do NOT mean 1-bit KV cache. The weights (W_K, W_V, W_Q, etc.) are 1-bit, but when they multiply against the input activations, the resulting Key and Value vectors are still full-precision floating point numbers. So even though Bonsai's weights are tiny, the KV cache during inference remains FP16-sized and grows linearly with context length — exactly the same memory wall problem.

This is where Turbo1Bit comes in: an open-source project that stacks TurboQuant-style KV cache quantization on top of Bonsai's 1-bit weights.

Turbo1Bit's KV cache compression findings for 1-bit models specifically:

Quantization Comparison Overview

FlashAttention — Making Hardware Work Smarter

FlashAttention is not a KV cache compression technique. But it changed how attention (and the KV cache) is read from GPU memory, making everything faster without touching the model architecture itself.

FlashAttention (NeurIPS 2022)

Tri Dao et al. observed that standard attention implementations waste enormous time moving data between GPU memory (HBM) and the fast on-chip memory (SRAM). FlashAttention uses a technique called "tiling": instead of computing the entire attention matrix at once (which requires materializing an enormous n×n matrix), it processes the attention in small blocks that fit in the fast on-chip SRAM.

Key result: 2–4× speedup over optimized baselines, with linear memory usage instead of quadratic. This means the n×n attention matrix is never fully materialized in GPU memory.

FlashAttention-2 (2023)

Dao improved upon the original with better parallelism and work partitioning across GPU thread blocks.

FlashAttention is now integrated into most major LLM frameworks (vLLM, Hugging Face Transformers, PyTorch) and is widely used for reducing memory overhead and improving throughput during both training and inference.

Model-by-Model Comparison — The Complete Picture

Let's bring everything together. The table below covers every major open model family from the original baseline through to mid-2026, showing exactly how each one handles the KV cache challenge.

Part A: Legacy & Foundational Models (2023–2024)

Part B: Frontier Models (2025–2026)

Exact KV Cache Memory Requirements — Calculated from config.json

Below are KV cache memory calculations derived from published config.json files where available. Rows marked with * are estimates due to hybrid architectures (mixed head dimensions, KV sharing, or sliding windows) where an exact single number is not straightforward.

What the Table Reveals

Visual Summary: How Each Technique Reduces KV Cache

Kimi K2.5 — How Reasoning Changes for AI Agents

Everything we've covered so far treats the model as a single entity: one prompt in, one response out. But the newest frontier in AI — agentic workflows — changes the game entirely. An AI agent doesn't just answer a question; it plans, breaks problems into steps, calls external tools (web search, code execution, APIs), reads the results, reasons about them, and repeats this cycle potentially hundreds of times. This creates a completely different stress profile for KV cache.

Moonshot AI's Kimi K2.5 (February 2026) is the first major open model specifically designed to handle this agentic paradigm at scale. Its technical report lays out how reasoning, memory, and KV cache management work differently when a model operates as an agent rather than a chatbot.

The Problem: Why Agents Break Traditional KV Cache

In a standard chatbot conversation, the context grows slowly — one user message, one model response, repeat. The KV cache grows predictably and stays within the context window.

An agent is nothing like this. Consider an agent researching a topic:

After 200 tool calls, the context might contain millions of tokens — most of which are tool outputs that the model only needed briefly. The KV cache balloons, latency spikes, and eventually the model hits its context limit and breaks down.

Kimi K2.5's Three-Part Solution

1. Agent Swarm — Parallel Decomposition

Instead of a single agent chewing through a problem step-by-step (which creates one enormous, ever-growing KV cache), K2.5 introduces Agent Swarm: a framework where a central orchestrator breaks a complex task into independent sub-problems, then spawns multiple sub-agents to work on them simultaneously.

The result: 4.5× latency reduction over single-agent baselines on complex web search tasks, and a 17.8% accuracy improvement on BrowseComp (60.6% → 78.4%).

2. PARL — Teaching the Model When to Parallelize

The orchestrator doesn't use hand-coded rules to decide when to spawn sub-agents. Instead, Moonshot trained it with a custom reinforcement learning method called Parallel-Agent RL (PARL). The reward function has three components:

A critical design choice: sub-agents are frozen (their weights don't update during PARL training). Only the orchestrator learns. This avoids a nasty problem in multi-agent RL where you can't tell which agent deserves credit for a good outcome.

3. Toggle — Budget-Aware Reasoning

Kimi K2.5 uses a training technique called Toggle that alternates between two modes:

Why this matters for KV cache: reasoning tokens are expensive. A "thinking" model like K2.5 might generate 36,000 internal reasoning tokens before producing a 200-token answer. Those 36K tokens all live in the KV cache. Toggle teaches the model to be frugal — use 7K tokens for easy sub-tasks, save the 36K budget for genuinely hard ones. At ~70 KB per token (MLA), that's the difference between 490 MB and 2.5 GB of KV cache just for the thinking phase.

Discard-All Context Management

For extremely long agent sessions, K2.5 uses a strategy called "Discard-all": after each major reasoning phase, the model discards the full context (including tool outputs) and starts fresh with only the accumulated conclusions. This effectively resets the KV cache periodically rather than letting it grow to the context limit.

On the BrowseComp benchmark, using Discard-all boosted accuracy from 60.6% to 74.9% — beating GPT-5.2's reported 65.8%, Claude Opus 4.5 (37.0%), and Gemini 3 Pro (37.8%). The surprising part: throwing away context helped, because a bloated context full of stale tool outputs actually confuses the model more than it helps.

The KV Cache Implications

The State of the Art (2026) & Future Directions

As of early 2026, the KV cache problem that once threatened to halt the scaling of large language models has been substantially addressed through a multi-pronged approach. Here's where things stand:

What's Working Today

The Emerging Frontier

1. Trillion-Parameter MLA: Moonshot AI's Kimi K2 pushed MLA to 1.04 trillion parameters with 384 experts — proving that DeepSeek's latent compression scales far beyond its original 236B deployment. With only 64 attention heads (vs DeepSeek-V3's 128), K2 squeezes even more efficiency out of MLA.

2. Ten-Million-Token Context: Meta's public LLaMA 4 Scout model card reports a 10 million token context window. That is roughly 5,000 pages of text in a single prompt, even though the public card does not spell out every attention/KV implementation detail behind it.

3. On-Device Intelligence: Gemma 4 E2B and E4B target mobile and edge deployments with 128K+ context. Their small KV budgets come from low KV-head counts, local-global attention, KV sharing across layers, and per-layer embeddings. Separately, Gemma 4 26B-A4B is the MoE family member that uses sparse routing (3.8B active of 26B); that MoE detail should not be attributed to the E2B/E4B variants. Post-training KV cache quantization methods (such as TurboQuant or KIVI) could further reduce memory, though Google's Gemma 4 documentation does not cite any specific KV quantization scheme as part of the on-device strategy.

4. Aggressive KV Head Reduction: Qwen3's MoE variant uses just 4 KV heads with 64 query heads — a 16:1 ratio. This trend toward extreme Q-to-KV ratios suggests the field is converging on the insight that you need very few KV heads, even for models with many query heads.

5. Data-Oblivious Quantization: TurboQuant requires no training data or model-specific tuning, making it a universal plug-in for any deployment. Combined with architectural improvements, effective KV cache reductions of 50–100× over the 2017 baseline are now routine.

The Journey in Numbers

References