Cache Atlas
note 01

Why KV cache papers matter

The important shift is this:

as context windows grow, cache memory becomes a product constraint, not just a systems detail

That is why the last stretch of serving papers keeps circling the same family of ideas:

I started with this map because the individual papers make more sense once the cache itself is treated as a product constraint.

first implementation MiniCache

paperarXiv:2405.14366
next noteReproducing MiniCache in PyTorch

One way to read the space: most cache work compresses or drops tokens along sequence length, while MiniCache asks what can be compressed across depth.

The KV cache is not optional

Autoregressive decoding without a cache is the obvious baseline and the wrong product system.

Every generated token would force the model to recompute key and value state for the entire existing prefix. That makes decode cost grow with sequence length even when almost all of the old state is unchanged.

The cache fixes that by storing old key and value tensors so later decode steps mainly append new state instead of rebuilding the past. The result is the modern serving stack:

The catch is that the cache itself gets expensive fast.

Why the memory bill gets ugly

Cache size grows with several things at once:

That is why serving teams keep running into the same practical problem:

the model still fits, but throughput collapses because memory traffic and cache footprint dominate decode

This is also why there is not one single “KV cache paper.” There is a whole cluster of them.

The main paper buckets

The easiest way to organize the space is by the axis they attack.

1. Token-axis compression and eviction

These papers ask:

which old tokens are worth keeping?

Examples:

This is especially natural for long prompts and vision-language models, where different parts of the prefix are not equally valuable.

2. Depth-axis compression

These papers ask:

how much redundancy exists across layers?

This is the lane that made MiniCache interesting enough to start with. Instead of only asking which tokens to keep, it asks whether nearby layers carry overlapping information that can be compressed more directly.

3. Architecture-specific cache policies

These papers ask:

if the model is multimodal or has uneven layer behavior, should the cache budget be allocated differently?

That is why PrefixKV is compelling even though it is not my first implementation target. It sits right at the intersection of cache policy and vision-language generation.

Why I started with MiniCache

MiniCache is a good first implementation target because the core mechanic can be isolated without reproducing an entire serving stack.

The paper:

Supporting links:

Why it works as a first reproduction:

Tradeoffs:

That tradeoff is acceptable for this project because the goal is to publish code that is readable, testable, and easy to extend.

The implementation intuition: if middle and deep layers are partially redundant, there is room to compress cache state across depth instead of only pruning tokens.

What the public artifacts add

The explanation site and implementation package have different jobs.

Hub repo: Cache Atlas

Cache Atlas adds:

Standalone repo: minicache-pytorch

minicache-pytorch adds:

That split keeps the notes readable while keeping the package focused on code, tests, and reproduction.

Example interactive demo

The first demo does not need to be perfect. It just needs to make the memory problem obvious.

inline demo

Tiny proxy for why cache-budget papers exist at all. Push layer count, kept prefix length, or per-token memory cost, and the memory bill climbs fast.

estimated cache memory14.4 MB

What comes after this

After MiniCache, the next paper track is PrefixKV.

That gives the overall project a clean progression:

  1. general note on why cache papers matter
  2. depth-axis compression with MiniCache
  3. token / layer budget allocation with PrefixKV

That order gives the project both:

Sources