On this page
The state that grows while the model runs
Model weights are the obvious memory bill. The KV cacheSaved attention keys and values from earlier tokens, reused when generating the next token. is the bill that keeps growing with the workload. Long contexts and more simultaneous requests add state even when the model itself has not changed.
That is what drew me to EigenKache. I wanted to understand how much old attention state could fit into a smaller representation, and what information would disappear when I compressed it. Whether the cache becomes the first bottleneck depends on the architecture, workload, precision, and serving implementation. It is a quantity to calculate, not an assumption to make from the model name.
Count keys and values, not query heads
During autoregressive decoding, each layer stores keys and values for previously processed tokens. A new token reuses those tensors instead of recomputing all the old K and V projections. It still attends to the cached history. With full attention, attention work per new token grows with that history, and cumulative attention over a growing sequence remains quadratic.
× bytes per element × total cached tokens
The leading 2 is for keys and values. For FP16 or BF16, each element is two bytes, so K plus V contributes four bytes. FP32 instead uses four bytes per element, or eight for the pair. In grouped-querySeveral query heads share each set of key and value heads. or multi-query attention, count KV heads, not the larger number of query heads.
If every sequence has the same length, total cached tokens is batch size times sequence length. For a ragged batch, sum the lengths. This estimates the unquantized tensor payload; allocation waste, block metadata, workspaces, sharing, and sharding change the memory a particular system reserves. How caching works
For an illustrative 32-layer architecture with eight KV heads, head dimension 128, and two-byte elements, four sequences of 8,192 tokens require 4 GiB of KV payload. Double the cached tokens and the payload doubles. No model run is needed to see the capacity pressure.
Summarize the cold middle
EigenKache keeps the beginning of the context and the recent tail exact. It scores the colder middle using attention from recent queries, keeps a small exact subset, and partitions the rest into segments with roughly equal attention mass. Each segment becomes a salience-weighted landmark key and value.
The attraction is coverage: a compact summary might preserve information spread across many similar tokens. The risk is equally concrete. Averaging changes the vectors used by softmax attention, so a landmark cannot be assumed to preserve what a future query needs. Selection and compression also take work. Fewer cache entries alone do not prove a faster decoder.
Two traces reverse the ranking
The committed paper reports a CPU-reference study of attention-output fidelity, comparing a sink-plus-window policy, an H2O-like salience-eviction policy, and landmarks at matched retention budgets. Its random trace has 96 tokens and dimension 128; its structured trace has 512 tokens and dimension 128, with a low-rank cold region. These are synthetic traces, not a long-context model-quality benchmark.
Reported synthetic attention-output cosine similarity
| Trace · retained | Window | H2O-like | Landmark |
|---|---|---|---|
| Random · 25% | 0.558 | 0.657 | 0.598 |
| Random · 33% | 0.653 | 0.804 | 0.683 |
| Random · 50% | 0.858 | 0.872 | 0.857 |
| Structured · 25% | 0.867 | 0.692 | 0.902 |
| Structured · 33% | 0.930 | 0.736 | 0.944 |
| Structured · 50% | 0.965 | 0.857 | 0.977 |
At 25% retention, H2O-like eviction leads on the random trace: 0.657 versus 0.598 for landmarks. On the structured trace, landmarks lead: 0.902 versus 0.692 for H2O-like and 0.867 for the window. The same budget favors different policies in the two setups.
That is the result worth keeping. Compression has a plausible advantage when there is useful structure to summarize, and it can lose when averaging destroys information. These two examples do not identify a universal boundary or prove that real model traces fall on the favorable side.
What would make this a serving result
The repository describes a CPU reference and a CUDA-oriented design. Real-model task quality and end-to-end GPU decode latency remain the decisive tests. Attention-output cosine is a local diagnostic, not a substitute for task accuracy, perplexity, or a user-visible latency distribution.
A fair next comparison would fix the model and workload, give every policy the same KV budget, and count compression overhead as well as attention time. It would report retained quality, peak memory, and decode latency together. That is where a smaller representation either earns its cost or does not.
I started with the memory wall. The useful question now is which information is worth the bytes, and whether the cost of choosing it survives contact with a real decoder.
Inspect the example
Change the architecture, dtype, or number of cached tokens to inspect the ideal payload size. This arithmetic example is separate from the synthetic attention-output experiment.
Python source
# Dense, unquantized KV payload only: no allocator or workspace overhead.
def kv_bytes(layers, kv_heads, head_dim, bytes_per_element, total_cached_tokens):
values = (layers, kv_heads, head_dim, bytes_per_element, total_cached_tokens)
if any(not isinstance(x, int) or isinstance(x, bool) or x < 0 for x in values):
raise ValueError("Use nonnegative integer dimensions and byte counts")
return 2 * layers * kv_heads * head_dim * bytes_per_element * total_cached_tokens
# Illustrative architecture, not a measured model configuration.
layers, kv_heads, head_dim, dtype_bytes = 32, 8, 128, 2
for sequence_length in (1024, 8192, 32768):
total_tokens = 4 * sequence_length
size = kv_bytes(layers, kv_heads, head_dim, dtype_bytes, total_tokens)
print(f"batch=4, sequence={sequence_length:5}: {size / 2**30:.2f} GiB")
assert kv_bytes(32, 8, 128, 2, 4 * 8192) == 4 * 2**30Output from the included example
batch=4, sequence= 1024: 0.50 GiB batch=4, sequence= 8192: 4.00 GiB batch=4, sequence=32768: 16.00 GiB
Run this example on the site · Download notebook Download Python