⚡ LLM Cache Lab

Measure — and actually verify — how prompt caching changes LLM serving cost, latency, and memory.

The headline experiment runs real inference on a local Ollama server, evicts the model, and compares a cold (uncached) prefix against a warm (cached) one. No API keys, no cloud bill.

Real result (measured, not simulated)

llama3.2:3b, 1,811-token prefix, 3 samples each:

Metric	Cold (evicted)	Warm (cached)	Speedup
Prefill (prompt_eval)	5,283.8 ms	32.5 ms	162.8x
Time to first token	6,990.5 ms	267.5 ms	26.1x
End-to-end	7,646.4 ms	1,131.5 ms	6.8x

• Effective cached prefix: 99.4% (~1,800 tokens skipped)
• Wall-clock saved per call: 6,514.8 ms
• Eviction after keep_alive=0s confirmed: True

Reproduce it yourself:

ollama serve                      # one terminal
ollama pull llama3.2:3b           # once
python scripts/run_experiment.py --model llama3.2:3b --prefix-tokens 2000

The full report is written to results/prefix_cache_llama3.2_3b.md. See results/prefix_cache_llama3.2_3b.json for the raw per-call timings.

How the cache is proven

Ollama returns real timing fields, and the cache signal is prompt_eval_duration — the time to prefill the prompt. When a repeated prefix is served from the KV cache it collapses even though prompt_eval_count (token count) is unchanged. This Ollama version ignores the cache_prompt option, so a call is only genuinely cold after the model is evicted with keep_alive=0s. The experiment performs that eviction before every cold sample.

---

What the project measures

Axis	What we measure	Where
⏱️ Real latency & caching	Cold vs warm prefill, TTFT, end-to-end	experiment.py + Ollama
💸 Cost	USD split by fresh / cache-write / cache-read / output tokens	cost.py + bench.py (modeled)
🧠 Memory	KV cache bytes for a model, context length, batch, dtype	kv_cache.py

Two providers:

• Ollama (providers/ollama.py) — real local inference; the source of truth.
• Mock (providers/mock.py) — an offline simulator so the suite, dashboard, and CI run anywhere without a server or downloads.

---

Why caching matters

Prompt caching (latency + cost)

When a request shares a long prefix with a previous one — a system prompt, a retrieved document, a few-shot block — the computed attention state can be reused. The first call pays a cache-write premium; later calls pay a cache-read discount (commonly ~10% of the normal input price).

KV cache (memory)

During decoding, every layer stores a Key and Value tensor per token:

KV bytes = 2 · n_layers · n_kv_heads · head_dim · dtype_bytes · batch · seq_len

Grouped-query attention (GQA/MQA) reduces n_kv_heads, and quantizing the cache (fp16 → fp8 → int4) cuts memory further. This project computes those numbers for real model configs.

---

Quickstart

git clone <your-repo-url> llm-cache-lab
cd llm-cache-lab

python -m venv .venv && source .venv/bin/activate
pip install -e ".[dev,dashboard]"

# 1) real cache experiment (needs `ollama serve` + a pulled model)
python scripts/run_experiment.py --model llama3.2:3b --prefix-tokens 2000

# 2) offline cost/latency benchmark (no server needed)
python scripts/run_bench.py --provider mock --pricing mock-mid --prefix-tokens 8000 --calls 5

# 3) explore interactively
streamlit run dashboard/app.py

# 4) tests
pytest

Modeled cost example (--prefix-tokens 8000, mock-mid)

strategy	call	fresh in	cached in	TTFT (ms)	cost
baseline	0	8000	0	4915	$0.02522
baseline	1	8000	0	4503	$0.02469
cached	0	8	0	273	$0.03070
cached	1	8	7992	268	$0.00344

The cache-write call costs slightly more; every cache-hit call is ~7x cheaper and ~17x faster to first token.

---

Dashboard

streamlit run dashboard/app.py gives five tabs:

• 💸 Cost vs hit rate — analytic sweep of cost at 0–100% cache hit rate.
• ⏱️ Latency — runs the mock benchmark and charts TTFT and per-call cost.
• 🦙 Real Ollama cache — runs the live cold-vs-warm experiment and shows the measured prefill speedup.
• 🧠 KV cache memory — KV size by model and context length, and the effect of fp16 → fp8 → int4.
• 🔢 Raw data — the underlying tables, downloadable as CSV.

---

Repo layout

llm-cache-lab/
├── configs/
│   ├── models.json             # architecture params (layers, kv heads, head dim)
│   └── pricing.json            # per-1M-token prices + cache multipliers
├── dashboard/app.py            # Streamlit dashboard
├── scripts/
│   ├── run_experiment.py       # real cold-vs-warm cache experiment (Ollama)
│   └── run_bench.py            # offline cost/latency benchmark
├── src/llm_cache_lab/
│   ├── experiment.py           # real cache metrics + Markdown report
│   ├── kv_cache.py             # KV memory math
│   ├── cost.py                 # cached vs uncached cost accounting
│   ├── models.py               # config/pricing loaders
│   ├── bench.py                # offline benchmark runner + CLI
│   ├── report.py               # analytic sweeps for charts
│   └── providers/
│       ├── base.py             # Provider protocol, CacheMode, Completion
│       ├── mock.py             # offline latency/cost simulator
│       └── ollama.py           # real local inference + eviction
├── tests/
└── results/                    # sample real run committed; rest gitignored

---

How the metrics are computed

• Real cache effect — experiment.py takes the median prompt_eval_duration of warm and cold samples; prefill_speedup = cold / warm and cached_fraction = 1 − warm / cold. Nothing is assumed.
• Cost — cost.py bills fresh input, cache-read, cache-write, and output tokens separately using configs/pricing.json (modeled).
• KV size — kv_cache.py implements the formula above with exact byte math; see tests/test_kv_cache.py for hand-checked values (e.g. Llama-3.1-8B is 128 KiB of KV per token at fp16).

Mock prices and timings are illustrative defaults, not billing-grade. The Ollama experiment is the ground truth for latency and caching on your own hardware.

---

Roadmap

• Add more models and context lengths to the experiment matrix.
• Compare cached vs uncached throughput at fixed memory.
• Optional real API adapters (OpenAI/Anthropic/DeepSeek) reading their actual cached_tokens fields.
• Export charts to PNG/SVG for the README.

License

MIT