Repository navigation
1.6.0: persistent cache writes ~2 TB/day to the internal SSD; about half of state write-behinds unneeded #720
Description
Environment: Splash 1.6.0, Apple M5 Max 128 GB, macOS 27.2, ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF:Q2_0, --max-context 256K --state-format bf16 --persistent-cache --max-cache-disk 256G, run under launchd KeepAlive. Load: about 1,300 requests an hour from coding agents, 4 concurrent, prompts averaging 41K tokens. --max-cache-disk-writes left at its default (0, no limit).
What we measured (/status after 22 h of uptime, 0 failed requests, 98% prefix-cache hit rate):
| counter | value |
|---|---|
disk.written_bytes |
1.85 TB |
disk.read_bytes |
0.93 TB |
disk.limit_bytes |
256 GiB |
disk.write_behind.durable / unneeded |
11,388 / 10,716 |
disk.taken_back.bytes |
139 GB |
disk.kv_demotions / kv_demotions_refused |
29,995 / 1,376 |
state.offloads |
24,571 |
That is about 2 TB/day written to the internal SSD, more than twice what is read back, and about 7× the cache's own size each day. Nearly half of the state write-behinds turned out to be unneeded. At this rate a 2 TB internal Apple SSD would use up its rated endurance within a few years, and an agent-serving Mac is exactly the kind of machine that runs 24/7.
Proposals:
- Write a state to disk only once it has survived a short time in memory, or once it is about to be evicted, rather than on publication. Most of the states counted as
unneededwere presumably superseded by the next turn of the same conversation. - Give
--max-cache-disk-writesa non-zero default, such as a fraction of the volume per day. - Add a projected daily write rate to
/statusand/metrics, so operators notice before the hardware does.
Glad to collect more counters or run an A/B with a build that changes the write-behind policy.
Activity
jianc99 commentedon Nov 8, 2026
Thanks for the detailed counters, we reproduced this.
Why it writes this much. These models have linear-attention layers, so resuming from a cached prefix needs a state snapshot as well as the KV: about 62 MB per point on Flash-Next. Most of your 1.85 TB is states, written in two ways:
One note:
write_behind.unneededcounts points that were skipped, because the conversation went on within those 10 s. They are writes already saved, not wasted ones.What we found. In our coding-agent simulations, two kinds of write are rarely read back:
In chat-style use, where someone edits or regenerates an earlier message, those older states are what gives the hit. So we want to cut these writes without hurting the hit rate.
Next release. We plan to write fewer superseded states, keeping branch points and one state every so many tokens, and to delay persistent copies longer. We will also add a write rate to
/status. Each change will be measured for hit rate and TTFT as well as bytes. PRs are welcome, and once we have a test build we'd be glad to have you run the A/B you offered.Until then,
--max-cache-disk-writescaps daily writes, at some cost in hit rate.