Problem
On high-traffic, long-lived conversations, the leaf compaction trigger (evaluateLeafTrigger() in compaction.ts:386-398) fires every turn because raw message tokens outside the fresh tail constantly exceed leafChunkTokens (default 20K).
Each leaf pass creates a new depth-0 summary inserted via replaceContextRangeWithSummary() (summary-store.ts:934-983), which resequences all ordinals. Since selectOldestLeafChunk() targets the oldest messages (near ordinal 0), this invalidates the Anthropic prompt cache prefix every turn.
Impact
- Prompt cache hit dropped from ~90%+ to 22% on a conversation with 18K messages / 7.5M tokens
- Every turn triggered a leaf pass (1-3% reduction, 60K→58K) that created a new summary, shifting the prefix
- Recovery only happened after
/new (which pruned shallow summaries via retainDepth=2) + config change
Root Cause
The leaf trigger threshold is fixed and doesn't account for:
- Prompt cache stability — small compaction gains (<5%) aren't worth the cache prefix invalidation
- Conversation throughput — high-traffic agents constantly exceed the threshold
- Assembly budget headroom — if context is well under budget (60K vs 200K), compaction provides no value
Suggested Fixes
- Cache-aware skip: if the estimated reduction is <5% of assembled tokens, skip compaction to preserve prefix stability
- Budget headroom check: skip leaf compaction if assembled tokens are well under
contextThreshold * budget (already below the full-sweep trigger)
- Batch leaf compaction: accumulate multiple chunks before compacting, reducing prefix-shifting frequency
- Adaptive threshold: scale
leafChunkTokens with conversation message count or throughput
Environment
Related
Problem
On high-traffic, long-lived conversations, the leaf compaction trigger (
evaluateLeafTrigger()incompaction.ts:386-398) fires every turn because raw message tokens outside the fresh tail constantly exceedleafChunkTokens(default 20K).Each leaf pass creates a new depth-0 summary inserted via
replaceContextRangeWithSummary()(summary-store.ts:934-983), which resequences all ordinals. SinceselectOldestLeafChunk()targets the oldest messages (near ordinal 0), this invalidates the Anthropic prompt cache prefix every turn.Impact
/new(which pruned shallow summaries viaretainDepth=2) + config changeRoot Cause
The leaf trigger threshold is fixed and doesn't account for:
Suggested Fixes
contextThreshold * budget(already below the full-sweep trigger)leafChunkTokenswith conversation message count or throughputEnvironment
Related