-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
[runtime] Add compaction rate-limit guardrail (minIntervalSeconds + maxPerHour) to prevent compaction storms #78367
Copy link
Copy link
Closed as not planned
Closed as not planned
Copy link
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Description
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.ClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.Good issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivityMarked as stale due to inactivity
Type
Fields
Priority
None yet
Summary
A single agent session can enter an unbounded compaction loop, consuming significant token budget and producing 4–7 MB checkpoint files within minutes. The runtime currently has no rate limit on compactions per session.
Repro
session.dmScope: "main"(the default in some older configs).agent:<id>:mainsession is at >70% context.Observed in production: one session accumulated 89 compactions in ~24 hours (~2.4MB transcript, ~700MB historic checkpoints).
Root cause
compactionSafeguardExtensionincompaction-successor-transcript-ZByj7D6a.js(around line 3592 in current dist) has no minimum-interval check. If the post-compaction state immediately re-trips the threshold, another compaction fires on the very next turn.Proposed fix
Two new config keys under
agents.defaults.compaction:{ "minIntervalSeconds": 60, // refuse compaction if Δt since previous < this "maxPerHour": 12 // refuse if rolling-hour count exceeds this }When either guard trips:
{ cancel: true, reason: "compaction throttled" }from the safeguard extensionwarn-level log entry with the session key and the trip reasonThe default values are conservative: 60s minimum and 12/hour both leave plenty of headroom for legitimate heavy-traffic sessions while turning a runaway loop into a clear, bounded failure mode that surfaces in logs instead of silently burning tokens.
Why "tune
maxHistorySharelower" is not enoughWe tested aggressive trimming (
maxHistoryShare: 0.4,reserveTokens: 24000). It reduces probability of re-trip but does not eliminate the loop — a session with persistent high inbound rate still compacts every few minutes. A first-class rate limit is the only way to bound it.Effort
~30 lines source change + the two schema entries. Tests: simulate compaction at t=0, attempt at t=10s with
minIntervalSeconds=60→ expect cancel; at t=70s → expect proceed.Context
Discovered during a P0 incident where Alfred's main session compacted 89× before manual intervention. Full incident notes available on request. Related but distinct from the
dmScope: "main"routing magnet (which we resolved withdmScope: "per-channel-peer"in our config); a guardrail at the compaction layer would have caught this even with bad routing config.