Skip to content

[runtime] Add compaction rate-limit guardrail (minIntervalSeconds + maxPerHour) to prevent compaction storms #78367

Description

@blazing-mj

Summary

A single agent session can enter an unbounded compaction loop, consuming significant token budget and producing 4–7 MB checkpoint files within minutes. The runtime currently has no rate limit on compactions per session.

Repro

  1. Configure session.dmScope: "main" (the default in some older configs).
  2. Have a single high-traffic peer (e.g. user on a primary channel) generate sustained inbound while the catch-all agent:<id>:main session is at >70% context.
  3. Each turn ends with a compaction; each compaction restores enough history that the next turn re-fills the budget; the cycle repeats every 1–3 minutes.

Observed in production: one session accumulated 89 compactions in ~24 hours (~2.4MB transcript, ~700MB historic checkpoints).

Root cause

compactionSafeguardExtension in compaction-successor-transcript-ZByj7D6a.js (around line 3592 in current dist) has no minimum-interval check. If the post-compaction state immediately re-trips the threshold, another compaction fires on the very next turn.

Proposed fix

Two new config keys under agents.defaults.compaction:

{
  "minIntervalSeconds": 60,   // refuse compaction if Δt since previous < this
  "maxPerHour": 12             // refuse if rolling-hour count exceeds this
}

When either guard trips:

  • Return { cancel: true, reason: "compaction throttled" } from the safeguard extension
  • Emit a warn-level log entry with the session key and the trip reason
  • Optionally fire a lifecycle event so monitoring can surface it

The default values are conservative: 60s minimum and 12/hour both leave plenty of headroom for legitimate heavy-traffic sessions while turning a runaway loop into a clear, bounded failure mode that surfaces in logs instead of silently burning tokens.

Why "tune maxHistoryShare lower" is not enough

We tested aggressive trimming (maxHistoryShare: 0.4, reserveTokens: 24000). It reduces probability of re-trip but does not eliminate the loop — a session with persistent high inbound rate still compacts every few minutes. A first-class rate limit is the only way to bound it.

Effort

~30 lines source change + the two schema entries. Tests: simulate compaction at t=0, attempt at t=10s with minIntervalSeconds=60 → expect cancel; at t=70s → expect proceed.

Context

Discovered during a P0 incident where Alfred's main session compacted 89× before manual intervention. Full incident notes available on request. Related but distinct from the dmScope: "main" routing magnet (which we resolved with dmScope: "per-channel-peer" in our config); a guardrail at the compaction layer would have caught this even with bad routing config.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions