Skip to content

fix(compaction): wire aggregate retry timeout to compaction.timeoutSeconds (#94391)#94413

Closed
SymbolStar wants to merge 2 commits into
openclaw:mainfrom
SymbolStar:fix/compaction-aggregate-timeout-94391
Closed

fix(compaction): wire aggregate retry timeout to compaction.timeoutSeconds (#94391)#94413
SymbolStar wants to merge 2 commits into
openclaw:mainfrom
SymbolStar:fix/compaction-aggregate-timeout-94391

Conversation

@SymbolStar

Copy link
Copy Markdown
Contributor

Fixes #94391

Root cause

In-turn compaction on ~200K-token sessions silently failed because the outer aggregate wait was a hardcoded COMPACTION_RETRY_AGGREGATE_TIMEOUT_MS = 60_000 const inside runEmbeddedAttempt (src/agents/embedded-agent-runner/run/attempt.ts). On anthropic/claude-sonnet-4-6 the summary model call runs ~143–187s. The 60s outer wait fired first, marked the attempt incomplete, and discarded the result — even though the trajectory shows model.completed aborted=False externalAbort=False timedOut=False at 174s/143s/187s with valid output the provider had already been paid for.

Subsequent retries then collided with the still-running model call holding the session write lock (DEFAULT_SESSION_WRITE_LOCK_ACQUIRE_TIMEOUT_MS = 60s) and gave up — the "self-compounding retry storm" Path 2 in the issue.

The inner safety wrapper agents.defaults.compaction.timeoutSeconds (default 180s, configurable; reporter set 300s) already governs how long a single compaction model call may run. The outer aggregate wait was simply not wired to it.

Fix

Adopt the reporter's proposed approach: derive the outer aggregate-wait budget from the inner compaction model-call timeout instead of adding a new config surface.

  • New helper resolveCompactionRetryAggregateTimeoutMs(compactionTimeoutMs) in compaction-retry-aggregate-timeout.ts returns max(60s floor, compactionTimeoutMs + 30s margin).
  • attempt.ts already resolves compactionTimeoutMs = resolveCompactionTimeoutMs(params.config) earlier in the function for the lock-hold/grace logic; the new outer wait reuses that value, no extra plumbing.
  • 60s historical floor preserved as a fallback when the inner timeout is missing/non-finite — installs that don't touch compaction.timeoutSeconds get the same behavior they had before.

Behavior matrix:

compaction.timeoutSeconds Old outer wait New outer wait
unset (default 180) 60s ❌ 210s ✅
300 (reporter's install) 60s ❌ 330s ✅
30 (below floor) 60s 60s (floor)

The +30s margin guarantees the inner safety timeout always fires first when a compaction call genuinely hangs — the outer wait never preempts a still-completing call (Path 1 cause), which in turn eliminates the lock contention next-retry sees (Path 2 effect).

Backwards compatibility

  • No config schema change. agents.defaults.compaction.timeoutSeconds is already a documented public knob (used by compaction-safety-timeout.ts).
  • Default install behavior changes from "60s outer wait abandons valid 150s+ compactions" to "210s outer wait waits for the inner cap to decide" — that is the intended fix; sessions that previously failed silently now succeed.
  • isCompactionStillInFlight extension behavior is unchanged: while compaction is still running, the timer is restarted on each idle iteration just like before.

Cancellation on abandon

The reporter also flagged that abandoning the wait without cancelling the model call wastes the API call. The existing inner wrapper compactWithSafetyTimeout already wires through an AbortSignal that fires when the inner timeout trips, and that signal is threaded into the underlying compact() call. With this PR the outer wait now waits long enough for that inner signal to be the one that fires on a true hang, so abandons happen with a real cancel rather than the outer timer leaving the call orphaned. Wiring an explicit cancel into the outer aggregate-wait abandon path would require crossing the waitForCompactionRetry abstraction (it does not currently expose a cancel) and is left for a follow-up if that path proves still reachable in practice.

Tests

pnpm vitest run src/agents/embedded-agent-runner/run/compaction-retry-aggregate-timeout.test.ts — 13 tests pass (was 8).

Related

@clawsweeper

clawsweeper Bot commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

Close as superseded: the branch is a focused attempt, but it remains a timeout-scaling fix while the open proof-positive maintainer candidate fixes the actual active-retry predicate and is the safer canonical landing path.

Root-cause cluster
Relationship: superseded
Canonical: #94421
Summary: This PR is superseded by the open proof-positive maintainer candidate that fixes the same linked compaction retry bug with the active session-streaming predicate.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Canonical path: Close this branch and continue review on #94421 as the canonical fix for the linked compaction retry bug.

So I’m closing this here and keeping the remaining discussion on #94421.

Review details

Best possible solution:

Close this branch and continue review on #94421 as the canonical fix for the linked compaction retry bug.

Do we have a high-confidence way to reproduce the issue?

Yes, at source level: current main races the aggregate wait against retry completion and only extends while compaction itself is in flight, while the linked issue supplies production trajectories with valid provider results after 60 seconds. I did not run a paid 200K-token provider reproduction in this read-only review.

Is this the best way to solve the issue?

No. Scaling the timer is a plausible mitigation, but the better fix is the active-work predicate in #94421 because it preserves the idle deadlock guard and tracks the actual retry stream.

Security review:

Security review cleared: Security review cleared: the diff only changes TypeScript agent runtime and test logic, with no dependency, workflow, secret, package, or code-execution surface changes.

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • steipete: Authored the viable open candidate fix that tracks active compaction retry work and supplied land-ready proof; also appears in recent history for the affected agent path. (role: current fix owner and recent area contributor; confidence: high; commits: ee568b5fc2ca, c61a6ed09a53; files: src/agents/embedded-agent-runner/run/attempt.ts, src/agents/embedded-agent-runner/run/compaction-retry-aggregate-timeout.ts)
  • cgdusek: Authored merged PR 40324, which added the bounded aggregate compaction retry wait behavior that this cluster refines. (role: introduced predecessor behavior; confidence: medium; commits: 54be30ef89f5; files: src/agents/pi-embedded-runner/run/attempt.ts, src/agents/pi-embedded-runner/run/compaction-retry-aggregate-timeout.ts)
  • Vincent Koc: Current-main blame maps the affected embedded runner helper and caller through a recent session-helper refactor, making this a routing hint rather than root-cause attribution. (role: recent current-main carrier; confidence: medium; commits: 0da706dbfb10; files: src/agents/embedded-agent-runner/run/attempt.ts, src/agents/embedded-agent-runner/run/compaction-retry-aggregate-timeout.ts)

Codex review notes: model internal, reasoning high; reviewed against 60d6a8a89d47.

@SymbolStar
SymbolStar force-pushed the fix/compaction-aggregate-timeout-94391 branch from 6da889f to c1f5d8e Compare June 18, 2026 06:52
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jun 18, 2026
@steipete

Copy link
Copy Markdown
Contributor

Superseded by #94421, landed as 9a57139. The landed fix follows both compaction and the active retry stream, preserving deadlock recovery when both lifecycle facts are idle.

Thank you @SymbolStar for investigating #94391 and contributing a fix.

@steipete steipete closed this Jun 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

2 participants