fix(compaction): wire aggregate retry timeout to compaction.timeoutSeconds (#94391)#94413
fix(compaction): wire aggregate retry timeout to compaction.timeoutSeconds (#94391)#94413SymbolStar wants to merge 2 commits into
Conversation
|
Thanks for the context here. I swept through the related work, and this is now duplicate or superseded. Close as superseded: the branch is a focused attempt, but it remains a timeout-scaling fix while the open proof-positive maintainer candidate fixes the actual active-retry predicate and is the safer canonical landing path. Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Canonical path: Close this branch and continue review on #94421 as the canonical fix for the linked compaction retry bug. So I’m closing this here and keeping the remaining discussion on #94421. Review detailsBest possible solution: Close this branch and continue review on #94421 as the canonical fix for the linked compaction retry bug. Do we have a high-confidence way to reproduce the issue? Yes, at source level: current main races the aggregate wait against retry completion and only extends while compaction itself is in flight, while the linked issue supplies production trajectories with valid provider results after 60 seconds. I did not run a paid 200K-token provider reproduction in this read-only review. Is this the best way to solve the issue? No. Scaling the timer is a plausible mitigation, but the better fix is the active-work predicate in #94421 because it preserves the idle deadlock guard and tracks the actual retry stream. Security review: Security review cleared: Security review cleared: the diff only changes TypeScript agent runtime and test logic, with no dependency, workflow, secret, package, or code-execution surface changes. AGENTS.md: found and applied where relevant. What I checked:
Likely related people:
Codex review notes: model internal, reasoning high; reviewed against 60d6a8a89d47. |
6da889f to
c1f5d8e
Compare
|
Superseded by #94421, landed as 9a57139. The landed fix follows both compaction and the active retry stream, preserving deadlock recovery when both lifecycle facts are idle. Thank you @SymbolStar for investigating #94391 and contributing a fix. |
Fixes #94391
Root cause
In-turn compaction on ~200K-token sessions silently failed because the outer aggregate wait was a hardcoded
COMPACTION_RETRY_AGGREGATE_TIMEOUT_MS = 60_000const insiderunEmbeddedAttempt(src/agents/embedded-agent-runner/run/attempt.ts). Onanthropic/claude-sonnet-4-6the summary model call runs ~143–187s. The 60s outer wait fired first, marked the attempt incomplete, and discarded the result — even though the trajectory showsmodel.completed aborted=False externalAbort=False timedOut=Falseat 174s/143s/187s with valid output the provider had already been paid for.Subsequent retries then collided with the still-running model call holding the session write lock (
DEFAULT_SESSION_WRITE_LOCK_ACQUIRE_TIMEOUT_MS= 60s) and gave up — the "self-compounding retry storm" Path 2 in the issue.The inner safety wrapper
agents.defaults.compaction.timeoutSeconds(default 180s, configurable; reporter set 300s) already governs how long a single compaction model call may run. The outer aggregate wait was simply not wired to it.Fix
Adopt the reporter's proposed approach: derive the outer aggregate-wait budget from the inner compaction model-call timeout instead of adding a new config surface.
resolveCompactionRetryAggregateTimeoutMs(compactionTimeoutMs)incompaction-retry-aggregate-timeout.tsreturnsmax(60s floor, compactionTimeoutMs + 30s margin).attempt.tsalready resolvescompactionTimeoutMs = resolveCompactionTimeoutMs(params.config)earlier in the function for the lock-hold/grace logic; the new outer wait reuses that value, no extra plumbing.compaction.timeoutSecondsget the same behavior they had before.Behavior matrix:
compaction.timeoutSecondsThe +30s margin guarantees the inner safety timeout always fires first when a compaction call genuinely hangs — the outer wait never preempts a still-completing call (Path 1 cause), which in turn eliminates the lock contention next-retry sees (Path 2 effect).
Backwards compatibility
agents.defaults.compaction.timeoutSecondsis already a documented public knob (used bycompaction-safety-timeout.ts).isCompactionStillInFlightextension behavior is unchanged: while compaction is still running, the timer is restarted on each idle iteration just like before.Cancellation on abandon
The reporter also flagged that abandoning the wait without cancelling the model call wastes the API call. The existing inner wrapper
compactWithSafetyTimeoutalready wires through anAbortSignalthat fires when the inner timeout trips, and that signal is threaded into the underlyingcompact()call. With this PR the outer wait now waits long enough for that inner signal to be the one that fires on a true hang, so abandons happen with a real cancel rather than the outer timer leaving the call orphaned. Wiring an explicit cancel into the outer aggregate-wait abandon path would require crossing thewaitForCompactionRetryabstraction (it does not currently expose a cancel) and is left for a follow-up if that path proves still reachable in practice.Tests
compaction-retry-aggregate-timeout.test.tsforresolveCompactionRetryAggregateTimeoutMs:compaction.timeoutSeconds=300must no longer trip the 60s legacy budget; the wait must complete cleanly withtimedOut=false.pnpm vitest run src/agents/embedded-agent-runner/run/compaction-retry-aggregate-timeout.test.ts— 13 tests pass (was 8).Related
EMBEDDED_COMPACTION_TIMEOUT_MSinner cap). Not a dup.