Repository navigation
fix(worker): discard a stalled session's buffered work once its resumes are spent - #5414
Merged
thedotmack merged 4 commits intoNov 6, 2026
Merged
Conversation
Contributor
[Medium impact] The PR appears safe to merge; the previous duplicate-send issue is fixed and no new blocking issues were found. SummaryThe PR discards buffered work once a session spends its bounded stall resumes.
The previous duplicate-send finding is fixed. No new actionable issues were found. Reviews (4) · Last reviewed commit: "fix(worker): keep the dedup set when a s..." · Reviewed by Greptile |
…ed-delay resumes are spent (thedotmack#5405)
0xABCD01
force-pushed
the
fix/5405-stall-backoff-resume
branch
from
November 6, 2026 02:47
42a1fe4 to
655e994
Compare
…n a timer (thedotmack#5405) The backoff resumes this PR added still ended in the same park, and a timer that resends stale buffered work is the backlog replay the live-only observer policy (thedotmack#5347) rules out. Once the bounded stall resumes are spent, the session's buffered work is now discarded at once with a WARN that states the pending and parked counts, so nothing waits on an event that may never come. The next captured event still starts fresh work. The teardown timer cleanup and its test are kept. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
…scarded clearPendingForSession also forgot every seen toolUseId, so a later hook or transcript replay could enqueue work that was already stored, or that was just discarded, and send it again. The stall discard now goes through discardBufferedWork, which drops the buffer and keeps the session's dedup set. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
thedotmack
added a commit
that referenced
this pull request
Nov 6, 2026
…iscard Main merged #5383 while the live-only contract was unlanded. It resumed auth-paused work once a later store succeeded, which is backlog replay. Auth now discards its inputs, so that resume path, its pause timestamp, auth scope and per-scope success ledger are removed. Quota and auth discards go through discardBufferedWork, so a replayed toolUseId is still suppressed and cannot be paid for again (Greptile P1). Releasing claims also forgets earliestPendingTimestamp, since released work can expire before it is claimed again (Greptile P2). AGENTS.md names "a later successful store" as one more thing that never authorizes backfill, and the lifecycle doc records #5383 and #5414. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8bpnb8tKFUmq6RZEX8
thedotmack
added a commit
that referenced
this pull request
Nov 6, 2026
* fix(observer): skip unavailable work instead of replaying a backlog Restore live-only observation capture, discard quota/auth backlogs, bound the RAM buffer, and document the observer architecture and queue history. Include regenerated runtime bundles. * fix(build): preserve executable modes on rebuilt runtime bundles * fix(observer): remove auth auto-resume and keep dedup on quota/auth discard Main merged #5383 while the live-only contract was unlanded. It resumed auth-paused work once a later store succeeded, which is backlog replay. Auth now discards its inputs, so that resume path, its pause timestamp, auth scope and per-scope success ledger are removed. Quota and auth discards go through discardBufferedWork, so a replayed toolUseId is still suppressed and cannot be paid for again (Greptile P1). Releasing claims also forgets earliestPendingTimestamp, since released work can expire before it is claimed again (Greptile P2). AGENTS.md names "a later successful store" as one more thing that never authorizes backfill, and the lifecycle doc records #5383 and #5414. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8bpnb8tKFUmq6RZEX8 * test(observer): move old-contract tests to the live-only observer rules 84 tests still asserted the contract this PR removes: quota/auth pauses keeping and later resuming buffered work, observer inputs landing in the disk hook spool, and buffers deeper than the 128-message cap. Rewrite them to assert the live-only contract while keeping what still holds (booking, breaker arming and tagging, admission waits, pacing, dedup, cwd attribution, body limits, transcript cursors). No test deleted. Test fakes that lacked SessionManager.discardBufferedWork now implement it faithfully, and fake sweeps now match the real one (pending work required, quota/auth pauses skipped), so several previously vacuous checks test something again. Remove the GeneratorRunner relabel of a stand-in's auth pause to provider_switch: auth now discards that work first, so the relabel could only ever mark an empty session resumable. Fix the comments that still described keep-and-wait. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8a4P6RjaWFnrGVZXnd * fix(observer): send each tool field once in the live handoff The live handoff spread the camelCase payload and then added the snake_case tool fields the route reads, so toolInput/toolResponse (and lastAssistantMessage) went over the wire twice. A 2.5-5 MiB result passed boundObservationDispatch but doubled past the worker's 5mb JSON cap, and the hook silently skipped it. Drop the camelCase copies. Greptile review. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_019FbY8a4P6RjaWFnrGVZXnd --------- Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #5405
Maintainer update (finished on this branch)
The original version retried a stalled session on a 60/120/240 s backoff timer once the fixed-delay resumes were spent. Two problems:
What this PR now does:
pendingCount,parkedCountanddiscarded. Nothing is left waiting for an event that may never come, and no timer replays it. The next captured event still starts fresh work. No new timer, retry cap or limit is added; the existingMAX_CONSECUTIVE_STALL_RESUMESis unchanged.SessionManager.discardBufferedWork, which keeps the session's seen-toolUseIdset. A later hook or transcript replay can't re-enqueue work that was already stored, or work that was just discarded.deleteSession/removeSessionImmediate) now clearsstallResumeTimerandscheduledResumeTimer, along with its test.Proof (
tests/worker/overflow-recycle-resume.test.ts):tests/worker/session-buffer-disposal-warnings.test.tscovers the dedup.npm run typecheckis clean.Original description
Fixes #5405
When a session's observer went unanswered four times in a row, the stall pacer spent its bounded fixed-delay resumes (3 × 30 s) and parked the session until the next captured event. If the content session had already ended, no event ever came: the preserved messages stayed in the in-memory buffer — still counted in queueDepth — until a worker restart discarded them.
This keeps the fixed-delay resumes, and once they are spent, retries on a backoff timer instead of parking: up to 3 more resumes at 60 s, 120 s, and 240 s, on the same
stallResumeTimerand through the sameensureGeneratorRunningpath. An answered turn resets both stall counters with the other resume debts, so a recovered session starts fresh. Once the backoff is spent too, the session parks exactly as before, with no timer pending. Session teardown (deleteSession/removeSessionImmediate) now clears the stall and scheduled resume timers alongsiderespawnTimer; neither was cleared before.Nothing is discarded and the buffer accounting is unchanged: the messages stay pending until a resumed generator stores them.
#5409 implements the issue's third option (reporting abandoned sessions on /api/processing-status); the two are complementary — a session is only reported abandoned once no stall-resume timer is pending, which is exactly the state this PR leaves a session in after the backoff budget is spent.
Tests:
tests/worker/overflow-recycle-resume.test.ts— the backoff chain (60/120/240 s delays, then a live generation), cancellation of a pending backoff resume by a generator start, the terminal park once both caps are spent, and teardown clearing the timers. The new tests fail on main (3 fail) and pass here; the response-pacing, session-routes, and generator-exit suites pass, andnpm run typecheckis clean.One existing test changed on purpose: "stops resuming once the consecutive stall cap is reached" became "stops resuming once the stall and backoff caps are both reached" — the fixed-delay cap alone no longer parks the session; that is the fix.
🤖 Generated with Claude Code
https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL