Skip to content

fix(worker): discard a stalled session's buffered work once its resumes are spent - #5414

Merged
thedotmack merged 4 commits into
thedotmack:mainfrom
0xABCD01:fix/5405-stall-backoff-resume
Nov 6, 2026
Merged

thedotmack merged 4 commits into
thedotmack:mainfrom
0xABCD01:fix/5405-stall-backoff-resume

Conversation

@0xABCD01

@0xABCD01 0xABCD01 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #5405

Maintainer update (finished on this branch)

The original version retried a stalled session on a 60/120/240 s backoff timer once the fixed-delay resumes were spent. Two problems:

What this PR now does:

  • Stall cap reached → discard. When the bounded stall resumes are spent, the session's buffered work is discarded at once, with a WARN that gives pendingCount, parkedCount and discarded. Nothing is left waiting for an event that may never come, and no timer replays it. The next captured event still starts fresh work. No new timer, retry cap or limit is added; the existing MAX_CONSECUTIVE_STALL_RESUMES is unchanged.
  • Dedup kept. The discard goes through SessionManager.discardBufferedWork, which keeps the session's seen-toolUseId set. A later hook or transcript replay can't re-enqueue work that was already stored, or work that was just discarded.
  • Kept from @0xABCD01's original commit: session teardown (deleteSession / removeSessionImmediate) now clears stallResumeTimer and scheduledResumeTimer, along with its test.
  • The backoff constants, counter and timer are removed.

Proof (tests/worker/overflow-recycle-resume.test.ts):


Original description

Fixes #5405

When a session's observer went unanswered four times in a row, the stall pacer spent its bounded fixed-delay resumes (3 × 30 s) and parked the session until the next captured event. If the content session had already ended, no event ever came: the preserved messages stayed in the in-memory buffer — still counted in queueDepth — until a worker restart discarded them.

This keeps the fixed-delay resumes, and once they are spent, retries on a backoff timer instead of parking: up to 3 more resumes at 60 s, 120 s, and 240 s, on the same stallResumeTimer and through the same ensureGeneratorRunning path. An answered turn resets both stall counters with the other resume debts, so a recovered session starts fresh. Once the backoff is spent too, the session parks exactly as before, with no timer pending. Session teardown (deleteSession / removeSessionImmediate) now clears the stall and scheduled resume timers alongside respawnTimer; neither was cleared before.

Nothing is discarded and the buffer accounting is unchanged: the messages stay pending until a resumed generator stores them.

#5409 implements the issue's third option (reporting abandoned sessions on /api/processing-status); the two are complementary — a session is only reported abandoned once no stall-resume timer is pending, which is exactly the state this PR leaves a session in after the backoff budget is spent.

Tests: tests/worker/overflow-recycle-resume.test.ts — the backoff chain (60/120/240 s delays, then a live generation), cancellation of a pending backoff resume by a generator start, the terminal park once both caps are spent, and teardown clearing the timers. The new tests fail on main (3 fail) and pass here; the response-pacing, session-routes, and generator-exit suites pass, and npm run typecheck is clean.

One existing test changed on purpose: "stops resuming once the consecutive stall cap is reached" became "stops resuming once the stall and backoff caps are both reached" — the fixed-delay cap alone no longer parks the session; that is the fix.

🤖 Generated with Claude Code

https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL

@greptile-apps

greptile-apps Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous duplicate-send issue is fixed and no new blocking issues were found.

Summary

The PR discards buffered work once a session spends its bounded stall resumes.

  • The worker discards a session’s buffered work after repeated unanswered stalls.
  • Session teardown cancels delayed resumes before removing the session.

The previous duplicate-send finding is fixed. No new actionable issues were found.

Reviews (4) · Last reviewed commit: "fix(worker): keep the dedup set when a s..." · Reviewed by Greptile

@0xABCD01
0xABCD01 force-pushed the fix/5405-stall-backoff-resume branch from 42a1fe4 to 655e994 Compare November 6, 2026 02:47
thedotmack and others added 2 commits November 5, 2026 20:51
…n a timer (thedotmack#5405)

The backoff resumes this PR added still ended in the same park, and a
timer that resends stale buffered work is the backlog replay the live-only
observer policy (thedotmack#5347) rules out. Once the bounded stall resumes are spent,
the session's buffered work is now discarded at once with a WARN that states
the pending and parked counts, so nothing waits on an event that may never
come. The next captured event still starts fresh work. The teardown timer
cleanup and its test are kept.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
@thedotmack thedotmack changed the title fix(worker): resume a stalled session on a backoff timer once the fixed-delay resumes are spent fix(worker): discard a stalled session's buffered work once its resumes are spent Nov 6, 2026
…scarded

clearPendingForSession also forgot every seen toolUseId, so a later hook or
transcript replay could enqueue work that was already stored, or that was
just discarded, and send it again. The stall discard now goes through
discardBufferedWork, which drops the buffer and keeps the session's dedup set.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8dT89upYQjUfJTcyFL
@thedotmack
thedotmack merged commit f090685 into thedotmack:main Nov 6, 2026
22 of 23 checks passed
thedotmack added a commit that referenced this pull request Nov 6, 2026
…iscard

Main merged #5383 while the live-only contract was unlanded. It resumed
auth-paused work once a later store succeeded, which is backlog replay.
Auth now discards its inputs, so that resume path, its pause timestamp,
auth scope and per-scope success ledger are removed.

Quota and auth discards go through discardBufferedWork, so a replayed
toolUseId is still suppressed and cannot be paid for again (Greptile P1).
Releasing claims also forgets earliestPendingTimestamp, since released
work can expire before it is claimed again (Greptile P2).

AGENTS.md names "a later successful store" as one more thing that never
authorizes backfill, and the lifecycle doc records #5383 and #5414.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8bpnb8tKFUmq6RZEX8
thedotmack added a commit that referenced this pull request Nov 6, 2026
* fix(observer): skip unavailable work instead of replaying a backlog

Restore live-only observation capture, discard quota/auth backlogs, bound the RAM buffer, and document the observer architecture and queue history. Include regenerated runtime bundles.

* fix(build): preserve executable modes on rebuilt runtime bundles

* fix(observer): remove auth auto-resume and keep dedup on quota/auth discard

Main merged #5383 while the live-only contract was unlanded. It resumed
auth-paused work once a later store succeeded, which is backlog replay.
Auth now discards its inputs, so that resume path, its pause timestamp,
auth scope and per-scope success ledger are removed.

Quota and auth discards go through discardBufferedWork, so a replayed
toolUseId is still suppressed and cannot be paid for again (Greptile P1).
Releasing claims also forgets earliestPendingTimestamp, since released
work can expire before it is claimed again (Greptile P2).

AGENTS.md names "a later successful store" as one more thing that never
authorizes backfill, and the lifecycle doc records #5383 and #5414.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8bpnb8tKFUmq6RZEX8

* test(observer): move old-contract tests to the live-only observer rules

84 tests still asserted the contract this PR removes: quota/auth pauses
keeping and later resuming buffered work, observer inputs landing in the
disk hook spool, and buffers deeper than the 128-message cap. Rewrite them
to assert the live-only contract while keeping what still holds (booking,
breaker arming and tagging, admission waits, pacing, dedup, cwd
attribution, body limits, transcript cursors). No test deleted.

Test fakes that lacked SessionManager.discardBufferedWork now implement it
faithfully, and fake sweeps now match the real one (pending work required,
quota/auth pauses skipped), so several previously vacuous checks test
something again.

Remove the GeneratorRunner relabel of a stand-in's auth pause to
provider_switch: auth now discards that work first, so the relabel could
only ever mark an empty session resumable. Fix the comments that still
described keep-and-wait.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8a4P6RjaWFnrGVZXnd

* fix(observer): send each tool field once in the live handoff

The live handoff spread the camelCase payload and then added the
snake_case tool fields the route reads, so toolInput/toolResponse (and
lastAssistantMessage) went over the wire twice. A 2.5-5 MiB result passed
boundObservationDispatch but doubled past the worker's 5mb JSON cap, and
the hook silently skipped it. Drop the camelCase copies. Greptile review.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_019FbY8a4P6RjaWFnrGVZXnd

---------

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A session abandoned after four unanswered observer prompts keeps its queued messages until the worker restarts

2 participants