Skip to content

fix(cli): stop stale session retry loops after failover#106284

Open
gangcaiyoule wants to merge 1 commit into
openclaw:mainfrom
gangcaiyoule:fix/cli-session-recovery-97887
Open

fix(cli): stop stale session retry loops after failover#106284
gangcaiyoule wants to merge 1 commit into
openclaw:mainfrom
gangcaiyoule:fix/cli-session-recovery-97887

Conversation

@gangcaiyoule

Copy link
Copy Markdown

Closes #97887

AI-assisted: implemented and reviewed with Codex.

What Problem This Solves

Fixes an issue where CLI-backed turns could keep reusing an invalid session ID after a recoverable failover. Because the stale binding remained in session state, later turns could repeatedly resume the same broken CLI session instead of recovering with a fresh one.

Why This Change Was Made

The existing CLI runner already supports a guarded fresh-session retry, but the direct reply and queued follow-up dispatch paths did not clear their stored binding before that retry. This change wires both paths to an atomic compare-and-clear helper. It clears only the expected stale CLI session across current memory and persistence, refuses to overwrite a newer concurrent binding, and does not recreate a session deleted during the failed attempt.

User Impact

Users with Claude CLI or another CLI backend can recover from an expired or otherwise retryable stale session without entering a repeated failover loop. Concurrent session resets or replacements remain protected.

Evidence

  • Focused CLI dispatch suite: 23 passed.
  • New queued follow-up recovery test: 1 passed with 149 unrelated tests skipped.
  • Direct and queued-path combined suite: 276 passed; one existing thinking-level assertion also fails unchanged on origin/main.
  • Follow-up full suite: 131 passed; 18 tests are blocked by the local Node 24.14 runtime because its embedded SQLite 3.51.2 is below the repository WAL-safety floor. The repository requires Node 24.15+ for this lane.
  • oxfmt check passed for all changed files.
  • oxlint passed for all changed files.
  • git diff --check passed.
  • Final Codex review: no actionable regression identified.

The added regression coverage verifies matching stale-state cleanup, protection of a newer persisted binding, missing-row lifecycle isolation, direct reply recovery, and queued follow-up recovery.

I understand the compare-and-clear behavior and verified that it only authorizes a fresh retry after the expected persisted stale binding was removed.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels Jul 13, 2026
@clawsweeper

clawsweeper Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 13, 2026, 7:17 AM ET / 11:17 UTC.

Summary
The PR adds atomic expected-value clearing of stale CLI session bindings before fresh-session retries in direct and queued reply paths, with concurrency and lifecycle regression tests.

PR surface: Source +50, Tests +224. Total +274 across 6 files.

Reproducibility: yes. at source level: current main retries an expired CLI session fresh without clearing the persisted binding, so the next turn can select the same stale ID again. A real current-main gateway reproduction was not supplied in the PR evidence.

Review metrics: 1 noteworthy metric.

  • Session mutation contract: 1 compare-and-clear invariant across 2 dispatch paths. The merge must preserve one atomic ownership rule for both direct and queued retries rather than allow the two paths to drift.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #97887
Summary: This PR is the explicit candidate fix for the stale persisted CLI binding loop; the separate helper-routing report overlaps the queued execution surface but describes a distinct routing defect.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🦐 gold shrimp
Result: blocked until real behavior proof from a real setup is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Rebase onto current main and port the mutation through the current storage-neutral session accessor and current direct/queued CLI execution paths.
  • Post a redacted real gateway failover transcript or terminal recording showing stale binding removal, successful fresh retry, and persisted state afterward.
  • Resolve the failing lint, type, and focused test checks on the updated head.

Proof guidance:

  • [P1] Needs real behavior proof before merge: The PR reports unit suites and static checks but no redacted after-fix gateway plus CLI-backend failover run showing the persisted stale binding removed and the conversation recovering; after adding proof, update the PR body to trigger review or ask a maintainer for @clawsweeper re-review.

Risk before merge

  • [P1] The branch's compare-and-clear mutation was validated against an older persistence topology; current main now requires the same invariant to be expressed through the SQLite-backed storage-neutral session accessor without clearing a concurrently rebound session.
  • [P1] The queued-path change targets code that has since been substantially refactored, so rebasing may require redesign rather than conflict-only resolution and must account for the adjacent current-main CLI follow-up routing gap.
  • [P1] No real gateway plus CLI-backend failover run demonstrates durable stale-binding removal, successful fresh retry, and preservation of a newer concurrent binding on the current storage backend.
  • [P1] Several current checks fail on the submitted head, including type and lint lanes, and their relationship to base drift versus patch defects has not yet been established.

Maintainer options:

  1. Rebase and port the invariant (recommended)
    Update the branch to current main, implement compare-and-clear through the current session accessor and dispatch topology, restore green focused checks, and add a redacted real failover transcript before merge.
  2. Pause for the queued-runtime refactor
    Pause this PR if the canonical queued CLI routing boundary is still moving, then resume once the adjacent routing work has a stable owner and interface.

Next step before merge

  • [P1] Keep the linked issue and this implementation PR open, but require the contributor to rebase/port the fix and supply real behavior proof; automation should not repair it because the current-main owner boundary changed and contributor-environment proof is a merge gate.

Security
Cleared: The diff adds guarded session-state mutation and tests without changing dependencies, permissions, secrets, downloaded code, workflows, or other supply-chain surfaces.

Review details

Best possible solution:

Port the expected-session compare-and-clear invariant onto current main's storage-neutral session accessor and canonical direct/queued CLI dispatch paths, then prove durable cleanup and successful continuation with a real expired-session failover while also proving that concurrent replacement and deletion remain untouched.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current main retries an expired CLI session fresh without clearing the persisted binding, so the next turn can select the same stale ID again. A real current-main gateway reproduction was not supplied in the PR evidence.

Is this the best way to solve the issue?

Yes in principle: an atomic expected-session compare-and-clear before authorizing the fresh retry is the narrowest concurrency-safe fix. The submitted branch still needs to be ported to current main's storage accessor and execution topology before that judgment applies to the merge result.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 4b6575636fd4.

Label changes

Label changes:

  • add P1: A persisted stale CLI binding can make every later turn in an affected conversation fail until users manually edit session state or reset the conversation.
  • add merge-risk: 🚨 session-state: The PR mutates persisted CLI session ownership during retry, and an incorrect port could clear a newer binding or leave memory and durable state inconsistent.
  • add rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • add status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR reports unit suites and static checks but no redacted after-fix gateway plus CLI-backend failover run showing the persisted stale binding removed and the conversation recovering; after adding proof, update the PR body to trigger review or ask a maintainer for @clawsweeper re-review.

Label justifications:

  • P1: A persisted stale CLI binding can make every later turn in an affected conversation fail until users manually edit session state or reset the conversation.
  • merge-risk: 🚨 session-state: The PR mutates persisted CLI session ownership during retry, and an incorrect port could clear a newer binding or leave memory and durable state inconsistent.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: The PR reports unit suites and static checks but no redacted after-fix gateway plus CLI-backend failover run showing the persisted stale binding removed and the conversation recovering; after adding proof, update the PR body to trigger review or ask a maintainer for @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +50, Tests +224. Total +274 across 6 files.

View PR surface stats
Area Files Added Removed Net
Source 3 55 5 +50
Tests 3 224 0 +224
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 6 279 5 +274

What I checked:

  • Current-main source reproduction: The current CLI runner retries a session-expired failure without the stale session ID, but its own comment states that the caller still needs to clear persisted session state; this leaves the linked issue's repeated-turn failure mechanism present on current main. (src/agents/cli-runner.ts:25, 4b6575636fd4)
  • Storage boundary changed after the PR base: Current main defines session-accessor.ts as the permanent storage-neutral boundary backed by SQLite and provides atomic patch/update APIs; the PR was authored against the older session-store topology and must be reviewed after rebasing onto this contract. (src/config/sessions/session-accessor.ts:1, 4b6575636fd4)
  • Queued execution topology changed: Current main's follow-up runner is substantially refactored and currently invokes the embedded agent path directly rather than containing the CLI dispatch block modified by this branch, so the queued-path patch and tests do not establish behavior against the present topology. (src/auto-reply/reply/followup-runner.ts:1, 4b6575636fd4)
  • Proposed implementation: The branch adds an expected-session compare-and-clear helper and wires it into direct and queued CLI fresh-retry callbacks, protecting newer replacement bindings and deleted-session lifecycle state. (src/auto-reply/reply/agent-runner-cli-dispatch.ts:298, c5ee5d025504)
  • Regression coverage: The branch adds focused tests for stale-state removal, a newer persisted binding, missing-row isolation, direct recovery, and queued follow-up recovery. (src/auto-reply/reply/agent-runner-cli-dispatch.test.ts:570, c5ee5d025504)
  • Real behavior proof gap: The PR body reports focused unit suites, lint, formatting, and static checks, but does not show an after-fix gateway/CLI run where an expired persisted binding is cleared and the same conversation successfully continues with a fresh CLI session. (c5ee5d025504)

Likely related people:

  • vincentkoc: Repository ownership documentation identifies this person with the Agents area, and current session-accessor architecture and adjacent agent-runtime history make them the strongest routing candidate for the storage-neutral compare-and-clear boundary. (role: agent and session-state owner; confidence: high; commits: 4b6575636fd4; files: src/config/sessions/session-accessor.ts, src/auto-reply/reply/agent-runner-execution.ts)
  • steipete: Recent merged history includes CLI turn runtime work and broad maintenance of the agent execution surface, making this person relevant to the fresh-session retry contract and current-main integration. (role: recent CLI runtime contributor; confidence: medium; commits: 5518ac9; files: src/agents/cli-runner.ts, src/auto-reply/reply/agent-runner-execution.ts)
  • RajvardhanPatil07: Recent merged history credits CLI turn output work in the same runtime area, providing an additional routing candidate for CLI execution behavior. (role: recent CLI runtime contributor; confidence: medium; commits: 5518ac9; files: src/agents/cli-runner.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper

clawsweeper Bot commented Jul 25, 2026

Copy link
Copy Markdown
Contributor

@gangcaiyoule thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward.

Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive.

Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FailoverError during cli-backend turn does not clear stale cliSessionIds from sessions.json, causing infinite retry loop

1 participant