Skip to content

CLI backend: retry fresh session on any FailoverError, not just session_expired #77089

Description

@clawdbotv2

Problem

When a CLI backend session hangs or crashes, OpenClaw only clears the stale session binding and retries fresh for FailoverError.reason === "session_expired". All other failure reasons (timeout, generic crash) keep the stale binding and immediately cascade to the model fallback chain (API Opus → API Sonnet).

This means a single stuck CLI session can take the entire CLI backend offline for hours — every subsequent turn tries to resume the dead session, fails instantly, and falls back to paid API models.

Observed incident (2026-05-03)

  1. 17:27 — CLI session 666dfaad8096 hung, produced no output for 180s, watchdog killed it. Reason: timeout. Fell to API Opus.
  2. 21:44 — Same session binding still stored. OpenClaw tried useResume=true, resumeSession=666dfaad8096. CLI crashed in 476ms with FailoverError. Fell to API Opus → timed out → fell to API Sonnet.
  3. 22:11 — A heartbeat came in with useResume=false, session=none (binding was cleared by a gateway restart). Fresh CLI session succeeded in 13s.

The binding was never cleared between steps 1 and 3 because the FailoverError reasons were timeout and unknown, not session_expired.

Root cause

In attempt-execution (dist: attempt-execution-BGTy1BCv.js:546), the catch block only handles session_expired:

if (err instanceof FailoverError && err.reason === "session_expired" && activeCliSessionBinding?.sessionId && ...) {

The retry-fresh logic (clear binding → runCliWithSession(void 0) → re-store new binding) is already implemented but gated behind this narrow condition.

Proposed fix

Widen the condition to catch any FailoverError when a session binding exists:

if (err instanceof FailoverError && activeCliSessionBinding?.sessionId && ...) {
    log.warn(`CLI session failed (reason=${err.reason ?? "unknown"}), clearing stale binding and retrying fresh: ...`);
    // existing clear + retry logic
}

This way:

  • Any CLI failure on a resumed session clears the stale binding
  • Retries once with a fresh session before falling back to API
  • If the fresh retry also fails, cascades to fallbacks as before
  • No behavior change for non-resume failures (no activeCliSessionBinding → falls through to throw err)

Impact

  • Prevents hours-long silent degradation from CLI → API Sonnet
  • Self-healing: one bad session gets cleared automatically instead of requiring a gateway restart
  • No change to fresh session failure behavior

Additional suggestions

  • Consider adding "timeout" and "unknown" to the explicit reason list if a broad catch feels too aggressive
  • The reliability.watchdog default of 180s may be too aggressive for Opus 4.7 — consider bumping the default or documenting tuning guidance

Environment

  • OpenClaw 2026.5.2 (8b2a6e5)
  • CLI backend: claude-cli → Claude CLI (Max subscription)
  • Primary model: claude-cli/claude-opus-4-7

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions