Skip to content

Isolated cron lanes leak on claude-cli backend: queued_work_without_active_run → release_lane released=0, lanes accumulate until restart #89766

Description

@jarvis-drakon

Summary

On a long-running gateway, isolated cron jobs using the claude-cli backend intermittently get classified as stuck while they are actively streaming, and the built-in lane recovery then fails to release the lane (released=0). Leaked lanes accumulate over hours until a full gateway restart clears them. Subsequent isolated cron runs fail at the pre-runner stage with stalled before execution start (last phase: runtime-plugins).

Environment

  • OpenClaw: 2026.5.28 (e932160)
  • Runtime: claude-cli (Claude Code CLI), OAuth; models route via agentRuntime: claude-cli
  • OS: Linux, systemd user service
  • Jobs: sessionTarget: isolated, payload.kind: agentTurn

Symptoms

  1. openclaw cron list shows isolated jobs in error.
  2. Cron records: cron: isolated agent run stalled before execution start (last phase: runtime-plugins).
  3. Only isolated-target jobs are affected; main-session jobs are fine.
  4. Builds up over hours; a gateway restart flushes it (temporarily).

Diagnostic evidence (gateway journald)

[diagnostic] stuck session: sessionId=<uuid> sessionKey=agent:main:cron:<jobId>:run:<runId> state=processing age=139s queueDepth=1 reason=queued_work_without_active_run classification=stale_session_state lastProgress=cli_live:stream_progress lastProgressAge=0s recovery=checking
[diagnostic] stuck session recovery: sessionId=<uuid> ... age=139s action=release_lane aborted=false drained=true released=0
[diagnostic] stuck session recovery outcome: status=released action=release_lane ... lane=session:agent:main:cron:<jobId>:run:<runId> released=0

Key contradictions:

  • lastProgress=cli_live:stream_progress lastProgressAge=0s — the run is streaming at the instant it is classified stuck.
  • reason=queued_work_without_active_run — the lane has queued work but no active run is bound to it, despite the CLI streaming.
  • Recovery action=release_lane ... released=0 — recovery runs but frees nothing, so the lane stays occupied. This is the leak.

Ruled out

  • timeoutSeconds — the pre-runner watchdog is capped independently (per docs/automation/cron-jobs.md), so raising per-job timeouts has no effect. Confirmed.
  • Concurrency — already cron.maxConcurrentRuns: 3; lowers frequency, doesn't stop it.
  • Plugin/MCP cold-start bloat — contributes to the trigger window (CLI startup loads plugins + bundle MCP), but removing a failing plugin barely changed startup time; not the root fault.

Hypothesis

For the cli_live (claude-cli streaming) runtime there is a window where a cron run is streaming but is not (yet / any longer) registered as the lane's active run. The stuck-session classifier sees queued_work_without_active_run and calls release_lane, but because the run↔lane linkage is inconsistent, released=0 and the lane is never reclaimed. Leaked lanes eventually exhaust isolated capacity, so new isolated runs stall at the pre-runner runtime-plugins phase.

Impact

Recurring failure of scheduled isolated jobs every few hours, only mitigated by periodic gateway restarts.

Repro conditions

claude-cli backend + isolated cron jobs + gateway uptime of several hours + light concurrency. Happy to provide a debug bundle.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions