-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
Isolated cron lanes leak on claude-cli backend: queued_work_without_active_run → release_lane released=0, lanes accumulate until restart #89766
Copy link
Copy link
Open
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Description
Metadata
Metadata
Assignees
Labels
P1High-priority user-facing bug, regression, or broken workflow.High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.ClawSweeper found a clear likely implementation shape for this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.Crash, hang, restart loop, or process-level availability failure.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.Session, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Priority
None yet
Summary
On a long-running gateway, isolated cron jobs using the
claude-clibackend intermittently get classified as stuck while they are actively streaming, and the built-in lane recovery then fails to release the lane (released=0). Leaked lanes accumulate over hours until a full gateway restart clears them. Subsequent isolated cron runs fail at the pre-runner stage withstalled before execution start (last phase: runtime-plugins).Environment
claude-cli(Claude Code CLI), OAuth; models route viaagentRuntime: claude-clisessionTarget: isolated,payload.kind: agentTurnSymptoms
openclaw cron listshows isolated jobs inerror.cron: isolated agent run stalled before execution start (last phase: runtime-plugins).isolated-target jobs are affected; main-session jobs are fine.Diagnostic evidence (gateway journald)
Key contradictions:
lastProgress=cli_live:stream_progress lastProgressAge=0s— the run is streaming at the instant it is classified stuck.reason=queued_work_without_active_run— the lane has queued work but no active run is bound to it, despite the CLI streaming.action=release_lane ... released=0— recovery runs but frees nothing, so the lane stays occupied. This is the leak.Ruled out
timeoutSeconds— the pre-runner watchdog is capped independently (perdocs/automation/cron-jobs.md), so raising per-job timeouts has no effect. Confirmed.cron.maxConcurrentRuns: 3; lowers frequency, doesn't stop it.Hypothesis
For the
cli_live(claude-cli streaming) runtime there is a window where a cron run is streaming but is not (yet / any longer) registered as the lane's active run. The stuck-session classifier seesqueued_work_without_active_runand callsrelease_lane, but because the run↔lane linkage is inconsistent,released=0and the lane is never reclaimed. Leaked lanes eventually exhaust isolated capacity, so new isolated runs stall at the pre-runnerruntime-pluginsphase.Impact
Recurring failure of scheduled isolated jobs every few hours, only mitigated by periodic gateway restarts.
Repro conditions
claude-cli backend + isolated cron jobs + gateway uptime of several hours + light concurrency. Happy to provide a debug bundle.