Skip to content

Embedded agent runs intermittently stall at embedded_run:started (no progress, no hook execution) — v2026.6.9 through v2026.6.11 #103683

Description

@martins-oss

Environment

  • OpenClaw v2026.6.9, reproduced also on v2026.6.11 (upgraded specifically for the embedded-run fixes in the release notes — improved frequency, did not eliminate)
  • Node v22.22.3, Ubuntu 24.04, systemd user service, single gateway process
  • Multi-agent setup (9 agents), Discord channels, mixed transports (codex app-server + embedded openai/anthropic)

Symptom

Embedded-transport agent runs intermittently hang at embedded_run:started and never progress. Diagnostics emit repeatedly:

[diagnostic] stalled session: state=processing reason=active_work_without_progress classification=stalled_agent_run activeWorkKind=embedded_run lastProgress=embedded_run:started lastProgressAge=200s+ recovery=none

The session queue backs up (queueDepth>=1) and the agent stops responding in its channel until the gateway is restarted. codex-transport runs are never affected.

Evidence gathered

  1. On 2026.6.9 it was deterministic when any plugin registered an agent_turn_prepare hook: every embedded run stalled before the handler executed. File-based tracers at the first line of the handler never fired — the hang is in the host between embedded_run:started and handler invocation, not in plugin code. Disabling the plugin restored embedded runs; A/B repeated twice.
  2. agent_turn_prepare is in PLUGIN_HOOK_NAMES; registration succeeds; runModifyingHook wraps handlers in try/catch and (when opts.timeoutMs is set) withHookTimeout — none of that engages because the handler is never reached.
  3. On 2026.6.11 the deterministic case is fixed (verified: same plugin, same session, embedded run completes with the hook executing in <1s), but intermittent stalls with the identical signature continue several times per day, including on sessions where the same message succeeds after a gateway restart.
  4. Ruled out: queued pluginNextTurnInjections (session entries show none at stall time), MCP server spawn (wrappers healthy / not spawned for embedded runs at all), model/provider (occurs on openai-transport gpt-5.x runs with no plugin fetch in flight).
  5. Possibly related observation: some long embedded runs emit no progress events at all while genuinely working (delivered successfully when left alone), so active_work_without_progress cannot distinguish a live long run from a dead one — which also makes external auto-recovery watchdogs risky.

Impact

Agents intermittently stop responding fleet-wide until a manual/cron gateway restart. recovery=none — no self-healing path.

Asks

  1. A timeout or liveness guard on the embedded-run startup path (between embedded_run:started and first hook/model progress).
  2. Progress events emitted during embedded model calls so stall classification can distinguish live from dead runs.
  3. If there is additional instrumentation we can enable to capture the hang point, happy to run it — this reproduces several times per day on our fleet.

Metadata

Metadata

Assignees

Labels

P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.maturity:stableIssue affects a taxonomy feature currently scored M4/M5.

Type

No type

Fields

Priority

None yet

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions