Summary

Agent runs today can hang without bound, die without a user-visible outcome, and strand their session's queued messages until a gateway restart. These are not independent bugs: ~160 tracker reports cluster into a small set of structural gaps in how we own a run's lifecycle. This issue proposes the long-term design — four invariants and four implementation tracks — that retires the class instead of patching instances.

Problem to solve

Users report two experiences, both costing us real deployments to competing frameworks:

1. "Long multi-step tasks hang partway and never complete; I have to handhold."
2. "I come back to a session and the agent died on a turn; everything I sent since is silently queued."

Cross-referencing the open tracker with a source audit, these decompose into three root causes:

Root cause 1 — stall detection is heuristic and self-defeating, and detection ≠ recovery.

• Liveness is inferred from proxies that wedged runs still satisfy. On claude-cli, a timer re-stamps lastProgress while the tool subprocess is wedged, defeating the detector (claude-cli backend: active-tool heartbeat masks wedged subprocesses, defeats stuck-session detection #96168). On the Codex app-server, the idle watchdog is gated on activeAppServerTurnRequests > 0, so a poisoned client is exactly the state it cannot see ([Bug]: Codex app-server session wedges indefinitely after a 429 rate-limit burst — terminal-idle watchdog still gated by activeAppServerTurnRequests > 0 (regression/incomplete fix of #82681) #89742, regression of Telegram session wedges indefinitely when codex turn-completion idle watchdog is gated by stuck activeAppServerTurnRequests counter #82681).
• The same heuristics also kill healthy work: hardcoded 120s stall threshold vs long vLLM/thinking calls ([Bug]: Agent stall detector hard-coded 120s threshold kills legitimate long model calls on local vLLM #85826), active cron runs aborted at ~6 min (Stuck-session recovery aborts long-but-active agent runs at warnMs×3 (~6min) with misleading "Reply operation aborted by user" reason #88870), liveness math skewed by event-loop stalls (Stuck-session liveness monitor misclassifies healthy processing runs as stale during event-loop stalls #101670).
• When detection does fire, abort does not release the session lane: release_lane released=0 ([Bug]: Stuck-session recovery never releases: status=aborted with released=0 on every event — wedges survive five correct-target aborts (~14h) and four ghost-target aborts (~16h) until an external restart or user /reset [2026.5.28] #92270, Conversational agent lane wedges indefinitely: ingress events never reconcile (re-delivery storm on reconnect) + abort_embedded_run leaves owner lock unreleased (released=0) #97538, [Bug]: Session write-lock held beyond run lifetime when agent run fails during tool execution #100872, Isolated cron lanes leak on claude-cli backend: queued_work_without_active_run → release_lane released=0, lanes accumulate until restart #89766). admitReplyTurn then returns active-run indefinitely; there is no wall-clock/staleness takeover of a "running" operation — src/auto-reply/reply/reply-run-registry.ts reclaims only terminal (failed/timeout/killed) leftovers. One hang becomes a dead session.
• Config footgun: timeoutSeconds: 0 (the natural setting for long tasks) maps the run budget to MAX_TIMER_TIMEOUT_MS, which resolveLlmIdleTimeoutMs treats as "disable the LLM idle watchdog entirely" (src/agents/embedded-agent-runner/run/llm-idle-timeout.ts, runTimeoutMs >= MAX_TIMER_TIMEOUT_MS → 0). "Unlimited budget" silently means "zero stall protection".

Root cause 2 — zero durability: a process death loses the in-flight turn and every queued followup, with no resume.

• The unhandled-rejection policy exits the process for any unclassified rejection (src/infra/unhandled-rejections.ts, final branch → exitWithTerminalRestore → process.exit(1)), and the turn pipeline contains many fire-and-forget promises. One stray rejection anywhere kills the gateway.
• The followup queue is a purely in-memory global map (src/auto-reply/reply/queue/state.ts, FOLLOWUP_QUEUES); active-run/abort state is in-memory; nothing persists "turn in progress". After restart, busy flags can even survive while dispatch never resumes ([Bug]: Session stuck in permanent busy state after mid-turn gateway restart — no auto-recovery, only /new or /reset clears it #92519: 301 assembles, 0 dispatches over 12h).

Root cause 3 — no completion drive: an early model stop is accepted as final, and some failures are silent.

• Mid-turn auto-compaction silently abandons the task (Auto-compaction during ongoing turn silently abandons task execution without resumption #59618); orphaned tool_use blocks after a timeout permanently poison the session (bug: compaction preserves orphaned tool_use blocks after request timeout, permanently breaking session #93321).
• A deterministically-throwing queued item is never removed and is rescheduled forever — drainNextQueueItem removes the item only after run(next) resolves, and there is no per-item failure cap (src/utils/queue-helpers.ts, src/auto-reply/reply/queue/drain.ts).
• Several terminal paths (fallback exhaustion, some abort routes) log without messaging the user, so honest deaths read as ghosting.

Proposed solution

Four invariants, then four tracks that implement them. The invariants are the contract; every track is judged against them.

I1 — Bounded liveness. Every run reaches a terminal state in bounded time, where "time" is measured as gap since last real forward-progress event, never raw wall clock and never synthetic heartbeats.
I2 — Terminal ⇒ released + reported. Reaching a terminal state atomically releases the session lane/locks and emits exactly one user-visible outcome (or an explicitly-silent one for room_event/silent modes). No path may terminate a run without doing both.
I3 — Restart-safe. The followup queue and a minimal turn-in-progress marker are persisted; boot reconciles interrupted runs into a resumable state instead of stranding them.
I4 — Early stop ≠ done. A model stopping prematurely triggers bounded continuation, not silent acceptance.

Track A — single run-lifecycle owner (abort ⇒ release)

• Give each reply run one closed state machine: admitted → running → terminal(completed | failed | aborted | timeout | interrupted), extending the existing terminal-outcome normalization (src/agents/agent-run-terminal-outcome.ts) rather than adding a parallel concept.
• Lane/lock/admission release becomes a side effect of the terminal transition, performed in exactly one place. Today's scattered finally releases (followup-runner.ts, dispatch-from-config.ts) become assertions that the transition already ran. This makes released=0 structurally impossible: an abort that reaches terminal is the release.
• Admission gains an evidence-staleness takeover: a "running" operation whose activity stamp (Track B) is older than the takeover threshold is reclaimable, extending the existing terminalRecovery path which today only covers already-terminal leftovers.
• Retires: [Bug]: Stuck-session recovery never releases: status=aborted with released=0 on every event — wedges survive five correct-target aborts (~14h) and four ghost-target aborts (~16h) until an external restart or user /reset [2026.5.28] #92270, Conversational agent lane wedges indefinitely: ingress events never reconcile (re-delivery storm on reconnect) + abort_embedded_run leaves owner lock unreleased (released=0) #97538, [Bug]: Session write-lock held beyond run lifetime when agent run fails during tool execution #100872, Isolated cron lanes leak on claude-cli backend: queued_work_without_active_run → release_lane released=0, lanes accumulate until restart #89766, and the amplification behind most "bot stopped responding until restart" reports.

Track B — evidence-based liveness (RunActivity)

• One RunActivity record per run, stamped only by real events: model stream deltas, tool call start/result, CLI child stdout frames, MCP responses, compaction progress. Delete the mechanisms that fake or hide liveness: the CLI timer heartbeat that re-stamps progress while a tool is wedged (claude-cli backend: active-tool heartbeat masks wedged subprocesses, defeats stuck-session detection #96168), the app-server activeAppServerTurnRequests watchdog gate ([Bug]: Codex app-server session wedges indefinitely after a 429 rate-limit burst — terminal-idle watchdog still gated by activeAppServerTurnRequests > 0 (regression/incomplete fix of #82681) #89742), and the hardcoded 120s threshold ([Bug]: Agent stall detector hard-coded 120s threshold kills legitimate long model calls on local vLLM #85826).
• One watchdog poller compares now - lastEvent against a single phase-aware threshold table: model-stream gap, tool-execution gap, per-reasoning-model floors (thinking models get minutes, not 120s). timeoutSeconds: 0 keeps the budget unlimited but must never disable liveness — fix the llm-idle-timeout coupling.
• On breach: warn the user first ("no activity for N min in tool X"), then interrupt through Track A's terminal transition — so recovery is guaranteed to release and report, with a diagnostic naming the stuck phase/tool instead of "aborted by user".
• Retires: claude-cli backend: active-tool heartbeat masks wedged subprocesses, defeats stuck-session detection #96168, [Bug]: Codex app-server session wedges indefinitely after a 429 rate-limit burst — terminal-idle watchdog still gated by activeAppServerTurnRequests > 0 (regression/incomplete fix of #82681) #89742, Codex app-server emits notification:turn/started then goes silent; embedded run wedges for the full stuck-session recovery window #85251, [Bug]: Agent stall detector hard-coded 120s threshold kills legitimate long model calls on local vLLM #85826, Stuck-session recovery aborts long-but-active agent runs at warnMs×3 (~6min) with misleading "Reply operation aborted by user" reason #88870, Stuck-session liveness monitor misclassifies healthy processing runs as stale during event-loop stalls #101670, Cron jobs stall during AI model calls (model_call:stream_progress never completes) #91892, [Bug]: Ollama remote provider streaming not consumed — model_call:started never progresses in chat sessions #94251, and the detection half of Diagnostic detects stalled_agent_run (recovery=none) after model_call:ended but never recovers #100213.

Track C — durability (crash resume)

• Move the followup queue and a minimal turn_in_progress marker into the shared state DB (state/openclaw.sqlite) per the repo storage mandate — the in-memory global map is existing migration debt.
• Boot reconcile: runs marked running at startup transition to interrupted (Track A ⇒ lane released, outcome recorded), then the session receives one synthesized continuation turn ("previous turn was interrupted; do not re-execute completed tool calls"), bounded by a restart-loop guard (e.g. 3 resume-boots/60s → suspend and notify).
• Rescope the unhandled-rejection policy: rejections attributable to an owning run fail that run terminally (Track A ⇒ release + notify) instead of process.exit(1); process exit remains only for truly unattributable/fatal states. The current classify-or-die policy makes every unannotated fire-and-forget promise a gateway-wide kill switch.
• Retires: [Bug]: Session stuck in permanent busy state after mid-turn gateway restart — no auto-recovery, only /new or /reset clears it #92519, [Bug] Restart-sentinel continuation can get stuck after transient Telegram sendMessage failure #76087, the "came back and it was dead" class, and the queued-message loss on every crash/restart.

Track D — completion drive

• Never-silent terminals: every terminal outcome produces a user-visible message unless the turn is explicitly silent (room_event/silent modes). Audit the suppressing branches in followup-runner.ts (fallback exhaustion, abort routes).
• Compaction-safe tasks: mid-turn compaction re-injects the active plan/remaining work and resumes the interrupted turn instead of abandoning it (Auto-compaction during ongoing turn silently abandons task execution without resumption #59618); orphaned tool_use blocks are closed with synthetic error results so the session stays resumable (bug: compaction preserves orphaned tool_use blocks after request timeout, permanently breaking session #93321).
• Bounded continuation nudges: when the model replies with a pure intent acknowledgement ("I'll do X") without tool calls mid-task, inject a bounded continue nudge (cap ~2/turn). Deliberately small and capped — this is a re-prompt, not an autonomy loop.
• Poison-item cap: per-item failure counter in the followup queue; after N consecutive failures drop the item with a user-visible notice instead of rescheduling forever.

Sequencing

A → B → C → D. A alone (abort ⇒ release + staleness takeover) removes the biggest amplifier and is small. B removes both false positives and false negatives at the root and deletes three ad-hoc mechanisms. C is the largest but rides existing SQLite/doctor machinery. D is independent polish that closes the loop on "works through to completion". Each track is separately PR-able and separately testable against the invariants.

Alternatives considered

• Keep fixing instances. We have been: the ingress stale-claim reaper (Stuck "claimed" rows in channel_ingress_events block all inbound delivery after restart (no stale-claim recovery) #96374), the initialization-conflicted wave (Bug: "reply session initialization conflicted" — session stuck in running status with GLM-5.2 cloud model #98745 et al.), transport reconnects. Each fix holds, but new variants of the same structural gaps arrive monthly because no component owns the invariants. Cost of instance-patching now exceeds the design fix.
• Per-provider/per-tool timeout tuning. Doesn't help: the field data shows both over-firing (killing healthy long calls) and under-firing (fake heartbeats) from the same heuristics. The fix is what counts as evidence of liveness, not the constants.
• Full workflow/checkpoint engine. Overkill for the reported pain. I3 needs a queue + one marker + a continuation turn, not a durable-execution framework.

Impact

• Directly addresses the two most-reported reliability complaints (hung multi-step tasks; dead sessions with silently-queued messages) — the top reason cited by users actively migrating to competing frameworks.
• Consolidates ~40+ open issues across six clusters into four owned tracks; each track lists the issues it retires so they can be closed with proof as tracks land.
• Deletes code and concepts on net: three liveness mechanisms → one, scattered lane releases → one transition, in-memory queue → the mandated SQLite store.

Evidence/examples

Cluster → canonical open issues (non-exhaustive; full sweep covered ~160):

• Lane never released after abort/death: [Bug]: Stuck-session recovery never releases: status=aborted with released=0 on every event — wedges survive five correct-target aborts (~14h) and four ghost-target aborts (~16h) until an external restart or user /reset [2026.5.28] #92270, Conversational agent lane wedges indefinitely: ingress events never reconcile (re-delivery storm on reconnect) + abort_embedded_run leaves owner lock unreleased (released=0) #97538, [Bug]: Session write-lock held beyond run lifetime when agent run fails during tool execution #100872, Isolated cron lanes leak on claude-cli backend: queued_work_without_active_run → release_lane released=0, lanes accumulate until restart #89766, embedded_run completes with empty final payload → reply silently dropped (lane goes dark, no auto-heal) #100162
• Watchdog false positives / false negatives / no-op recovery: Stuck-session recovery aborts long-but-active agent runs at warnMs×3 (~6min) with misleading "Reply operation aborted by user" reason #88870, [Bug]: Agent stall detector hard-coded 120s threshold kills legitimate long model calls on local vLLM #85826, Stuck-session liveness monitor misclassifies healthy processing runs as stale during event-loop stalls #101670, Stale diagnostic tool_call activity can survive recovery/reset and re-block sessions as blocked_tool_call #87310, Diagnostic detects stalled_agent_run (recovery=none) after model_call:ended but never recovers #100213
• Codex app-server wedge (watchdog gated): [Bug]: Codex app-server session wedges indefinitely after a 429 rate-limit burst — terminal-idle watchdog still gated by activeAppServerTurnRequests > 0 (regression/incomplete fix of #82681) #89742, Codex app-server emits notification:turn/started then goes silent; embedded run wedges for the full stuck-session recovery window #85251, Codex app-server stalls after inter-session sessions_send timeout #90673, [Bug]: Codex bundled harness initialize still hangs in 2026.5.18 isolated cron — surfaces via #64744 timeout-wrapping as 'isolated agent setup timed out before runner start' #84567
• claude-cli heartbeat masking / synthetic results: claude-cli backend: active-tool heartbeat masks wedged subprocesses, defeats stuck-session detection #96168, [Bug]: Claude CLI reused session emits synthetic empty result before real response, causing OpenClaw session discard and background task cascade #99131, In-turn background work (Workflow / run_in_background / Workflow-backed skills) is SIGKILLed at the turn boundary under the claude-cli runtime #97919
• Mid-turn restart loses turn / busy flag persists: [Bug]: Session stuck in permanent busy state after mid-turn gateway restart — no auto-recovery, only /new or /reset clears it #92519, [Bug] Restart-sentinel continuation can get stuck after transient Telegram sendMessage failure #76087, Restart race condition with LaunchAgent KeepAlive causes shutdown instead of restart #88309
• Compaction abandons/poisons task: Auto-compaction during ongoing turn silently abandons task execution without resumption #59618, bug: compaction preserves orphaned tool_use blocks after request timeout, permanently breaking session #93321, Bug: Preemptive context overflow silently kills embedded sessions without notifying user #84536, Session hangs indefinitely when compaction times out, causing repeated duplicate message sends #43661, Session lock stuck in 'processing' after context overflow compaction succeeds #70334
• Subagent wake/termination deadlocks (touched by I2/D, may need follow-up design): Subagent announce give-up (retry-limit/expiry) does not wake parent waiting on sessions_yield → permanent deadlock #90178, [Bug]: Agent / sessions_spawn tool_use can emit with no matching tool_result and no parent-visible signal — silent subagent drop (~3.8% historical rate) #88856, Bug: Subagent child sessions receive no termination signal when parent dies mid-run #77720
• Model call unbounded / stream stall: Cron jobs stall during AI model calls (model_call:stream_progress never completes) #91892, [Bug]: Ollama remote provider streaming not consumed — model_call:started never progresses in chat sessions #94251, [Bug]: Stalled direct chat model call surfaces LLM request failed instead of auto-resetting/retrying wedged session #90516, WhatsApp session stalls on long model_call: incomplete turn with payloads=0, reply never delivered #84569 (fixed instance: Embedded agent run: model generation call has no per-call timeout (hangs until stuck-session watchdog) #95865)
• Silent terminal failures: [Bug]: Long-running tasks complete but agent doesn't reply to user #96263, [Bug]: Session status blind spot | Sessions in done state excluded from context saturation monitoring — agents run dead for hours undetected #91832, [Bug]: EmbeddedAttemptSessionTakeoverError causes silent message loss when session lock is released during API retries #87180

Code evidence (verified on current main):

• src/infra/unhandled-rejections.ts — final unclassified branch exits the process.
• src/agents/embedded-agent-runner/run/llm-idle-timeout.ts — runTimeoutMs >= MAX_TIMER_TIMEOUT_MS returns 0 (idle watchdog disabled when budget is "unlimited").
• src/auto-reply/reply/reply-run-registry.ts — takeover restricted to terminal leftovers; no staleness reclaim of a hung "running" op.
• src/auto-reply/reply/queue/state.ts — FOLLOWUP_QUEUES in-memory global map; no persistence.
• src/utils/queue-helpers.ts + src/auto-reply/reply/queue/drain.ts — item removed only after successful run; no per-item failure cap.

Additional information

Prior art: competing agent frameworks (e.g. Hermes) implement exactly this shape — event-stamped activity heartbeats with per-reasoning-model floors, staged user warning before interrupt, SQLite task claims with lease TTL + dead-worker reclaim, resume_pending sessions that receive a synthesized continuation turn on boot, and bounded anti-premature-stop nudges. Their users cite "never drops a task" as the primary switching reason; this design closes that gap with mechanisms consistent with our architecture rules (SQLite-first state, one canonical path, doctor-owned migration).