Environment

• OpenClaw: 2026.6.9 (c645ec4)
• Backend: cliBackends.claude-cli (Anthropic Claude Code CLI)
• Channel: Telegram (long-polling); direct + group-topic lanes both observed in the same incident
• Agent: agents.list[main], model claude-cli/claude-opus-4-7
• Host: single-VPS Linux, systemd --user (openclaw-gateway.service, Restart=always, StartLimitBurst=5, StartLimitIntervalSec=60)
• Storage: per-agent sqlite + JSON sessions.json; no custom plugins on the restart-recovery path

TL;DR

When a channel main-session turn wedges (no settlement event within a long window), and a side-effect trigger fires a gateway SIGUSR1 restart, the restart drain marks the wedged session entries in sessions.json with abortedLastRun: true plus an entry appended to restartRecoveryRuns. On the next boot, the recovery pass calls back into the same agent method for those entries unconditionally. There is no persistent per-session retry counter, no quarantine, no backoff across boots. The result is a deterministic death loop until systemd's StartLimitBurst is exhausted and operator intervention is required.

In-process retry orchestration exists (MAX_RECOVERY_RETRIES = 3 with RETRY_BACKOFF_MULTIPLIER = 2 and DEFAULT_RECOVERY_DELAY_MS = 5_000) but bounds retries within a single process invocation only — the counter is not persisted to the store. Each fresh boot is treated as a fresh retry budget.

Same architecture (and same gap) appears in the subagent-orphan-recovery path. An explicit log line there documents the intent: resume failed for ${childSessionKey}; abortedLastRun flag preserved for retry on next restart. The cross-boot preservation IS the retry mechanism by design — the missing piece is the ceiling.

Production incident (2026-06-21)

• Two concurrent Telegram channel main-session turns started ~1s apart (a DM lane + a group-topic lane). Both claude-cli / claude-opus-4-7.
• Both turns hung 559 s with no settlement. Gateway emitted cli_live:stream_progress ticks from claude-cli every ~30 s through the entire hang — the runtime kept reporting stream progress but no tool-call and no text block ever settled.
• eventLoopDelayP99Ms climbed to ~68 s during the hang (steady state pre-incident was sub-second).
• A scheduled isolated cron's setup-timeout watchdog fired with reason: "cron.isolated_agent_setup_timeout". That triggered a gateway SIGUSR1 restart.
• Restart drain marked both wedged channel sessions: abortedLastRun: true, one {runId, lifecycleGeneration} appended to restartRecoveryRuns.
• Next boot: same hang. Same SIGUSR1. Same re-mark. restartRecoveryRuns.length incremented to 2.
• 6 cycles total before systemd exhausted StartLimitBurst and the unit refused to restart.
• Recovery: offline edit of sessions.json to strip restartRecovery* fields on both entries, systemctl --user reset-failed openclaw-gateway, restart.

Field-level evidence in sessions.json (post-incident, pre-repair)

Each affected entry (anonymized):

"agent:main:<channel>:<lane>": {
  "status": "running",
  "abortedLastRun": true,
  "restartRecoveryRuns": [
    { "runId": "...", "lifecycleGeneration": "..." },
    { "runId": "...", "lifecycleGeneration": "..." }
  ],
  "restartRecoveryDeliveryContext": { ... },
  "restartRecoveryDeliveryRunId": "...",
  // ...
}

• restartRecoveryRuns.length increments by 1 per failed resume.
• It is not used as a retry-ceiling input. The resume gate checks only status === "running" && abortedLastRun === true.
• The success path clears abortedLastRun = false. The failure path is log-only — no field increment, no quarantine timestamp, no persistent attempt counter.

Reproducer (no patches required)

1. Run the gateway under systemd --user (or any supervisor with auto-restart enabled).
2. Schedule an isolated cron that natural-fires every ~5 minutes.
3. Trigger a channel main-session turn with a sufficiently large prompt to encourage Opus to deliberate ≥30 s without emitting either a tool call or text.
4. While that turn is hanging, the cron's setup-timeout watchdog will eventually fire on event-loop wedge → SIGUSR1.
5. Inspect sessions.json after the first restart: the channel entry has abortedLastRun: true and one restartRecoveryRuns entry.
6. On subsequent boots, observe restartRecoveryRuns.length increment monotonically until StartLimitBurst is exhausted.

The reproduction is deterministic. The original cant_reproduce close on #86239 may have predated either the cron isolated-setup-timeout watchdog SIGUSR1 trigger or the default cron runMode flip from force → due in 2026.6.9 — both raise the probability of the coincidence required to trigger the loop.

Proposed fix shape

Add a persistent abortedLastRunAttempts integer field on session entries, parallel to abortedLastRun. Behavior:

1. Mark phase (every site that sets abortedLastRun = true — markRestartAbortedMainSessions, the from-locks scan, the expired-pending-delivery path): increment abortedLastRunAttempts by 1, default-initializing from 0.
2. Resume gate (the loop that picks entries to retry): require abortedLastRunAttempts < MAX_RECOVERY_RETRIES in addition to the current status === "running" && abortedLastRun === true predicate.
3. Overrun (when the resume gate excludes an entry that would otherwise be eligible): clear abortedLastRun = false, set quarantinedAt = <ISO timestamp>, set quarantineReason = "exceeded_restart_retry_budget", emit a structured WARN log line with the session key + the count.
4. Quarantine cleanup: an entry with quarantinedAt set is no longer eligible for restart-recovery. It may still receive new inbound traffic — the next user-driven turn clears quarantinedAt and resets abortedLastRunAttempts to 0 (or some operator-tunable behavior).

The same shape needs to land in the subagent-orphan-recovery path. Both share the architecture and both have the same gap.

MAX_RECOVERY_RETRIES (currently 3 in-process) is a reasonable cross-boot value too. Optionally make it configurable via recovery.maxAcrossBoots.

Behavioral contract

The quarantine semantic prevents the death loop without losing the session transcript pointer (which manual deletion of the entry would lose).

Coordination concerns

• [Feature]: Wall-clock deadline on channel-lane agent turns (no-settlement bound, dispatcher-level) #95630 (just filed; dispatcher wall-clock deadline on channel lanes) attacks the same family at the trigger layer — it prevents the wedge that starts the chain. This issue attacks the survivability layer — it bounds what happens if a wedge does occur. Both should land; they're complementary, not substitutes.
• fix: restart gateway after isolated cron setup timeout #89055 (open PR; restart gateway after isolated cron setup timeout) would amplify the death loop family without a circuit breaker upstream of the restart-recovery mark phase. Worth coordinating timing: ideally this issue's fix lands before fix: restart gateway after isolated cron setup timeout #89055 merges, or #89055's gateway-restart path is gated on a flag.
• [Bug] sessions.json can become 0 bytes (empty), breaking session aggregation with no recovery mechanism #77783 + open PR fix(sessions): recover store from stale temp artifacts safely #79910 (sessions.json recovery / corruption) is adjacent — both touch the same store. The patches should land in the same release window if possible to avoid operators having to apply offline repair twice.

Related upstream issues

• [Bug]: MissingAgentHarnessError on inbound dispatch under event-loop starvation — harness IS registered (2026.5.22, distinct root cause from #86227) #86239 — closed cant_reproduce 2026-05-28, locked. Same root mechanism. This issue supersedes that conversation with a deterministic reproducer on 2026.6.9 c645ec4.
• Telegram/WhatsApp channel recovery can starve gateway event loop and trigger supervisor restart loop #95288 — open P1 near-twin (channel-recovery framing). The structural gap underneath both reports is what's described here.
• [Feature]: Wall-clock deadline on channel-lane agent turns (no-settlement bound, dispatcher-level) #95630 — just filed; dispatcher wall-clock deadline (trigger-layer fix; complementary to this survivability-layer fix).
• Long streaming model responses cause event loop starvation #89392, [Bug]: event-loop starvation delays fetch timeouts during gateway load #84835, [Bug]: 2026.5.22 Docker/WSL2 gateway event-loop starvation, 284s provider-auth prewarm, slow Telegram turn, and local RPC timeouts #86752, Gateway event loop starvation and HTTP/WS outage during sessions usage/cost under memory pressure #86718 — event-loop starvation family (the triggering condition).
• fix: restart gateway after isolated cron setup timeout #89055 — open PR; restart-on-isolated-setup-timeout (would amplify this family without coordination).
• [Bug] sessions.json can become 0 bytes (empty), breaking session aggregation with no recovery mechanism #77783, fix(sessions): recover store from stale temp artifacts safely #79910 — sessions.json recovery / corruption family (adjacent operational concern).

Scope sanity check

The narrow change is:

• One new integer field in the session-entry shape (abortedLastRunAttempts).
• Increment at every mark-true site (small number, documented in mark-phase functions).
• One predicate addition in the resume gate.
• One quarantine path (clear + timestamp + WARN).
• Same shape applied to the subagent-orphan-recovery surface.

No new config required if MAX_RECOVERY_RETRIES is reused as the cross-boot ceiling. Add recovery.maxAcrossBoots config if you want operator override (low priority — the in-process value is reasonable).

Happy to scope a PR if the maintainers want concrete code rather than a description. If you'd prefer a different fix shape (e.g. wall-clock TTL on the restartRecovery* flag instead of an attempt counter), open to discussing — the operator-facing contract is what matters: don't death-loop the gateway when a session is genuinely wedged.

Sanity questions

• Is there an existing operator knob I missed that would bound the cross-boot retries? I checked recovery.* config; MAX_RECOVERY_RETRIES exists but is in-process only. Did I overlook a persistence path?
• For the subagent-orphan-recovery surface: same gap, or does that path already have a counter I missed?