Skip to content

fix(agents): mark interrupted sessions before restart#82772

Merged
joshavant merged 3 commits into
mainfrom
fix/restart-aborted-main-session-markers
May 16, 2026
Merged

fix(agents): mark interrupted sessions before restart#82772
joshavant merged 3 commits into
mainfrom
fix/restart-aborted-main-session-markers

Conversation

@joshavant

Copy link
Copy Markdown
Contributor

Summary

  • Problem: main sessions interrupted by a forced gateway restart could remain running without abortedLastRun, so startup recovery did not resume or fail the interrupted turn.
  • Why it matters: a user-visible agent response can be stranded after a config-triggered restart while model/tool work is still in flight.
  • What changed: restart deferral and forced restart paths now mark active main sessions for restart recovery using active embedded/reply run session ids, including configured custom session stores.
  • What did NOT change (scope boundary): subagent/cron/ACP recovery behavior remains excluded, and detached background task requester keys are not used as main-session selectors.

Change Type (select all)

  • Bug fix
  • Feature
  • Refactor required for the fix
  • Docs
  • Security hardening
  • Chore/infra

Scope (select all touched areas)

  • Gateway / orchestration
  • Skills / tool execution
  • Auth / tokens
  • Memory / storage
  • Integrations
  • API / contracts
  • UI / DX
  • CI/CD / infra

Linked Issue/PR

Real behavior proof (required for external PRs)

Behavior addressed: forced gateway restart while a main agent turn is active leaves the session recoverable instead of stranded without abortedLastRun.
Real environment tested: AWS Crabbox, provider aws, lease cbx_12b1656eac41, run run_0c7d40c7e54e, Linux c7a.8xlarge, Node 22.22.2 bootstrapped in the run.
Exact steps or command run after this patch: synced the patched dirty branch to Crabbox, built OpenClaw, started a gateway with isolated state and a local slow OpenAI-compatible model endpoint, began a main agent call, changed config to trigger a restart with gateway.reload.deferralTimeoutMs=1200, then inspected the persisted session row after the forced restart marker.
Evidence after fix: run run_0c7d40c7e54e printed before_restart status=running abortedLastRun=false hasActiveRun=null, then after_restart status=running abortedLastRun=true hasActiveRun=null, plus [main-session-restart-recovery] marked 1 interrupted main session(s) for restart recovery (config reload forced restart).
Observed result after fix: the harness printed FIX_CONFIRMED issue=82433 active session marked restart-aborted and exited 0.
What was not tested: actual provider auth and channel delivery were not exercised; the model endpoint was a local OpenAI-compatible slow-response server to isolate restart/session behavior.
Before evidence: AWS Crabbox main-branch repro cbx_3da5ccc6840a, run run_206ed4b243f8, showed after_restart status=running abortedLastRun=false hasActiveRun=null.

Root Cause (if applicable)

  • Root cause: stale transcript-lock cleanup could only mark interrupted sessions after lock cleanup, but the forced restart path did not mark the active main session before aborting/restarting, so a running session could survive without abortedLastRun.
  • Missing detection / guardrail: coverage did not exercise forced restart with active embedded/reply run session metadata or custom session stores.
  • Contributing context (if known): restart deferral already knew active embedded run counts, but not enough prepared session identity was carried into the pre-restart recovery marker.

Regression Test Plan (if applicable)

  • Coverage level that should have caught this:
    • Unit test
    • Seam / integration test
    • End-to-end test
    • Existing coverage already sufficient
  • Target test or file: src/agents/main-session-restart-recovery.test.ts, src/gateway/server-reload-handlers.test.ts, src/cli/gateway-cli/run-loop.test.ts
  • Scenario the test should lock in: active main sessions are marked by session id/key during forced restart, custom stores are scanned and recovered, duplicate keys across stores do not mark stale sessions, and marker failures do not block server close.
  • Why this is the smallest reliable guardrail: the behavior crosses gateway restart orchestration and persisted session recovery; focused seam tests cover each boundary without requiring live provider auth.
  • Existing test that already covers this (if any): stale lock recovery tests covered only lock-cleanup marking, not pre-restart active run marking.

User-visible / Behavior Changes

Interrupted main agent sessions are now marked for restart recovery when a forced gateway restart cuts off active model/tool work.

Diagram (if applicable)

Before:
[active main turn] -> [forced gateway restart] -> [session remains running, abortedLastRun=false]

After:
[active main turn] -> [forced gateway restart] -> [session marked abortedLastRun=true] -> [startup recovery can resume/fail it]

Security Impact (required)

  • New permissions/capabilities? No
  • Secrets/tokens handling changed? No
  • New/changed network calls? No
  • Command/tool execution surface changed? No
  • Data access scope changed? No
  • If any Yes, explain risk + mitigation: N/A

Repro + Verification

Environment

  • OS: macOS local focused tests; AWS Crabbox Linux for live repro/fix proof
  • Runtime/container: Node 22.22.2 in AWS proof
  • Model/provider: local OpenAI-compatible slow-response test server
  • Integration/channel (if any): gateway agent RPC with channels skipped
  • Relevant config (redacted): isolated OPENCLAW_STATE_DIR, custom session.store, gateway.reload.mode=restart, gateway.reload.deferralTimeoutMs=1200

Steps

  1. Start gateway with isolated state and a slow local model endpoint.
  2. Start a main agent request and wait for the session row to become running.
  3. Change config to trigger a gateway restart while the run is active.
  4. Wait for forced restart marker and inspect the session row.

Expected

  • Active interrupted main session has abortedLastRun=true after the forced restart marker.

Actual

  • After fix: status=running abortedLastRun=true; harness exited 0.

Evidence

  • Failing test/log before + passing after
  • Trace/log snippets
  • Screenshot/recording
  • Perf numbers (if relevant)

Human Verification (required)

What you personally verified (not just CI), and how:

  • Verified scenarios: focused local regression tests, $codex-review, and AWS Crabbox before/after repro path for Gateway restart timeout can interrupt active agent work without marking sessions for continuation #82433.
  • Edge cases checked: custom session stores, duplicate session keys across stores, session-id-only marking, skipped subagent/cron sessions, marker failure still allowing server close, old/current config store resolution during config reload.
  • What you did not verify: live external model provider auth and post-restart final delivery through a real channel.

Review Conversations

  • I replied to or resolved every bot review conversation I addressed in this PR.
  • I left unresolved only the conversations that still need reviewer or maintainer judgment.

Compatibility / Migration

  • Backward compatible? Yes
  • Config/env changes? No
  • Migration needed? No
  • If yes, exact upgrade steps: N/A

Risks and Mitigations

  • Risk: over-marking sessions in unrelated stores that reuse a session key.
    • Mitigation: restart callers pass active session ids and the marker prefers session-id matching when available; regression test covers duplicate keys across stores.
  • Risk: marker failure could interrupt restart cleanup.
    • Mitigation: run-loop marker failures are caught and logged before server close proceeds.

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime cli CLI command changes agents Agent runtime and tooling labels May 16, 2026
@openclaw-barnacle openclaw-barnacle Bot added size: L maintainer Maintainer-authored PR labels May 16, 2026
@clawsweeper

clawsweeper Bot commented May 16, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge.

Summary
The PR marks active main sessions as restart-aborted before forced gateway restarts and extends recovery to configured custom session stores.

Reproducibility: yes. The linked report plus PR body provide a concrete forced-restart reproduction, and current-main source shows recovery skips running sessions unless abortedLastRun is already true.

Real behavior proof
Sufficient (live_output): The PR body includes copied Crabbox before/after live output showing the same session left unmarked on main and marked abortedLastRun=true after the patch.

Next step before merge
No ClawSweeper repair lane: the PR has a protected maintainer label and no narrow repair finding from this review.

Security
Cleared: The diff changes restart/session-store marking, tests, and changelog only; no new dependency, permission, secret, network, or code-execution surface was found.

Review details

Best possible solution:

Review and land this PR after maintainer approval and CI, keeping the linked issue open until the merge closes it.

Do we have a high-confidence way to reproduce the issue?

Yes. The linked report plus PR body provide a concrete forced-restart reproduction, and current-main source shows recovery skips running sessions unless abortedLastRun is already true.

Is this the best way to solve the issue?

Yes. Marking known active main sessions at the restart boundary, then letting existing startup recovery consume that marker, is a narrow fix that avoids broadening recovery to subagent, cron, ACP, or detached background-task sessions.

What I checked:

  • Current-main bug path: Current main only recovers sessions with status === "running" and abortedLastRun === true, so a forced restart that leaves a running session unmarked is skipped by startup recovery. (src/agents/main-session-restart-recovery.ts:288, 3b2cd0dd1a85)
  • Current-main restart gap: Current main's config reload restart deferral emits a restart after timeout without a durable marker for the active main session identities. (src/gateway/server-reload-handlers.ts:448, 3b2cd0dd1a85)
  • PR marker implementation: The PR adds markRestartAbortedMainSessions, resolving configured/default session stores and setting abortedLastRun only on matching running main sessions while skipping subagent, cron, and ACP sessions. (src/agents/main-session-restart-recovery.ts:91, 34139a42f32f)
  • PR restart-boundary integration: The PR collects active embedded/reply session keys and ids, then marks active main sessions before forced config-reload restart emission and before forced/drain-timeout run-loop restarts. (src/cli/gateway-cli/run-loop.ts:463, 34139a42f32f)
  • PR test coverage: The diff adds focused tests for matching active sessions, custom session stores, duplicate keys across stores, config reload forced restart marking, and run-loop forced/drain-timeout marker behavior. (src/agents/main-session-restart-recovery.test.ts:79, 34139a42f32f)
  • Real behavior proof: The PR body reports AWS Crabbox main-branch before proof with abortedLastRun=false, then patched-branch proof with abortedLastRun=true and FIX_CONFIRMED issue=82433 active session marked restart-aborted. (34139a42f32f)

Likely related people:

  • Peter Steinberger: Recent history on gateway restart/run-loop and session recovery-adjacent paths includes restart intent tracing and many gateway/restart maintenance commits. (role: recent area contributor; confidence: high; commits: 6369bf64cd2b, 680eff63fbf8, f066dd2f31c2; files: src/infra/restart.ts, src/cli/gateway-cli/run-loop.ts, src/gateway/server-reload-handlers.ts)
  • Charles Dusek: Commit history shows work on bounding compaction retry waits and draining embedded runs on restart, which is directly adjacent to this PR's drain-timeout marker path. (role: adjacent restart-drain contributor; confidence: medium; commits: 54be30ef89f5; files: src/cli/gateway-cli/run-loop.ts, src/agents/pi-embedded-runner/run-state.ts, src/agents/pi-embedded-runner/runs.ts)
  • Bridgerz: History shows the reply/restart deferral behavior was previously changed to defer gateway restarts until replies are sent, which is part of the same forced-restart boundary. (role: restart deferral contributor; confidence: medium; commits: ab4a08a82acc; files: src/infra/restart.ts, src/gateway/server-reload-handlers.ts, src/auto-reply/reply/reply-run-registry.ts)

Remaining risk / open question:

  • This pass did not execute the changed tests because the target checkout was constrained to read-only inspection; maintainers should confirm CI or focused Testbox results before merge.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 3b2cd0dd1a85.

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. P1 High-priority user-facing bug, regression, or broken workflow. labels May 16, 2026
@joshavant
joshavant merged commit 99a269f into main May 16, 2026
186 of 196 checks passed
@joshavant
joshavant deleted the fix/restart-aborted-main-session-markers branch May 16, 2026 23:11
galiniliev pushed a commit to galiniliev/openclaw that referenced this pull request May 20, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 24, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 24, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 24, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request May 24, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
galiniliev pushed a commit to galiniliev/openclaw that referenced this pull request May 25, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 26, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 26, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
SebTardif pushed a commit to SebTardif/openclaw that referenced this pull request May 26, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
jameslcowan pushed a commit to jameslcowan/openclaw that referenced this pull request Jun 2, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
SYU8384 pushed a commit to SYU8384/openclaw that referenced this pull request Jun 3, 2026
* fix(agents): mark interrupted sessions before restart

* docs: add restart recovery changelog

* fix(agents): satisfy restart recovery type checks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling cli CLI command changes gateway Gateway runtime maintainer Maintainer-authored PR P1 High-priority user-facing bug, regression, or broken workflow. proof: sufficient ClawSweeper judged the real behavior proof convincing. size: L

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gateway restart timeout can interrupt active agent work without marking sessions for continuation

1 participant