feat(gateway): add cooperative host suspension by steipete · Pull Request #103618 · openclaw/openclaw

steipete · 2026-07-10T09:59:21Z

Partially addresses #86876.

What Problem This Solves

External hosting controllers had no atomic way to verify that a Gateway had stopped accepting new tracked user work and was idle before freezing or snapshotting its process.

A read-only preflight can race with a new RPC, session turn, cron tick, HTTP request, or detached persistence/delivery task. Restart draining is one-way, so it cannot safely represent an abandoned or reversible host snapshot operation.

Why This Change Was Made

This adds a host-neutral Gateway contract instead of a platform-specific API:

gateway.suspend.prepare
gateway.suspend.status
gateway.suspend.resume

prepare closes process-wide root, session, and command admission; pauses automatic cron scheduling; and synchronously inspects tracked active work. It returns ready only when the snapshot is idle. If work is active, it restores scheduling and admission before returning busy.

A ready lease lasts two minutes, can be renewed with the same stable requestId, auto-resumes on expiry, and requires the matching suspensionId for explicit resume. Scheduler-resume failures remain fail-closed and retryable. Restart signaling shares the same admission state, preventing a signal from racing a ready lease or reopening work during restart drain.

The generic fence lives in core. The Admin HTTP plugin exposes the same RPC methods as an alternate authenticated control path. The protocol change is additive and introduces no new config or environment variable.

This is complementary to the durable queue work in #51620 and #82572: the queue preserves accepted work, while this contract creates a bounded cooperative idle window for host snapshots.

User Impact

Host controllers can now coordinate a bounded idle window:

Fence host-controlled ingress.
Call gateway.suspend.prepare.
Retry a busy result while leaving the process running.
Freeze or snapshot before expiresAtMs after a ready result.
Call gateway.suspend.resume after thaw or abandonment.

While prepared, new WebSocket handshakes are rejected; existing connections can call only suspension control methods; other RPCs return retryable UNAVAILABLE; automatic cron ticks remain paused; built-in HTTP user-work routes return 503; /healthz remains live; and /readyz reports not ready. A retained authenticated WebSocket or pre-enabled Admin HTTP RPC route is required for explicit resume.

This is a cooperative tracked-work fence, not general process quiescence. It does not persist incoming messages, stop third-party channel transports, fence external ingress, or prove arbitrary plugin/background work idle. The host remains responsible for fencing ingress and consistently freezing or snapshotting the full process tree and filesystem. If the control connection is lost, admission reopens when the lease expires.

Evidence

Exact final head e475996d946af6810c053f14a69684a6f9ebfb46; Git tree 35b61e79b5b2a8f1a535125081936ca4551afff0.
Source-blind AWS Crabbox run run_45c5519361af built that exact tree and passed the persistent WebSocket/Admin HTTP contract: baseline health/readiness, ready lease, retained status, prepared 200/503 probes, retryable unrelated-RPC refusal, fresh-WebSocket 503, mismatch preservation, matching resume, restored readiness, and idempotent repeat resume.
Exact-tree AWS Crabbox run run_5d1bf9ff69fa passed 159 focused protocol, Gateway, coordinator, cron, admission, detached agent/chat, and followup-drain tests across six Vitest shards.
Fresh high-reasoning autoreview found no actionable defect (patch is correct, confidence 0.88).
Final range-diff maps all 15 pre/post-rebase commits exactly; the detached-work patch-id is unchanged, current-main file overlap is zero, and merge-tree is clean.

Generated TypeBox protocol schemas, Swift client models, and the docs map are current. vgateway.quiesce is not introduced; the public API is the host-neutral gateway.suspend.* family.

clawsweeper · 2026-07-10T12:10:26Z

Codex review: needs real behavior proof before merge. Reviewed July 10, 2026, 3:19 PM ET / 19:19 UTC.

Summary
Adds three cooperative Gateway suspension RPCs plus a reversible process-wide admission fence integrated with cron, restart, HTTP/WebSocket ingress, detached lifecycle work, Admin HTTP, protocol clients, and documentation.

Reproducibility: not applicable. This PR introduces a new lifecycle capability rather than reproducing an established current-main bug.

Review metrics: 3 noteworthy metrics.

Public lifecycle methods: 3 added. They establish a new operator wire contract over WebSocket and Admin HTTP.
Tracked blocker categories: 12 kinds. A ready result depends on complete accounting across queues, replies, runs, tasks, sessions, and persistence.
Configuration surface: 0 added, 0 changed, 0 removed. Upgrade risk is behavioral and operational rather than config-migration based.

Stored data model
Persistent data-model change detected: serialized state: src/auto-reply/reply/session-hooks-context.test.ts, serialized state: src/auto-reply/reply/session-updates.lifecycle.test.ts, serialized state: src/auto-reply/reply/session-updates.ts, serialized state: src/auto-reply/reply/session.ts, serialized state: src/gateway/session-reset-service.ts, serialized state: src/sessions/session-lifecycle-admission.test.ts, and 9 more. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🐚 platinum hermit
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

[P1] Add redacted exact-head live transcripts for both suspension control paths.
Obtain explicit Gateway owner approval for the public lifecycle contract.

Proof guidance:

[P1] Needs stronger real behavior proof before merge: The linked workflows are cancelled and target older SHAs; add redacted exact-head terminal or live output for both WebSocket and Admin HTTP paths, then update the PR body to trigger a fresh ClawSweeper review.

Risk before merge

[P1] A missed detached state-changing path could allow a false-ready snapshot, while an incorrectly retained root or failed scheduler recovery could prolong Gateway unavailability.
[P1] Prepared suspension intentionally rejects new connections and ordinary RPC/HTTP work; operators need a retained authenticated control path or must wait for lease expiry.
[P1] The three public RPCs and their fail-closed semantics can become long-term host-controller and plugin compatibility obligations.

Maintainer options:

Add exact-head operational proof (recommended)
Provide redacted current-head output covering busy and ready prepare, ordinary RPC rejection, health/readiness probes, mismatched and matching resume, expiry, WebSocket, and Admin HTTP.
Narrow the public commitment
Remove or defer the RPC exposure while retaining internal safety primitives if maintainers are not ready to own the compatibility contract.
Pause the draft
Defer the XL lifecycle change if its availability semantics and ongoing audit burden are not accepted.

Next step before merge

[P1] The remaining blockers are contributor-supplied exact-head proof and maintainer product acceptance, not a narrow automated code repair.

Maintainer decision needed

Question: Should OpenClaw support gateway.suspend.prepare, gateway.suspend.status, and gateway.suspend.resume with these reversible process-wide admission semantics as a public Gateway lifecycle contract?
Rationale: The API deliberately changes service availability and creates a durable operator-facing compatibility commitment; correctness review cannot decide that product boundary.
Likely owner: steipete — Current-main history most strongly connects Peter to the affected safe-restart, Gateway lifecycle, and protocol surfaces.
Options:
- Adopt after exact-head proof (recommended): Accept the public contract once both authenticated transports and all recovery states are demonstrated on the reviewed SHA.
- Keep the primitives internal: Retain generic admission and coordinator code while deferring public RPC exposure to a separately approved API design.
- Defer cooperative suspension: Pause the branch until host snapshot lifecycle support is an explicit product priority.

Security
Cleared: No dependency, workflow, package-resolution, secret, or permission expansion concern was found; the suspension methods retain explicit authenticated operator scopes and control-plane policy.

Review details

Best possible solution:

Keep one canonical core admission/coordinator path, but merge only after redacted exact-head live transcripts prove both WebSocket and Admin HTTP flows and the Gateway owner explicitly accepts the cooperative—not total-process—quiescence contract.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR introduces a new lifecycle capability rather than reproducing an established current-main bug.

Is this the best way to solve the issue?

Yes, provisionally: an atomic core fence is the narrowest way to prevent races that an observational plugin cannot close, and one canonical coordinator is preferable to platform-specific snapshot hooks; exact-head proof and product acceptance remain required.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against e1934d968e43.

Label changes

Label changes:

add rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
add feature: ✨ showcase: ClawSweeper spotlight: unusually compelling feature idea for maintainer attention. An atomic, bounded idle lease could unlock safe host snapshots and reversible scale-down coordination without embedding platform-specific hosting policy in core.
remove rating: 🧂 unranked krab: Current PR rating is rating: 🦪 silver shellfish, so this older rating label is no longer current.

Label justifications:

P2: This is a broad, normal-priority Gateway lifecycle feature rather than an urgent user regression.
merge-risk: 🚨 compatibility: The new public RPC names and result semantics can become stable host-controller and plugin dependencies.
merge-risk: 🚨 availability: Admission and scheduler-recovery behavior can reject normal traffic or keep the Gateway unavailable.
rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🐚 platinum hermit.
feature: ✨ showcase: ClawSweeper spotlight: unusually compelling feature idea for maintainer attention. An atomic, bounded idle lease could unlock safe host snapshots and reversible scale-down coordination without embedding platform-specific hosting policy in core.
status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The linked workflows are cancelled and target older SHAs; add redacted exact-head terminal or live output for both WebSocket and Admin HTTP paths, then update the PR body to trigger a fresh ClawSweeper review.

Evidence reviewed

What I checked:

Current-main necessity: The reviewed current main does not contain the suspension coordinator, work-admission owner, or gateway.suspend RPC implementation, so the capability remains unique to this PR. (src/infra/restart-coordinator.ts:1, e1934d968e43)
Canonical admission owner: The branch centralizes nested roots, independent detached work, retained continuations, restart-signal fencing, reversible preparation, and generation-bound release in one process-wide state owner. (src/process/gateway-work-admission.ts:106, e475996d946a)
Atomic coordinator: Preparation closes admission, pauses scheduling, synchronously inspects tracked work, rolls busy attempts back, commits idle leases, auto-expires, and stays fail-closed when scheduler resume fails. (src/infra/gateway-suspend-coordinator.ts:260, e475996d946a)
Authorization and recovery surface: Prepare is operator.admin and rate-limited, status is operator.read, and resume is an operator.admin escape hatch deliberately excluded from write throttling. (src/gateway/methods/core-descriptors.ts:292, e475996d946a)
Focused invariant coverage: Tests cover concurrent root blockers, exclusion of the prepare caller, rejection of ordinary read/write methods, reachable suspension status, and rejection of nested prepare calls. (src/gateway/server-methods.suspension-admission.test.ts:140, e475996d946a)
Proof gap: The cited focused Testbox run was cancelled and executed older SHA 7fe004d rather than the reviewed head. (7fe004d85251)

Likely related people:

steipete: Current-main blame and concentrated history connect Peter to safe restart activity inspection, Gateway lifecycle, and protocol behavior central to this PR. (role: feature owner and recent area contributor; confidence: high; commits: e43f225c814d, 5aea34ad8c21; files: src/infra/restart-coordinator.ts, src/infra/restart.ts, src/process/command-queue.ts)
vincentkoc: Area shortlog shows repeated current-main contributions across the sampled Gateway, restart, and protocol surfaces relevant to owner review. (role: recent adjacent contributor; confidence: medium; commits: e085fa1a3ffd; files: src/infra/restart.ts, src/infra/restart-coordinator.ts, packages/gateway-protocol/src/index.ts)
xingzhou: The latest merged work identified in prior review context touched command queue and restart coordination, providing useful review context for the admission changes. (role: recent adjacent contributor; confidence: low; commits: 4d004e2054ba; files: src/process/command-queue.ts, src/infra/restart.ts, src/infra/restart-coordinator.ts)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

Review history (2 earlier review cycles)

reviewed 2026-07-10T12:08:52.340Z sha f915754 :: needs real behavior proof before merge. :: none
reviewed 2026-07-10T18:05:10.291Z sha 0309c02 :: needs real behavior proof before merge. :: none

steipete · 2026-07-10T19:24:05Z

Maintainer proof complete for exact head e475996d946af6810c053f14a69684a6f9ebfb46 (tree 35b61e79b5b2a8f1a535125081936ca4551afff0).

Review fixes now included:

transactional cron re-arm and explicit fail-closed scheduler recovery;
stale AsyncLocalStorage descendants rejected after root retirement;
detached agent and chat.send dispatch retain their request root through cleanup;
followup/collect drains own an independent root across debounce and retries;
clearing a debouncing queue aborts the timer and releases its root immediately;
dispatch-error lifecycle projection stays tracked when restart drain overlaps.

Exact proof:

corepack pnpm test packages/gateway-protocol/src/gateway-suspend.test.ts src/gateway/server-methods/suspend.test.ts src/gateway/server-methods.suspension-admission.test.ts src/infra/gateway-suspend-coordinator.test.ts src/cron/service.every-jobs-fire.test.ts src/process/gateway-work-admission.test.ts src/gateway/server.agent.gateway-server-agent-a.test.ts src/gateway/server.chat.gateway-server-chat.test.ts src/auto-reply/reply/queue.drain-restart.test.ts
- 159 tests, six Vitest shards, clean fresh-PR checkout: AWS Crabbox run_5d1bf9ff69fa
corepack pnpm build plus source-blind persistent WebSocket/Admin HTTP validation: AWS Crabbox run_45c5519361af
- baseline /healthz and /readyz 200;
- idle prepare ready, activeCount: 0, no blockers;
- while prepared: health 200, readiness 503, unrelated RPC retryable UNAVAILABLE, fresh WebSocket 503;
- mismatched resume 400 and lease preserved;
- matching resume 200/resumed: true, readiness restored;
- repeated resume resumed: false, ordinary RPC recovered.
Fresh high-reasoning autoreview: no actionable findings (patch is correct, confidence 0.88).
Final range-diff: all 15 commits preserved exactly; current-main overlap zero; merge-tree clean.

Known scope boundary: this is a cooperative tracked-work fence, not third-party channel shutdown or an OS/process-tree snapshot adapter. #86876 remains open for those broader controller/plugin concerns.

@clawsweeper re-review

clawsweeper · 2026-07-10T19:24:09Z

🦞👀
ClawSweeper picked this up.

Command router queued. I will update this comment with the next step.

steipete · 2026-07-10T19:24:58Z

Merged via squash.

Prepared head SHA: e475996d946af6810c053f14a69684a6f9ebfb46
Landed commit: 1bcc4c5e7063e2614208e7408be181ade80e0335

* feat(gateway): add cooperative suspension preparation * style: satisfy suspension lint checks * test(gateway): reset work admission between shared suites * fix(gateway): reject upgrades during suspension * fix(gateway): preserve admitted work during suspension * test(gateway): isolate suspension and restart state * fix(gateway): close suspension false-ready gaps * refactor(protocol): slim suspension declaration graph * refactor(plugin-sdk): sever protocol registry edges * fix(gateway): preserve admitted restart follow-ups * fix(gateway): make suspension recovery fail closed * fix(protocol): keep validation formatter re-export only * test(gateway): simplify deferred fixture type * style(gateway): clarify suspension entry name * fix(gateway): retain detached work admission

openclaw-barnacle Bot added docs Improvements or additions to documentation app: web-ui App: web-ui gateway Gateway runtime cli CLI command changes scripts Repository scripts agents Agent runtime and tooling extensions: admin-http-rpc size: XL maintainer Maintainer-authored PR labels Jul 10, 2026

steipete self-assigned this Jul 10, 2026

steipete force-pushed the codex/gateway-suspend-prepare branch 3 times, most recently from 296414e to 0309c02 Compare July 10, 2026 17:39

steipete added 11 commits July 10, 2026 14:49

feat(gateway): add cooperative suspension preparation

e6c8d36

style: satisfy suspension lint checks

c4b00cf

test(gateway): reset work admission between shared suites

6b6c1e2

fix(gateway): reject upgrades during suspension

ba3d7fe

fix(gateway): preserve admitted work during suspension

f7d97a4

test(gateway): isolate suspension and restart state

c80a91f

fix(gateway): close suspension false-ready gaps

714e817

refactor(protocol): slim suspension declaration graph

e0eae82

refactor(plugin-sdk): sever protocol registry edges

9f50209

fix(gateway): preserve admitted restart follow-ups

f73d500

fix(gateway): make suspension recovery fail closed

fce8cc5

steipete force-pushed the codex/gateway-suspend-prepare branch from c18474c to e475996 Compare July 10, 2026 18:49

steipete marked this pull request as ready for review July 10, 2026 19:24

steipete requested a review from a team as a code owner July 10, 2026 19:24

steipete merged commit 1bcc4c5 into main Jul 10, 2026
116 of 120 checks passed

steipete deleted the codex/gateway-suspend-prepare branch July 10, 2026 19:24

steipete mentioned this pull request Jul 10, 2026

fix: restore validation after suspension coverage #103924

Closed

vincentkoc mentioned this pull request Jul 10, 2026

fix(gateway): restore cooperative suspension validation #103925

Merged

steipete mentioned this pull request Jul 10, 2026

fix(gateway): restore suspension CI gates #103926

Closed

github-actions Bot mentioned this pull request Jul 10, 2026

📡 Upstream Digest — 2026-07-10 21:00 UTC curtismercier/openclaw-mods#1254

Open

clawsweeper Bot mentioned this pull request Jul 10, 2026

fix(gateway): reload plugin code across in-process restarts (#103571) #103688

Open

This was referenced Jul 11, 2026

[Bug]: Plugin HTTP work bypasses Gateway suspension admission #104077

Closed

[Feature]: Add a packaged Gateway host-lifecycle Docker contract #104081

Closed

This was referenced Jul 11, 2026

fix(gateway): admit plugin HTTP work during suspension #104112

Merged

fix: suspension and restart wait for background execs #104172

Merged

clawsweeper Bot mentioned this pull request Jul 11, 2026

[Feature]: Global agent killswitch (pause/abort) triggerable via Signal and CLI #104315

Open

This was referenced Jul 12, 2026

fix(gateway): suspension waits for accepted node work #105134

Merged

[Feature]: Add plugin-provided busy-state and lease signals for on-demand container runtimes #86876

Closed

This was referenced Jul 21, 2026

RFC 0013: SQLite snapshot backup artifacts openclaw/rfcs#20

Open

RFC 0013 addendum: recovery lifecycle sidecars openclaw/rfcs#46

Open

ekinnee mentioned this pull request Jul 22, 2026

fix: keep streaming OpenAI-compatible requests admitted #111779

Closed

steipete mentioned this pull request Jul 25, 2026

[Bug]: All stream:true /v1/chat/completions requests fail with GatewayDrainingError (2026.7.2-beta.3) — streaming continuation outlives its root-work admission #111752

Closed

This was referenced Jul 26, 2026

[Feature]: Define a safe scale-to-zero recovery contract for Gateway hosts #114145

Open

feat(snapshot): capture final recovery points #112865

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

feat(gateway): add cooperative host suspension#103618

feat(gateway): add cooperative host suspension#103618
steipete merged 15 commits into
mainfrom
codex/gateway-suspend-prepare

steipete commented Jul 10, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

steipete commented Jul 10, 2026

Uh oh!

clawsweeper Bot commented Jul 10, 2026

Uh oh!

Uh oh!

steipete commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Uh oh!

Conversation

steipete commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Problem This Solves

Why This Change Was Made

User Impact

Evidence

Uh oh!

clawsweeper Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

steipete commented Jul 10, 2026

Uh oh!

clawsweeper Bot commented Jul 10, 2026

Uh oh!

Uh oh!

steipete commented Jul 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

steipete commented Jul 10, 2026 •

edited

Loading

clawsweeper Bot commented Jul 10, 2026 •

edited

Loading