Skip to content

fix(update): restart managed gateway when update handoff fails after stop#92111

Merged
vincentkoc merged 3 commits into
openclaw:mainfrom
yetval:fix/update-handoff-gateway-recovery
Jun 16, 2026
Merged

fix(update): restart managed gateway when update handoff fails after stop#92111
vincentkoc merged 3 commits into
openclaw:mainfrom
yetval:fix/update-handoff-gateway-recovery

Conversation

@yetval

@yetval yetval commented Jun 11, 2026 •
edited by vincentkoc
Loading

Copy link
Copy Markdown
Contributor

Summary

Fixes #92088. When a managed-service update handoff fails after the update command has already stopped openclaw-gateway.service, nothing restarted the gateway, leaving the host with no live gateway until manual recovery (loaded inactive dead, reported on Ubuntu systemd user services).

The detached handoff helper is the only process that survives the update command, so it now performs a best-effort gateway service start whenever the managed update command fails to spawn, exits nonzero, or dies on a signal (covers OOM kills and exit 127 style wrapper failures that bypass the CLI's own restore path):

  • systemd: systemctl --user start <unit> (a no-op when the unit is already active, so failures that happen before the stop, or after the CLI already restored the gateway, do not bounce a healthy gateway)
  • launchd: launchctl kickstart, falling back to enable plus bootstrap with the agent plist (the CLI stop path uses bootout)
  • Windows: schtasks /Run on the gateway task

The service identity (unit/label/task name) is resolved at handoff start from the same env vars and daemon/constants.js resolvers the rest of the daemon code uses, and passed to the helper via its params file as a serviceRecovery descriptor.

Also closes the one CLI failure path that exited without restoring a gateway it had stopped: the plugin post-update sync error branch in src/cli/update-cli/update-command.ts now calls maybeRestartServiceAfterFailedMutableUpdate like the sibling error/skipped branches.

The issue's stale update-restart-pending sentinel note is already handled on main: the helper marks the pending sentinel as error via markUpdateSentinelFailureIfPending on the same failure paths.

Real behavior proof

Behavior addressed: a failed managed update handoff left openclaw-gateway.service stopped (loaded inactive dead, gateway HTTP unreachable) with no automatic recovery.

Real environment tested: Ubuntu Linux (kernel 6.8), systemd user manager, real openclaw-gateway.service installed by openclaw gateway install from this checkout's dist build, gateway serving HTTP 200 on port 18789. The real generated handoff.cjs ran inside a real systemd-run --user --scope transient unit via startManagedServiceUpdateHandoff, with the managed update command replaced by a stub that reproduces the reported timeline: stop openclaw-gateway.service, then exit 127.

Exact steps or command run after this patch: systemctl --user start openclaw-gateway.service, confirm active and HTTP 200, then run a driver that calls startManagedServiceUpdateHandoff({ supervisor: "systemd", env: { ...process.env, OPENCLAW_SYSTEMD_UNIT: "openclaw-gateway.service" }, ... }) against the patched module with the failing update stub, wait for the handoff to finish, then check systemctl --user is-active openclaw-gateway.service, the unit list, HTTP on 18789, and the handoff log.

Evidence after fix:

=== AFTER fix, after failed update:
unit: active
  openclaw-gateway.service loaded active running OpenClaw Gateway (v2026.6.2)
HTTP: 200
=== handoff.log:
[2026-06-11T07:28:36.078Z] starting managed update command: openclaw update --yes
[2026-06-11T07:28:36.086Z] managed update command pid=46776
[2026-06-11T07:28:36.132Z] Stopping openclaw-gateway...
[2026-06-11T07:28:37.081Z] Running openclaw update...
[2026-06-11T07:28:37.086Z] managed update command exited code=127 signal=null
[2026-06-11T07:28:37.108Z] gateway service recovery succeeded target=openclaw-gateway.service

Same scenario on unpatched main (418d7e1) for contrast:

=== BEFORE fix, after failed update:
unit: inactive
  openclaw-gateway.service loaded inactive dead OpenClaw Gateway (v2026.6.2)
HTTP: 000
=== handoff.log:
[2026-06-11T07:27:53.379Z] starting managed update command: openclaw update --yes
[2026-06-11T07:27:53.385Z] managed update command pid=46685
[2026-06-11T07:27:53.434Z] Stopping openclaw-gateway...
[2026-06-11T07:27:57.154Z] Running openclaw update...
[2026-06-11T07:27:57.161Z] managed update command exited code=127 signal=null

No-bounce check on the patched build: an update command that fails with exit 1 before stopping the gateway leaves the running gateway untouched, MainPID before=46787 after=46787 unit=active, and the helper logs gateway service recovery succeeded from the idempotent systemctl --user start.

Observed result after fix: the gateway service is automatically started again within about 20 ms of the failed update command exiting; the host keeps a live gateway instead of a full outage, and successful updates and pre-stop failures see no extra restart.

What was not tested: launchd and schtasks recovery commands were not executed against real macOS/Windows service managers (covered by unit tests asserting the descriptor and the command wiring; the command sequences mirror src/cli/update-cli/restart-helper.ts); the plugin post-update sync error branch in update-command.ts has no existing test harness and was verified by code reading against the sibling error/skipped branches.

Maintainer follow-up

Autoreview found one launchd edge case after the contributor proof: bootstrap can report an already-loaded/in-progress label while the service still needs a start. The final commit retries start-only kickstart after that failure, preserving the no-bounce policy and adding a regression test for the full kickstart → enable → failed bootstrap → kickstart sequence.

Verification

  • node scripts/run-vitest.mjs src/infra/update-managed-service-handoff.test.ts passes (9 tests); the contributor's 3 core regression tests fail on unpatched main and pass with this change
  • pnpm tsgo:core and pnpm tsgo:core:test clean
  • node scripts/run-oxlint.mjs and oxfmt --check clean on the touched files
  • pnpm check:changed passes in Testbox tbx_01kv78qyb9v89ffrtesqekjdgb (run)
  • Fresh exact-head Crabbox systemd proof passes on Ubuntu/Node 24: the real installed gateway stops inside the failing detached update, exits 127, recovers through systemctl --user start, and returns active with HTTP 200 (run)
  • Live systemd before/after repro above

Reported by @ofan.

@openclaw-barnacle openclaw-barnacle Bot added cli CLI command changes size: M proof: supplied External PR includes structured after-fix real behavior proof. labels Jun 11, 2026
@clawsweeper

clawsweeper Bot commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 15, 2026, 12:24 PM ET / 16:24 UTC.

Summary
The PR adds best-effort managed gateway service recovery to the detached update handoff helper and regression coverage for systemd recovery plus launchd and Windows recovery descriptors.

PR surface: Source +93, Tests +160. Total +253 across 2 files.

Reproducibility: yes. Current main clearly exits failed detached handoffs without service recovery, and the PR body supplies live before/after systemd output showing the outage and recovery; I did not rerun the live scenario in this read-only review.

Review metrics: none identified.

Stored data model
Persistent data-model change detected: serialized state: src/infra/update-managed-service-handoff.test.ts. Confirm migration or upgrade compatibility proof before merge.

Merge readiness
Overall: 🦞 diamond lobster
Proof: 🦞 diamond lobster
Patch quality: 🦞 diamond lobster
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Risk before merge

  • [P1] The supplied real-environment proof covers systemd only; launchd and Windows recovery are supported by descriptor and command-path tests plus source-contract comparison, not native service-manager execution.
  • [P1] The patch intentionally uses start-only recovery semantics to avoid bouncing a gateway that was already restored or never stopped; maintainers should explicitly accept that policy across systemd, launchd, and Windows before merge.

Maintainer options:

  1. Accept Focused Platform Evidence (recommended)
    Merge after normal maintainer checks, accepting the live systemd proof plus source-aligned launchd and Windows command coverage for this narrow best-effort recovery path.
  2. Request Native Supervisor Proof
    Pause merge until the failed-handoff scenario is executed against real launchd and Windows Task Scheduler environments, without asking for code churn first.

Next step before merge

  • No automated repair is needed; maintainers should decide whether the systemd live proof plus command-level launchd/Windows coverage is sufficient before merge.

Security
Cleared: No concrete security or supply-chain regression was found; the patch adds no dependencies, workflow changes, lockfile changes, secret handling, or shell-interpolated command execution.

Review details

Best possible solution:

Land the detached-helper recovery after normal checks if maintainers accept the platform-proof gap; otherwise request native launchd and Windows failed-handoff proof without moving the fix out of the helper owner.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main clearly exits failed detached handoffs without service recovery, and the PR body supplies live before/after systemd output showing the outage and recovery; I did not rerun the live scenario in this read-only review.

Is this the best way to solve the issue?

Yes. The detached helper is the best fix location because it is the process that survives after the gateway is stopped; the CLI restart path is not sufficient for child update failures that bypass the normal restore path.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 865e4db1cd0a.

Label changes

Label justifications:

  • P1: A failed managed update can leave the primary gateway stopped, making agent and channel workflows unavailable until manual recovery.
  • merge-risk: 🚨 availability: The patch changes the supervisor recovery path that decides whether a failed managed update leaves the gateway stopped or returns it to service.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (live_output): The PR body provides credible before-and-after live systemd output from a real generated handoff helper, installed gateway service, transient user scope, and HTTP 200 recovery after the fix.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides credible before-and-after live systemd output from a real generated handoff helper, installed gateway service, transient user scope, and HTTP 200 recovery after the fix.
Evidence reviewed

PR surface:

Source +93, Tests +160. Total +253 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 94 1 +93
Tests 1 160 0 +160
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 254 1 +253

What I checked:

Likely related people:

  • shakkernerd: Recently changed the same managed update handoff and gateway update service-identity path that this PR extends. (role: recent area contributor; confidence: high; commits: 6b3bcc986f66, cf6572dea42e, 3394a4ad2ce9; files: src/infra/update-managed-service-handoff.ts, src/gateway/server-methods/update.ts)
  • vincentkoc: Added the supervised auto-update handoff surface that shares this detached helper and is covered by the PR's recovery location. (role: introduced adjacent behavior; confidence: high; commits: fc2a7be0bc95; files: src/infra/update-managed-service-handoff.ts, src/infra/update-startup.ts)
  • steipete: Has repeated recent history in the update restart helper and daemon service command contracts that this recovery path mirrors. (role: adjacent owner; confidence: medium; commits: 4af066b01352, 1ace6a0d6a0d, 81390c643b96; files: src/cli/update-cli/restart-helper.ts, src/daemon/systemd.ts, src/daemon/launchd.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 11, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the cli CLI command changes label Jun 11, 2026
@yetval

yetval commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Jun 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper clawsweeper Bot added rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. and removed proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 11, 2026
@vincentkoc vincentkoc self-assigned this Jun 11, 2026
@yetval

yetval commented Jun 11, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🌊 off-meta tidepool PR readiness rating does not apply to this item. labels Jun 11, 2026
@yetval
yetval force-pushed the fix/update-handoff-gateway-recovery branch from 47cddd9 to 36ec65d Compare June 12, 2026 00:09
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 12, 2026
@clawsweeper clawsweeper Bot added proof: sufficient ClawSweeper judged the real behavior proof convincing. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. labels Jun 12, 2026
@clawsweeper clawsweeper Bot mentioned this pull request Jun 15, 2026
Closed
@vincentkoc
vincentkoc force-pushed the fix/update-handoff-gateway-recovery branch from 36ec65d to bd21e48 Compare June 16, 2026 03:49
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label Jun 16, 2026
@vincentkoc

vincentkoc commented Jun 16, 2026

Copy link
Copy Markdown
Member

Maintainer verification complete on exact head bd21e48da472ae340c2b3c927368b4abfd5b68e3.

  • Best-fix verdict: best. The detached handoff helper is the process that survives the gateway stop, so it owns recovery for child spawn, nonzero-exit, signal, and helper failures. CLI-only recovery cannot cover those failures.
  • Autoreview found one launchd edge case: failed bootstrap could mean an already-loaded label still needed a start. Commit bd21e48da47 adds a final start-only kickstart retry and a regression test; full-branch autoreview is clean afterward.
  • Focused proof: node scripts/run-vitest.mjs src/infra/update-managed-service-handoff.test.ts — 9/9 passed.
  • Changed gate: Testbox tbx_01kv78qyb9v89ffrtesqekjdgb — pnpm check:changed passed (run).
  • Real behavior proof: Crabbox cbx_82afdfb6034e, run run_98bf7b1d0a71 — installed a real systemd user gateway, confirmed active + HTTP 200, stopped it from a detached update stub that exited 127, then confirmed recovery to a new active PID + HTTP 200 (run).
  • Exact-head GitHub checks: 131 passed, 0 failed, 0 pending; 24 inapplicable checks skipped.
  • Remaining proof gap: launchd and Windows are command-contract/unit tested, not rerun on native service managers. Accepted for this narrow best-effort recovery path.

The PR remains three commits, preserving the contributor’s two commits plus the maintainer launchd correction.

@vincentkoc
vincentkoc merged commit 48d96cd into openclaw:main Jun 16, 2026
173 checks passed
@clawsweeper clawsweeper Bot mentioned this pull request Jun 20, 2026
Open
This was referenced Jun 30, 2026
Closed
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

@vincentkoc vincentkoc

Labels

merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. P1 High-priority user-facing bug, regression, or broken workflow. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. size: M status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Updater can leave managed gateway stopped when transient update handoff fails

2 participants

@yetval @vincentkoc