Skip to content

fix(gateway): re-read config on in-process restart instead of replaying stale snapshot#80041

Closed
jimdawdy-hub wants to merge 1 commit into
openclaw:mainfrom
jimdawdy-hub:codex/fix-gateway-restart-config-drop
Closed

fix(gateway): re-read config on in-process restart instead of replaying stale snapshot#80041
jimdawdy-hub wants to merge 1 commit into
openclaw:mainfrom
jimdawdy-hub:codex/fix-gateway-restart-config-drop

Conversation

@jimdawdy-hub

@jimdawdy-hub jimdawdy-hub commented May 10, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #79947.

The gateway's CLI runner (src/cli/gateway-cli/run.ts) captured the pre-read startup config snapshot once before entering the restart loop and then reused that same snapshot for every iteration. After a SIGUSR1 in-process restart, downstream startup writes (auth bootstrap, plugin auto-enable persistence, control-ui seed) projected the stale snapshot back to disk and silently clobbered any user config written between boot and the restart.

The fix consumes the pre-read snapshot exactly once. Subsequent in-process restart iterations re-read openclaw.json from disk so user writes between restarts survive.

// src/cli/gateway-cli/run.ts
let pendingStartupSnapshot: typeof startupConfigSnapshotRead | undefined =
  startupConfigSnapshotRead;
const startLoop = async () =>
  await runGatewayLoop({
    runtime: defaultRuntime,
    lockPort: port,
    healthHost,
    start: async ({ startupStartedAt } = {}) => {
      const snapshotForThisStart = pendingStartupSnapshot;
      pendingStartupSnapshot = undefined;
      return await startGatewayServer(port, {
        bind,
        auth: authOverride,
        tailscale: tailscaleOverride,
        startupStartedAt,
        ...(snapshotForThisStart ? { startupConfigSnapshotRead: snapshotForThisStart } : {}),
      });
    },
  });

This matches the proposed fix in the issue body — preserve the cold-start optimization on the first iteration, force disk re-read on every iteration after.

Real behavior proof

Behavior or issue addressed: When the OpenClaw gateway runs in an environment without a supervisor marker (OPENCLAW_SYSTEMD_UNIT, INVOCATION_ID, etc. unset — the default in container deployments per src/infra/process-respawn.ts), a SIGUSR1 in-process restart silently overwrites openclaw.json with a snapshot captured at the original boot, dropping any user config written between boot and the restart. Visible to users as "the bot disappeared after I configured it" — bot tokens revert to empty, auth profiles vanish, channels become disabled. See #79947 for the original report including pre/post sha256s and the exact Config overwrite log line.

Real environment tested: Live Docker reproduction on a contributor host (Linux 7.0.3-zen1-2-zen, Arch Linux), Node 24, Docker image node:24-alpine. OpenClaw built locally from this branch (package version 2026.5.8). All SUPERVISOR_HINT_ENV_VARS (OPENCLAW_SYSTEMD_UNIT, INVOCATION_ID, SYSTEMD_EXEC_PID, JOURNAL_STREAM, the launchd hints, OPENCLAW_SERVICE_MARKER, OPENCLAW_SERVICE_KIND) explicitly unset inside the container so detectRespawnSupervisor returns null and the gateway falls through to the in-process restart path documented at src/infra/process-respawn.ts:70 ("container: use in-process restart to keep PID 1 alive"). Gateway invoked with OPENCLAW_ALLOW_ROOT=1 since the container runs as root. Two variants of the same dist/run-D9To-U2i.js were compared: the pre-fix closure shape (BUGGY) and this PR's shape (FIXED). Patch verified to match the source pattern exactly once before swapping.

Exact steps or command run after this patch: Inside node:24-alpine, for each variant of dist/run-D9To-U2i.js: (1) write a minimal openclaw.json with gateway.mode=local, gateway.port=18790, gateway.auth.mode=none. (2) Start the gateway in the background: node dist/index.js gateway run --port 18790 --bind loopback --allow-unconfigured and wait for the [gateway] ready log line (about 6 seconds on the test host). (3) sha256sum /root/.openclaw/openclaw.json to record the baseline. (4) Use node -e to write a restart-required key directly to disk — auth.profiles.repro_79947 = {id, marker: "USER_WROTE_THIS"} and channels.telegram = {enabled: true, botToken: "USER_WROTE_TOKEN"}. (5) sha256sum again to confirm the user write landed. (6) kill -USR1 $GW_PID to trigger the in-process SIGUSR1 restart. (7) sleep 8 for the restart cycle. (8) sha256sum and grep -q USER_WROTE_THIS / grep -q USER_WROTE_TOKEN on /root/.openclaw/openclaw.json to see whether the user write survived. The full driver shell script bind-mounted the source dist/ into /openclaw read-only and copied to a writable /opt/openclaw inside the container.

Evidence after fix: The contributor pasted the full terminal output of the live docker run invocation. Verbatim console output captured from the host (real, non-mock):

BUGGY variant — sha256 + size of openclaw.json from sha256sum and stat -c%s:

disk before user write:  sha=fac21eeb67b3b5f8  size=274
disk after user write:   sha=1a65ebefc47845d8  size=515
sent SIGUSR1; waiting for restart...
disk after restart:      sha=a40253354fc96b38  size=274

Smoking-gun runtime log line printed by the openclaw gateway during the restart iteration (real output captured live, redacted log path is the in-container /root/.openclaw):

2026-05-10T00:52:43.252+00:00 [gateway] signal SIGUSR1 received
2026-05-10T00:52:43.261+00:00 [gateway] received SIGUSR1; restarting
2026-05-10T00:52:43.294+00:00 [gateway] restart mode: in-process restart (container: use in-process restart to keep PID 1 alive)
Config overwrite: /root/.openclaw/openclaw.json
  (sha256 1a65ebefc47845d8495559a2a9f1e9293a866abeb0cf7a80dfdfca4038789fe0
       -> a40253354fc96b38fa5fd39dcd3a7bd3b0d3ffcdac681d56010f5ffdb4137bc9,
   backup=/root/.openclaw/openclaw.json.bak)

Result line emitted by the test driver:

RESULT: auth.profiles.repro_79947 WAS DROPPED (bug present)
RESULT: channels.telegram.botToken WAS DROPPED (bug present)

FIXED variant on the same dist tree, only the closure shape in runGatewayCommand differs:

disk before user write:  sha=2fbd0d8f025788db  size=274
disk after user write:   sha=409d59518ef13671  size=515
sent SIGUSR1; waiting for restart...
disk after restart:      sha=409d59518ef13671  size=515
RESULT: auth.profiles.repro_79947 SURVIVED
RESULT: channels.telegram.botToken SURVIVED

Independent confirmation that the FIXED gateway re-read disk on the restart iteration: the same gateway logged a config validation error against the new auth.profiles.repro_79947 payload that we wrote between boot and SIGUSR1. The buggy gateway never saw that payload because it overwrote the file with the stale snapshot before validation ran. Stack trace excerpt printed live by the gateway runtime:

[gateway] startup failed: Invalid config at /root/.openclaw/openclaw.json.
auth.profiles.repro_79947.provider: Invalid input: expected string, received undefined
auth.profiles.repro_79947.mode: Invalid input (allowed: "api_key", "aws-sdk", "oauth", "token")
auth.profiles.repro_79947: Unrecognized keys: "id", "marker"
    at assertValidGatewayStartupConfigSnapshot (file:///opt/openclaw/dist/server-startup-config--ehPahs8.js:104:8)
    at loadGatewayStartupConfigSnapshot (file:///opt/openclaw/dist/server-startup-config--ehPahs8.js:19:29)
    at async Object.start (file:///opt/openclaw/dist/run-D9To-U2i.js:882:11)
    at async runGatewayLoop (file:///opt/openclaw/dist/run-D9To-U2i.js:403:14)
    at async Object.startLoop (file:///opt/openclaw/dist/run-D9To-U2i.js:875:32)

Observed result after fix: Post-restart sha256 and byte size of openclaw.json match the post-write state exactly (409d59518ef13671 / 515 bytes), versus the BUGGY run where the post-restart sha (a40253354fc96b38) and size (274 bytes) regressed all the way back to the pre-write baseline. The fixed gateway reads openclaw.json from disk on the restart iteration — confirmed both by the surviving content and by the fact that the runtime now surfaces a config validation stack trace for the user-written profile rather than silently overwriting it. Same hardware, same image, same dist tree — only the closure shape in runGatewayCommand differs between the two runs.

What was not tested: End-to-end macOS launchd reproduction. The bug is platform-agnostic — the in-process restart fallback runs the same code on every platform when no supervisor marker is set, per src/infra/process-respawn.ts — but I did not run a live launchd reproduction. I also did not exercise the real openclaw config set CLI flow; the reproduction triggers SIGUSR1 with kill -USR1 and writes the user config directly. The reporter's path used openclaw config set and pairing approve to trigger the same code path; I verified the bug at the SIGUSR1 boundary which is what the fix targets. Three-or-more consecutive in-process restarts in a single gateway run were not exercised live, but the fix is shaped to handle N restarts (snapshot consumed on first, fresh disk read on every iteration after) and the unit regression covers two iterations.

Test plan

  • pnpm test src/cli/gateway-cli/run.option-collisions.test.ts — 19/19 pass, including the new regression that verifies startupConfigSnapshotRead is passed only on the first start call within a single runGatewayLoop
  • pnpm test src/cli/gateway-cli/run-loop.test.ts src/gateway/server-startup-config.recovery.test.ts src/cli/gateway-cli/run.supervised-lock.test.ts — 36/36 pass
  • pnpm check:changed — exit 0 (lint, import-cycle, sidecar-loader, webhook/pairing/account guards)
  • pnpm exec oxfmt --check --threads=1 on touched files — clean
  • Live Docker reproduction (above) — bug present without the patch, gone with the patch

AI-assisted disclosure

  • AI-assisted PR (Claude Haiku 4.5 → Opus 4.7 in Claude Code, on @jimdawdy-hub's machine)
  • Human-driven Docker reproduction performed on the contributor's host (not just AI-generated unit tests)
  • Author confirms understanding of the closure change and the snapshot consumption semantics
  • codex review --base origin/main not run — Codex CLI was not installed on the contributor's machine when this PR was opened. Happy to run it if a maintainer wants that pass before review.
  • Will resolve any bot review conversations as they land

@openclaw-barnacle openclaw-barnacle Bot added cli CLI command changes size: XS triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 10, 2026
@clawsweeper

clawsweeper Bot commented May 10, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I swept through the related work, and this is now duplicate or superseded.

This PR is a valid focused bug fix, but it is now superseded because #80161 merged the same one-shot startup snapshot behavior, regression coverage, and changelog entry into current main.

So I’m closing this here and keeping the remaining discussion on the canonical linked item.

Review details

Best possible solution:

Keep the merged #80161 implementation as the canonical fix and close this duplicate branch.

Do we have a high-confidence way to reproduce the issue?

Yes. The original failure has a high-confidence source path and the PR body supplies live Docker before/after logs; current main no longer fails because the canonical fix has merged.

Is this the best way to solve the issue?

No as a merge path. The code shape is sound, but the same fix is already on main through #80161, so merging this branch would only duplicate that work.

Security review:

Security review cleared: Cleared: the PR only changes gateway startup option plumbing, a focused unit test, and changelog text, with no new dependencies, permissions, or code execution surface.

What I checked:

Likely related people:

  • scoootscooob: Authored and merged fix: reread config on in-process gateway restart #80161, which put the one-shot startup snapshot fix and regression test on current main. (role: merged canonical fix author; confidence: high; commits: e5fe9bdef085, 3eb650944f58; files: src/cli/gateway-cli/run.ts, src/cli/gateway-cli/run.option-collisions.test.ts, CHANGELOG.md)
  • steipete: The behavior appears to date to commit 598004089454fa589c7532e96cef05004f963790, which introduced the duplicate gateway config-load optimization and touched the central startup snapshot files. (role: introduced startup snapshot optimization; confidence: high; commits: 598004089454; files: src/cli/gateway-cli/run.ts, src/gateway/server-startup-config.ts, src/gateway/server.impl.ts)
  • vincentkoc: Recent restart lifecycle work touched the gateway restart loop area that repeatedly invokes the startup callback after SIGUSR1 restarts. (role: adjacent restart lifecycle contributor; confidence: medium; commits: f6f8d74419a1; files: src/cli/gateway-cli/run-loop.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against 9e77a41fcb99; fix evidence: release v2026.5.10-beta.3, commit e5fe9bdef085.

@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 10, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 10, 2026
…ng stale snapshot

The gateway's CLI runner captured the pre-read startup config snapshot once
before entering the restart loop and reused it on every iteration. After a
SIGUSR1 in-process restart, downstream startup writes (auth bootstrap, plugin
auto-enable persistence, control-ui seed) projected the stale snapshot back
to disk, silently clobbering any user config written between gateway boot
and the restart.

Consume the pre-read snapshot exactly once. Subsequent in-process restart
iterations now re-read openclaw.json from disk so user writes between
restarts survive.

Code review
-----------

Approach. The fix introduces a closure-scoped `pendingStartupSnapshot`
variable that holds the pre-read snapshot, then clears it on first read
inside the start callback. This matches the proposed fix in the issue
("only use startupConfigSnapshotRead on the first start call") while
keeping the cold-start optimization intact for the first iteration. An
alternative `isFirstStart` boolean would have been equivalent; the
consume-once shape was chosen because it makes the lifecycle of the
snapshot explicit and avoids a separate flag that has to stay in sync
with the snapshot variable.

Concurrency. `runGatewayLoop`'s `for(;;)` loop
(src/cli/gateway-cli/run-loop.ts:537) calls `params.start` sequentially,
awaiting each call before the next iteration. There is no concurrency on
the closure variable, so no synchronization is needed.

Lock-recovery interaction. `runGatewayLoopWithSupervisedLockRecovery`
(src/cli/gateway-cli/run.ts:382) may re-invoke `startLoop` on a
`GatewayAlreadyRunningLockError`. With this fix, the second invocation has
no snapshot and falls through to a fresh disk read inside the new
`runGatewayLoop` invocation. That is the correct behavior on a retry of a
failed first start; the original buggy code would have replayed the stale
snapshot on the retry too, so this path is also incidentally improved.

Downstream contract. `loadGatewayStartupConfigSnapshot`
(src/gateway/server-startup-config.ts:67) honors `initialSnapshotRead`
when supplied and falls through to `readConfigFileSnapshotWithPluginMetadata`
when absent. The second iteration now exercises the disk-read path that
the existing recovery test
(src/gateway/server-startup-config.recovery.test.ts:244) already covers
in isolation.

Test coverage. New regression in
src/cli/gateway-cli/run.option-collisions.test.ts uses
`runGatewayLoop.mockImplementationOnce` to invoke the start callback twice
and asserts `startupConfigSnapshotRead` is present on the first
`startGatewayServer` call and absent on the second. Composition of this
test with the existing recovery test exercises the full fixed path.

Live behavior proof. Reproduced and verified on `node:24-alpine` with all
`SUPERVISOR_HINT_ENV_VARS` unset to force the in-process restart fallback.
With the buggy closure, an `auth.profiles` entry written to
`openclaw.json` between gateway boot and SIGUSR1 was clobbered by the
gateway's own writeback during restart startup; sha256 and size of the
file reverted to the pre-write state and the gateway logged
"Config overwrite ... -> [stale snapshot]". With the fix applied to the
same `dist/run-D9To-U2i.js`, the user write survived byte-for-byte across
restart, and the gateway surfaced a config validation error on the
user-written profile, independently confirming that the disk re-read
happened. Full evidence (sha256s, sizes, log lines) is in the PR body.

Notes for reviewers. The TypeScript annotation
`typeof startupConfigSnapshotRead | undefined` is technically redundant
because the source type is already nullable, but it reads cleaner at the
declaration site than relying on inference. Happy to drop the annotation
if maintainers prefer. No public seam, gateway protocol surface, or
plugin SDK contract is touched. Codex CLI was not installed on the
contributor's machine, so the recommended `codex review --base origin/main`
pass was skipped; GitHub Codex review will run on the PR.

Fixes openclaw#79947.
@jimdawdy-hub
jimdawdy-hub force-pushed the codex/fix-gateway-restart-config-drop branch from 80c7863 to e11f8af Compare May 10, 2026 01:19
@openclaw-barnacle openclaw-barnacle Bot added proof: supplied External PR includes structured after-fix real behavior proof. and removed triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 10, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 10, 2026
@clawsweeper clawsweeper Bot closed this May 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cli CLI command changes proof: sufficient ClawSweeper judged the real behavior proof convincing. proof: supplied External PR includes structured after-fix real behavior proof. size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: in-process gateway restart silently drops user config (stale startupConfigSnapshotRead reuse)

1 participant