Skip to content

Commit b81666c

Browse files
sallyomsallyom
authored
Fix container image upgrade migrations before gateway readiness (#101881)
* run all 'openclaw upgrade' migrations with container image upgrades Signed-off-by: sallyom <[email protected]> * fix: block gateway startup on plugin repair warnings Signed-off-by: sallyom <[email protected]> --------- Signed-off-by: sallyom <[email protected]>
1 parent aa27ae9 commit b81666c

13 files changed

Lines changed: 995 additions & 108 deletions

‎docs/cli/doctor.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -168,6 +168,13 @@ openclaw doctor --lint --all --skip core/doctor/session-locks
168168

169169
`openclaw doctor --post-upgrade` runs plugin compatibility probes for chaining after a build or upgrade. Findings go to stdout; exit code is 1 if any finding has `level: "error"`. Add `--json` for a machine-readable envelope (`{ probesRun, findings }`), suitable for CI, the community `fork-upgrade` skill, and other post-upgrade smoke tooling. If the installed plugin index is missing or malformed, JSON mode still emits the envelope with a `plugin.index_unavailable` error finding.
170170

171+
Container image startup is the exception to the usual "run doctor after
172+
updating" flow. When `openclaw gateway run` starts on a new OpenClaw version, it
173+
runs safe state and plugin repairs before reporting ready. If repair cannot
174+
finish safely, startup exits and tells you to run the same image once with
175+
`openclaw doctor --fix` against the same mounted state/config before restarting
176+
the container normally.
177+
171178
## Notes
172179

173180
- In Nix mode (`OPENCLAW_NIX_MODE=1`), read-only doctor checks still work, but `doctor --fix`, `doctor --repair`, `doctor --yes`, and `doctor --generate-gateway-token` are disabled because `openclaw.json` is immutable. Edit the Nix source for this install instead; for nix-openclaw, use the agent-first [Quick Start](https://github.com/openclaw/nix-openclaw#quick-start).

‎docs/docs_map.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4200,6 +4200,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
42004200
- H2: Prerequisites
42014201
- H2: Containerized gateway
42024202
- H3: Manual flow
4203+
- H3: Upgrading container images
42034204
- H3: Environment variables
42044205
- H3: Observability
42054206
- H3: Health checks
@@ -4468,6 +4469,7 @@ Do not edit it by hand; run `pnpm docs:map:gen`.
44684469
- H2: Podman and Tailscale
44694470
- H2: Systemd (Quadlet, optional)
44704471
- H2: Config, env, and storage
4472+
- H2: Upgrading images
44714473
- H2: Useful commands
44724474
- H2: Troubleshooting
44734475
- H2: Related

‎docs/gateway/doctor.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -412,7 +412,9 @@ That stages grounded durable candidates into the short-term dreaming store while
412412
<Accordion title="7b. Plugin install cleanup">
413413
Doctor removes legacy OpenClaw-generated plugin dependency staging state in `openclaw doctor --fix` / `openclaw doctor --repair` mode: stale generated dependency roots, old install-stage directories, package-local debris from earlier bundled-plugin dependency repair code, and orphaned or recovered managed npm copies of bundled `@openclaw/*` plugins that can shadow the current bundled manifest. Doctor also relinks the host `openclaw` package into managed npm plugins that declare `peerDependencies.openclaw`, so package-local runtime imports such as `openclaw/plugin-sdk/*` keep resolving after updates or npm repairs.
414414

415-
Doctor can also reinstall missing downloadable plugins when config references them but the local plugin registry cannot find them (material `plugins.entries`, configured channel/provider/search settings, configured agent runtimes). During package updates, doctor avoids running package-manager plugin repair while the core package is being swapped; run `openclaw doctor --fix` again after the update if a configured plugin still needs recovery. Gateway startup and config reload do not run package managers; plugin installs remain explicit doctor/install/update work.
415+
Doctor can also reinstall missing downloadable plugins when config references them but the local plugin registry cannot find them (material `plugins.entries`, configured channel/provider/search settings, configured agent runtimes). During package updates, doctor avoids reinstalling plugin packages while the core package is being swapped; run `openclaw doctor --fix` again after the update if a configured plugin still needs recovery. Outside the container image startup exception below, gateway startup and config reload do not run package repair; plugin installs remain explicit doctor/install/update work.
416+
417+
Containerized gateway startup has a narrow upgrade exception: when `openclaw gateway run` starts on a new OpenClaw version, it runs safe state migrations and the existing post-core plugin convergence before readiness, then records a per-version checkpoint. This startup pass can clean stale bundled-plugin records, repair local plugin links, reinstall configured plugin packages when the convergence path requires it, and check active plugin payloads. If startup cannot repair safely, run the same image once with `openclaw doctor --fix` against the same mounted state/config before restarting the container normally.
416418

417419
</Accordion>
418420
<Accordion title="8. Gateway service migrations and cleanup hints">

‎docs/install/docker.md‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -115,6 +115,28 @@ docker compose up -d openclaw-gateway
115115
Run `docker compose` from the repo root. If you enabled `OPENCLAW_EXTRA_MOUNTS` or `OPENCLAW_HOME_VOLUME`, the setup script writes `docker-compose.extra.yml`; include it after any `docker-compose.override.yml` you maintain yourself, e.g. `-f docker-compose.yml -f docker-compose.override.yml -f docker-compose.extra.yml`.
116116
</Note>
117117

118+
### Upgrading container images
119+
120+
When you replace the OpenClaw image but keep the same mounted state/config, the
121+
new gateway runs startup-safe upgrade migrations and plugin convergence before
122+
readiness. Routine image upgrades should not require a separate
123+
`openclaw doctor --fix` pass.
124+
125+
If startup cannot complete those repairs safely, the gateway exits instead of
126+
reporting healthy. With a restart policy, Docker, Podman, or Kubernetes may show
127+
the gateway container restarting. Keep the mounted state volume, then run the
128+
same image once with `openclaw doctor --fix` as the container command, using the
129+
same state/config mounts the gateway uses:
130+
131+
```bash
132+
docker run --rm -v <openclaw-state>:/home/node/.openclaw <image> openclaw doctor --fix
133+
podman run --rm -v <openclaw-state>:/home/node/.openclaw <image> openclaw doctor --fix
134+
```
135+
136+
After doctor finishes, restart the gateway container with its default command.
137+
In Kubernetes, run the same command in a one-off Job or debug pod mounted to the
138+
same PVC, then restart the Deployment or StatefulSet.
139+
118140
### Environment variables
119141

120142
Optional variables accepted by `scripts/docker/setup.sh` (and, for the gateway container, by `docker-compose.yml` directly):

‎docs/install/podman.md‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -153,6 +153,35 @@ Useful env vars for the manual launcher (persist these in `~/.openclaw/.env`; th
153153

154154
If you use a non-default `OPENCLAW_CONFIG_DIR` or `OPENCLAW_WORKSPACE_DIR`, set the same variables for both `./scripts/podman/setup.sh` and later `./scripts/run-openclaw-podman.sh launch` commands -- the repo-local launcher does not persist custom path overrides across shells.
155155

156+
## Upgrading images
157+
158+
After you rebuild or pull a new image, restart the container or Quadlet service.
159+
On first startup for a new OpenClaw version, the gateway runs safe state and
160+
plugin repairs before reporting ready.
161+
162+
If the gateway exits instead of becoming ready, run the same image once with
163+
`openclaw doctor --fix` against the same mounted state/config, then restart the
164+
gateway normally:
165+
166+
```bash
167+
OPENCLAW_CONFIG_DIR="${OPENCLAW_CONFIG_DIR:-$HOME/.openclaw}"
168+
OPENCLAW_WORKSPACE_DIR="${OPENCLAW_WORKSPACE_DIR:-$OPENCLAW_CONFIG_DIR/workspace}"
169+
OPENCLAW_PODMAN_IMAGE="${OPENCLAW_PODMAN_IMAGE:-${OPENCLAW_IMAGE:-openclaw:local}}"
170+
171+
podman run --rm -it \
172+
--userns=keep-id \
173+
--user "$(id -u):$(id -g)" \
174+
-e HOME=/home/node \
175+
-e NPM_CONFIG_CACHE=/home/node/.openclaw/.npm \
176+
-v "$OPENCLAW_CONFIG_DIR:/home/node/.openclaw:rw" \
177+
-v "$OPENCLAW_WORKSPACE_DIR:/home/node/.openclaw/workspace:rw" \
178+
"$OPENCLAW_PODMAN_IMAGE" \
179+
openclaw doctor --fix
180+
```
181+
182+
On SELinux hosts, add `,Z` to both bind mounts if Podman blocks access to the
183+
mounted state.
184+
156185
## Useful commands
157186

158187
- **Container logs:** `podman logs -f openclaw`
@@ -165,6 +194,7 @@ If you use a non-default `OPENCLAW_CONFIG_DIR` or `OPENCLAW_WORKSPACE_DIR`, set
165194

166195
- **Permission denied (EACCES) on config or workspace:** The container runs with `--userns=keep-id` and `--user <your uid>:<your gid>` by default. Ensure the host config/workspace paths are owned by your current user.
167196
- **Gateway start blocked (missing `gateway.mode=local`):** Ensure `~/.openclaw/openclaw.json` exists and sets `gateway.mode="local"`. `scripts/podman/setup.sh` creates this if missing.
197+
- **Container restarts after an image update:** Run the one-off `openclaw doctor --fix` command in [Upgrading images](#upgrading-images), then start the gateway again.
168198
- **Container CLI commands hit the wrong target:** Use `openclaw --container <name> ...` explicitly, or export `OPENCLAW_CONTAINER=<name>` in your shell.
169199
- **`openclaw update` fails with `--container`:** Expected. Rebuild/pull the image, then restart the container or the Quadlet service.
170200
- **Quadlet service does not start:** Run `systemctl --user daemon-reload`, then `systemctl --user start openclaw.service`. On headless systems you may also need `sudo loginctl enable-linger "$(whoami)"`.

‎docs/install/updating.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,11 @@ title: "Updating"
88

99
Keep OpenClaw up to date.
1010

11+
For Docker, Podman, and Kubernetes image replacements, see
12+
[Upgrading container images](/install/docker#upgrading-container-images). The
13+
gateway runs startup-safe upgrade work before readiness and exits if mounted
14+
state needs manual repair.
15+
1116
## Recommended: `openclaw update`
1217

1318
Detects your install type (npm or git), fetches the latest version, runs `openclaw doctor`, and restarts the gateway.

‎src/cli/program/config-guard.test.ts‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -206,6 +206,27 @@ describe("ensureConfigReady", () => {
206206
});
207207
});
208208

209+
it("requires a startup migration checkpoint for foreground gateway startup", async () => {
210+
await runEnsureConfigReady(["gateway"]);
211+
212+
expect(loadAndMaybeMigrateDoctorConfigMock).toHaveBeenCalledWith({
213+
migrateState: true,
214+
migrateLegacyConfig: false,
215+
invalidConfigNote: false,
216+
requireStartupMigrationCheckpoint: true,
217+
});
218+
});
219+
220+
it("does not require a startup migration checkpoint for gateway probes", async () => {
221+
await runEnsureConfigReady(["gateway", "health"]);
222+
223+
expect(loadAndMaybeMigrateDoctorConfigMock).toHaveBeenCalledWith({
224+
migrateState: true,
225+
migrateLegacyConfig: false,
226+
invalidConfigNote: false,
227+
});
228+
});
229+
209230
it("runs doctor flow for legacy sessions without task sidecars", async () => {
210231
const root = useTempOpenClawHome();
211232
fs.mkdirSync(path.join(root, ".openclaw", "sessions"), { recursive: true });

‎src/cli/program/config-guard.ts‎

Lines changed: 13 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -10,14 +10,7 @@ import { resolveRequiredHomeDir } from "../../infra/home-dir.js";
1010
import type { RuntimeEnv } from "../../runtime.js";
1111
import { shouldMigrateStateFromPath } from "../argv.js";
1212

13-
const ALLOWED_INVALID_COMMANDS = new Set([
14-
"audit",
15-
"doctor",
16-
"logs",
17-
"health",
18-
"help",
19-
"status",
20-
]);
13+
const ALLOWED_INVALID_COMMANDS = new Set(["audit", "doctor", "logs", "health", "help", "status"]);
2114
const ALLOWED_INVALID_GATEWAY_SUBCOMMANDS = new Set([
2215
"run",
2316
"status",
@@ -177,6 +170,15 @@ function snapshotHasConfiguredSessionStore(
177170
return typeof store === "string" && store.trim().length > 0;
178171
}
179172

173+
function shouldRequireStartupMigrationCheckpoint(commandPath: string[]): boolean {
174+
const commandName = commandPath[0];
175+
const subcommandName = commandPath[1];
176+
return (
177+
commandName === "gateway" &&
178+
(subcommandName === undefined || subcommandName === "run" || subcommandName.trim() === "")
179+
);
180+
}
181+
180182
async function getConfigSnapshot() {
181183
// Tests often mutate config fixtures; caching can make those flaky.
182184
if (process.env.VITEST === "true") {
@@ -212,6 +214,9 @@ export async function ensureConfigReady(params: {
212214
migrateState: true,
213215
migrateLegacyConfig: false,
214216
invalidConfigNote: false,
217+
...(shouldRequireStartupMigrationCheckpoint(commandPath)
218+
? { requireStartupMigrationCheckpoint: true }
219+
: {}),
215220
...(params.beforeStateMigrations
216221
? { beforeStateMigrations: params.beforeStateMigrations }
217222
: {}),

0 commit comments

Comments
 (0)