Skip to content

Commit 564eb5f

Browse files
steipetesteipete
committed
fix(updater): build snapshot control client safely
1 parent 4139938 commit 564eb5f

3 files changed

Lines changed: 141 additions & 13 deletions

File tree

‎.agents/skills/openclaw-live-updater/SKILL.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,7 +30,7 @@ Keep `/Users/steipete/openclaw` a read-only-to-the-agent deployment mirror: clea
3030
- Every successful update sets `actions.gatewayBuild` and rebuilds exact new `main` before any restart.
3131
- Missing, invalid, or stale build output also forces a build, even when Git did not move.
3232
- A dependency-input change, absent `node_modules`, or missing/invalid build provenance requires `pnpm install --frozen-lockfile`. When a build is required, do not install before acquiring the maintenance suspension and stopping the managed Gateway.
33-
- Before any dependency or build mutation, invoke the Gateway's existing built CLI to acquire `gateway.suspend.prepare`, binding both prepare and resume to this checkout's managed LaunchAgent loopback port and service auth even when normal CLI configuration points at a remote Gateway. The LaunchAgent may execute either this checkout's `dist/index.js` or a clean detached canonical snapshot under `~/.openclaw/runtime/gateway-<sha>` whose commit is an ancestor of the checkout; reject every other entrypoint. Never execute snapshot code: capture an exact source control build before the Git fast-forward for its prepare and failure-resume calls, preserve any validated generated service-environment wrapper, and stop the managed LaunchAgent with native launchd bootout. If that control build is missing, proceed only when native proof shows the snapshot job is already booted out with its port free, allowing a failed-build retry without executing unverified generated code. This atomically pauses cron scheduling, closes new work admission, and refuses while active work remains. A busy result defers the build to the next heartbeat without installing dependencies or stopping the Gateway; never replace this fence with `cron list` polling. Once ready, stop directly without a source launcher, install frozen dependencies when required, then build; source launchers can auto-build stale output before dispatching the stop. Resume the suspension if stop fails. If suspension RPC is unavailable on macOS, proceed only when native inspection proves this checkout's managed LaunchAgent is booted out and its configured port has no listener; never accept a loaded KeepAlive job's transient stopped state. On other platforms, require the existing CLI to prove the managed service is stopped with no PID, listener, or RPC. This preserves retry after a post-stop failure without weakening the live-work fence. Preserve `dist/OpenClaw.app` outside `dist` for the build and restore it even when the build fails, because the JS build cleans `dist` regardless of Mac impact classification. Never mutate the live `dist` tree while an old Gateway can dynamically import from it. `pnpm build` must leave both canonical stamp heads and `dist/build-info.json.commit` equal to post-update `afterSha`; any missing/mismatched stamp or required artifact blocks restart.
33+
- Before mutating any entrypoint currently executed by the Gateway, invoke an exact trusted source CLI to acquire `gateway.suspend.prepare`, binding both prepare and resume to this checkout's managed LaunchAgent loopback port and service auth even when normal CLI configuration points at a remote Gateway. The LaunchAgent may execute either this checkout's `dist/index.js` or a clean detached canonical snapshot under `~/.openclaw/runtime/gateway-<sha>` whose commit is an ancestor of the checkout; reject every other entrypoint. Never execute snapshot code: capture an exact source control build before the Git fast-forward for its prepare and failure-resume calls, preserve any validated generated service-environment wrapper, and stop the managed LaunchAgent with native launchd bootout. If that control build is missing, first accept native proof that the snapshot job is already booted out with its port free; when the isolated snapshot is still running, build the verified clean source checkout to obtain an exact suspension client, then use only that source client for prepare and failure-resume. Never use this recovery build while launchd targets the source checkout. This atomically pauses cron scheduling, closes new work admission, and refuses while active work remains. A busy result defers further mutation to the next heartbeat; never replace this fence with `cron list` polling. Once ready, stop directly without a source launcher, install frozen dependencies when required, then build unless the exact recovery build already produced the deployment artifact; source launchers can auto-build stale output before dispatching the stop. Resume the suspension if stop fails. If suspension RPC is unavailable on macOS, proceed only when native inspection proves this checkout's managed LaunchAgent is booted out and its configured port has no listener; never accept a loaded KeepAlive job's transient stopped state. On other platforms, require the existing CLI to prove the managed service is stopped with no PID, listener, or RPC. This preserves retry after a post-stop failure without weakening the live-work fence. Preserve `dist/OpenClaw.app` outside `dist` for the build and restore it even when the build fails, because the JS build cleans `dist` regardless of Mac impact classification. Never mutate the live `dist` tree while an old Gateway can dynamically import from it. `pnpm build` must leave both canonical stamp heads and `dist/build-info.json.commit` equal to post-update `afterSha`; any missing/mismatched stamp or required artifact blocks restart.
3434
- Only after exact-SHA build proof may it restart the managed Gateway and require `gateway status --deep --require-rpc --json` plus `health --verbose --json`. A validated ancestor snapshot is suspension-only: prove the old launchd job is booted out with its port free, then atomically retarget only the owned LaunchAgent entrypoint to this checkout's exact `dist/index.js`, including on a retry where the build is already current. Preserve all other service arguments and environment unchanged. After restart, prove the loaded launchd PID owns the configured listener.
3535
- After every managed restart, query Gateway logs through RPC, restrict the audit to entries emitted since that restart began, report warning summaries, and fail the pass on any error/fatal entry. If RPC verification or log retrieval fails, still inspect the local structured log for that restart window. Never accept supervisor or RPC health without this restart-window log audit.
3636

‎.agents/skills/openclaw-live-updater/scripts/update-main.mjs‎

Lines changed: 53 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1569,7 +1569,7 @@ export function maintainMain(options, dependencies = {}) {
15691569
verifiedBefore.checkout,
15701570
verifiedBefore.headSha,
15711571
);
1572-
const gatewayControlDeployment = resolveGatewayControlDeployment(
1572+
let gatewayControlDeployment = resolveGatewayControlDeployment(
15731573
verifiedBefore.checkout,
15741574
gatewayDeploymentBefore,
15751575
sourceBuildBeforeUpdate,
@@ -1620,6 +1620,8 @@ export function maintainMain(options, dependencies = {}) {
16201620

16211621
if (actions.gatewayBuild || actions.dependencyInstall || gatewayRuntimeRepointRequired) {
16221622
actions.gatewayRestart = true;
1623+
let controlBuildPrepared = false;
1624+
let controlDependenciesInstalled = false;
16231625
let gatewaySuspension;
16241626
const controlUnavailable =
16251627
gatewayDeploymentBefore !== null && gatewayControlDeployment === null;
@@ -1630,16 +1632,55 @@ export function maintainMain(options, dependencies = {}) {
16301632
proof: proveGatewayStopped(update.checkout),
16311633
};
16321634
} catch (proofError) {
1633-
throw new AggregateError(
1634-
[
1635-
new UpdateInvariantError(
1635+
try {
1636+
if (!gatewayRuntimeRepointRequired) {
1637+
throw new UpdateInvariantError(
1638+
"gateway_live_source_build_forbidden",
1639+
"refusing to rebuild the source entrypoint while its managed Gateway is still running",
1640+
);
1641+
}
1642+
// The running Gateway is isolated in its immutable snapshot, so a
1643+
// clean source build cannot mutate its code. Build only to obtain
1644+
// an exact trusted client for the suspension RPC.
1645+
if (actions.dependencyInstall) {
1646+
runCommand("pnpm", ["install", "--frozen-lockfile"], update.checkout);
1647+
controlDependenciesInstalled = true;
1648+
}
1649+
if (!actions.gatewayBuild) {
1650+
throw new UpdateInvariantError(
16361651
"gateway_snapshot_control_unavailable",
1637-
"managed Gateway uses a snapshot but the source checkout has no exact trusted control build",
1638-
),
1639-
proofError,
1640-
],
1641-
"Gateway control is unavailable and the managed Gateway could not be proven stopped",
1642-
);
1652+
"managed Gateway snapshot has no exact trusted source control build",
1653+
);
1654+
}
1655+
runBuildWithPreservedMacApp(runCommand, update.checkout, sleep);
1656+
assertExactBuild(update.checkout, update.afterSha);
1657+
controlBuildPrepared = true;
1658+
gatewayControlDeployment = resolveGatewayControlDeployment(
1659+
update.checkout,
1660+
gatewayDeploymentBefore,
1661+
inspectBuildState(update.checkout, update.afterSha),
1662+
update.afterSha,
1663+
);
1664+
if (!gatewayControlDeployment) {
1665+
throw new UpdateInvariantError(
1666+
"gateway_snapshot_control_unavailable",
1667+
"source build did not produce an exact trusted Gateway control client",
1668+
);
1669+
}
1670+
gatewaySuspension = prepareSuspension(update.checkout, gatewayControlDeployment);
1671+
} catch (controlError) {
1672+
throw new AggregateError(
1673+
[
1674+
new UpdateInvariantError(
1675+
"gateway_snapshot_control_unavailable",
1676+
"managed Gateway uses a snapshot but the source checkout has no exact trusted control build",
1677+
),
1678+
proofError,
1679+
controlError,
1680+
],
1681+
"Gateway control is unavailable and the managed Gateway could not be proven stopped",
1682+
);
1683+
}
16431684
}
16441685
} else {
16451686
try {
@@ -1695,10 +1736,10 @@ export function maintainMain(options, dependencies = {}) {
16951736
throw error;
16961737
}
16971738
}
1698-
if (actions.dependencyInstall) {
1739+
if (actions.dependencyInstall && !controlDependenciesInstalled) {
16991740
runCommand("pnpm", ["install", "--frozen-lockfile"], update.checkout);
17001741
}
1701-
if (actions.gatewayBuild) {
1742+
if (actions.gatewayBuild && !controlBuildPrepared) {
17021743
runBuildWithPreservedMacApp(runCommand, update.checkout, sleep);
17031744
}
17041745
assertExactBuild(update.checkout, update.afterSha);

‎test/scripts/openclaw-live-updater.test.ts‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1050,6 +1050,93 @@ describe("openclaw live updater", () => {
10501050
]);
10511051
});
10521052

1053+
test("builds a trusted source control client while a snapshot is still running", () => {
1054+
const { root, mirror } = makeFixture();
1055+
mkdirSync(path.join(mirror, "node_modules"));
1056+
const commands = fakeCommands(mirror);
1057+
const snapshot = path.join(root, "gateway-ancestor/dist/index.js");
1058+
const source = path.join(mirror, "dist/index.js");
1059+
const configPath = path.join(root, "openclaw.json");
1060+
const plistPath = path.join(root, "ai.openclaw.gateway.plist");
1061+
writeFileSync(configPath, "{}\n");
1062+
writeFileSync(plistPath, "plist\n", { mode: 0o600 });
1063+
let deployedEntrypoint = snapshot;
1064+
let controlEntrypoint = "";
1065+
let stoppedProofAttempts = 0;
1066+
1067+
const output = maintainFixture(
1068+
{ checkout: mirror, remote: "origin", lockPath: path.join(root, "maintenance.lock") },
1069+
{
1070+
runCommand: commands.runCommand,
1071+
prepareGatewaySuspension: (_checkout: string, deployment: { entrypoint: string }) => {
1072+
controlEntrypoint = deployment.entrypoint;
1073+
return { status: "ready", suspensionId: "fixture-suspension" };
1074+
},
1075+
inspectGatewayDeployment: () => ({
1076+
configPath,
1077+
entrypoint: deployedEntrypoint,
1078+
entrypointIndex: 1,
1079+
executable: process.execPath,
1080+
invocationPrefix: [deployedEntrypoint],
1081+
label: "ai.openclaw.gateway",
1082+
plistPath,
1083+
port: 18789,
1084+
runtime: process.execPath,
1085+
}),
1086+
proveGatewayStopped: () => {
1087+
stoppedProofAttempts += 1;
1088+
commands.calls.push("prove gateway stopped");
1089+
if (stoppedProofAttempts === 1) {
1090+
throw new Error("snapshot still owns its listener");
1091+
}
1092+
return {
1093+
runtimeStatus: "stopped",
1094+
port: 18789,
1095+
portStatus: "free",
1096+
proofSource: "fixture",
1097+
};
1098+
},
1099+
repointGatewayDeployment: (
1100+
_checkout: string,
1101+
deployment: { entrypoint: string; invocationPrefix: string[] },
1102+
) => {
1103+
deployedEntrypoint = source;
1104+
return {
1105+
changed: true,
1106+
...deployment,
1107+
entrypoint: source,
1108+
invocationPrefix: [source],
1109+
previousEntrypoint: deployment.entrypoint,
1110+
};
1111+
},
1112+
verifyGatewayRuntime: () => ({
1113+
commit: git(mirror, "rev-parse", "HEAD"),
1114+
entrypoint: source,
1115+
pid: 123,
1116+
port: 18789,
1117+
}),
1118+
},
1119+
);
1120+
1121+
expect(controlEntrypoint).toBe(source);
1122+
expect(stoppedProofAttempts).toBe(2);
1123+
expect(output.gatewayDeployment).toMatchObject({
1124+
changed: true,
1125+
entrypoint: source,
1126+
previousEntrypoint: snapshot,
1127+
});
1128+
const uid = process.getuid?.() ?? 501;
1129+
expect(commands.calls).toEqual([
1130+
"prove gateway stopped",
1131+
"pnpm install --frozen-lockfile",
1132+
"pnpm build",
1133+
`/bin/launchctl bootout gui/${uid}/ai.openclaw.gateway`,
1134+
"prove gateway stopped",
1135+
`/bin/launchctl enable gui/${uid}/ai.openclaw.gateway`,
1136+
`/bin/launchctl bootstrap gui/${uid} ${plistPath}`,
1137+
]);
1138+
});
1139+
10531140
test("recovers a stopped snapshot when the source control build is missing", () => {
10541141
const { root, mirror } = makeFixture();
10551142
mkdirSync(path.join(mirror, "node_modules"));

0 commit comments

Comments
 (0)