Skip to content

Commit 3f98fc7

Browse files
authored
fix(rebuild): distinguish unverified versions and restore recovery guidance (#12061)
## Outcome When a rebuild cannot read the replacement agent’s version, NemoClaw now says that verification failed instead of reporting a version mismatch with `observed unverified`. It also prints the preserved backup location and applicable recovery commands before exiting. Fixes #12004 ## Why A failed version probe does not mean the replacement has the wrong version. The previous message pointed troubleshooting in the wrong direction and omitted the recovery instructions. ## Changes - Distinguish an unavailable version from an observed mismatch, keeping expected/observed details for actual mismatches. - Print backup and applicable MCP recovery guidance before exiting. For eligible Hermes failures after restoration, suggest `gateway restart`, then `recover` if health remains unverified. - Preserve mandatory recovery for cron-gated and resumed recovery paths, nonzero exits, and cleared unverified version metadata. - Extend the existing post-restore tests. No new test files, E2E scenarios, dependencies, retries, or runtime probes. ## Verification Tested commit: `d400d32eb17536b6092736450834fb8160a813ba`. ```bash npx vitest run --project cli src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts src/lib/sandbox/version.test.ts ``` **84 tests passed across three files.** The six regression cases from the first commit also failed against the original implementation and passed with the fix. The follow-up adds coverage for resumed recovery. CLI and plugin type-checking passed with Node.js 24 and `NODE_OPTIONS=--max-old-space-size=8192`. Commit and push hooks passed. ## Live validation I tested a disposable Hermes sandbox on macOS with Docker Desktop, OpenShell 0.0.116, Hermes 0.20.6, and local Ollama using `llama3.2:3b`. A normal rebuild completed with exit code 0, restored 16 directories and five files, and preserved the test marker. ![1. Normal rebuild succeeds and preserves the marker](https://github.com/user-attachments/assets/c9b86361-60a9-4e4f-8a67-87ab0a5c199c) For the interruption test, a local helper stopped only the replacement sandbox container immediately after `State restored` appeared. The rebuild exited with code 1 and printed the new verification message, backup path, and recovery commands. ![2. Interrupted rebuild prints the corrected diagnostics and recovery guidance](https://github.com/user-attachments/assets/c73510a5-c7c3-4be7-83c2-803c4c1f8270) While following the printed instructions, I also encountered recovery failures: - `gateway restart` failed without returning command output. - `recover` failed during Hermes cron preparation because the container was not running. - `start` failed because OpenShell reported `Error` and required `Stopped`. ![3. Gateway restart fails after the container stop](https://github.com/user-attachments/assets/7100550f-b204-4739-8c13-a0cb8379e04c) ![4. Recovery fails during cron preparation in the stopped container](https://github.com/user-attachments/assets/4b749118-cceb-4964-9740-a3f39c71b30c) ![5. OpenShell rejects sandbox start from Error state](https://github.com/user-attachments/assets/dd43098c-9dbb-454c-b760-b87a6951ffd3) Further attempts are recorded in terminal output, beyond these screenshots: starting the container directly left OpenShell in `Error`, and another rebuild failed while inspecting Hermes MCP configuration with `sandbox is not ready`. The marker was still present when checked directly inside the container. The live test confirms the diagnostic behavior requested in #12004. Recovery didn't succeed, and no DGX Spark validation was done. ## Related recovery issues After encountering these failures, I searche reports and found other issues that describe similar problems: - [NemoClaw #11885](#11885) reports container stop/start leaving a sandbox in `Error`, with both `start` and `recover` unable to restore it. [OpenShell #3440](NVIDIA/OpenShell#3440) also reports a running container remaining stuck in `Error`. - [NemoClaw #11898](#11898) includes the same start rejection and MCP inspection error (`sandbox is not ready`), following an upgrade rather than a rebuild interruption. - [A follow-up comment on #8649](#8649 (comment)) reports Hermes cron preparation failing during stopped-container recovery and identifies the same possible ordering problem: cron preparation runs before container recovery. The parent issue is closed, but I encountered this failure during this test. These reports overlap with the symptoms I observed; I have not established that every failure has the same root cause. This PR addresses the misleading diagnostics and missing recovery guidance in #12004. It does not change the recovery implementation or claim to fix these additional failures. ## Other test findings A separate local comparison produced 87 passes and the same `destroy-flow.test.ts` failure on both the base and patched source. The reported `auto-pair-warmup.test.ts` missing-log failure did not reproduce in those comparison runs; that does not establish that it is resolved. Neither file is changed here. --- Signed-off-by: Adrian Osorio <[email protected]> <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved rebuild handling when restored versions cannot be verified or do not match expected versions. * Recovery guidance now distinguishes verification failures from version mismatches and provides clearer next steps. * Recovery flows now report when gateway health remains unverified after restoration. * Preserved backups and recovery state during failed recovery scenarios. * **Tests** * Expanded coverage for prepared and non-prepared recovery, restore failures, version checks, and gateway recovery scenarios. <!-- end of auto-generated comment: release notes by coderabbit.ai --> --------- Signed-off-by: Adrian Osorio Blanchard <[email protected]>
1 parent e14faec commit 3f98fc7

2 files changed

Lines changed: 226 additions & 30 deletions

File tree

‎src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts‎

Lines changed: 145 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -565,7 +565,7 @@ describe("rebuild post-restore phase", () => {
565565
expect(verification).toEqual({ mutableConfigPermissionsVerified: true });
566566
});
567567

568-
it("rejects a replacement whose live version does not match the rebuild target", async () => {
568+
it.each([false, true])("rejects a version mismatch during recovery=%s", async (recovery) => {
569569
agentName = "hermes";
570570
vi.mocked(agentDefs.loadAgent).mockReturnValue({
571571
name: "hermes",
@@ -581,12 +581,15 @@ describe("rebuild post-restore phase", () => {
581581
});
582582
const args = {
583583
...input(),
584+
preparedBackupRecovery: recovery,
584585
versionCheck: { expectedVersion: "0.20.6" } as never,
585-
hermesCronRestoreIdentity: {
586-
pid: 41,
587-
start_time: 902,
588-
drain_token: "restore-token",
589-
},
586+
hermesCronRestoreIdentity: !recovery
587+
? {
588+
pid: 41,
589+
start_time: 902,
590+
drain_token: "restore-token",
591+
}
592+
: undefined,
590593
};
591594

592595
await runRebuildPostRestorePhase(args);
@@ -609,14 +612,144 @@ describe("rebuild post-restore phase", () => {
609612
rebuildHermesPostRestore.completeHermesCronRestoreAfterGatewayReplacement,
610613
).not.toHaveBeenCalled();
611614
expect(messagingHostForward.ensureMessagingHostForwardAfterRebuild).not.toHaveBeenCalled();
612-
expect(vi.mocked(console.error).mock.calls.flat().join("\n")).toContain(
613-
"Hermes cron dispatch remains drained",
614-
);
615-
expect(vi.mocked(console.log).mock.calls.flat().join("\n")).not.toContain(
616-
"rebuilt successfully",
617-
);
615+
const errors = vi.mocked(console.error).mock.calls.flat().join("\n");
616+
expect(errors.includes("Hermes cron dispatch remains drained")).toBe(!recovery);
617+
expect(errors).toContain("nemoclaw alpha recover");
618+
expect(errors).toContain("expected 0.20.6, observed 0.19.0");
619+
expect(errors).not.toContain("gateway restart");
620+
expect(vi.mocked(console.log).mock.calls.flat().join("\n")).not.toContain("rebuild completed");
618621
});
619622

623+
it.each([
624+
{
625+
cronGate: false,
626+
preparedBackupRecovery: false,
627+
restoreSucceeded: true,
628+
gatewayDetail: "nemoclaw alpha gateway restart",
629+
recoveryDetail: "If gateway health is still unverified",
630+
},
631+
{
632+
cronGate: true,
633+
preparedBackupRecovery: false,
634+
restoreSucceeded: true,
635+
gatewayDetail: "Hermes gateway health was not verified after state restore.",
636+
recoveryDetail: "Hermes cron dispatch remains drained",
637+
},
638+
{
639+
cronGate: false,
640+
preparedBackupRecovery: true,
641+
restoreSucceeded: false,
642+
gatewayDetail: "Correct the reported restore problem",
643+
recoveryDetail: "nemoclaw alpha recover",
644+
},
645+
])(
646+
"reports unavailable verification with cron gate=$cronGate, prepared recovery=$preparedBackupRecovery, and restore=$restoreSucceeded (#12004)",
647+
async ({
648+
cronGate,
649+
preparedBackupRecovery,
650+
restoreSucceeded,
651+
gatewayDetail,
652+
recoveryDetail,
653+
}) => {
654+
agentName = "hermes";
655+
vi.mocked(sandboxVersion.checkAgentVersion).mockResolvedValue({
656+
sandboxVersion: null,
657+
expectedVersion: "0.20.6",
658+
isStale: false,
659+
verificationFailed: true,
660+
detectionMethod: "unknown",
661+
unavailableReason: "probe-failed",
662+
});
663+
vi.mocked(rebuildHermesPostRestore.restartHermesGatewayAfterStateRestore).mockResolvedValue(
664+
"restart-failed",
665+
);
666+
vi.mocked(
667+
rebuildHermesPostRestore.verifyHermesGatewayAfterStateRestoreForCronGate,
668+
).mockResolvedValue({ state: "unverified" });
669+
vi.mocked(rebuildMcp.restoreMcpAfterRebuild).mockResolvedValue(false);
670+
const failure = new Error("rebuild stopped");
671+
const args = {
672+
...input(),
673+
preparedBackupRecovery,
674+
restoreSucceeded,
675+
bail: vi.fn((): never => {
676+
throw failure;
677+
}),
678+
backupManifest: { backupPath: "/tmp/alpha-backup" } as never,
679+
versionCheck: { expectedVersion: "0.20.6" } as never,
680+
hermesCronRestoreIdentity: cronGate
681+
? { pid: 41, start_time: 902, drain_token: "restore-token" }
682+
: undefined,
683+
};
684+
685+
await expect(runRebuildPostRestorePhase(args)).rejects.toBe(failure);
686+
687+
expect(args.bail).toHaveBeenCalledExactlyOnceWith(
688+
restoreSucceeded
689+
? "Replacement agent version could not be verified after rebuild."
690+
: "State restore remained incomplete after rebuilding 'alpha'.",
691+
);
692+
const errors = vi.mocked(console.error).mock.calls.flat().join("\n");
693+
const logs = vi.mocked(console.log).mock.calls.flat().join("\n");
694+
expect(errors).toContain(
695+
"Replacement agent version could not be verified (expected 0.20.6).",
696+
);
697+
expect(errors).not.toContain("did not match");
698+
expect(errors).toContain("Backup is preserved at: /tmp/alpha-backup");
699+
expect(errors).toContain("nemoclaw alpha recover");
700+
expect(logs).toContain("nemoclaw alpha mcp restart");
701+
expect(errors).toContain(gatewayDetail);
702+
expect(errors).toContain(recoveryDetail);
703+
expect(errors.includes("gateway restart")).toBe(!cronGate && !preparedBackupRecovery);
704+
expect(errors.indexOf(gatewayDetail)).toBeLessThan(errors.indexOf("alpha recover"));
705+
expect(registry.updateSandbox).toHaveBeenLastCalledWith("alpha", { agentVersion: null });
706+
expect(
707+
rebuildHermesPostRestore.completeHermesCronRestoreAfterGatewayReplacement,
708+
).not.toHaveBeenCalled();
709+
},
710+
);
711+
712+
it.each([
713+
{ target: "hermes", restoreSucceeded: true, reason: "invalid-gateway-binding" },
714+
{ target: "hermes", restoreSucceeded: false, reason: "probe-failed" },
715+
{ target: "openclaw", restoreSucceeded: true, reason: "probe-failed" },
716+
] as const)(
717+
"omits Hermes restart guidance for $target with restore=$restoreSucceeded and $reason (#12004)",
718+
async ({ target, restoreSucceeded, reason }) => {
719+
agentName = target;
720+
vi.mocked(sandboxVersion.checkAgentVersion).mockResolvedValue({
721+
sandboxVersion: null,
722+
expectedVersion: "0.20.6",
723+
isStale: false,
724+
verificationFailed: true,
725+
detectionMethod: reason === "probe-failed" ? "unknown" : "unavailable",
726+
unavailableReason: reason,
727+
});
728+
const failure = new Error("rebuild stopped");
729+
const args = {
730+
...input(),
731+
restoreSucceeded,
732+
bail: vi.fn((): never => {
733+
throw failure;
734+
}),
735+
versionCheck: { expectedVersion: "0.20.6" } as never,
736+
};
737+
await expect(runRebuildPostRestorePhase(args)).rejects.toBe(failure);
738+
expect(args.bail).toHaveBeenCalledExactlyOnceWith(
739+
restoreSucceeded
740+
? "Replacement agent version could not be verified after rebuild."
741+
: "State restore remained incomplete after rebuilding 'alpha'.",
742+
);
743+
const errors = vi.mocked(console.error).mock.calls.flat().join("\n");
744+
expect(errors).not.toContain("gateway restart");
745+
expect(
746+
errors.includes(
747+
"State recovery remains incomplete. Correct the restore error, then run `nemoclaw alpha rebuild` again.",
748+
),
749+
).toBe(!restoreSucceeded);
750+
},
751+
);
752+
620753
it("records the live replacement version only after an exact forced probe", async () => {
621754
agentName = "hermes";
622755
vi.mocked(agentDefs.loadAgent).mockReturnValue({

‎src/lib/actions/sandbox/rebuild-post-restore-phase.ts‎

Lines changed: 81 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -55,13 +55,6 @@ export {
5555
runHermesCronRestoreTransaction,
5656
} from "./rebuild-hermes-post-restore";
5757

58-
/** Probe the recreated runtime instead of accepting its requested version metadata. */
59-
function probeRebuiltAgentVersion(
60-
sandboxName: string,
61-
): ReturnType<typeof sandboxVersion.checkAgentVersion> {
62-
return sandboxVersion.checkAgentVersion(sandboxName, { forceProbe: true });
63-
}
64-
6558
export function printHermesCronRestoreRecoveryCommand(
6659
sandboxName: string,
6760
writeLine: (message: string) => void = console.error,
@@ -113,6 +106,61 @@ export interface RebuildPostRestoreVerification {
113106
readonly mutableConfigPermissionsVerified: boolean;
114107
}
115108

109+
function describeRebuiltVersionFailure(
110+
expectedVersion: string,
111+
rebuiltVersion: sandboxVersion.VersionCheckResult,
112+
): { detail: string; bailMessage: string } | null {
113+
const versionUnverified =
114+
rebuiltVersion.verificationFailed || rebuiltVersion.sandboxVersion === null;
115+
if (!versionUnverified && rebuiltVersion.sandboxVersion === expectedVersion) return null;
116+
return {
117+
detail: versionUnverified
118+
? ` Replacement agent version could not be verified (expected ${expectedVersion}).`
119+
: ` Replacement agent version did not match the rebuild target (expected ${expectedVersion}, observed ${rebuiltVersion.sandboxVersion}).`,
120+
bailMessage: versionUnverified
121+
? "Replacement agent version could not be verified after rebuild."
122+
: "Replacement agent version did not match the authoritative rebuild target.",
123+
};
124+
}
125+
126+
function printRebuildVersionFailureRecovery(
127+
input: RebuildPostRestorePhaseInput,
128+
rebuiltVersion: sandboxVersion.VersionCheckResult,
129+
mcpBridgeRestoreUnverified: boolean,
130+
versionFailureMessage: string,
131+
): string {
132+
const { sandboxName, backupManifest, targetAgentName, restoreSucceeded } = input;
133+
const failureMessage = restoreSucceeded
134+
? versionFailureMessage
135+
: `State restore remained incomplete after rebuilding '${sandboxName}'.`;
136+
if (backupManifest) {
137+
console.error(` Backup is preserved at: ${backupManifest.backupPath}`);
138+
}
139+
printMcpRestoreRecovery(sandboxName, mcpBridgeRestoreUnverified);
140+
// Resumed replacements can retain a cron gate without a new restore identity.
141+
if (targetAgentName === "hermes" && input.preparedBackupRecovery) {
142+
printHermesCronRestoreRecoveryCommand(sandboxName);
143+
return failureMessage;
144+
}
145+
if (!restoreSucceeded) {
146+
console.error(
147+
` State recovery remains incomplete. Correct the restore error, then run \`${CLI_NAME} ${sandboxName} rebuild\` again.`,
148+
);
149+
return failureMessage;
150+
}
151+
if (
152+
targetAgentName === "hermes" &&
153+
rebuiltVersion.verificationFailed &&
154+
rebuiltVersion.unavailableReason === "probe-failed"
155+
) {
156+
console.error(` Run \`${CLI_NAME} ${sandboxName} gateway restart\`.`);
157+
console.error(
158+
` If gateway health is still unverified, run \`${CLI_NAME} ${sandboxName} recover\`.`,
159+
);
160+
}
161+
return failureMessage;
162+
}
163+
116164
export function printHermesOperatorConfigRestoreReport(
117165
targetAgentName: string,
118166
report: HermesOperatorConfigRestoreReport | undefined,
@@ -421,34 +469,49 @@ export async function runRebuildPostRestorePhase(
421469
: { state: "not-applicable" as const, replacementIdentity: undefined };
422470
const hermesGatewayRestoreState = hermesGatewayVerification.state;
423471
const hermesGatewayRestoreUnverified = hermesGatewayRestoreState === "unverified";
472+
const reportMcpRestoreFailure = mcpBridgeRestoreUnverified
473+
? () => printMcpRestoreRecovery(sandboxName, true)
474+
: undefined;
424475
let verifiedAgentVersion: string | null = null;
425476
if (versionCheck.expectedVersion) {
426477
// The replacement runtime is the only authority for the completed rebuild
427478
// version. Clear create-time bookkeeping before the forced live probe so a
428479
// failed probe cannot leave the requested version recorded as observed.
429480
registry.updateSandbox(sandboxName, { agentVersion: null });
430-
const rebuiltVersion = await probeRebuiltAgentVersion(sandboxName);
431-
if (
432-
rebuiltVersion.verificationFailed ||
433-
rebuiltVersion.sandboxVersion !== versionCheck.expectedVersion
434-
) {
481+
const rebuiltVersion = await sandboxVersion.checkAgentVersion(sandboxName, {
482+
forceProbe: true,
483+
});
484+
const versionFailure = describeRebuiltVersionFailure(
485+
versionCheck.expectedVersion,
486+
rebuiltVersion,
487+
);
488+
if (versionFailure) {
435489
// checkAgentVersion caches a successful probe. Do not retain metadata
436490
// from a replacement that this rebuild rejects.
437491
registry.updateSandbox(sandboxName, { agentVersion: null });
438-
const observed = rebuiltVersion.sandboxVersion ?? "unverified";
439-
const detail = ` Replacement agent version did not match the rebuild target (expected ${versionCheck.expectedVersion}, observed ${observed}).`;
492+
const { detail, bailMessage } = versionFailure;
440493
if (hermesCronRestoreIdentity) {
494+
if (hermesGatewayRestoreUnverified) {
495+
console.error(" Hermes gateway health was not verified after state restore.");
496+
}
441497
return bailAfterHermesCronRestoreFailure(
442498
sandboxName,
443499
backupManifest,
444500
`${detail} Hermes cron dispatch remains drained.`,
445-
"Replacement agent version did not match the authoritative rebuild target.",
501+
bailMessage,
446502
bail,
447-
mcpBridgeRestoreUnverified ? () => printMcpRestoreRecovery(sandboxName, true) : undefined,
503+
reportMcpRestoreFailure,
448504
);
449505
}
450506
console.error(detail);
451-
bail("Replacement agent version did not match the authoritative rebuild target.");
507+
bail(
508+
printRebuildVersionFailureRecovery(
509+
input,
510+
rebuiltVersion,
511+
mcpBridgeRestoreUnverified,
512+
bailMessage,
513+
),
514+
);
452515
return;
453516
}
454517
verifiedAgentVersion = rebuiltVersion.sandboxVersion;
@@ -477,7 +540,7 @@ export async function runRebuildPostRestorePhase(
477540
" Hermes cron dispatch remains drained because the replacement gateway was not verified.",
478541
"Hermes cron restore validation failed; dispatch was not re-enabled.",
479542
bail,
480-
mcpBridgeRestoreUnverified ? () => printMcpRestoreRecovery(sandboxName, true) : undefined,
543+
reportMcpRestoreFailure,
481544
);
482545
}
483546
if (mcpBridgeRestoreUnverified) {

0 commit comments

Comments
 (0)