Repository navigation
Commit 3f98fc7
authored
fix(rebuild): distinguish unverified versions and restore recovery guidance (#12061)
## Outcome
When a rebuild cannot read the replacement agent’s version, NemoClaw now
says that verification failed instead of reporting a version mismatch
with `observed unverified`. It also prints the preserved backup location
and applicable recovery commands before exiting.
Fixes #12004
## Why
A failed version probe does not mean the replacement has the wrong
version. The previous message pointed troubleshooting in the wrong
direction and omitted the recovery instructions.
## Changes
- Distinguish an unavailable version from an observed mismatch, keeping
expected/observed details for actual mismatches.
- Print backup and applicable MCP recovery guidance before exiting. For
eligible Hermes failures after restoration, suggest `gateway restart`,
then `recover` if health remains unverified.
- Preserve mandatory recovery for cron-gated and resumed recovery paths,
nonzero exits, and cleared unverified version metadata.
- Extend the existing post-restore tests. No new test files, E2E
scenarios, dependencies, retries, or runtime probes.
## Verification
Tested commit: `d400d32eb17536b6092736450834fb8160a813ba`.
```bash
npx vitest run --project cli src/lib/actions/sandbox/rebuild-post-restore-phase.test.ts src/lib/actions/sandbox/rebuild-hermes-post-restore.test.ts src/lib/sandbox/version.test.ts
```
**84 tests passed across three files.** The six regression cases from
the first commit also failed against the original implementation and
passed with the fix. The follow-up adds coverage for resumed recovery.
CLI and plugin type-checking passed with Node.js 24 and
`NODE_OPTIONS=--max-old-space-size=8192`. Commit and push hooks passed.
## Live validation
I tested a disposable Hermes sandbox on macOS with Docker Desktop,
OpenShell 0.0.116, Hermes 0.20.6, and local Ollama using `llama3.2:3b`.
A normal rebuild completed with exit code 0, restored 16 directories and
five files, and preserved the test marker.

For the interruption test, a local helper stopped only the replacement
sandbox container immediately after `State restored` appeared. The
rebuild exited with code 1 and printed the new verification message,
backup path, and recovery commands.

While following the printed instructions, I also encountered recovery
failures:
- `gateway restart` failed without returning command output.
- `recover` failed during Hermes cron preparation because the container
was not running.
- `start` failed because OpenShell reported `Error` and required
`Stopped`.



Further attempts are recorded in terminal output, beyond these
screenshots: starting the container directly left OpenShell in `Error`,
and another rebuild failed while inspecting Hermes MCP configuration
with `sandbox is not ready`. The marker was still present when checked
directly inside the container.
The live test confirms the diagnostic behavior requested in #12004.
Recovery didn't succeed, and no DGX Spark validation was done.
## Related recovery issues
After encountering these failures, I searche reports and found other
issues that describe similar problems:
- [NemoClaw #11885](#11885)
reports container stop/start leaving a sandbox in `Error`, with both
`start` and `recover` unable to restore it. [OpenShell
#3440](NVIDIA/OpenShell#3440) also reports a
running container remaining stuck in `Error`.
- [NemoClaw #11898](#11898)
includes the same start rejection and MCP inspection error (`sandbox is
not ready`), following an upgrade rather than a rebuild interruption.
- [A follow-up comment on
#8649](#8649 (comment))
reports Hermes cron preparation failing during stopped-container
recovery and identifies the same possible ordering problem: cron
preparation runs before container recovery. The parent issue is closed,
but I encountered this failure during this test.
These reports overlap with the symptoms I observed; I have not
established that every failure has the same root cause. This PR
addresses the misleading diagnostics and missing recovery guidance in
#12004. It does not change the recovery implementation or claim to fix
these additional failures.
## Other test findings
A separate local comparison produced 87 passes and the same
`destroy-flow.test.ts` failure on both the base and patched source. The
reported `auto-pair-warmup.test.ts` missing-log failure did not
reproduce in those comparison runs; that does not establish that it is
resolved. Neither file is changed here.
---
Signed-off-by: Adrian Osorio <[email protected]>
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved rebuild handling when restored versions cannot be verified or
do not match expected versions.
* Recovery guidance now distinguishes verification failures from version
mismatches and provides clearer next steps.
* Recovery flows now report when gateway health remains unverified after
restoration.
* Preserved backups and recovery state during failed recovery scenarios.
* **Tests**
* Expanded coverage for prepared and non-prepared recovery, restore
failures, version checks, and gateway recovery scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Signed-off-by: Adrian Osorio Blanchard <[email protected]>1 parent e14faec commit 3f98fc7
2 files changed
Lines changed: 226 additions & 30 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
565 | 565 | | |
566 | 566 | | |
567 | 567 | | |
568 | | - | |
| 568 | + | |
569 | 569 | | |
570 | 570 | | |
571 | 571 | | |
| |||
581 | 581 | | |
582 | 582 | | |
583 | 583 | | |
| 584 | + | |
584 | 585 | | |
585 | | - | |
586 | | - | |
587 | | - | |
588 | | - | |
589 | | - | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
590 | 593 | | |
591 | 594 | | |
592 | 595 | | |
| |||
609 | 612 | | |
610 | 613 | | |
611 | 614 | | |
612 | | - | |
613 | | - | |
614 | | - | |
615 | | - | |
616 | | - | |
617 | | - | |
| 615 | + | |
| 616 | + | |
| 617 | + | |
| 618 | + | |
| 619 | + | |
| 620 | + | |
618 | 621 | | |
619 | 622 | | |
| 623 | + | |
| 624 | + | |
| 625 | + | |
| 626 | + | |
| 627 | + | |
| 628 | + | |
| 629 | + | |
| 630 | + | |
| 631 | + | |
| 632 | + | |
| 633 | + | |
| 634 | + | |
| 635 | + | |
| 636 | + | |
| 637 | + | |
| 638 | + | |
| 639 | + | |
| 640 | + | |
| 641 | + | |
| 642 | + | |
| 643 | + | |
| 644 | + | |
| 645 | + | |
| 646 | + | |
| 647 | + | |
| 648 | + | |
| 649 | + | |
| 650 | + | |
| 651 | + | |
| 652 | + | |
| 653 | + | |
| 654 | + | |
| 655 | + | |
| 656 | + | |
| 657 | + | |
| 658 | + | |
| 659 | + | |
| 660 | + | |
| 661 | + | |
| 662 | + | |
| 663 | + | |
| 664 | + | |
| 665 | + | |
| 666 | + | |
| 667 | + | |
| 668 | + | |
| 669 | + | |
| 670 | + | |
| 671 | + | |
| 672 | + | |
| 673 | + | |
| 674 | + | |
| 675 | + | |
| 676 | + | |
| 677 | + | |
| 678 | + | |
| 679 | + | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
| 686 | + | |
| 687 | + | |
| 688 | + | |
| 689 | + | |
| 690 | + | |
| 691 | + | |
| 692 | + | |
| 693 | + | |
| 694 | + | |
| 695 | + | |
| 696 | + | |
| 697 | + | |
| 698 | + | |
| 699 | + | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
| 705 | + | |
| 706 | + | |
| 707 | + | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
| 719 | + | |
| 720 | + | |
| 721 | + | |
| 722 | + | |
| 723 | + | |
| 724 | + | |
| 725 | + | |
| 726 | + | |
| 727 | + | |
| 728 | + | |
| 729 | + | |
| 730 | + | |
| 731 | + | |
| 732 | + | |
| 733 | + | |
| 734 | + | |
| 735 | + | |
| 736 | + | |
| 737 | + | |
| 738 | + | |
| 739 | + | |
| 740 | + | |
| 741 | + | |
| 742 | + | |
| 743 | + | |
| 744 | + | |
| 745 | + | |
| 746 | + | |
| 747 | + | |
| 748 | + | |
| 749 | + | |
| 750 | + | |
| 751 | + | |
| 752 | + | |
620 | 753 | | |
621 | 754 | | |
622 | 755 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
59 | | - | |
60 | | - | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | 58 | | |
66 | 59 | | |
67 | 60 | | |
| |||
113 | 106 | | |
114 | 107 | | |
115 | 108 | | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
116 | 164 | | |
117 | 165 | | |
118 | 166 | | |
| |||
421 | 469 | | |
422 | 470 | | |
423 | 471 | | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
424 | 475 | | |
425 | 476 | | |
426 | 477 | | |
427 | 478 | | |
428 | 479 | | |
429 | 480 | | |
430 | | - | |
431 | | - | |
432 | | - | |
433 | | - | |
434 | | - | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
435 | 489 | | |
436 | 490 | | |
437 | 491 | | |
438 | | - | |
439 | | - | |
| 492 | + | |
440 | 493 | | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
441 | 497 | | |
442 | 498 | | |
443 | 499 | | |
444 | 500 | | |
445 | | - | |
| 501 | + | |
446 | 502 | | |
447 | | - | |
| 503 | + | |
448 | 504 | | |
449 | 505 | | |
450 | 506 | | |
451 | | - | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
452 | 515 | | |
453 | 516 | | |
454 | 517 | | |
| |||
477 | 540 | | |
478 | 541 | | |
479 | 542 | | |
480 | | - | |
| 543 | + | |
481 | 544 | | |
482 | 545 | | |
483 | 546 | | |
| |||
0 commit comments