Skip to content

[Ubuntu 26.04][Sandbox] Hermes does not restart automatically after its Docker container restarts #13165

New issue
New issue
Open
Bug
Open
[Ubuntu 26.04][Sandbox] Hermes does not restart automatically after its Docker container restarts#13165
Bug
Assignees
chengjiew
Labels
NV QABugs found by the NVIDIA QA Teamarea: sandboxOpenShell sandbox lifecycle, runtime, config, or recoveryintegration: hermesHermes integration behaviorplatform: containerAffects Docker, containerd, Podman, or images

Description

@mercl-lau
mercl-lau
opened on Oct 14, 2026

Description

A healthy Ready Hermes sandbox stops serving its container-local gateway after the exact sandbox Docker container is restarted. The gateway does not recover automatically during a complete 120-second observation window.

Platform scope: Reproduced on Ubuntu 26.04 GPU only; other platforms were not tested for this exact container-restart flow.

Regression: Unknown — earlier versions were not tested with this exact Docker restart contract.

OpenShell issue: No — the affected behavior is the managed Hermes sandbox lifecycle after its container restarts.

Environment

Device:        Ubuntu GPU validation host
OS:            Ubuntu 26.04
Architecture:  x86_64
Node.js:       v22.x (exact patch was not captured)
npm:           10.x (exact patch was not captured)
Docker:        29.1.3
OpenShell CLI: 0.0.116
NemoClaw:      v0.0.133
OpenClaw:      N/A (Hermes runtime)

Steps to Reproduce

  1. Install NemoClaw v0.0.133 and onboard a fresh Hermes sandbox.
  2. Confirm the sandbox is Ready, a real Hermes turn succeeds, and the container-local Hermes health endpoint is reachable.
  3. Resolve the exact Docker container owned by that sandbox.
  4. Run docker restart CONTAINER_ID.
  5. Without invoking a public recovery command, poll the container-local Hermes health endpoint for 120 seconds.

Expected Result

The managed Hermes gateway starts automatically after the sandbox container restarts and its health endpoint becomes reachable within the recovery window.

Actual Result

The Docker restart succeeds, but the container-local Hermes health endpoint remains unreachable for the full 120-second window. Every final probe exits 7.

Logs

Precondition: real Hermes turn and health check passed
docker restart: succeeded
Recovery window: 120 seconds
Final health probe exit: 7
Result: container-local Hermes health did not recover automatically

Activity

  1. mercl-lau
    added
    NV QABugs found by the NVIDIA QA Team
    needs: triageAwaiting maintainer classification
    on Oct 14, 2026
  2. mercl-lau
    added the
    Bug
    issue type on Oct 14, 2026
  3. yimoj
    assigned
    yimoj
    and unassigned
    yimoj
    on Oct 15, 2026
  4. yanyunl1991
    self-assigned this
    on Oct 21, 2026

  5. mercl-lau commented on Oct 21, 2026

    @mercl-lau
    mercl-lau
    on Oct 21, 2026
    ContributorAuthor

    Still reproduces on a second platform and two releases later — so this is not specific to Ubuntu 26.04 GPU or to v0.0.133.

    Ubuntu 22.04 (x86_64), NemoClaw v0.0.135 — same behavior as originally reported:

    • Pre-restart checks all pass: a real Hermes turn succeeds, and the exact container plus both forwards are healthy.
    • After docker restart of the sandbox's own container, the container-local Hermes health endpoint never recovers automatically — it stays unreachable for the full 120-second window and every probe exits 7.

    Environment:

    OS:            Ubuntu 22.04 (Linux 5.15.0-173-generic)
Architecture:  x86_64
Node.js:       v22.22.3
npm:           10.9.8
Docker:        Docker Engine 29.4.0
OpenShell CLI: openshell 0.0.116
NemoClaw:      nemoclaw v0.0.135

    Observed:

    PASS  Preconditions: real turn, marker, exact container and both forwards are healthy  (11641ms)
FAIL  exact container restart triggers automatic health before any public command  (120479ms)
      container-local Hermes health did not recover automatically; last exit=7

    A manual public recovery command still restores the gateway; only the automatic post-restart recovery is affected.

  6. mercl-lau commented on Oct 21, 2026

    @mercl-lau
    mercl-lau
    on Oct 21, 2026
    ContributorAuthor

    Now also reproduced on macOS with NemoClaw v0.0.136 — a third platform and a third consecutive release, confirming this is general, not platform- or version-specific.

    Onboard was fully healthy on this run (Hermes gateway healthy, inference smoke passed), so the failure is the post-restart recovery itself, not setup:

    ✓ Preconditions 2-4: real turn, marker, exact container and both forwards are healthy  (5191ms)
× Steps 1-2: exact container restart triggers automatic health before any public command  (120171ms)
   container-local Hermes health did not recover automatically; last exit=7

    Reproduction matrix so far:

    • Ubuntu 26.04 (GPU), NemoClaw v0.0.133
    • Ubuntu 22.04 (x86_64), NemoClaw v0.0.135
    • macOS (Apple silicon), NemoClaw v0.0.136

  7. yanyunl1991 commented on Oct 22, 2026

    @yanyunl1991
    yanyunl1991
    on Oct 22, 2026
    Contributor

    Root cause is in OpenShell, not Hermes. I filed it upstream as NVIDIA/OpenShell#4828.

    Reproduced on NemoClaw main (86d3c83) on Ubuntu 24.04 x86_64 with OpenShell 0.0.116 and NVIDIA Endpoints, starting from a fresh environment:

    • Hermes sandbox: Ready, and in-sandbox /health returned 200. After docker restart of its exact container, the OpenShell phase was Error for the full 120 seconds.
    • OpenClaw sandbox, same flow: also Error, although Docker reported the container healthy. The problem is not specific to Hermes.
    • Inside the restarted Hermes container, the gateway (hermes gateway run --external-supervisor), the 8642 forward and the dashboard all run again. The agent recovers; the OpenShell phase does not.
    • The gateway log records one ReportMainProcessExit per restart, then the supervisor reconnects (ConnectSupervisor). After that the phase stays Error.

    No NemoClaw command recovers the sandbox, because OpenShell rejects each lifecycle step while the phase is Error:

    Command Result
    nemoclaw <sb> start / stop OpenShell rejected the start/stop request because the sandbox is in Error state
    nemoclaw <sb> recover fails, because the sandbox is in Error
    nemoclaw <sb> rebuild Sandbox '<sb>' is unreachable: the pre-recreate backup needs exec, and exec is blocked
    restart the OpenShell gateway phase is still Error (persisted)

    The same behavior reproduces with plain OpenShell and no NemoClaw: a sleep infinity sandbox goes to Error after docker restart, and openshell sandbox stop / start reject it ("sandbox must be Ready to stop" / "sandbox must be Stopped to start"). Details are in NVIDIA/OpenShell#4828.

    NemoClaw cannot implement automatic recovery: it has no host-side watcher, and OpenShell owns the sandbox phase. The fix belongs in OpenShell. Either the sandbox returns to Ready when its supervisor reconnects, or stop/start are accepted from this Error. Until then, the only path that works is nemoclaw <sb> destroy, then nemoclaw onboard, which loses the sandbox workspace.

    I am unassigning myself, because this needs the upstream OpenShell fix.

  8. yanyunl1991
    removed their assignment
    on Oct 22, 2026
  9. wscurran
    added
    area: sandboxOpenShell sandbox lifecycle, runtime, config, or recovery
    integration: hermesHermes integration behavior
    platform: containerAffects Docker, containerd, Podman, or images
    and removed
    needs: triageAwaiting maintainer classification
    on Oct 22, 2026
  10. chengjiew
    self-assigned this
    on Oct 23, 2026

  11. chengjiew commented on Oct 23, 2026

    @chengjiew
    chengjiew
    on Oct 23, 2026
    Contributor

    try to fix openshell

  12. chengjiew commented on Oct 23, 2026

    @chengjiew
    chengjiew
    on Oct 23, 2026
    Contributor

    PR created to fix the upstream OpenShell root cause: NVIDIA/OpenShell#4865

    fix(server): recover a sandbox to Ready after its container restarts — when a sandbox backing container is restarted, the reconnecting supervisor is a fresh instance and now restores the sandbox to Ready instead of leaving it in Error (which previously blocked exec/stop/start/recover/rebuild). Also fixes NVIDIA/OpenShell#4828. Regression tests added for the container-restart reconnect and for the no-resurrection cases (same-instance reconnect, Completed).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

NV QABugs found by the NVIDIA QA Teamarea: sandboxOpenShell sandbox lifecycle, runtime, config, or recoveryintegration: hermesHermes integration behaviorplatform: containerAffects Docker, containerd, Podman, or images

Type

Bug

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions