Sandbox non-persistence, unstable gateway lifecycle, and apparent one-request-at-a-time -m LLM path in VM deployment

Description

NemoClaw is not behaving like a stable, repeatable agent runtime in my VM-based deployment.

I am seeing four related problems:

Sandbox non-persistence

After restart/reboot, sandbox state is lost or becomes unusable
Recovery appears to require reinitialization instead of normal restart

Gateway lifecycle instability

I do not see a reliable non-destructive start/stop/restart workflow
Recovery appears to require destructive teardown/reset

Unclear or inconsistent CLI behavior

Commands such as:
nemoclaw connect
did not behave as expected
It is not clear what the supported/reliable command path is for reconnecting to an existing setup

LLM interaction appears limited to one request at a time

In practice, we could only get one request at a time to the LLM by using the -m argument
That makes the system feel like a one-shot command path rather than a persistent multi-step agent workflow

Together, these issues block progression from an early proof-of-concept to a repeatable system.

Expected Behavior

Sandbox should persist across restart/reboot, or there should be a clearly documented restore path
Gateway should support a reliable non-destructive lifecycle:
- start
- stop
- restart
CLI commands for connect/reconnect should be deterministic and documented
The system should support persistent multi-step agent workflows, not only one request at a time via -m
There should be a minimal "hello world" validation path for:
- gateway health
- sandbox readiness
- model connectivity

Actual Behavior

Sandbox state is lost or becomes unusable after restart/reboot
Gateway recovery appears to require destructive teardown/reset
CLI reconnect path is unclear or inconsistent
Practical LLM interaction appears limited to one request at a time using -m
There is no obvious minimal validation path to separate gateway issues, sandbox issues, and model connectivity issues

Addtional Context
This effort is currently around NASA Technology Readiness Level (TRL) 2-3:

concept and early proof-of-concept have been demonstrated

The goal is to progress toward TRL 4-5:

validated, repeatable system behavior

Right now, sandbox non-persistence, lifecycle fragility, and the apparent one-request-at-a-time -m interaction path are blocking that transition.

Target architecture:

isolated VM sandbox
host-served local model over HTTP
repeatable agent workflow
no dependence on external API-key-only flow for basic local testing

Happy to test patches, provide logs, or validate fixes in a controlled VM environment.

Reproduction Steps

Deploy NemoClaw in Ubuntu 22.04 VM under KVM/QEMU
Use Docker-based NemoClaw setup
Start gateway and sandbox
Attempt agent workflow / connection workflow
Restart VM and/or services
Observe that sandbox state is lost or unusable, and normal recovery is unclear
Attempt reconnect / reuse workflow with commands such as:
nemoclaw connect
Observe unclear or inconsistent behavior
Attempt LLM requests
Observe that practical operation appears limited to one request at a time using the -m argument

Environment

Host: Fedora
Hardware: Minisforum MS-S1
Memory split target/use case: 128 GB UMA system, targeting 32 GB CPU / 96 GB GPU
VM: Ubuntu 22.04
Hypervisor: KVM/QEMU
Runtime: Docker + NemoClaw
Architecture: VM sandbox -> host local model over HTTP
Use case: secure isolated agent workflows using a local model endpoint

Debug Output

Attempted to run the requested debug command:

  nemoclaw debug --quick

Result:

  Unknown command: debug

  Registered sandboxes: nemoclaw-baseline, <redacted-user-sandbox>
  Try: nemoclaw <sandbox-name> connect

  Run 'nemoclaw help' for usage.

This suggests the installed CLI does not include the debug subcommand referenced by the issue template.

Collected diagnostics using available commands:

1. nemoclaw list

  Sandboxes:
    nemoclaw-baseline *
      model: unknown  provider: unknown  CPU  policies: none
    <redacted-user-sandbox>
      model: nvidia/nemotron-3-super-120b-a12b  provider: nvidia-nim  CPU  policies: none

  * = default sandbox

2. nemoclaw nemoclaw-baseline status

  Sandbox: nemoclaw-baseline
    Model:    unknown
    Provider: unknown
    GPU:      no
    Policies: none
    NIM:      not running

3. nemoclaw nemoclaw-baseline logs

  error: unrecognized subcommand 'logs'

  Usage: openshell sandbox [OPTIONS] [COMMAND]

  For more information, try '--help'.
    Command failed (exit 2): openshell sandbox logs nemoclaw-baseline

4. nemoclaw status

  Sandboxes:
    nemoclaw-baseline *
    <redacted-user-sandbox> (nvidia/nemotron-3-super-120b-a12b)

  ● telegram-bridge  (stopped)
  ● cloudflared  (stopped)

Notes:
- The issue template references a debug command that is not present in the installed CLI
- The CLI help advertises a logs command, but it fails due to an underlying openshell subcommand error
- Default sandbox reports unknown model/provider and NIM not running

Logs

Unable to provide logs.

The CLI advertises:
  nemoclaw <name> logs

However, invoking it results in:

  error: unrecognized subcommand 'logs'
  Command failed (exit 2): openshell sandbox logs <sandbox-name>

This appears to be a broken or unimplemented logs pathway in the current CLI.

Checklist

I confirmed this bug is reproducible
I searched existing issues and this is not a duplicate

Do you have a stack trace or associated logs?

Description

NemoClaw is not behaving like a stable, repeatable agent runtime in my VM-based deployment.

I am seeing four related problems:
1. Sandbox non-persistence


* After restart/reboot, sandbox state is lost or becomes unusable

* Recovery appears to require reinitialization instead of normal restart


2. Gateway lifecycle instability


* I do not see a reliable non-destructive start/stop/restart workflow

* Recovery appears to require destructive teardown/reset


3. Unclear or inconsistent CLI behavior


* Commands such as:
  nemoclaw  connect
  did not behave as expected

* It is not clear what the supported/reliable command path is for reconnecting to an existing setup


4. LLM interaction appears limited to one request at a time


* In practice, we could only get one request at a time to the LLM by using the -m argument

* That makes the system feel like a one-shot command path rather than a persistent multi-step agent workflow
Together, these issues block progression from an early proof-of-concept to a repeatable system.

Expected Behavior
* Sandbox should persist across restart/reboot, or there should be a clearly documented restore path

* Gateway should support a reliable non-destructive lifecycle:
  
  * start
  * stop
  * restart

* CLI commands for connect/reconnect should be deterministic and documented

* The system should support persistent multi-step agent workflows, not only one request at a time via -m

* There should be a minimal "hello world" validation path for:
  
  * gateway health
  * sandbox readiness
  * model connectivity
Actual Behavior
* Sandbox state is lost or becomes unusable after restart/reboot

* Gateway recovery appears to require destructive teardown/reset

* CLI reconnect path is unclear or inconsistent

* Practical LLM interaction appears limited to one request at a time using -m

* There is no obvious minimal validation path to separate gateway issues, sandbox issues, and model connectivity issues
Addtional Context This effort is currently around NASA Technology Readiness Level (TRL) 2-3:
* concept and early proof-of-concept have been demonstrated
The goal is to progress toward TRL 4-5:
* validated, repeatable system behavior
Right now, sandbox non-persistence, lifecycle fragility, and the apparent one-request-at-a-time -m interaction path are blocking that transition.

Target architecture:
* isolated VM sandbox

* host-served local model over HTTP

* repeatable agent workflow

* no dependence on external API-key-only flow for basic local testing
Happy to test patches, provide logs, or validate fixes in a controlled VM environment.

Reproduction Steps
1. Deploy NemoClaw in Ubuntu 22.04 VM under KVM/QEMU

2. Use Docker-based NemoClaw setup

3. Start gateway and sandbox

4. Attempt agent workflow / connection workflow

5. Restart VM and/or services

6. Observe that sandbox state is lost or unusable, and normal recovery is unclear

7. Attempt reconnect / reuse workflow with commands such as:
   nemoclaw  connect

8. Observe unclear or inconsistent behavior

9. Attempt LLM requests

10. Observe that practical operation appears limited to one request at a time using the -m argument
Environment
* Host: Fedora

* Hardware: Minisforum MS-S1

* Memory split target/use case: 128 GB UMA system, targeting 32 GB CPU / 96 GB GPU

* VM: Ubuntu 22.04

* Hypervisor: KVM/QEMU

* Runtime: Docker + NemoClaw

* Architecture: VM sandbox -> host local model over HTTP

* Use case: secure isolated agent workflows using a local model endpoint
Debug Output

Attempted to run the requested debug command:

nemoclaw debug --quick

Result:

Unknown command: debug

Registered sandboxes: nemoclaw-baseline,
Try: nemoclaw connect

Run 'nemoclaw help' for usage.

This suggests the installed CLI does not include the debug subcommand referenced by the issue template.

Collected diagnostics using available commands:

nemoclaw list

Sandboxes:
nemoclaw-baseline *
model: unknown provider: unknown CPU policies: none

model: nvidia/nemotron-3-super-120b-a12b provider: nvidia-nim CPU policies: none

= default sandbox

nemoclaw nemoclaw-baseline status

Sandbox: nemoclaw-baseline
Model: unknown
Provider: unknown
GPU: no
Policies: none
NIM: not running

nemoclaw nemoclaw-baseline logs

error: unrecognized subcommand 'logs'

Usage: openshell sandbox [OPTIONS] [COMMAND]

For more information, try '--help'.
Command failed (exit 2): openshell sandbox logs nemoclaw-baseline

nemoclaw status

Sandboxes:
nemoclaw-baseline *
(nvidia/nemotron-3-super-120b-a12b)

● telegram-bridge (stopped)
● cloudflared (stopped)

Notes:

The issue template references a debug command that is not present in the installed CLI

The CLI help advertises a logs command, but it fails due to an underlying openshell subcommand error

Default sandbox reports unknown model/provider and NIM not running

Logs

Unable to provide logs.

The CLI advertises:
nemoclaw logs

However, invoking it results in:

error: unrecognized subcommand 'logs'
Command failed (exit 2): openshell sandbox logs

This appears to be a broken or unimplemented logs pathway in the current CLI.

Checklist
* [x]  I confirmed this bug is reproducible[x]  I searched existing issues and this is not a duplicate

Agreed, nemoclaw is broken. There is no persistence and anytime docker decides to crash you have to rebuild from scratch. Been trying to get a stable deployment on my Jetson Orin Nano Super for the past week, when it works its great but god forbid you need to reboot. Back to square one each time, openshell refuses to start a stopped gateway without destroying and recreating . I know we all work for free for Nvidia and just wasting our tokens and overcharging our AI subscriptiosn but if this is how Nvidia wants to drive adoption...

Workaround: restore-sandbox.sh — push all customizations back after every wipe

Sandbox non-persistence is a real pain. Our solution is a restore-sandbox.sh script that runs on every startup and re-pushes everything needed via SSH. We keep a ~/nemoclaw-persist/ directory on the host with everything that needs to survive a wipe.

The script pushes:

Network policy YAML (openshell policy set)
.openclaw/.env (API keys, NODE_OPTIONS)
SOUL.md (agent personality)
models.json + auth-profiles.json (model config, bypasses the root-owned openclaw.json problem)
Custom skills directories
sessions.json (conversation history)
memories.json (agent memory)
Re-sets inference: openclaw models set provider/model
Starts any background services (memory server, etc.)

Core pattern:

push_file() {
  local src="$1" dst="$2"
  ssh -T -F "$SSH_CONF" "openshell-${SANDBOX}" "mkdir -p $(dirname $dst)"
  ssh -T -F "$SSH_CONF" "openshell-${SANDBOX}" "cat > $dst" < "$src"
}

# Apply policy
openshell policy set "$SANDBOX" --policy "$PERSIST_DIR/policy.yaml" --wait

# Push files
push_file "$PERSIST_DIR/openclaw.env" "/sandbox/.openclaw/.env"
push_file "$PERSIST_DIR/SOUL.md"      "/sandbox/.openclaw-data/workspace/SOUL.md"
push_file "$PERSIST_DIR/models.json"  "/sandbox/.openclaw/agents/main/agent/models.json"

We also back up sessions.json and memories.json every 5-30 min from the sandbox back to the host persist dir, so nothing is lost even when Docker decides to wipe things.

Combined with the startup recovery script in #910, this gives full persistence across crashes, recreates, and reboots with no manual steps.

fix(onboard): upgrade OpenShell to 0.0.24 for sandbox survival across…

All four issues described here have been addressed. Sandbox survival across gateway restarts landed in feat: sandbox survival across gateway restarts and was hardened further when OpenShell was upgraded to 0.0.24 (fix(onboard): upgrade OpenShell to 0.0.24 for sandbox survival across restarts). OpenClaw processes now auto-recover after sandbox restart (feat(cli): auto-recover OpenClaw processes after sandbox restart). If you're still seeing non-persistence or lifecycle instability on the current version, please reopen with logs from nemoclaw <name> logs — that scenario is now well-tested and regressions are worth investigating.

test(e2e): migrate test-sandbox-survival.sh to vitest (#5332)

Navigation Menu

Repository navigation

Sandbox non-persistence, unstable gateway lifecycle, and apparent one-request-at-a-time -m LLM path in VM deployment #859

Description

Reproduction Steps

Environment

Debug Output

Logs

Checklist

Description

Reproduction Steps

Environment

Debug Output

Logs

Checklist

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Sandbox non-persistence, unstable gateway lifecycle, and apparent one-request-at-a-time -m LLM path in VM deployment #859

Description

Description

Reproduction Steps

Environment

Debug Output

Logs

Checklist

Activity

8r4n commented on Mar 25, 2026

ACCGAGTT commented on Mar 25, 2026

Description

Reproduction Steps

Environment

Debug Output

Logs

Checklist

dknos commented on Mar 30, 2026

wscurran commented on Apr 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions