bug(rebuild): --force cannot recover an unreachable sandbox with managed MCP state #7062

Investigation Summary

• Reproduced with NemoClaw 0.0.81, OpenShell 0.0.72 (Docker driver), and an OpenClaw 2026.6.10 sandbox containing one healthy managed MCP registration.
• After the sandbox exec relay became unavailable, ordinary rebuild failed its backup safely, but rebuild --force still required an in-sandbox MCP adapter scrub and could not reach the destructive recreation path.
• Generic onboard --recreate-sandbox refused the same sandbox because its registry entry still owned managed MCP state; mcp remove --force could not converge through the broken relay and preserved the ownership manifest.
• Reviewed current v0.0.83 source. The live-sandbox rebuild path still calls prepareMcpBridgesForRebuild, which verifies and scrubs the in-sandbox adapter before detaching the provider and deleting the sandbox.
• Searched open and closed issues. [DGX Spark][Sandbox] rebuild exits 1 silently when MCP destroy markers are present; mcp remove --force cannot clear the incomplete-destroy state (no non-destructive recovery) #6376 concerns synthetic incomplete-destroy markers, refactor(rebuild): journal the destructive boundary and resume absent-sandbox recovery #6435/refactor(rebuild): resume a matching replacement through restore and finalization #6436 concern resuming an interrupted destructive transaction, and Sandbox gateway relay token expires (~15-20 min) with "invalid token: ExpiredSignature", breaking exec/agent/logs/rebuild #6287 concerns relay-token expiration. None provides a supported recovery path when the sandbox is still reported present but cannot execute the managed-adapter teardown.

Description

A registered sandbox with managed MCP state has no supported destructive recovery path when its OpenShell exec relay is permanently unavailable.

This is most visible after a supervisor failure that leaves OpenShell reporting the sandbox as Ready while every openshell sandbox exec fails. The OpenShell failure and deterministic QA fault injection are tracked in NVIDIA/OpenShell#2337.

The normal rebuild behavior is safe: the backup cannot run and NemoClaw aborts before deletion. However, the documented --force path only skips a total backup failure. It still requires the old sandbox to execute the managed MCP adapter cleanup (for OpenClaw, mcporter config remove) before it will detach the provider and delete/recreate the sandbox.

That leaves all supported paths blocked:

• nemoclaw <name> rebuild --yes cannot back up through the failed relay.
• nemoclaw <name> rebuild --force cannot scrub the managed MCP adapter through the failed relay.
• nemoclaw onboard --recreate-sandbox refuses a sandbox that owns managed MCP servers.
• nemoclaw <name> mcp remove <server> --force cannot prove complete adapter/provider/policy cleanup and preserves the ownership manifest.

Recovery then requires direct OpenShell provider manipulation plus manual edits to NemoClaw's registry-owned MCP metadata. Operators should not need to edit ~/.nemoclaw/sandboxes.json to recover a disposable or otherwise explicitly abandoned sandbox.

Reproduction Steps

Use a disposable sandbox and a QA-owned Streamable HTTP MCP fixture. The fault injection in step 3 intentionally makes the sandbox unreachable.

1. Onboard an OpenClaw sandbox:

   nemoclaw onboard
   # Select OpenClaw and name the sandbox qa-worker.

2. Register one managed MCP server and confirm it is healthy:

   export QA_MCP_TOKEN='<disposable test credential>'
   nemoclaw qa-worker mcp add qa-mcp \
     --url https://<qa-mcp-host>/mcp \
     --env QA_MCP_TOKEN
   nemoclaw qa-worker mcp status qa-mcp --json

   Confirm provider attachment, credential readiness, policy readiness, and adapter registration all report healthy.

3. Make the existing sandbox supervisor unable to accept new proxy/relay descriptors while leaving the control-plane phase Ready. The deterministic Docker-driver fault injection is documented in bug(proxy): EMFILE stops the sandbox proxy while phase remains Ready OpenShell#2337. Verify the precondition:

   openshell sandbox list
   openshell sandbox exec --name qa-worker -- true

   The list still reports Ready; exec fails with a relay pipe/EOF error.

4. Run the safe rebuild:

   nemoclaw qa-worker rebuild --yes --verbose

   Backup fails and rebuild aborts with the rebuild --force recovery hint.

5. Accept loss of unsnapshotted sandbox state and run the documented force path:

   nemoclaw qa-worker rebuild --force --verbose

6. Try both other public recovery paths:

   nemoclaw onboard --recreate-sandbox
   nemoclaw qa-worker mcp remove qa-mcp --force

Actual Behavior

The force rebuild fails before recreation because managed MCP preparation still needs the unavailable exec relay:

Deleting old sandbox...
Failed to preserve MCP bridges before rebuild: mcporter config remove failed for 'qa-mcp': ...

The Deleting old sandbox... headline is also misleading at this point: MCP adapter preparation happens before the actual openshell sandbox delete call.

Generic recreation refuses the registry entry:

Sandbox 'qa-worker' has managed MCP servers. Refusing the generic onboard recreation path.
Run nemoclaw qa-worker rebuild to preserve and restore MCP bridges.

Forced MCP removal cannot prove cleanup and preserves durable ownership state:

Provider detach state for '<managed-provider>' is unknown; preserved the MCP policy and ownership manifest.

The exact final detail varies according to whether provider detach succeeds before adapter cleanup fails, but no public command converges the sandbox to a recreatable state.

Expected Behavior

After the operator explicitly accepts data loss with rebuild --force, NemoClaw should provide a transactional recovery path for a registered-but-unreachable sandbox with managed MCP state.

A safe implementation could:

1. persist the exact managed MCP recovery intent and provider identities;
2. prove ownership of each managed provider and prevent credential reuse;
3. detach provider references through the host control plane;
4. delete the abandoned sandbox without requiring its adapter to execute;
5. recreate the sandbox from registry metadata; and
6. restore provider attachment, generated policy, and adapter configuration from the durable MCP entries.

If that sequence cannot be made safe, the command should fail before any external mutation and print one supported, ownership-safe cleanup command that can converge without hand-editing the registry.

It should remain fail-closed for ambiguous provider ownership and must not delete a same-name provider owned by another workflow.

Environment

• OS: Ubuntu Linux, x86_64
• Compute driver: Docker
• NemoClaw: 0.0.81
• OpenShell: 0.0.72
• OpenClaw: 2026.6.10
• Latest NemoClaw checked: 0.0.83 source and release notes
• Existing issues checked: open and closed searches across MCP rebuild, force rebuild, provider detach, recreate-sandbox, and exec relay terms

Debug Output

The original sandbox had to be recreated to restore service, so a pre-recovery nemoclaw debug --quick bundle is not available. The minimal diagnostic state was:

$ nemoclaw --version
0.0.81

$ openshell --version
0.0.72

$ openshell sandbox list
# qa-worker: Ready

$ openshell sandbox exec --name qa-worker -- true
Error: exec relay closed before the command reported an exit status

$ nemoclaw qa-worker rebuild --yes --verbose
# Total backup failure; aborts before deletion and suggests --force.

$ nemoclaw qa-worker rebuild --force --verbose
Deleting old sandbox...
Failed to preserve MCP bridges before rebuild: mcporter config remove failed for 'qa-mcp'

No credentials, provider values, endpoint tokens, or sandbox-specific user data are required to reproduce.

Suggested Acceptance Coverage

• A live registry entry with one managed MCP server and a fault-injected failing exec adapter.
• rebuild --force either recreates from durable MCP intent or exits with the original sandbox and all external resources unchanged.
• Process interruption at each provider-detach/delete/recreate boundary remains resumable.
• A failed adapter scrub never strands provider, policy, and registry ownership in different states.
• Generic onboarding continues to refuse unsafe managed-MCP replacement, but its message points to a recovery command that actually works for an unreachable sandbox.

Checklist

• The failure was reproduced on a disposable managed sandbox.
• Existing open and closed issues were searched; related issues are not duplicates.