Skip to content

[All Platforms][Inference] Multi-sandbox parallel routing test fails: both sandboxes hit gateway scope-upgrade block and fall back to embedded mode #5343

Closed

Description

@PrachiShevate-nv

Description

The documented multi-sandbox routing test requires two NemoClaw sandboxes with different inference providers to handle prompts in parallel via the gateway (inference.local). When executed on this build, both sandboxes correctly discover their respective providers, but all gateway-backed agent requests fail with scope upgrade pending approval and pairing required errors. OpenClaw falls back to embedded mode in each sandbox. This prevents the "two sandboxes, two providers, both via gateway, in parallel" scenario from running as specified.

Environment

Field Value
Host OS Ubuntu 24.04.4 LTS (aarch64)
Docker Engine - Community 29.2.1
NemoClaw v0.0.64 (NVIDIA Cloud + local Ollama)
OpenClaw (in sandboxes) 2026.5.27 (27ae826)
sandbox-A NVIDIA Endpoints, model nvidia/nemotron-3-super-120b-a12b, dashboard http://127.0.0.1:18789/
sandbox-B Local Ollama, model qwen3.5:9b, dashboard http://127.0.0.1:18790/

Pre-conditions

Two NemoClaw sandboxes onboarded and healthy. Each reports its provider via the NemoClaw plugin:

sandbox-A:

NemoClaw registered
Endpoint: Managed Inference Route (inference.local)
Provider: NVIDIA Endpoints
Model:    nvidia/nemotron-3-super-120b-a12b
Slash:    /nemoclaw

sandbox-B:

NemoClaw registered
Endpoint: Managed Inference Route (inference.local)
Provider: Local Ollama
Model:    qwen3.5:9b
Slash:    /nemoclaw

Gateway URLs inside each sandbox:

  • sandbox-A: OPENCLAW_GATEWAY_URL=ws://127.0.0.1:18789
  • sandbox-B: OPENCLAW_GATEWAY_URL=ws://127.0.0.1:18790

Steps to Reproduce

Terminal 1:

nemoclaw sandbox-A connect
openclaw tui
# In TUI, session A1: ask "What is 2+2?"

Terminal 2 (in parallel):

nemoclaw sandbox-B connect
openclaw tui
# In TUI, session B1: ask "What is 3+3?"

Or via non-interactive agent calls:

sandbox-A:

openclaw agent --agent main --session-id cloud-id -m "Hello"

sandbox-B:

openclaw agent --agent main --session-id ollama-id -m "Hello"

Expected Result

Both sandboxes run openclaw tui or openclaw agent via the gateway concurrently:

  • sandbox-A → inference.local → NVIDIA Cloud model nvidia/nemotron-3-super-120b-a12b
  • sandbox-B → inference.local → local Ollama model qwen3.5:9b

No scope upgrade pending approval / pairing required errors during normal operation using only the default NemoClaw/OpenClaw scopes. No EMBEDDED FALLBACK — the intent is to verify gateway-backed routing for two sandboxes and two providers in parallel.

Actual Result

Both sandboxes reject gateway connections and fall back to embedded mode.

sandbox-A (nvidia/nemotron-3-super-120b-a12b):

gateway connect failed: GatewayClientRequestError: scope upgrade pending approval
(requestId: 9d81fd6b-60f7-430b-8e96-b3ac8556436a)
EMBEDDED FALLBACK: Gateway agent failed; running embedded agent:
GatewayTransportError: gateway closed (1008): pairing required:
device is asking for more scopes than currently approved
(requestId: 9d81fd6b-60f7-430b-8e96-b3ac85564)
Gateway target: ws://127.0.0.1:18789
Source: env OPENCLAW_GATEWAY_URL
Config: /sandbox/.openclaw/openclaw.json

sandbox-B (qwen3.5:9b):

gateway connect failed: GatewayClientRequestError: scope upgrade pending approval
(requestId: 1978e739-dbd0-4e43-81ab-6b69c9a1c07f)
EMBEDDED FALLBACK: Gateway agent failed; running embedded agent:
GatewayTransportError: gateway closed (1008): pairing required:
device is asking for more scopes than currently approved
(requestId: 1978e739-dbd0-4e43-81ab-6b69c9a1c)
Gateway target: ws://127.0.0.1:18790
Source: env OPENCLAW_GATEWAY_URL
Config: /sandbox/.openclaw/openclaw.json

Both sandboxes register the NemoClaw plugin and discover their provider correctly, but all gateway-backed agent calls are rejected. Running prompts in parallel now only tests two independent embedded agents, not the intended multi-sandbox gateway scenario.

Impact

  • The multi-sandbox routing test ("sandbox-A Cloud + sandbox-B Ollama, both via gateway, in parallel") cannot be verified on this build.
  • Behavior only exercises embedded agents inside each sandbox, not inference.local gateway routing.
  • Blocks validation of correct multi-provider routing and isolation for the shared gateway.

Notes / Hypothesis

The gateway is treating standard NemoClaw/OpenClaw agent scopes as a "scope upgrade" requiring manual approval, even though the sandbox is already onboarded and configured. Either:

  1. The default gateway pairing/approval state is too strict for NemoClaw's own agent scopes, or
  2. NemoClaw/OpenClaw are requesting additional scopes during this test that are not covered by the initial approval, causing all gateway agent sessions to be rejected.

A fix should ensure that once NemoClaw and the sandboxes are onboarded with their configured providers, openclaw tui and openclaw agent can use the gateway from both sandboxes in parallel without triggering scope upgrade pending approval for the standard test scenario.

Activity

  1. self-assigned this
    on Jun 13, 2026
  2. added
    area: inferenceInference routing, serving, model selection, or outputs
    area: routingRequest routing, policy routing, model selection, or fallback logic
    and removed on Jun 23, 2026
  3. added theissue type on Jun 23, 2026
  4. added a commit that references this issue on Jun 30, 2026
    dca3a49
  5. PrachiShevate-nv commented on Jul 1, 2026

    @PrachiShevate-nv
    Author

    [QA Verification — FAIL] v0.0.71 / DGX Spark (spark-8158, aarch64)

    Environment:

    • Host: DGX Spark (spark-8158), Ubuntu 24.04, aarch64
    • NemoClaw: v0.0.71
    • OpenClaw: 2026.5.27 (27ae826)

    Sandboxes tested:

    • new-sbox: nvidia-prod / nvidia/nemotron-3-nano-omni-30b-a3b-reasoning, dashboard ws://10.200.0.2:18794
    • my-assistant: ollama-local / qwen2.5:1.5b, dashboard ws://10.200.0.2:18791

    Result:
    Both sandboxes hit EMBEDDED FALLBACK with device pairing required when running openclaw agent in parallel. Neither sandbox routed through inference.local via the gateway.

    gateway connect failed: GatewayClientRequestError: device pairing required
    EMBEDDED FALLBACK: GatewayTransportError: gateway closed (1008): pairing required: device is not approved yet

    Expected: Both sandboxes route via gateway (inference.local) with no pairing errors and no embedded fallback.

    Status: NOT FIXED — issue reproduces on v0.0.71. Reopening.

  6. laitingsheng commented on Jul 2, 2026

    @laitingsheng
    Contributor

    I think the new issue should be tracked in #6113 instead since the symptom changed.

  7. added a commit that references this issue on Jul 12, 2026
    11a5cdb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

NV QABugs found by the NVIDIA QA Teamarea: inferenceInference routing, serving, model selection, or outputsarea: routingRequest routing, policy routing, model selection, or fallback logic

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions