[DGX Spark][Sandbox] managed MCP unusable at runtime — OpenShell never resolves the openshell:resolve:env: credential placeholder (sends it literally); mcp status stays all-green

Description

On DGX Spark (v0.0.74 / OpenShell 0.0.72), a managed MCP server that reports fully healthy via mcp status --json cannot be used at runtime by any agent. Root cause is in the OpenShell credential-resolution layer: the stored credential reference openshell:resolve:env:GITHUB_PERSONAL_ACCESS_TOKEN is never resolved to the real secret when the MCP request goes out on the wire — the literal placeholder string is sent as the bearer, so the MCP server rejects it (400/401/403) and the bridge pre-flight fails.

Egress and auth themselves work: an unauthenticated probe to the endpoint returns 401 (reachable); a hand-built MCP initialize with the real token returns 200 with a full handshake, both from inside the sandbox and from the host. Only the placeholder substitution is dead.

Host-wide, agent-neutral: reproduced on both an openclaw sandbox (mcporter proxy path → CONNECT 403) and a Deep Agents sandbox (direct egress → MCP server skipped: pre-flight failed ... 403 Forbidden, zero MCP tools loaded). The identical NemoClaw/OpenShell stack on DGX Station resolves the placeholder correctly and loads MCP tools end-to-end (Loaded 44 MCP tools, ghmcp_get_me RC=0) — so this is specific to this Spark host / OpenShell build, not the agent bridge code.

Critically, mcp status <server> --json reports all-green throughout (provider attached=true, credentialReady=true, adapter registered) while every real tool invocation fails — the health probe does not exercise credential resolution, so it is misleading.

Environment

Device:        DGX Spark (spark-6087)
OS:            Ubuntu 24.04
Architecture:  aarch64
Node.js:       v22.x
npm:           10.x
Docker:        27.x
OpenShell CLI: 0.0.72
NemoClaw:      v0.0.74
OpenClaw:      2026.5.27 / Deep Agents dcode v0.1.30

Steps to Reproduce

On DGX Spark, onboard a sandbox (openclaw or langchain-deepagents-code).
Register a managed MCP server, supplying the token via --env (v0.0.74 reads the value from the invoking env and stores it as an OpenShell credential provider referenced as openshell:resolve:env:GITHUB_PERSONAL_ACCESS_TOKEN):

export GITHUB_PERSONAL_ACCESS_TOKEN=<real PAT>
nemoclaw <name> mcp add ghmcp --url https://api.githubcopilot.com/mcp/ --env GITHUB_PERSONAL_ACCESS_TOKEN

nemoclaw <name> mcp status ghmcp --json → observe all-green (attached, credentialReady, adapter registered).
Drive the agent to use a tool from that MCP server (dcode -n "use your github MCP tool to show my github user profile", or the openclaw equivalent).

Expected Result

The credential reference resolves to the real token on the wire; the MCP server authenticates; the agent loads the MCP tools and the tool call succeeds end-to-end (as on DGX Station: Loaded 44 MCP tools, ghmcp_get_me RC=0).

Actual Result

The MCP bridge pre-flight fails and the server is skipped; no MCP tools are loaded:

MCP server 'ghmcp' skipped: pre-flight failed: ... 403 Forbidden

(openclaw path shows the same failure as an OpenShell proxy CONNECT 403.)

Isolation probes from inside the sandbox (container ID verified stable across all steps):

GET https://api.githubcopilot.com/mcp/  (unauthenticated)              -> 401 (endpoint reachable)
MCP initialize with the LITERAL placeholder as bearer                 -> 400
MCP initialize with the REAL token as bearer                          -> 200, full MCP handshake
Same real-token initialize from the host                              -> 200

⇒ egress + auth are fine; the credential placeholder is simply never substituted at request time. mcp status stays all-green the entire time.

Logs

$ nemoclaw dcode-xcheck mcp status ghmcp --json
  ...provider attached=true, credentialReady=true, adapter registered=true...   (all green)

$ docker exec -u sandbox <container> dcode -n "use your github MCP tool to show my github user profile"
  MCP server 'ghmcp' skipped: pre-flight failed: <...> 403 Forbidden
  (no "Loaded N MCP tools" line; zero ghmcp tools in the session)

Contrast — DGX Station, identical stack/version, same MCP:
  ✓ Loaded 44 MCP tools ; ghmcp_get_me invoked RC=0  (placeholder resolved correctly)

NVB#6422643

Triage findings

Confirmed the failure surface split after walking the MCP bridge code path. Summary of where responsibility lands:

What NemoClaw does today

mcp add --env <NAME> stores the real secret in an OpenShell credential provider and writes only the placeholder header into the adapter config — Authorization: Bearer openshell:resolve:env:<NAME> (constructed in src/lib/actions/sandbox/mcp-bridge-adapter-status.ts, authPlaceholder/authorizationValue). The real token intentionally never lands inside the sandbox; substitution on the wire is owned by the OpenShell gateway (this boundary is documented in src/lib/actions/sandbox/mcp-bridge-status.ts — "OpenShell owns provider attachment and HTTP rewrite binding").
mcp status derives its green checks purely from metadata:
- credentialReady ← providerMatchesCredential(...) — provider exists, credential key name matches, provider ID matches
- attached ← providerAttached(...) — attachment metadata only
- adapter.registered — adapter config present inside the sandbox

None of these exercises the actual placeholder substitution, which is exactly why status stays all-green while every real request fails. The isolation probes in this report (unauthenticated → 401, literal placeholder → 400, real token → 200) cleanly pin the dead link to the substitution step itself.

Root cause split

OpenShell (primary) — on this Spark host the gateway never rewrites the placeholder on egress; the literal string goes out on the wire. Host-specific (identical stack works on DGX Station). NemoClaw has no interception point on the wire (HTTPS header injection requires the gateway's TLS termination), so this cannot be fixed in this repo. Tracking issue filed on the OpenShell side: [DGX Spark] gateway never rewrites openshell:resolve:env: credential placeholder on sandbox egress — literal placeholder sent on the wire (works on DGX Station, same versions) OpenShell#2161
NemoClaw (secondary, actionable here) — the health surface is misleading. mcp status should exercise credential resolution end-to-end and fail loudly when substitution is broken.

Planned NemoClaw fix

Add a wire-level resolution probe to statusMcpBridge: run an in-sandbox MCP initialize against the registered URL with the placeholder Authorization header (same path the agent runtime takes, via executeSandboxCommand). 2xx → resolution confirmed; 4xx → new provider.credentialResolution field reports the failure with the HTTP status and a warning pointing at the OpenShell host installation, and overall status is no longer green.
Run the probe by default for single-server mcp status <server>; opt-in for the list form to avoid N network calls.
Optionally run the same verification at the tail of mcp add so the failure surfaces at registration time instead of as a silent runtime skip.
Output redacted via the existing redactBridgeSecretsForDisplay path; tests pinned in mcp-bridge-status-boundaries.test.ts.

Note on expectations: the NemoClaw fix restores diagnostic honesty, not runtime functionality — MCP tools on this host stay unusable until the OpenShell substitution issue is resolved. This issue should stay open (or be relabeled as blocked on OpenShell) after the probe PR lands.

test(mcp): keep credential-resolution probe test bodies linear (NVIDI…

Fix PRs are up on both sides:

NemoClaw (diagnostic surface): fix(mcp): probe wire-level credential resolution in mcp status and add (#6379) #6382 — wire-level credential-resolution probe in mcp status / mcp add. CI is green. This makes the failure loud and attributable but does not restore MCP functionality on the affected host.
OpenShell (root cause): fix(network): fail closed when credential placeholders cannot be rewritten OpenShell#2162 — fail-closed hardening for the two degradations that leak the literal placeholder onto the wire: (1) rewrite_http_header_block forwarded header blocks verbatim when the secret resolver was absent (empty provider environment), and (2) a TLS-detected CONNECT fell back to a raw tunnel when ephemeral-CA initialization failed at proxy startup, bypassing rewrite entirely. Both now refuse instead of silently passing the placeholder through, which also turns the runtime failure into an explicit, diagnosable error on hosts like this one.

Suggest keeping this issue open until OpenShell#2162 (or an equivalent upstream fix) lands and the Spark host's CA-init/provider-env failure is confirmed resolved.

fix(mcp): address probe review findings on 400 handling and temp path (…

fix(mcp): make the resolution probe differential, body-free, and URL-…

fix(mcp): probe wire-level credential resolution in mcp status and add (

test(mcp): keep credential-resolution probe test bodies linear (NVIDI…

fix(mcp): address probe review findings on 400 handling and temp path (…

fix(mcp): make the resolution probe differential, body-free, and URL-…

fix(mcp): treat identical 5xx probe responses as indeterminate (NVIDI…

fix(mcp): report identical probe rejections as inconclusive evidence (N…

fix(mcp): probe wire-level credential resolution in mcp status and add (

fix(ci): harden trusted OpenShell release upgrades (#6744)

chore(openshell): upgrade supported version to 0.0.85 (NVIDIA#6726)

[QA] Verified fixed on DGX Spark, NemoClaw v0.0.88 / OpenShell 0.0.85. mcp add reports "Credential resolution verified on the wire (HTTP 200)", and mcp status --json shows credentialResolution: { ok: true, httpStatus: 200, controlHttpStatus: 400 } — the placeholder now resolves to the real token on the wire (200) while the literal-placeholder control returns 400. The agent loads the ghmcp MCP tool with no "pre-flight failed 403 / server skipped" — the original runtime failure is gone. OpenShell#2162 (in OpenShell 0.0.85) + the #6382 diagnostic probe both confirmed on this Spark host.

Navigation Menu

Repository navigation

[DGX Spark][Sandbox] managed MCP unusable at runtime — OpenShell never resolves the openshell:resolve:env: credential placeholder (sends it literally); mcp status stays all-green #6379

Description

Environment

Steps to Reproduce

Expected Result

Actual Result

Logs

Triage findings

What NemoClaw does today

Root cause split

Planned NemoClaw fix

33 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[DGX Spark][Sandbox] managed MCP unusable at runtime — OpenShell never resolves the openshell:resolve:env: credential placeholder (sends it literally); mcp status stays all-green #6379

Description

Description

Environment

Steps to Reproduce

Expected Result

Actual Result

Logs

Activity

TonyLuo-NV commented on Jul 7, 2026

Triage findings

What NemoClaw does today

Root cause split

Planned NemoClaw fix

TonyLuo-NV commented on Jul 7, 2026

33 remaining items

wangericnv commented on Jul 20, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions