Repository navigation
Closed
Labels
NV QABugs found by the NVIDIA QA TeamBugs found by the NVIDIA QA Teamarea: cliCommand line interface, flags, terminal UX, or outputCommand line interface, flags, terminal UX, or outputarea: local-modelsLocal model providers, downloads, launch, or connectivityLocal model providers, downloads, launch, or connectivityarea: providersInference provider integrations and provider behaviorInference provider integrations and provider behaviorfixed-on-latestVerified by verify-stale skill: bug not reproducible on latestVerified by verify-stale skill: bug not reproducible on latestprovider: ollamaOllama local model provider behaviorOllama local model provider behavior
Description
Description
Description
On Linux, after `nemoclaw onboard` with provider option 7 (Local Ollama), the in-sandbox OpenClaw TUI cannot get any inference response. Sandbox logs show every agent call returning HTTP 401 Unauthorized from the inference proxy route, even though Ollama itself (running on localhost:11434) has no auth. Ollama directly answers requests fine when called outside the sandbox. The wizard's Review configuration explicitly shows: API key: OPENAI_API_KEY (stored in ~/.nemoclaw/credentials.json) So the wizard reads the host's OPENAI_API_KEY env var and stores it as the bearer credential for the ollama-local provider. This is semantically wrong — local Ollama does not need OpenAI auth — and operationally fragile, because: (a) If the user's OPENAI_API_KEY in env is expired/invalid (a common case, since OpenAI keys get rotated), the cached value triggers HTTP 401 in the inference path even though the user is using a local model. (b) Once cached in credentials.json, simply `unset OPENAI_API_KEY` in the host shell and re-onboarding does NOT clear the bad value — the wizard reads from credentials.json on subsequent onboards. There is no obvious wizard-level path to "use this Ollama with no/empty bearer". Independently verified the host's OPENAI_API_KEY is invalid (curl https://api.openai.com/v1/models with the same env var also returns 401). The same 401 surfaces in inference.local — strongly suggesting the gateway routes / forwards using that bearer.Environment
Device: Linux dev host (host: 2u1g-b650-0082, no GPU) OS: Ubuntu 24.10 (Linux 6.11.0-18-generic) Architecture: x86_64 Node.js: v22.22.2 (installer-managed via nvm) npm: 10.9.7 Docker: Docker version 28.4.0 OpenShell CLI: openshell 0.0.36 NemoClaw: v0.0.26 OpenClaw: 2026.4.9 (0512059) Local Ollama: 0.x (installed by wizard's "Install Ollama (Linux)" path during T5882218 verification), running on localhost:11434, model qwen2.5:7b loaded.Steps to Reproduce
1. On a Linux host where OPENAI_API_KEY is set in env (and is invalid / expired — common in dev environments):
curl -s -o /dev/null -w "%{http_code}\n" https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"
(returns 401, confirming the env key is stale)
2. Install Ollama if not present, ensure `ollama list` shows at least one model (e.g. qwen2.5:7b):
curl -fsSL https://ollama.com/install.sh | sh
ollama pull qwen2.5:7b # or use wizard's "Install Ollama (Linux)" path
3. Run nemoclaw onboard:
nemoclaw onboard --recreate-sandbox
# Provider [1]: 7 (Local Ollama)
# Model [1]: 1 (qwen2.5:7b)
# Sandbox name: any
# Apply [Y/n]: Y
# Brave [y/N]: N
# Messaging: Enter (skip)
4. Note the wizard's Review configuration line:
API key: OPENAI_API_KEY (stored in ~/.nemoclaw/credentials.json)
The wizard captured OPENAI_API_KEY from host env.
5. After onboard finishes, connect and try to use the agent:
nemoclaw connect
openclaw tui
# type "hello"
6. The TUI shows "kerfuffling" / "conjuring" status indefinitely; agent never responds.
In another terminal, `nemoclaw logs --follow` shows:
[agent] embedded run agent end: runId=... isError=true
model=qwen2.5:7b provider=inference
error=HTTP 401: Unauthorized rawError=401 Unauthorized
[agent] embedded run failover decision: runId=... stage=assistant
decision=surface_error reason=auth provider=inference/qwen2.5:7b
Reproducible on every prompt.
7. Workaround attempt that should work (but does NOT):
unset OPENAI_API_KEY # in host shell
nemoclaw onboard --recreate-sandbox # repeat with same defaults
The wizard still says "API key: OPENAI_API_KEY (stored in ~/.nemoclaw/credentials.json)"
even though host env no longer has it. Inference still returns 401 — the stale value
persists in credentials.json from the first onboard.
Expected Result
For provider option 7 (Local Ollama), the wizard should: (a) Either NOT capture OPENAI_API_KEY at all — Ollama doesn't need auth; the gateway should forward to localhost:11434 with no bearer or with an empty/static value. (b) Or, if a bearer header is required by the gateway proxy plumbing (e.g. an internal validation step), it should be a separate, independent credential not coupled to the user's host OPENAI_API_KEY env var. Additionally, on re-onboard with `--recreate-sandbox` and a host env where the key has been cleared, the wizard should refresh the cached credentials.json (or at least allow the user a path like `nemoclaw credentials reset OPENAI_API_KEY` discoverable from the wizard). Currently the user has no obvious way to escape the cached bad value through the onboard flow.Actual Result
Wizard Review configuration shows:
Provider: ollama-local
Model: qwen2.5:7b
API key: OPENAI_API_KEY (stored in ~/.nemoclaw/credentials.json)
...
Note: Sandbox build takes ~6 minutes on this host.
Onboard completes successfully (sandbox build, gateway, presets all OK).
Sandbox status: Inference: healthy (http://127.0.0.1:11434/api/tags) ← health probe goes direct to Ollama, OK
Agent inference: HTTP 401 Unauthorized (reproducible every prompt).
Sandbox logs excerpt:
2026-04-27T11:13:04 [agent] embedded run agent end:
runId=eb7e3a05-... isError=true model=qwen2.5:7b provider=inference
error=HTTP 401: Unauthorized rawError=401 Unauthorized
2026-04-27T11:13:04 [agent] embedded run failover decision:
runId=eb7e3a05-... stage=assistant decision=surface_error
reason=auth provider=inference/qwen2.5:7b profile=-
Reproduces both with provider re-creation via `--recreate-sandbox`. After
`unset OPENAI_API_KEY` + re-onboard, behavior unchanged.
Logs
Discovered during day-0 manual validation of T5882212 (project 1462 / task 508562) on Linux dev host (host: 2u1g-b650-0082, Ubuntu 24.10). Two independent reproductions in the same session: - Run 1 (host has stale OPENAI_API_KEY in env): 401 from inference path. - Run 2 (host had OPENAI_API_KEY unset, --recreate-sandbox to rebuild from scratch): same 401. Distinct from related bugs: - 6115797 (Brev/Ubuntu24 cloud TUI Connection error — security guard blocking inference.local with "Blocked hostname or private/internal/special-use IP address"): this bug shows the routing reaches the upstream and gets a real HTTP 401 response, not a guard block. Different failure mode. - 6107668 (TLS handshake corruption after `openshell provider update`): not triggered here (we did not run provider update; first onboard surfaces 401 immediately). - 6081485 (HTTP 503 inference service unavailable on Jetson Orin): different status code, different upstream. Suspected root cause: The gateway's ollama-local provider config is created with `OPENAI_API_KEY=$value` as the bearer credential. Either (a) the gateway forwards this bearer to the actual ollama proxy at :11435 (which validates and rejects), or (b) the gateway mis-routes the call to api.openai.com (real OpenAI) — the symmetric 401 on direct api.openai.com vs inference.local with the same bearer hints at (b). Engineering should verify which of (a)/(b) is actually happening and fix the wizard to not capture OPENAI_API_KEY for Ollama at all.
Bug Details
| Field | Value |
|---|---|
| Priority | Unprioritized |
| Action | Dev - Open - To fix |
| Disposition | Open issue |
| Module | Machine Learning - NemoClaw |
| Keyword | NemoClaw, NEMOCLAW_GH_SYNC_APPROVAL, NemoClaw_Inference, NemoClaw_Onboard |
[NVB#6116337]
Metadata
Metadata
Assignees
Labels
NV QABugs found by the NVIDIA QA TeamBugs found by the NVIDIA QA Teamarea: cliCommand line interface, flags, terminal UX, or outputCommand line interface, flags, terminal UX, or outputarea: local-modelsLocal model providers, downloads, launch, or connectivityLocal model providers, downloads, launch, or connectivityarea: providersInference provider integrations and provider behaviorInference provider integrations and provider behaviorfixed-on-latestVerified by verify-stale skill: bug not reproducible on latestVerified by verify-stale skill: bug not reproducible on latestprovider: ollamaOllama local model provider behaviorOllama local model provider behavior
Activity
[-][Linux][Inference] Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset[/-][+][Linux]Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset[/+]fix(onboard): also decouple sandbox-side config from OPENAI_API_KEY
latenighthackathon commentedon Apr 28, 2026
Filed #2625 for what looks like the upgrade-sandboxes side of the same broken assumption you hit here — wizard caches
OPENAI_API_KEYfor local providers (this issue), and the rebuild preflight then demands it after a provider switch from remote to local (the new one). Both stem from local providers being treated as if they need remote credentials. Closing one direction does not close the other, but a canonical pattern (clearsession.credentialEnvon provider switch + skip remote-credential capture in the wizard for local providers) would close both. Cheers!fix(onboard): decouple local inference providers from OPENAI_API_KEY (#…
fix(onboard): decouple local inference providers from OPENAI_API_KEY (N…
[-][Linux]Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset[/-][+][Linux][Brev]Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset[/+]hulynn commentedon Apr 30, 2026
[Repro] v0.0.31, no OPENAI_API_KEY, clean machine
Reproduced on nemoclaw v0.0.31 (Brev cloud, Ubuntu 22.04, A100-SXM4-80GB, Ollama v0.22.0, model qwen2.5:0.5b).
Trigger: No OPENAI_API_KEY set. Wizard shows API key: (not required for ollama-local) — nothing stored in credentials.json. Onboard completes, nemoclaw status reports Inference: healthy. TUI spins indefinitely, no response.
Sandbox logs:
Host:
The gateway routes to :11435 but injects no Authorization header. openshell provider get ollama-local shows NEMOCLAW_OLLAMA_PROXY_TOKEN as the credential key, but it is never injected as Bearer when routing.
Same root cause as this bug — different trigger: no stale OPENAI_API_KEY needed, just a clean machine with no OPENAI_API_KEY at all.
jyaunches commentedon May 7, 2026
Status update — closing against original 401 scope
PR #2580 landed the wizard-side fix (merged 2026-04-28, released in v0.0.30+):
OPENAI_API_KEYforollama-local/vllm-local— recordscredentialEnv = nulland shows(not required for ollama-local).NEMOCLAW_OLLAMA_PROXY_TOKENfor ollama-local,NEMOCLAW_VLLM_LOCAL_TOKENfor vllm-local) instead of reading the host key.OPENAI_API_KEYentries are pruned from~/.nemoclaw/credentials.jsonon next onboard.That resolves the original HTTP 401 symptom described in this report (stale host
OPENAI_API_KEYbleeding into the inference path).However, the underlying credential-injection path is still broken at runtime — per @hulynn's v0.0.31 reproduction on 2026-04-30: on a clean machine with no
OPENAI_API_KEYat all, the gateway routes to:11435but never injectsNEMOCLAW_OLLAMA_PROXY_TOKENas a Bearer header, so the call surfaces as a silent hang instead of a 401.That follow-on scope is tracked separately in a new issue so we don't re-open and re-title this one and so NV QA can carry a fresh bug ID against the v0.0.31 repro.
→ Continuation: #3198
Closing this as the original 401 scope is fixed.
Note: PR #2580's body used
Refs #2519rather thanFixes #2519, which is why this issue wasn't auto-closed on merge and why the "Development" field stayed empty. Team reminder to useFixes #/Closes #going forward.fix(verify-stale): echo resolved nemoclaw version after install, fix …
fix(onboard): persist credentialEnv clears when switching remote→local (
fix(verify-stale): re-check state == OPEN before posting
hulynn commentedon May 11, 2026
Verified Fixed in v0.0.38 — 2026-05-09
Steps
Result — wizard Review configuration
Persistence + provider-credential evidence
Root cause definitively addressed: wizard no longer captures host's
OPENAI_API_KEYinto the Ollama provider's bearer credential. A separateNEMOCLAW_OLLAMA_PROXY_TOKENis generated for the local Ollama proxy at :11435, independent of any host env var. The HTTP 401 chain shown in the original bug (gateway forwarding stale OPENAI_API_KEY → upstream rejects) cannot occur because the bearer source has been decoupled.(End-to-end TUI inference smoke test not run on this verification because a deliberately bogus
NEMOCLAW_PROXY_HOST=1.2.3.4was still set in parallel for 6045052 verification and blocks outgoing connections inside the sandbox. The config-level evidence above is sufficient to confirm the root cause is fixed; the wizard's prior behavior of capturing OPENAI_API_KEY has been removed entirely.)feat(skills): add nemoclaw-maintainer-verify-stale (#3327)