Skip to content

[Linux][Brev]Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset #2519

Closed

Description

@hulynn

Description

Description

On Linux, after `nemoclaw onboard` with provider option 7 (Local Ollama), the in-sandbox OpenClaw TUI cannot get any inference response. Sandbox logs show every agent call returning HTTP 401 Unauthorized from the inference proxy route, even though Ollama itself (running on localhost:11434) has no auth. Ollama directly answers requests fine when called outside the sandbox.

The wizard's Review configuration explicitly shows:
  API key:  OPENAI_API_KEY (stored in ~/.nemoclaw/credentials.json)

So the wizard reads the host's OPENAI_API_KEY env var and stores it as the bearer credential for the ollama-local provider. This is semantically wrong — local Ollama does not need OpenAI auth — and operationally fragile, because:

(a) If the user's OPENAI_API_KEY in env is expired/invalid (a common case, since OpenAI keys get rotated), the cached value triggers HTTP 401 in the inference path even though the user is using a local model.
(b) Once cached in credentials.json, simply `unset OPENAI_API_KEY` in the host shell and re-onboarding does NOT clear the bad value — the wizard reads from credentials.json on subsequent onboards. There is no obvious wizard-level path to "use this Ollama with no/empty bearer".

Independently verified the host's OPENAI_API_KEY is invalid (curl https://api.openai.com/v1/models with the same env var also returns 401). The same 401 surfaces in inference.local — strongly suggesting the gateway routes / forwards using that bearer.
Environment
Device:        Linux dev host (host: 2u1g-b650-0082, no GPU)
OS:            Ubuntu 24.10 (Linux 6.11.0-18-generic)
Architecture:  x86_64
Node.js:       v22.22.2 (installer-managed via nvm)
npm:           10.9.7
Docker:        Docker version 28.4.0
OpenShell CLI: openshell 0.0.36
NemoClaw:      v0.0.26
OpenClaw:      2026.4.9 (0512059)

Local Ollama: 0.x (installed by wizard's "Install Ollama (Linux)" path during T5882218 verification), running on localhost:11434, model qwen2.5:7b loaded.
Steps to Reproduce
1. On a Linux host where OPENAI_API_KEY is set in env (and is invalid / expired — common in dev environments):
     curl -s -o /dev/null -w "%{http_code}\n" https://api.openai.com/v1/models -H "Authorization: Bearer $OPENAI_API_KEY"
   (returns 401, confirming the env key is stale)

2. Install Ollama if not present, ensure `ollama list` shows at least one model (e.g. qwen2.5:7b):
     curl -fsSL https://ollama.com/install.sh | sh
     ollama pull qwen2.5:7b   # or use wizard's "Install Ollama (Linux)" path

3. Run nemoclaw onboard:
     nemoclaw onboard --recreate-sandbox
     # Provider [1]: 7   (Local Ollama)
     # Model [1]: 1     (qwen2.5:7b)
     # Sandbox name: any
     # Apply [Y/n]: Y
     # Brave [y/N]: N
     # Messaging: Enter (skip)

4. Note the wizard's Review configuration line:
     API key:  OPENAI_API_KEY (stored in ~/.nemoclaw/credentials.json)
   The wizard captured OPENAI_API_KEY from host env.

5. After onboard finishes, connect and try to use the agent:
     nemoclaw  connect
     openclaw tui
     # type "hello"

6. The TUI shows "kerfuffling" / "conjuring" status indefinitely; agent never responds.
   In another terminal, `nemoclaw  logs --follow` shows:
     [agent] embedded run agent end: runId=... isError=true
       model=qwen2.5:7b provider=inference
       error=HTTP 401: Unauthorized rawError=401 Unauthorized
     [agent] embedded run failover decision: runId=... stage=assistant
       decision=surface_error reason=auth provider=inference/qwen2.5:7b
   Reproducible on every prompt.

7. Workaround attempt that should work (but does NOT):
     unset OPENAI_API_KEY   # in host shell
     nemoclaw onboard --recreate-sandbox  # repeat with same defaults
   The wizard still says "API key: OPENAI_API_KEY (stored in ~/.nemoclaw/credentials.json)"
   even though host env no longer has it. Inference still returns 401 — the stale value
   persists in credentials.json from the first onboard.
Expected Result
For provider option 7 (Local Ollama), the wizard should:

(a) Either NOT capture OPENAI_API_KEY at all — Ollama doesn't need auth; the gateway should forward to localhost:11434 with no bearer or with an empty/static value.

(b) Or, if a bearer header is required by the gateway proxy plumbing (e.g. an internal validation step), it should be a separate, independent credential not coupled to the user's host OPENAI_API_KEY env var.

Additionally, on re-onboard with `--recreate-sandbox` and a host env where the key has been cleared, the wizard should refresh the cached credentials.json (or at least allow the user a path like `nemoclaw credentials reset OPENAI_API_KEY` discoverable from the wizard). Currently the user has no obvious way to escape the cached bad value through the onboard flow.
Actual Result
Wizard Review configuration shows:
  Provider:      ollama-local
  Model:         qwen2.5:7b
  API key:       OPENAI_API_KEY (stored in ~/.nemoclaw/credentials.json)
  ...
  Note:          Sandbox build takes ~6 minutes on this host.

Onboard completes successfully (sandbox build, gateway, presets all OK).
Sandbox status: Inference: healthy (http://127.0.0.1:11434/api/tags)   ← health probe goes direct to Ollama, OK
Agent inference: HTTP 401 Unauthorized (reproducible every prompt).

Sandbox logs excerpt:
  2026-04-27T11:13:04 [agent] embedded run agent end:
    runId=eb7e3a05-... isError=true model=qwen2.5:7b provider=inference
    error=HTTP 401: Unauthorized rawError=401 Unauthorized
  2026-04-27T11:13:04 [agent] embedded run failover decision:
    runId=eb7e3a05-... stage=assistant decision=surface_error
    reason=auth provider=inference/qwen2.5:7b profile=-

Reproduces both with provider re-creation via `--recreate-sandbox`. After
`unset OPENAI_API_KEY` + re-onboard, behavior unchanged.
Logs
Discovered during day-0 manual validation of T5882212 (project 1462 / task 508562)
on Linux dev host (host: 2u1g-b650-0082, Ubuntu 24.10).

Two independent reproductions in the same session:
- Run 1 (host has stale OPENAI_API_KEY in env): 401 from inference path.
- Run 2 (host had OPENAI_API_KEY unset, --recreate-sandbox to rebuild from scratch): same 401.

Distinct from related bugs:
- 6115797 (Brev/Ubuntu24 cloud TUI Connection error — security guard blocking inference.local with "Blocked hostname or private/internal/special-use IP address"): this bug shows the routing reaches the upstream and gets a real HTTP 401 response, not a guard block. Different failure mode.
- 6107668 (TLS handshake corruption after `openshell provider update`): not triggered here (we did not run provider update; first onboard surfaces 401 immediately).
- 6081485 (HTTP 503 inference service unavailable on Jetson Orin): different status code, different upstream.

Suspected root cause:
The gateway's ollama-local provider config is created with `OPENAI_API_KEY=$value` as the bearer credential. Either (a) the gateway forwards this bearer to the actual ollama proxy at :11435 (which validates and rejects), or (b) the gateway mis-routes the call to api.openai.com (real OpenAI) — the symmetric 401 on direct api.openai.com vs inference.local with the same bearer hints at (b). Engineering should verify which of (a)/(b) is actually happening and fix the wizard to not capture OPENAI_API_KEY for Ollama at all.

Bug Details

Field Value
Priority Unprioritized
Action Dev - Open - To fix
Disposition Open issue
Module Machine Learning - NemoClaw
Keyword NemoClaw, NEMOCLAW_GH_SYNC_APPROVAL, NemoClaw_Inference, NemoClaw_Onboard

[NVB#6116337]

Activity

  1. changed the title [-][Linux][Inference] Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset[/-] [+][Linux]Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset[/+] on Apr 27, 2026
  2. latenighthackathon commented on Apr 28, 2026

    @latenighthackathon
    Collaborator

    Filed #2625 for what looks like the upgrade-sandboxes side of the same broken assumption you hit here — wizard caches OPENAI_API_KEY for local providers (this issue), and the rebuild preflight then demands it after a provider switch from remote to local (the new one). Both stem from local providers being treated as if they need remote credentials. Closing one direction does not close the other, but a canonical pattern (clear session.credentialEnv on provider switch + skip remote-credential capture in the wizard for local providers) would close both. Cheers!

  3. changed the title [-][Linux]Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset[/-] [+][Linux][Brev]Ollama-local provider returns HTTP 401 Unauthorized via inference.local — wizard caches OPENAI_API_KEY env var into credentials.json and not refreshable via env unset[/+] on Apr 30, 2026
  4. hulynn commented on Apr 30, 2026

    @hulynn
    ContributorAuthor

    [Repro] v0.0.31, no OPENAI_API_KEY, clean machine

    Reproduced on nemoclaw v0.0.31 (Brev cloud, Ubuntu 22.04, A100-SXM4-80GB, Ollama v0.22.0, model qwen2.5:0.5b).

    Trigger: No OPENAI_API_KEY set. Wizard shows API key: (not required for ollama-local) — nothing stored in credentials.json. Onboard completes, nemoclaw status reports Inference: healthy. TUI spins indefinitely, no response.

    Sandbox logs:

    [openshell_router] routing proxy inference request
      endpoint=http://host.openshell.internal:11435/v1 path=/v1/chat/completions
    [sandbox] NET:FAIL [LOW] inference.local:443   ← repeats every ~15s
    

    Host:

    $ curl http://localhost:11435/
    HTTP 401 Unauthorized
    

    The gateway routes to :11435 but injects no Authorization header. openshell provider get ollama-local shows NEMOCLAW_OLLAMA_PROXY_TOKEN as the credential key, but it is never injected as Bearer when routing.

    Same root cause as this bug — different trigger: no stale OPENAI_API_KEY needed, just a clean machine with no OPENAI_API_KEY at all.

  5. jyaunches commented on May 7, 2026

    @jyaunches
    Contributor

    Status update — closing against original 401 scope

    PR #2580 landed the wizard-side fix (merged 2026-04-28, released in v0.0.30+):

    • Wizard no longer captures host OPENAI_API_KEY for ollama-local / vllm-local — records credentialEnv = null and shows (not required for ollama-local).
    • Gateway config registers dedicated internal tokens (NEMOCLAW_OLLAMA_PROXY_TOKEN for ollama-local, NEMOCLAW_VLLM_LOCAL_TOKEN for vllm-local) instead of reading the host key.
    • Pre-fix OPENAI_API_KEY entries are pruned from ~/.nemoclaw/credentials.json on next onboard.

    That resolves the original HTTP 401 symptom described in this report (stale host OPENAI_API_KEY bleeding into the inference path).

    However, the underlying credential-injection path is still broken at runtime — per @hulynn's v0.0.31 reproduction on 2026-04-30: on a clean machine with no OPENAI_API_KEY at all, the gateway routes to :11435 but never injects NEMOCLAW_OLLAMA_PROXY_TOKEN as a Bearer header, so the call surfaces as a silent hang instead of a 401.

    That follow-on scope is tracked separately in a new issue so we don't re-open and re-title this one and so NV QA can carry a fresh bug ID against the v0.0.31 repro.

    → Continuation: #3198

    Closing this as the original 401 scope is fixed.

    Note: PR #2580's body used Refs #2519 rather than Fixes #2519, which is why this issue wasn't auto-closed on merge and why the "Development" field stayed empty. Team reminder to use Fixes # / Closes # going forward.

  6. added
    fixed-on-latestVerified by verify-stale skill: bug not reproducible on latest
    on May 7, 2026
  7. hulynn commented on May 11, 2026

    @hulynn
    ContributorAuthor

    Verified Fixed in v0.0.38 — 2026-05-09

    Host:     local-mercl@2u1g-b650-0197 (Ubuntu 26.04 LTS, no NVIDIA GPU — same class as original repro 2u1g-b650-0082)
    NemoClaw: v0.0.38
    OpenShell: 0.0.36
    OpenClaw:  2026.4.24 (cbcfdf6)
    Sandbox:   r3
    

    Steps

    1. export OPENAI_API_KEY=sk-test-invalid-cached     # exact bug premise: stale OpenAI key in host env
    2. nemoclaw onboard --recreate-sandbox
    3. Choose provider 7 (Install Ollama).
    4. Wizard installs Ollama, pulls qwen2.5:7b, then arrives at Review configuration.
    

    Result — wizard Review configuration

    Provider:      ollama-local
    Model:         qwen2.5:7b
    API key:       (not required for ollama-local)             ← was "OPENAI_API_KEY (stored in ~/.nemoclaw/credentials.json)"
    Web search:    disabled
    Messaging:     none
    Sandbox name:  r3
    

    Persistence + provider-credential evidence

    ~/.nemoclaw/credentials.json
      → "No such file or directory"               ← was: file containing OPENAI_API_KEY
    
    openshell provider get ollama-local -g nemoclaw
      Name:            ollama-local
      Type:            openai
      Credential keys: NEMOCLAW_OLLAMA_PROXY_TOKEN ← independent token, not OPENAI_API_KEY
      Config keys:     OPENAI_BASE_URL
    

    Root cause definitively addressed: wizard no longer captures host's OPENAI_API_KEY into the Ollama provider's bearer credential. A separate NEMOCLAW_OLLAMA_PROXY_TOKEN is generated for the local Ollama proxy at :11435, independent of any host env var. The HTTP 401 chain shown in the original bug (gateway forwarding stale OPENAI_API_KEY → upstream rejects) cannot occur because the bearer source has been decoupled.

    (End-to-end TUI inference smoke test not run on this verification because a deliberately bogus NEMOCLAW_PROXY_HOST=1.2.3.4 was still set in parallel for 6045052 verification and blocks outgoing connections inside the sandbox. The config-level evidence above is sufficient to confirm the root cause is fixed; the wizard's prior behavior of capturing OPENAI_API_KEY has been removed entirely.)

  8. added
    area: cliCommand line interface, flags, terminal UX, or output
    area: local-modelsLocal model providers, downloads, launch, or connectivity
    area: providersInference provider integrations and provider behavior
    and removed on Jun 3, 2026
  9. added theissue type on Jun 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

NV QABugs found by the NVIDIA QA Teamarea: cliCommand line interface, flags, terminal UX, or outputarea: local-modelsLocal model providers, downloads, launch, or connectivityarea: providersInference provider integrations and provider behaviorfixed-on-latestVerified by verify-stale skill: bug not reproducible on latestprovider: ollamaOllama local model provider behavior

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions