Skip to content

Commit b79b7e8

Browse files
paritoshd-nvclaude
andauthored
docs: add local Ollama inference setup section (#822)
* docs: add local Ollama inference setup section Includes changes done in PR #735. Fixes #314, #385 Signed-off-by: Paritosh Dixit <[email protected]> * docs: fix style and remove stale experimental gate Remove numbered section titles and colons from the Ollama setup section in spark-install.md per doc style guide. Remove the NEMOCLAW_EXPERIMENTAL=1 gate from inference-profiles.md — local providers are GA and the gate no longer exists in source. Co-authored-by: Claude Opus 4.6 <[email protected]> Signed-off-by: Paritosh Dixit <[email protected]> --------- Signed-off-by: Paritosh Dixit <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]>
1 parent a461606 commit b79b7e8

2 files changed

Lines changed: 134 additions & 51 deletions

File tree

‎docs/reference/inference-profiles.md‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -66,8 +66,7 @@ If validation fails, the wizard does not continue to sandbox creation.
6666

6767
## Local Providers
6868

69-
Local providers are available behind the `NEMOCLAW_EXPERIMENTAL=1` gate.
70-
These use the same routed `inference.local` pattern, but the upstream runtime is local to the host.
69+
Local providers use the same routed `inference.local` pattern, but the upstream runtime runs on the host rather than in the cloud.
7170

7271
- Local Ollama
7372
- Local NVIDIA NIM

‎spark-install.md‎

Lines changed: 133 additions & 49 deletions
Original file line numberDiff line numberDiff line change
@@ -2,19 +2,147 @@
22

33
> **WIP** — This page is actively being updated as we work through Spark installs. Expect changes.
44
5+
## Prerequisites
6+
7+
- **Docker** (pre-installed, v28.x)
8+
- **Node.js 22** (installed by the install.sh)
9+
- **OpenShell CLI** (installed via the Quick Start steps below)
10+
- **NVIDIA API Key** from [build.nvidia.com](https://build.nvidia.com) — prompted on first run
11+
512
## Quick Start
613

714
```bash
8-
# Clone and install
15+
# Install OpenShell:
16+
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
17+
18+
# Clone NemoClaw:
919
git clone https://github.com/NVIDIA/NemoClaw.git
1020
cd NemoClaw
11-
sudo npm install -g .
1221

13-
# Spark-specific setup (configures Docker for cgroup v2, then runs normal setup)
14-
nemoclaw setup-spark
22+
# Spark-specific setup (For details see [What's Different on Spark](#whats-different-on-spark))
23+
sudo ./scripts/setup-spark.sh
24+
25+
# Install NemoClaw using the NemoClaw/install.sh:
26+
./install.sh
27+
28+
# Alternatively, you can use the hosted install script:
29+
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
30+
```
31+
32+
## Verifying Your Install
33+
34+
```bash
35+
# Check sandbox is running
36+
nemoclaw my-assistant connect
37+
38+
# Inside the sandbox, talk to the agent:
39+
openclaw agent --agent main --local -m "hello" --session-id test
40+
```
41+
42+
## Uninstall (perform this before re-installing)
43+
44+
```bash
45+
# Uninstall NemoClaw (Remove OpenShell sandboxes, gateway, NemoClaw providers, related Docker containers, images, volumes and configs)
46+
nemoclaw uninstall
47+
```
48+
49+
## Setup Local Inference (Ollama)
50+
51+
Use this to run inference locally on the DGX Spark's GPU instead of routing to cloud.
52+
53+
### Verify the NVIDIA Container Runtime
54+
55+
```bash
56+
docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
1557
```
1658

17-
That's it. `setup-spark` handles everything below automatically.
59+
If this fails, configure the NVIDIA runtime and restart Docker:
60+
61+
```bash
62+
sudo nvidia-ctk runtime configure --runtime=docker
63+
sudo systemctl restart docker
64+
```
65+
66+
### Install Ollama
67+
68+
```bash
69+
curl -fsSL https://ollama.com/install.sh | sh
70+
```
71+
72+
Verify it is running:
73+
74+
```bash
75+
curl http://localhost:11434
76+
```
77+
78+
### Pull and Pre-load a Model
79+
80+
Download Nemotron 3 Super 120B (~87 GB; may take several minutes):
81+
82+
```bash
83+
ollama pull nemotron-3-super:120b
84+
```
85+
86+
Run it briefly to pre-load weights into unified memory, then exit:
87+
88+
```bash
89+
ollama run nemotron-3-super:120b
90+
# type /bye to exit
91+
```
92+
93+
### Configure Ollama to Listen on All Interfaces
94+
95+
By default Ollama binds to `127.0.0.1`, which is not reachable from inside the sandbox container. Configure it to listen on all interfaces:
96+
97+
> **Note:** `OLLAMA_HOST=0.0.0.0` exposes Ollama on your network. If you're not on a trusted LAN, restrict access with host firewall rules (`ufw`, `iptables`, etc.).
98+
99+
```bash
100+
sudo mkdir -p /etc/systemd/system/ollama.service.d
101+
printf '[Service]\nEnvironment="OLLAMA_HOST=0.0.0.0"\n' | sudo tee /etc/systemd/system/ollama.service.d/override.conf
102+
103+
sudo systemctl daemon-reload
104+
sudo systemctl restart ollama
105+
```
106+
107+
Verify Ollama is listening on all interfaces:
108+
109+
```bash
110+
sudo ss -tlnp | grep 11434
111+
```
112+
113+
### Install OpenShell and NemoClaw
114+
115+
```bash
116+
# If the OpenShell and NemoClaw are already installed, uninstall them. A fresh NemoClaw install will run onboard with local inference options.
117+
nemoclaw uninstall
118+
119+
# Install OpenShell and NemoClaw
120+
curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
121+
curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash
122+
```
123+
124+
When prompted for **Inference options**, select **Local Ollama**, then select the model you pulled.
125+
126+
### Connect and Test
127+
128+
```bash
129+
# Connect to the sandbox
130+
nemoclaw my-assistant connect
131+
```
132+
133+
Inside the sandbox, first verify `inference.local` is reachable directly (must use HTTPS — the proxy intercepts `CONNECT inference.local:443`):
134+
135+
```bash
136+
curl -sf https://inference.local/v1/models
137+
# Expected: JSON response listing the configured model
138+
# Exits non-zero on HTTP errors (403, 503, etc.) — failure here indicates a proxy routing regression
139+
```
140+
141+
Then talk to the agent:
142+
143+
```bash
144+
openclaw agent --agent main --local -m "Which model and GPU are in use?" --session-id test
145+
```
18146

19147
## What's Different on Spark
20148

@@ -42,28 +170,6 @@ Failed to start ContainerManager: failed to initialize top level QOS containers
42170

43171
**Fix**: `setup-spark` sets `"default-cgroupns-mode": "host"` in `/etc/docker/daemon.json` and restarts Docker. This makes all containers use the host cgroup namespace, which is what k3s needs.
44172

45-
## Prerequisites
46-
47-
These should already be on your Spark:
48-
49-
- **Docker** (pre-installed, v28.x)
50-
- **Node.js 22** — if not installed:
51-
52-
```bash
53-
curl -fsSL https://deb.nodesource.com/setup_22.x | sudo -E bash -
54-
sudo apt-get install -y nodejs
55-
```
56-
57-
- **OpenShell CLI**:
58-
59-
```bash
60-
ARCH=$(uname -m) # aarch64 on Spark
61-
curl -fsSL "https://github.com/NVIDIA/OpenShell/releases/latest/download/openshell-linux-${ARCH}" -o /usr/local/bin/openshell
62-
chmod +x /usr/local/bin/openshell
63-
```
64-
65-
- **NVIDIA API Key** from [build.nvidia.com](https://build.nvidia.com) — prompted on first run
66-
67173
## Manual Setup (if setup-spark doesn't work)
68174

69175
### Fix Docker cgroup namespace
@@ -93,12 +199,6 @@ sudo usermod -aG docker $USER
93199
newgrp docker # or log out and back in
94200
```
95201

96-
### Then run the onboard wizard
97-
98-
```bash
99-
nemoclaw onboard
100-
```
101-
102202
## Known Issues
103203

104204
| Issue | Status | Workaround |
@@ -109,22 +209,6 @@ nemoclaw onboard
109209
| Image pull failure (k3s can't find built image) | OpenShell bug | `openshell gateway destroy && openshell gateway start`, re-run setup |
110210
| GPU passthrough | Untested on Spark | Should work with `--gpu` flag if NVIDIA Container Toolkit is configured |
111211

112-
## Verifying Your Install
113-
114-
```bash
115-
# Check sandbox is running
116-
openshell sandbox list
117-
# Should show: nemoclaw Ready
118-
119-
# Test the agent
120-
openshell sandbox connect nemoclaw
121-
# Inside sandbox:
122-
nemoclaw-start openclaw agent --agent main --local -m 'hello' --session-id test
123-
124-
# Monitor network egress (separate terminal)
125-
openshell term
126-
```
127-
128212
## Architecture Notes
129213

130214
```text

0 commit comments

Comments
 (0)