Skip to content

Commit 929389f

Browse files
authored
fix(installer): enable stock DGX OS on Station GB300 (#7108)
## Summary - Enables the Station express recipe on physical Station GB300 systems running stock DGX OS `7.2.0`, `7.4.0`, or `7.5.0`. - Requires bounded `Station` and `GB300` DMI identifiers. Generic Ubuntu preparation also requires the exact NVIDIA GB300 PCI identity (`10de:31c2`) before any host mutation. - Accepts only a bounded, root-owned, non-writable `/etc/dgx-release` marker with a strict schema, well-formed OTA history, `DGX_OTA_PRETTY_NAME="DGX OS"`, and `DGX_PLATFORM="DGX Server for GALAXY-GB300"`. - Validates the factory GB300 driver, zero volatile ECC errors, local Docker/containerd/Buildx, `nvidia-ctk` CDI, and GPU visibility through both CDI and `--gpus all` before continuing. - Selects the local default Docker context before Station preparation and does not replace stock host packages, restart services, generate CDI configuration, or rewrite the Docker runtime. - On Station GB300 only, grants direct-GPU sandboxes the exact read-only BDF directory for each discovered NVIDIA display-class PCI device plus only existing CPU, memory, NUMA, and NVIDIA module-initialization paths. It does not grant `/sys`, the PCI parent subtree, or sysfs write access. - Keeps bare `P3830`, partial product matches, unknown DGX OS releases, NVIDIA BaseOS, unsafe markers, and other Station generations fail-closed. - Keeps DGX Station support status **Deferred** pending repeat clean-host qualification and CI coverage. ## Why Issue #7103 reports that the released express installer rejects the stock DGX OS image shipped on Station. The prior qualification also exposed the OpenShell sandbox CUDA failure tracked in NVIDIA/OpenShell#2343. NemoClaw can safely address that runtime behavior at its policy boundary while retaining released OpenShell `0.0.85`. This does **not** claim that NVIDIA/OpenShell#2343 is fixed upstream, promote Station support, or cover the dual-Station work in #7091. Fixes #7103. ## Physical Station evidence A clean-application-baseline run at `58c11e9d6` passed end to end on physical DGX OS `7.5.0` Station GB300: - zero containers, GPU processes, `8000`/`8080` listeners, installed NemoClaw/OpenShell commands, and failed systemd units before installation; - exact installer source SHA verified after installation; - released OpenShell `0.0.85` and local `nvidia/nemotron-3-ultra-550b-a55b` serving; - sandbox CUDA validation on GB300 with driver `595.71.05` and volatile ECC `0/0`; - Hermes one-shot and real TUI tool use with successful `write_file` and `read_file` operations. A separate physical A/B policy test established the least-privilege sysfs boundary encoded at `2db0ea0bc`: - no sysfs exception: sandbox CUDA initialization failed; - exact GPU BDF plus existing CPU, memory, NUMA, and NVIDIA module-init paths: sandbox CUDA initialization passed; - `/sys/fs/cgroup/cgroup.controllers` and `/sys/class/net/lo/address` remained denied; - neither `/sys` nor `/sys/bus/pci/devices` is added to the generated policy. The exact `2db0ea0bc` full clean-install acceptance then passed in 910 seconds: - resolved source SHA matched `2db0ea0bcc3ecce1406ffea497465070d6ec4b7f`; - the effective policy contained only GPU BDF `0009:06:00.0`, CPU, memory, NUMA, and NVIDIA module-init sysfs paths, with no broad `/sys` or PCI parent entry; - sandbox CUDA, local Nemotron Ultra, and Hermes Agent health passed; - Hermes used `write_file` and `read_file`, and the resulting two-line artifact was independently verified; - the exact GPU vendor path was readable while unrelated cgroup and loopback-network sysfs reads remained denied. After merging current `main`, the true review head is `1874ac2ae9356d24fb6c2f8c340b8d9a8f31ae0a`. Its full cold-application-baseline acceptance rerun is in progress; this body will be updated with that result before re-review is requested. ## Verification - merged Station/resume/policy regression set: **326 passed, 1 skipped** across 13 files; - focused installer and direct-GPU policy set: **191 passed, 1 skipped**; - post-split Station identity/host-preparation set: **75 passed**; - CLI and JS-config TypeScript checks: passed; - ShellCheck, shell syntax, Biome, repository checks, docs generation/sync, test-size budget, source-shape budget, test-conditional scanner, and diff checks: passed; - branch is based on current `main`. Physical validation here is intentionally single-Station and does not change the platform matrix from Deferred. Signed-off-by: Aaron Erickson <[email protected]> --------- Signed-off-by: Aaron Erickson <[email protected]>
1 parent 43a0080 commit 929389f

20 files changed

Lines changed: 1639 additions & 158 deletions

‎ci/platform-matrix.json‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{
22
"$comment": "Single source of truth for NemoClaw launch claims and platform support. Covers platforms, inference providers, supported agents, messaging integrations, and deployment paths. Scripts read this to generate README and docs tables. QA/CI update platform/provider rows; the engineering owner reviews other rows. Docs are derived.",
33
"version": "1.1",
4-
"updated": "2026-07-15",
4+
"updated": "2026-07-17",
55

66
"project_status": {
77
"stage": "alpha",
@@ -64,7 +64,7 @@
6464
"status": "deferred",
6565
"prd_priority": "P1",
6666
"ci_tested": false,
67-
"notes": "The PRD marks this platform as P1. Workstation form-factor with NVIDIA GPUs and the same Docker + NVIDIA Container Toolkit + CDI requirements as DGX Spark. The installer detects DGX Station and offers express install with the pinned `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` recipe, including an approximately 352 GB model download, without follow-up provider, model, policy, or sandbox-name choices. Pass `--station-deepseek` to use `deepseek-ai/DeepSeek-V4-Flash` for a Station demo while retaining the one-confirmation express flow. The flag requires an interactive terminal, and `/dev/tty` must be available when the installer runs through `curl | bash`. For headless setup, select `NEMOCLAW_PROVIDER=install-vllm` and `NEMOCLAW_VLLM_MODEL=deepseek-v4-flash` instead. Direct managed-vLLM onboarding still defaults to `deepseek-ai/DeepSeek-V4-Flash` when no model override is set. The full NemoClaw onboarding path, including the express recipe, has not been validated end-to-end on physical DGX Station hardware and remains `deferred` until that run is signed off."
67+
"notes": "The PRD marks this platform as P1. Workstation form-factor with NVIDIA GPUs and the same Docker + NVIDIA Container Toolkit + CDI requirements as DGX Spark. Station remains Deferred. On a Station GB300, the installer accepts either a clean generic Ubuntu 24.04 ARM64 image or stock DGX OS `7.2.0`, `7.4.0`, or `7.5.0` when a safe, root-owned `/etc/dgx-release` marker identifies `DGX Server for GALAXY-GB300`. The stock DGX OS path validates the factory GB300 driver and volatile ECC state, Docker and containerd, Buildx, `nvidia-ctk`, CDI, and GPU visibility through both CDI and `--gpus all` before it proceeds; it does not replace host packages or rewrite the runtime. Unknown DGX OS versions, NVIDIA BaseOS images, unsafe release markers, and other Station generations stop before host preparation. The installer offers express install with the pinned `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` recipe, including an approximately 352 GB model download, without follow-up provider, model, policy, or sandbox-name choices. On Station GB300 only, direct-GPU sandbox policies add the exact read-only BDF directory for each host-discovered NVIDIA display-class PCI device, plus only existing CPU, memory, NUMA topology, and NVIDIA module initialization paths; they do not add `/sys`, the PCI parent subtree, or sysfs write access. One physical DGX OS `7.5.0` GB300 validation completed with released OpenShell `0.0.85`, local Nemotron Ultra serving, sandbox `cuInit(0)`, and a Hermes write/read file-tool task. In that validation, reads of `/sys/fs/cgroup/cgroup.controllers` and `/sys/class/net/lo/address` remained denied. DGX Station remains Deferred pending repeat qualification from a clean host and CI coverage. Pass `--station-deepseek` to use `deepseek-ai/DeepSeek-V4-Flash` while retaining the one-confirmation express flow. The flag requires an interactive terminal, and `/dev/tty` must be available when the installer runs through `curl | bash`. For headless setup on a qualifying Station, select `NEMOCLAW_PROVIDER=install-vllm` and `NEMOCLAW_VLLM_MODEL=deepseek-v4-flash` instead. Direct managed-vLLM onboarding still defaults to `deepseek-ai/DeepSeek-V4-Flash` when no model override is set."
6868
},
6969
{
7070
"name": "NVIDIA RTX (consumer and Pro workstation GPUs)",
@@ -147,7 +147,7 @@
147147
"name": "Local vLLM (managed install/start)",
148148
"status": "caveated",
149149
"endpoint_type": "Local OpenAI-compatible",
150-
"notes": "Appears by default on DGX Spark and DGX Station. DGX Station remains deferred until its managed-vLLM onboarding path is validated end-to-end on physical hardware. Generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`. Host must have the NVIDIA Container Toolkit installed and a CDI spec present (`onboard` asserts CDI presence). NemoClaw pins runtime images to immutable digests. DGX Spark and DGX Station models without a model-specific runtime use the `linux/arm64` digest published under `nvcr.io/nvidia/vllm:26.05.post1-py3`; generic Linux NVIDIA GPU hosts use the matching `linux/arm64` or `linux/amd64` digest published under `nvcr.io/nvidia/vllm:26.03.post1-py3`. The DGX Station express installer selects `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` with a pinned Hugging Face revision and the multi-platform index digest published under `vllm/vllm-openai:v0.22.0`; that index resolves to the `linux/arm64` manifest on Station. Direct managed-vLLM profile defaults are listed in `src/lib/inference/vllm-models.ts`: DGX Spark uses `nvidia/Qwen3.6-35B-A3B-NVFP4`, DGX Station uses `deepseek-ai/DeepSeek-V4-Flash`, and Linux NVIDIA GPU uses `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8`. Image pulls from `nvcr.io` require NGC registry login (`docker login nvcr.io`); onboard prompts for the NGC API key when authentication is missing."
150+
"notes": "Appears by default on DGX Spark and qualifying DGX Station GB300 hosts. DGX Station remains Deferred. The Station path accepts a clean generic Ubuntu 24.04 ARM64 image or stock DGX OS `7.2.0`, `7.4.0`, or `7.5.0` after the release marker and factory GPU runtime pass validation. On Station GB300 only, direct-GPU sandbox policies add the exact read-only BDF directory for each host-discovered NVIDIA display-class PCI device, plus only existing CPU, memory, NUMA topology, and NVIDIA module initialization paths; they do not add `/sys`, the PCI parent subtree, or sysfs write access. One physical DGX OS `7.5.0` GB300 validation completed with released OpenShell `0.0.85`, local Nemotron Ultra serving, sandbox `cuInit(0)`, and a Hermes write/read file-tool task. In that validation, reads of `/sys/fs/cgroup/cgroup.controllers` and `/sys/class/net/lo/address` remained denied. DGX Station remains Deferred pending repeat qualification from a clean host and CI coverage. Generic Linux NVIDIA GPU hosts require `NEMOCLAW_EXPERIMENTAL=1` or `NEMOCLAW_PROVIDER=install-vllm`. Host must have the NVIDIA Container Toolkit installed and a CDI spec present (`onboard` asserts CDI presence). NemoClaw pins runtime images to immutable digests. DGX Spark and DGX Station models without a model-specific runtime use the `linux/arm64` digest published under `nvcr.io/nvidia/vllm:26.05.post1-py3`; generic Linux NVIDIA GPU hosts use the matching `linux/arm64` or `linux/amd64` digest published under `nvcr.io/nvidia/vllm:26.03.post1-py3`. The DGX Station express installer selects `nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-NVFP4` with a pinned Hugging Face revision and the multi-platform index digest published under `vllm/vllm-openai:v0.22.0`; that index resolves to the `linux/arm64` manifest on Station. Direct managed-vLLM profile defaults are listed in `src/lib/inference/vllm-models.ts`: DGX Spark uses `nvidia/Qwen3.6-35B-A3B-NVFP4`, DGX Station uses `deepseek-ai/DeepSeek-V4-Flash`, and Linux NVIDIA GPU uses `nvidia/NVIDIA-Nemotron-3-Nano-4B-FP8`. Image pulls from `nvcr.io` require NGC registry login (`docker login nvcr.io`); onboard prompts for the NGC API key when authentication is missing."
151151
}
152152
],
153153

‎docs/get-started/prerequisites.mdx‎

Lines changed: 21 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -41,10 +41,13 @@ If the group change is not active in the current shell, the installer exits with
4141
If you choose the native Linux Ollama install path, the onboard wizard also requires `zstd` for Ollama archive extraction.
4242
The installer also requires `strings` from `binutils` to verify the OpenShell binary before it continues with OpenShell install work.
4343

44-
On a DGX Station GB300 running the generic Ubuntu 24.04 ARM64 image, accepting express install prepares the host with NVIDIA open driver `610.43.02`, Docker CE `29.6.1` with Buildx, and NVIDIA Container Toolkit `1.19.1`.
45-
DGX OS, NVIDIA BaseOS images, and other Station generations are outside this automatic preparation boundary and stop before host preparation.
46-
On those systems, set `NEMOCLAW_PROVIDER` or `NEMOCLAW_NO_EXPRESS=1` explicitly to continue without Station host automation.
47-
The preparation probes package and runtime state first, reuses exact matches, and installs only missing pinned packages, including the NVIDIA Container Toolkit libraries and `nvidia-ctk` CLI.
44+
DGX Station GB300 express install accepts either the generic Ubuntu 24.04 ARM64 image or stock DGX OS `7.2.0`, `7.4.0`, or `7.5.0`.
45+
A stock DGX OS image qualifies only when a safe, root-owned `/etc/dgx-release` marker identifies `DGX Server for GALAXY-GB300`.
46+
An unknown version, malformed or unsafe marker, NVIDIA BaseOS image, or other Station generation stops before host preparation.
47+
On an unqualified system, set `NEMOCLAW_PROVIDER` or `NEMOCLAW_NO_EXPRESS=1` explicitly to continue without Station host automation.
48+
49+
On the generic Ubuntu path, accepting express install prepares the host with NVIDIA open driver `610.43.02`, Docker CE `29.6.1` with Buildx, and NVIDIA Container Toolkit `1.19.1`.
50+
Preparation probes package and runtime state first, reuses exact matches, and installs only missing pinned packages, including the NVIDIA Container Toolkit libraries and `nvidia-ctk` CLI.
4851
It permits only the reviewed factory transition from `dkms` `3.0.11-1ubuntu13` to `1:3.4.0-1ubuntu1`.
4952
After reboot, preparation enables NVIDIA's packaged CDI refresh path and service, requires the `nvidia.com/gpu=all` device, and verifies it with a real container launch.
5053
If the packaged refresh fails or does not produce that device, preparation prints service diagnostics and stops for administrator repair.
@@ -59,9 +62,22 @@ It does not install a host CUDA toolkit or Docker Compose.
5962
If any other existing prerequisite version differs, preparation stops instead of changing it automatically.
6063
After changing pinned packages, the installer exits with status `10`; reboot, sign in, and run the printed command, which pins the exact accepted NemoClaw commit before resuming express setup.
6164

65+
On a qualifying stock DGX OS image, express install validates the factory stack in place against the local default Docker daemon.
66+
It requires a loaded GB300 driver with zero volatile corrected and uncorrected ECC errors, active Docker and containerd services, working Docker Buildx, `nvidia-ctk`, the `nvidia.com/gpu=all` CDI device, and successful `nvidia-smi` device-visibility probes through both CDI and `--gpus all`.
67+
This path does not install or replace host packages, enable or restart services, generate CDI configuration, or rewrite the Docker runtime.
68+
The preparation helper writes a private audit log under `~/station-bootstrap-logs`.
69+
Aside from the installer's separately disclosed `docker`-group membership grant when the current trusted account needs Docker access, its only persistent stock-runtime effect is that the digest-pinned visibility-test image can remain in the Docker cache.
70+
One physical DGX OS `7.5.0` GB300 validation completed with released OpenShell `0.0.85`, local Nemotron Ultra serving, sandbox `cuInit(0)`, and a Hermes write/read file-tool task.
71+
Repeat qualification from a clean host and CI coverage remain pending.
72+
On Station GB300 direct GPU sandboxes, NemoClaw validates the vendor and display-class metadata for each NVIDIA PCI device and adds only that device's exact BDF directory as read-only.
73+
It also adds only the CPU, memory, NUMA topology, and NVIDIA module initialization paths that exist on the host.
74+
NemoClaw does not add `/sys`, the `/sys/bus/pci/devices` parent subtree, or any sysfs write grant.
75+
This scoped addition is not applied on WSL, Jetson, or other non-Station hosts.
76+
In the physical DGX OS `7.5.0` validation, the exact-BDF policy passed sandbox `cuInit(0)` while reads of `/sys/fs/cgroup/cgroup.controllers` and `/sys/class/net/lo/address` remained denied.
77+
6278
<Warning title="DGX Station Support Status">
6379
DGX Station remains Deferred.
64-
Full NemoClaw onboarding with this recipe has not completed end-to-end validation on physical DGX Station hardware.
80+
Repeat qualification from a clean host and CI coverage remain pending before any support-status promotion.
6581
</Warning>
6682

6783
<Warning title="Docker Group Access">

0 commit comments

Comments
 (0)