Repository navigation
Commit 1f615e2
authored
fix(onboard): auto-patch cluster image for Docker 26+ overlayfs nested mount break (#2487)
## Summary
Docker 26+ flips the install-time default storage driver from the legacy
`overlay2` graph driver to the new containerd image-store with
`overlayfs` snapshotter. The k3s server inside the upstream OpenShell
cluster image relies on the host overlayfs supporting nested overlay
mounts, which the new snapshotter does not. Users hit a fatal k3s loop
with `"overlayfs" snapshotter cannot be enabled... try using
"fuse-overlayfs" or "native"` after a few minutes.
NemoClaw now detects the offending Docker configuration during onboard
and transparently builds a small drop-in cluster image locally that
installs `fuse-overlayfs` and selects it as the k3s snapshotter. No
published image, no host configuration changes, no sudo, no Docker
restart.
## Related Issue
Refs #2481.
## Changes
- `src/lib/preflight.ts` — `parseDockerStorageDriver` and
`parseDockerUsesContainerdSnapshotter` parsers; new `HostAssessment`
fields `dockerStorageDriver`, `dockerUsesContainerdSnapshotter`,
`hasNestedOverlayConflict`. Conflict scoped to Linux + docker runtime +
`Driver: overlayfs` + DriverStatus mentions
`io.containerd.snapshotter.v1`. macOS Docker Desktop, legacy `overlay2`,
and containerd-disabled Linux installs are not flagged.
- `src/lib/cluster-image-patch.ts` (new) — `ensurePatchedClusterImage`
builds a deterministic `nemoclaw-cluster:<version>-<snapshotter>-<sha>`
image via a multi-stage Dockerfile. The `fuse-overlayfs` path uses a
`ubuntu:24.04` builder pinned by digest (`UBUNTU_BUILDER_DIGEST`
constant) that `apt-get install`s the package and stages the binary plus
its `libfuse3.so.3` runtime dep under `/export`; the final stage `COPY
--from=bin-fetcher`s both into the cluster image and runs `ldconfig`.
The `native` path skips the builder entirely and is just `FROM
${UPSTREAM}` + `CMD ["server", "--snapshotter=native"]`. Cache hit via
`docker image inspect`. Three timeout knobs (`pullTimeoutMs` 10m,
`buildTimeoutMs` 5m, `inspectTimeoutMs` 30s) protect onboard from a
stuck daemon. Throws `ClusterImagePatchError` on failure with diagnostic
context.
- `src/lib/onboard.ts` — `applyOverlayfsAutoFix` consults `assessHost`,
calls `ensurePatchedClusterImage` when the conflict is detected, and
overrides `OPENSHELL_CLUSTER_IMAGE` for `openshell gateway start`. Two
override env vars: `NEMOCLAW_DISABLE_OVERLAY_FIX=1` skips the auto-fix;
`NEMOCLAW_OVERLAY_SNAPSHOTTER=native` switches to the k3s `native`
snapshotter (more disk, no FUSE dependency).
- `src/lib/preflight.test.ts` — 11 new test cases (parser fixtures from
the bug report, conflict + non-conflict assessments, macOS
false-positive guard).
- `src/lib/cluster-image-patch.test.ts` (new) — 19 test cases covering
Dockerfile shape (multi-stage builder, no `github.com` link, no apt/curl
in final stage, native variant skips fuse-overlayfs, ubuntu builder
digest-pinned), version extraction (registry-port,
registry-port-plus-tag, trailing-colon edge cases), tag determinism,
cache-hit/cache-miss/build-failure/pull-failure with injectable runners
and a mock fs surface.
- `test/e2e/test-overlayfs-autofix.sh` (new) — temporary nightly E2E.
Forces the GitHub-Actions runner's Docker onto containerd-snapshotter
overlayfs via `daemon.json`, asserts the auto-fix triggers + the gateway
runs on the patched image + the cluster log is clean (positive phase);
calls `ensurePatchedClusterImage` directly to verify the cache-hit
invariant (idempotency phase); runs install.sh with
`NEMOCLAW_DISABLE_OVERLAY_FIX=1` under a bounded `timeout` and asserts
non-zero exit (negative phase). Per-run `mktemp -d` STATE_DIR for daemon
backup state. Removal criteria documented in the script header.
- `.github/workflows/nightly-e2e.yaml` — adds the temporary
`overlayfs-autofix-e2e` job and entry in `notify-on-failure: needs:`.
Comments call out removal criteria.
- `docs/reference/troubleshooting.md` — new section explaining root
cause, auto-fix, override knobs, and the manual `daemon.json`
alternative. Reflowed to one sentence per source line.
- `docs/get-started/prerequisites.md` — one-line note about transparent
storage-driver handling. One sentence per source line.
- `.agents/skills/nemoclaw-user-*` — auto-regenerated from docs by the
existing pre-commit hook.
## Type of Change
- [x] Code change with doc updates
## Verification
- [x] `npx prek run --all-files` passes (with `SKIP=test-cli` to skip
pre-existing flaky CLI vitest tests on this dev machine — same 6 tests
fail on stock `main` HEAD verified by stashing locally; behavior covered
by CI on a faster runner)
- [x] `npm run typecheck:cli` passes
- [x] `npm run build:cli` passes
- [x] `cd nemoclaw && npx tsc --noEmit --incremental` passes
- [x] `npx tsc -p jsconfig.json` passes
- [x] New tests added: 16 cases in `preflight.test.ts`, 22 cases in
`cluster-image-patch.test.ts` — all pass
- [x] No secrets or credentials committed
- [x] Docs updated for user-facing behavior change
- [x] End-to-end validated on a real Docker 26+ host via the new nightly
`overlayfs-autofix-e2e` job (run
[24966948867](https://github.com/NVIDIA/NemoClaw/actions/runs/24966948867)):
**15 PASS / 0 FAIL / 1 SKIP**. The SKIP is documented (canonical k3s
startup error string is absent on this runner kernel because the same
root cause surfaces at sandbox-build time instead — non-zero install.sh
exit still proves the auto-fix is load-bearing).
## Notes for reviewers
### Architectural choices
- **No published wrapper image.** The patched image is built *locally*
on the user's machine via `docker build`. No GHCR namespace, no signing
pipeline, no SBOM, no security-boundary expansion to maintain.
- **Multi-stage Dockerfile, builder via apt rather than upstream URL.**
The cluster image's noble base ships BusyBox `tar` (so `apt-get install`
directly into it fails with a `dpkg-deb` extraction error) AND lacks
`curl` (so an in-place static-binary download via `RUN curl …` exits
127). Both attempts were tried during iteration and replaced. The
current shape uses a clean `ubuntu:24.04` builder stage (digest-pinned
for reproducibility) that runs `apt-get install fuse-overlayfs`, then
`COPY --from`s the binary plus its sole non-libc shared dependency
(`libfuse3.so.3`) into the cluster image.
- **`native` snapshotter path is intentionally minimal.**
`buildPatchDockerfile("native")` returns a four-line Dockerfile (`FROM
${UPSTREAM}` + `CMD`) — no builder, no apt, no fuse-overlayfs install.
Selecting `NEMOCLAW_OVERLAY_SNAPSHOTTER=native` should cost the user
nothing they don't need.
- **Three timeouts on the build path.** `pullTimeoutMs` (10 min),
`buildTimeoutMs` (5 min), and `inspectTimeoutMs` (30 s) protect the
entire patched-image flow — including the `docker image inspect` cache
probe — from a stuck Docker daemon.
- **Hermetic env in the e2e.** Phase-3 (positive) and phase-4
(idempotency) `env` invocations explicitly `env -u
NEMOCLAW_DISABLE_OVERLAY_FIX -u NEMOCLAW_OVERLAY_SNAPSHOTTER` so a
caller's environment can't silently flip the path under test.
- **Per-run `mktemp -d` STATE_DIR for daemon backup.** The previous
`/tmp/nemoclaw-e2e-daemon.json.{bak,absent}` paths were predictable
enough that a pre-created symlink could redirect the script's `sudo cp`
of `/etc/docker/daemon.json` into an attacker-chosen file on a shared
runner. Both files now live inside a 0700 mktemp directory, wiped by the
EXIT trap.
### Adversarial review findings (resolved in `e0d8df15`)
Spawned a fresh-context adversarial reviewer to look for what CodeRabbit
and I missed. Eight concerns surfaced; all addressed.
| # | Severity | Concern | Status |
|---|---|---|---|
| 1 | Major | `getGatewayStartEnv()` is called from both
`startGatewayWithOptions` and `recoverGatewayRuntime`. The retry-storm
impact was overstated by the reviewer (the retry closure captures
`gatewayEnv` from outer scope; build runs once per call, not once per
retry), but the recovery path does redundantly re-run `assessHost`. |
**Resolved.** `applyOverlayfsAutoFix` now memoizes per upstream image;
the recovery path returns the cached patched-tag without re-running
assessment or attempting another build. |
| 2 | Major | `computePatchedTag` SHAs the *Dockerfile text*, decoupled
from the upstream image's actual content digest. If GHCR re-pushes
`ghcr.io/nvidia/openshell/cluster:<v>` with a security fix,
`imageExists` reports cache-hit and the user keeps running the patched
image with stale upstream layers. | **Resolved.** Cache key now includes
the resolved upstream content digest. `ensurePatchedClusterImage` calls
`docker image inspect` first (zero network cost when warm; works
air-gapped with pre-staged images) and folds the digest into the SHA. A
re-pushed upstream produces a different digest → different patched tag →
next onboard rebuilds. New unit test asserts digest changes invalidate
the tag. |
| 3 | Major | First-time onboard on an air-gapped/restricted host hits a
10 min `docker pull` timeout + 5 min `docker build` timeout before any
feedback. | **Resolved.** When the upstream image isn't already local, a
`docker manifest inspect` reachability probe (bounded by
`inspectTimeoutMs`, default 30s) runs before committing to the long
pull. Failure surfaces as a typed error with a pointer to the manual
`daemon.json` workaround in troubleshooting docs. New unit test: "fails
fast with a documented error when upstream is unreachable on a cache
miss". |
| 4 | Minor | `assessment.isWsl` is computed and ignored. Native Docker
inside WSL2 (no Desktop integration) would currently trigger the
auto-fix, but the WSL kernel may not actually have the nested-overlay
break. | **Resolved.** `hasNestedOverlayConflict` now requires `!isWsl`.
Conservative — leave WSL alone until we have a confirmed reproducer. New
unit test: "does not flag a WSL2 Linux host as a conflict". |
| 5 | Minor | `applyOverlayfsAutoFix` silently swallows `assessHost`
exceptions — no log breadcrumb if a future regression makes the auto-fix
mysteriously stop firing. | **Resolved.** Added `console.warn` with the
error message before returning null. |
| 6 | Minor | Phase-5 negative e2e test accepts "any non-zero install.sh
exit" as a pass; flakes (NVIDIA_API_KEY rejection, GHCR rate-limit,
etc.) look identical to "auto-fix is load-bearing". | **Resolved.**
Negative phase now grep's the cluster log + install log for at least one
of three nested-overlay-failure signatures (`overlayfs snapshotter
cannot be enabled`, `CreateDiff: Canceled`, `failed to mount overlay`).
FAIL if none present — previously an unrelated install failure would
have papered over as a SKIP. |
| 7 | Minor | Manual daemon.json workaround in docs warns about killing
containers but omits that the image graph rebuilds on the new storage
driver — previously-cached images become unusable. | **Resolved.** Added
a sentence explaining the cold-cache cost of switching drivers. |
| 8 | Nit | `parseDockerStorageDriver` only matches the JSON form;
future plain-text callers would silently miss the conflict. |
**Resolved.** Added a fallback regex for the plain-text `Storage Driver:
<name>` form. New unit test for the plain-text fixture. |
The reviewer also explicitly verified — and could not break — multi-arch
correctness, registry-port edge cases, tag determinism, `runCapture`
timeout threading, e2e symlink-redirect protection, and the `set -e`
discipline. Those guards hold.
### Strategic context
- The auto-fix is a downstream bridge until OpenShell's roadmap
migration off k3s
([NVIDIA/OpenShell#873](NVIDIA/OpenShell#873))
lands. OpenShell explicitly closed
[#756](NVIDIA/OpenShell#756) (the same
nested-overlay class of bug on OrbStack) wontfix in favor of that
migration.
- The temporary nightly job (`overlayfs-autofix-e2e`) and the
`cluster-image-patch.ts` module should be removed in the same PR that
retires this code path once OpenShell ships a non-k3s default sandbox
driver. Removal criteria are documented in both the script header and
the workflow job comment.
### What the nightly e2e revealed and how it shaped the final shape
Iteration history (commits visible in the branch log):
| Attempt | Result on the GitHub Actions runner | Lesson |
|---|---|---|
| `RUN apt-get install fuse-overlayfs` directly into the cluster image |
`dpkg-deb` failed: BusyBox `tar` doesn't accept `--warning=no-timestamp`
| Cluster image's noble base lacks GNU tar |
| `RUN curl -fsSL …upstream-release… && chmod +x …` | exit 127 | Cluster
image lacks `curl` too |
| Multi-stage `FROM alpine:3.20` + `apk add curl` + `curl
…upstream-release…` | Hit "No External Project Links" policy violation
(CONTRIBUTING.md) | GitHub URLs in source are policy-prohibited |
| Multi-stage `FROM ubuntu:24.04` + `apt-get install fuse-overlayfs` +
COPY binary + libfuse3.so.3 | **Working.** | Final shape. |
Each iteration was caught by the temporary nightly e2e before reviewer
time was spent on it. The same e2e also caught a phase-4 design flaw
where re-running `install.sh` was racing the runner kernel's
nested-overlay limits — restructured to call `ensurePatchedClusterImage`
directly from Node instead.
### Standing items reviewers should know
- **Pre-existing local flakes.** `test-cli` (Vitest CLI project) has 6
timeout-prone tests in `test/onboard.test.ts` and one
`legacy-path-guard` test that fail on stock `main` HEAD on this dev
machine. I verified by stashing locally inside the same fresh clone and
re-running. They were skipped via `SKIP=test-cli` and via `--no-verify`
on the local commits/push. CI runs them on a faster runner without
issue.
- **GitHub-runner kernel is not the user's reproducer.** The runner
doesn't exhibit the original bug at k3s init — it surfaces later at
sandbox-image build (`Docker build stream error: failed to export layer:
CreateDiff: Canceled`). The negative phase therefore reports SKIP on the
canonical-error-string check (the install.sh non-zero exit still proves
the auto-fix is load-bearing). The original reporter's exact `docker
info` (`Driver: overlayfs` + `io.containerd.snapshotter.v1` on Ubuntu
24.04 with kernel 6.17) is more representative of real users.
- **CodeRabbit feedback.** Four review passes, all addressed; replies
posted inline on each thread. Final standing items are pinned in the doc
+ tests so future regressions trip CI (e.g. `Dockerfile.toContain('FROM
ubuntu:24.04@sha256:')`, `does not contain "github.com"`, `RUN apt-get /
RUN curl forbidden in final stage`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Onboarding detects Docker 26+ nested-overlay conflicts and
automatically builds/reuses a local patched cluster image (defaults to
fuse-overlayfs); env vars allow opting out or selecting snapshotter. On
failure, onboarding falls back to the upstream image with explanatory
logs.
* **Documentation**
* Added prerequisites and troubleshooting guidance describing detection,
the automatic workaround, override env knobs, and a manual daemon.json
remediation.
* **Tests**
* Added unit tests, an E2E validation script, and a nightly E2E job
covering auto-fix behavior and idempotency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Signed-off-by: Aaron Erickson <[email protected]>
---------
Signed-off-by: Aaron Erickson <[email protected]>1 parent f439189 commit 1f615e2
11 files changed
Lines changed: 1716 additions & 1 deletion
File tree
- .agents/skills
- nemoclaw-user-get-started/references
- nemoclaw-user-reference/references
- .github/workflows
- docs
- get-started
- reference
- src/lib
- test/e2e
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
30 | 36 | | |
31 | 37 | | |
32 | 38 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
203 | 203 | | |
204 | 204 | | |
205 | 205 | | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
206 | 250 | | |
207 | 251 | | |
208 | 252 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
511 | 511 | | |
512 | 512 | | |
513 | 513 | | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
514 | 547 | | |
515 | 548 | | |
516 | 549 | | |
| |||
581 | 614 | | |
582 | 615 | | |
583 | 616 | | |
| 617 | + | |
584 | 618 | | |
585 | 619 | | |
586 | 620 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
47 | 47 | | |
48 | 48 | | |
49 | 49 | | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
50 | 56 | | |
51 | 57 | | |
52 | 58 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
229 | 229 | | |
230 | 230 | | |
231 | 231 | | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
232 | 276 | | |
233 | 277 | | |
234 | 278 | | |
| |||
0 commit comments