Skip to content

Commit 1f615e2

Browse files
authored
fix(onboard): auto-patch cluster image for Docker 26+ overlayfs nested mount break (#2487)
## Summary Docker 26+ flips the install-time default storage driver from the legacy `overlay2` graph driver to the new containerd image-store with `overlayfs` snapshotter. The k3s server inside the upstream OpenShell cluster image relies on the host overlayfs supporting nested overlay mounts, which the new snapshotter does not. Users hit a fatal k3s loop with `"overlayfs" snapshotter cannot be enabled... try using "fuse-overlayfs" or "native"` after a few minutes. NemoClaw now detects the offending Docker configuration during onboard and transparently builds a small drop-in cluster image locally that installs `fuse-overlayfs` and selects it as the k3s snapshotter. No published image, no host configuration changes, no sudo, no Docker restart. ## Related Issue Refs #2481. ## Changes - `src/lib/preflight.ts` — `parseDockerStorageDriver` and `parseDockerUsesContainerdSnapshotter` parsers; new `HostAssessment` fields `dockerStorageDriver`, `dockerUsesContainerdSnapshotter`, `hasNestedOverlayConflict`. Conflict scoped to Linux + docker runtime + `Driver: overlayfs` + DriverStatus mentions `io.containerd.snapshotter.v1`. macOS Docker Desktop, legacy `overlay2`, and containerd-disabled Linux installs are not flagged. - `src/lib/cluster-image-patch.ts` (new) — `ensurePatchedClusterImage` builds a deterministic `nemoclaw-cluster:<version>-<snapshotter>-<sha>` image via a multi-stage Dockerfile. The `fuse-overlayfs` path uses a `ubuntu:24.04` builder pinned by digest (`UBUNTU_BUILDER_DIGEST` constant) that `apt-get install`s the package and stages the binary plus its `libfuse3.so.3` runtime dep under `/export`; the final stage `COPY --from=bin-fetcher`s both into the cluster image and runs `ldconfig`. The `native` path skips the builder entirely and is just `FROM ${UPSTREAM}` + `CMD ["server", "--snapshotter=native"]`. Cache hit via `docker image inspect`. Three timeout knobs (`pullTimeoutMs` 10m, `buildTimeoutMs` 5m, `inspectTimeoutMs` 30s) protect onboard from a stuck daemon. Throws `ClusterImagePatchError` on failure with diagnostic context. - `src/lib/onboard.ts` — `applyOverlayfsAutoFix` consults `assessHost`, calls `ensurePatchedClusterImage` when the conflict is detected, and overrides `OPENSHELL_CLUSTER_IMAGE` for `openshell gateway start`. Two override env vars: `NEMOCLAW_DISABLE_OVERLAY_FIX=1` skips the auto-fix; `NEMOCLAW_OVERLAY_SNAPSHOTTER=native` switches to the k3s `native` snapshotter (more disk, no FUSE dependency). - `src/lib/preflight.test.ts` — 11 new test cases (parser fixtures from the bug report, conflict + non-conflict assessments, macOS false-positive guard). - `src/lib/cluster-image-patch.test.ts` (new) — 19 test cases covering Dockerfile shape (multi-stage builder, no `github.com` link, no apt/curl in final stage, native variant skips fuse-overlayfs, ubuntu builder digest-pinned), version extraction (registry-port, registry-port-plus-tag, trailing-colon edge cases), tag determinism, cache-hit/cache-miss/build-failure/pull-failure with injectable runners and a mock fs surface. - `test/e2e/test-overlayfs-autofix.sh` (new) — temporary nightly E2E. Forces the GitHub-Actions runner's Docker onto containerd-snapshotter overlayfs via `daemon.json`, asserts the auto-fix triggers + the gateway runs on the patched image + the cluster log is clean (positive phase); calls `ensurePatchedClusterImage` directly to verify the cache-hit invariant (idempotency phase); runs install.sh with `NEMOCLAW_DISABLE_OVERLAY_FIX=1` under a bounded `timeout` and asserts non-zero exit (negative phase). Per-run `mktemp -d` STATE_DIR for daemon backup state. Removal criteria documented in the script header. - `.github/workflows/nightly-e2e.yaml` — adds the temporary `overlayfs-autofix-e2e` job and entry in `notify-on-failure: needs:`. Comments call out removal criteria. - `docs/reference/troubleshooting.md` — new section explaining root cause, auto-fix, override knobs, and the manual `daemon.json` alternative. Reflowed to one sentence per source line. - `docs/get-started/prerequisites.md` — one-line note about transparent storage-driver handling. One sentence per source line. - `.agents/skills/nemoclaw-user-*` — auto-regenerated from docs by the existing pre-commit hook. ## Type of Change - [x] Code change with doc updates ## Verification - [x] `npx prek run --all-files` passes (with `SKIP=test-cli` to skip pre-existing flaky CLI vitest tests on this dev machine — same 6 tests fail on stock `main` HEAD verified by stashing locally; behavior covered by CI on a faster runner) - [x] `npm run typecheck:cli` passes - [x] `npm run build:cli` passes - [x] `cd nemoclaw && npx tsc --noEmit --incremental` passes - [x] `npx tsc -p jsconfig.json` passes - [x] New tests added: 16 cases in `preflight.test.ts`, 22 cases in `cluster-image-patch.test.ts` — all pass - [x] No secrets or credentials committed - [x] Docs updated for user-facing behavior change - [x] End-to-end validated on a real Docker 26+ host via the new nightly `overlayfs-autofix-e2e` job (run [24966948867](https://github.com/NVIDIA/NemoClaw/actions/runs/24966948867)): **15 PASS / 0 FAIL / 1 SKIP**. The SKIP is documented (canonical k3s startup error string is absent on this runner kernel because the same root cause surfaces at sandbox-build time instead — non-zero install.sh exit still proves the auto-fix is load-bearing). ## Notes for reviewers ### Architectural choices - **No published wrapper image.** The patched image is built *locally* on the user's machine via `docker build`. No GHCR namespace, no signing pipeline, no SBOM, no security-boundary expansion to maintain. - **Multi-stage Dockerfile, builder via apt rather than upstream URL.** The cluster image's noble base ships BusyBox `tar` (so `apt-get install` directly into it fails with a `dpkg-deb` extraction error) AND lacks `curl` (so an in-place static-binary download via `RUN curl …` exits 127). Both attempts were tried during iteration and replaced. The current shape uses a clean `ubuntu:24.04` builder stage (digest-pinned for reproducibility) that runs `apt-get install fuse-overlayfs`, then `COPY --from`s the binary plus its sole non-libc shared dependency (`libfuse3.so.3`) into the cluster image. - **`native` snapshotter path is intentionally minimal.** `buildPatchDockerfile("native")` returns a four-line Dockerfile (`FROM ${UPSTREAM}` + `CMD`) — no builder, no apt, no fuse-overlayfs install. Selecting `NEMOCLAW_OVERLAY_SNAPSHOTTER=native` should cost the user nothing they don't need. - **Three timeouts on the build path.** `pullTimeoutMs` (10 min), `buildTimeoutMs` (5 min), and `inspectTimeoutMs` (30 s) protect the entire patched-image flow — including the `docker image inspect` cache probe — from a stuck Docker daemon. - **Hermetic env in the e2e.** Phase-3 (positive) and phase-4 (idempotency) `env` invocations explicitly `env -u NEMOCLAW_DISABLE_OVERLAY_FIX -u NEMOCLAW_OVERLAY_SNAPSHOTTER` so a caller's environment can't silently flip the path under test. - **Per-run `mktemp -d` STATE_DIR for daemon backup.** The previous `/tmp/nemoclaw-e2e-daemon.json.{bak,absent}` paths were predictable enough that a pre-created symlink could redirect the script's `sudo cp` of `/etc/docker/daemon.json` into an attacker-chosen file on a shared runner. Both files now live inside a 0700 mktemp directory, wiped by the EXIT trap. ### Adversarial review findings (resolved in `e0d8df15`) Spawned a fresh-context adversarial reviewer to look for what CodeRabbit and I missed. Eight concerns surfaced; all addressed. | # | Severity | Concern | Status | |---|---|---|---| | 1 | Major | `getGatewayStartEnv()` is called from both `startGatewayWithOptions` and `recoverGatewayRuntime`. The retry-storm impact was overstated by the reviewer (the retry closure captures `gatewayEnv` from outer scope; build runs once per call, not once per retry), but the recovery path does redundantly re-run `assessHost`. | **Resolved.** `applyOverlayfsAutoFix` now memoizes per upstream image; the recovery path returns the cached patched-tag without re-running assessment or attempting another build. | | 2 | Major | `computePatchedTag` SHAs the *Dockerfile text*, decoupled from the upstream image's actual content digest. If GHCR re-pushes `ghcr.io/nvidia/openshell/cluster:<v>` with a security fix, `imageExists` reports cache-hit and the user keeps running the patched image with stale upstream layers. | **Resolved.** Cache key now includes the resolved upstream content digest. `ensurePatchedClusterImage` calls `docker image inspect` first (zero network cost when warm; works air-gapped with pre-staged images) and folds the digest into the SHA. A re-pushed upstream produces a different digest → different patched tag → next onboard rebuilds. New unit test asserts digest changes invalidate the tag. | | 3 | Major | First-time onboard on an air-gapped/restricted host hits a 10 min `docker pull` timeout + 5 min `docker build` timeout before any feedback. | **Resolved.** When the upstream image isn't already local, a `docker manifest inspect` reachability probe (bounded by `inspectTimeoutMs`, default 30s) runs before committing to the long pull. Failure surfaces as a typed error with a pointer to the manual `daemon.json` workaround in troubleshooting docs. New unit test: "fails fast with a documented error when upstream is unreachable on a cache miss". | | 4 | Minor | `assessment.isWsl` is computed and ignored. Native Docker inside WSL2 (no Desktop integration) would currently trigger the auto-fix, but the WSL kernel may not actually have the nested-overlay break. | **Resolved.** `hasNestedOverlayConflict` now requires `!isWsl`. Conservative — leave WSL alone until we have a confirmed reproducer. New unit test: "does not flag a WSL2 Linux host as a conflict". | | 5 | Minor | `applyOverlayfsAutoFix` silently swallows `assessHost` exceptions — no log breadcrumb if a future regression makes the auto-fix mysteriously stop firing. | **Resolved.** Added `console.warn` with the error message before returning null. | | 6 | Minor | Phase-5 negative e2e test accepts "any non-zero install.sh exit" as a pass; flakes (NVIDIA_API_KEY rejection, GHCR rate-limit, etc.) look identical to "auto-fix is load-bearing". | **Resolved.** Negative phase now grep's the cluster log + install log for at least one of three nested-overlay-failure signatures (`overlayfs snapshotter cannot be enabled`, `CreateDiff: Canceled`, `failed to mount overlay`). FAIL if none present — previously an unrelated install failure would have papered over as a SKIP. | | 7 | Minor | Manual daemon.json workaround in docs warns about killing containers but omits that the image graph rebuilds on the new storage driver — previously-cached images become unusable. | **Resolved.** Added a sentence explaining the cold-cache cost of switching drivers. | | 8 | Nit | `parseDockerStorageDriver` only matches the JSON form; future plain-text callers would silently miss the conflict. | **Resolved.** Added a fallback regex for the plain-text `Storage Driver: <name>` form. New unit test for the plain-text fixture. | The reviewer also explicitly verified — and could not break — multi-arch correctness, registry-port edge cases, tag determinism, `runCapture` timeout threading, e2e symlink-redirect protection, and the `set -e` discipline. Those guards hold. ### Strategic context - The auto-fix is a downstream bridge until OpenShell's roadmap migration off k3s ([NVIDIA/OpenShell#873](NVIDIA/OpenShell#873)) lands. OpenShell explicitly closed [#756](NVIDIA/OpenShell#756) (the same nested-overlay class of bug on OrbStack) wontfix in favor of that migration. - The temporary nightly job (`overlayfs-autofix-e2e`) and the `cluster-image-patch.ts` module should be removed in the same PR that retires this code path once OpenShell ships a non-k3s default sandbox driver. Removal criteria are documented in both the script header and the workflow job comment. ### What the nightly e2e revealed and how it shaped the final shape Iteration history (commits visible in the branch log): | Attempt | Result on the GitHub Actions runner | Lesson | |---|---|---| | `RUN apt-get install fuse-overlayfs` directly into the cluster image | `dpkg-deb` failed: BusyBox `tar` doesn't accept `--warning=no-timestamp` | Cluster image's noble base lacks GNU tar | | `RUN curl -fsSL …upstream-release… && chmod +x …` | exit 127 | Cluster image lacks `curl` too | | Multi-stage `FROM alpine:3.20` + `apk add curl` + `curl …upstream-release…` | Hit "No External Project Links" policy violation (CONTRIBUTING.md) | GitHub URLs in source are policy-prohibited | | Multi-stage `FROM ubuntu:24.04` + `apt-get install fuse-overlayfs` + COPY binary + libfuse3.so.3 | **Working.** | Final shape. | Each iteration was caught by the temporary nightly e2e before reviewer time was spent on it. The same e2e also caught a phase-4 design flaw where re-running `install.sh` was racing the runner kernel's nested-overlay limits — restructured to call `ensurePatchedClusterImage` directly from Node instead. ### Standing items reviewers should know - **Pre-existing local flakes.** `test-cli` (Vitest CLI project) has 6 timeout-prone tests in `test/onboard.test.ts` and one `legacy-path-guard` test that fail on stock `main` HEAD on this dev machine. I verified by stashing locally inside the same fresh clone and re-running. They were skipped via `SKIP=test-cli` and via `--no-verify` on the local commits/push. CI runs them on a faster runner without issue. - **GitHub-runner kernel is not the user's reproducer.** The runner doesn't exhibit the original bug at k3s init — it surfaces later at sandbox-image build (`Docker build stream error: failed to export layer: CreateDiff: Canceled`). The negative phase therefore reports SKIP on the canonical-error-string check (the install.sh non-zero exit still proves the auto-fix is load-bearing). The original reporter's exact `docker info` (`Driver: overlayfs` + `io.containerd.snapshotter.v1` on Ubuntu 24.04 with kernel 6.17) is more representative of real users. - **CodeRabbit feedback.** Four review passes, all addressed; replies posted inline on each thread. Final standing items are pinned in the doc + tests so future regressions trip CI (e.g. `Dockerfile.toContain('FROM ubuntu:24.04@sha256:')`, `does not contain "github.com"`, `RUN apt-get / RUN curl forbidden in final stage`). <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Onboarding detects Docker 26+ nested-overlay conflicts and automatically builds/reuses a local patched cluster image (defaults to fuse-overlayfs); env vars allow opting out or selecting snapshotter. On failure, onboarding falls back to the upstream image with explanatory logs. * **Documentation** * Added prerequisites and troubleshooting guidance describing detection, the automatic workaround, override env knobs, and a manual daemon.json remediation. * **Tests** * Added unit tests, an E2E validation script, and a nightly E2E job covering auto-fix behavior and idempotency. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Signed-off-by: Aaron Erickson <[email protected]> --------- Signed-off-by: Aaron Erickson <[email protected]>
1 parent f439189 commit 1f615e2

11 files changed

Lines changed: 1716 additions & 1 deletion

File tree

‎.agents/skills/nemoclaw-user-get-started/references/prerequisites.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,12 @@ For NemoClaw-managed environments, use `nemoclaw onboard` when you need to creat
2727
Avoid `openshell self-update`, `npm update -g openshell`, `openshell gateway start --recreate`, or `openshell sandbox create` directly unless you intend to manage OpenShell separately and then rerun `nemoclaw onboard`.
2828
:::
2929

30+
:::{note} Docker storage driver
31+
On Linux hosts running Docker 26 or later with the [containerd image store](https://docs.docker.com/engine/storage/containerd/) enabled (the install-time default for fresh `docker-ce` installations on Ubuntu 24.04 and similar distros), `nemoclaw onboard` transparently builds a `fuse-overlayfs`-enabled cluster image to bypass a kernel-level nested-overlay limitation in k3s.
32+
No manual setup is required.
33+
See the troubleshooting guide (use the `nemoclaw-user-reference` skill) for the override knobs and a manual `daemon.json` alternative.
34+
:::
35+
3036
## Platforms
3137

3238
The following table lists tested platform and runtime combinations.

‎.agents/skills/nemoclaw-user-reference/references/troubleshooting.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,50 @@ $ nemoclaw onboard
203203
Podman is not a tested runtime.
204204
If onboarding or sandbox lifecycle fails, switch to a tested runtime (Docker Desktop, Colima, or Docker Engine) and rerun onboarding.
205205

206+
### Cluster fails with `overlayfs snapshotter cannot be enabled` on Docker 26+
207+
208+
Docker Engine 26 and later default fresh installations to the [containerd image store](https://docs.docker.com/engine/storage/containerd/), which exposes its layers via the `overlayfs` snapshotter rather than the legacy `overlay2` graph driver.
209+
The k3s server inside the OpenShell cluster image needs to mount its own overlay filesystem on top, and the kernel rejects nesting two non-trivial overlay mounts.
210+
The cluster container then loops with:
211+
212+
```text
213+
"overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd",
214+
try using "fuse-overlayfs" or "native":
215+
failed to mount overlay: ... err: invalid argument
216+
```
217+
218+
This is a Docker default-driver change, not a NemoClaw or OpenShell regression.
219+
The same hardware running Docker 25 or earlier — or any Docker version with the containerd image store disabled — uses the legacy `overlay2` driver and is unaffected.
220+
221+
NemoClaw detects the Docker 26+ containerd-snapshotter overlayfs configuration during onboarding and transparently builds a small drop-in replacement for the cluster image on the local Docker engine.
222+
The patched image installs `fuse-overlayfs` and selects it as the k3s snapshotter, bypassing the kernel-level nested-overlay limitation.
223+
No host configuration changes, sudo, or Docker restart required.
224+
225+
The auto-fix runs once per OpenShell version on the affected host.
226+
Subsequent onboarding runs reuse the cached patched image.
227+
Hosts without the conflict (`Driver: overlay2` in `docker info`, macOS Docker Desktop, or Linux installations that disable the containerd image store) see no change in behavior.
228+
229+
Override knobs:
230+
231+
- `NEMOCLAW_DISABLE_OVERLAY_FIX=1` — skip the auto-fix and run against the unmodified upstream cluster image.
232+
Useful for diagnosis or when you have already applied the manual workaround below.
233+
- `NEMOCLAW_OVERLAY_SNAPSHOTTER=native` — build the patched image with k3s's `native` snapshotter instead of `fuse-overlayfs`.
234+
The `native` snapshotter copies image layers instead of overlaying them, so it uses more disk but does not depend on FUSE.
235+
Default is `fuse-overlayfs`.
236+
237+
If you prefer to disable the new Docker storage driver instead of running the patched image, edit `/etc/docker/daemon.json`:
238+
239+
```json
240+
{
241+
"storage-driver": "overlay2",
242+
"features": { "containerd-snapshotter": false }
243+
}
244+
```
245+
246+
Then restart Docker (`sudo systemctl restart docker`) and re-run `nemoclaw onboard`.
247+
This restores the legacy `overlay2` driver host-wide, which kills any other running containers — prefer the auto-fix unless you need the change for unrelated reasons.
248+
Switching storage drivers also rebuilds the entire local image graph: previously-pulled images become unusable and Docker re-pulls them on first reference, so expect a cold cache and additional disk usage right after the restart.
249+
206250
### OpenShell version above maximum
207251

208252
Each NemoClaw release validates against a range of tested OpenShell versions.

‎.github/workflows/nightly-e2e.yaml‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -511,6 +511,39 @@ jobs:
511511
path: /tmp/nemoclaw-e2e-install.log
512512
if-no-files-found: ignore
513513

514+
# ── Docker 26+ overlayfs nested-mount auto-fix (#2481) ──────
515+
# TEMPORARY: validates the auto-fix in src/lib/cluster-image-patch.ts.
516+
# Remove this job — and the matching notify-on-failure entry — in the
517+
# same PR that deletes cluster-image-patch.ts when the OpenShell
518+
# roadmap migration off k3s (NVIDIA/OpenShell#873) lands.
519+
overlayfs-autofix-e2e:
520+
if: github.repository == 'NVIDIA/NemoClaw'
521+
runs-on: ubuntu-latest
522+
timeout-minutes: 45
523+
steps:
524+
- name: Checkout
525+
uses: actions/checkout@v6
526+
527+
- name: Run overlayfs auto-fix E2E test
528+
env:
529+
NVIDIA_API_KEY: ${{ secrets.NVIDIA_API_KEY }}
530+
NEMOCLAW_NON_INTERACTIVE: "1"
531+
NEMOCLAW_ACCEPT_THIRD_PARTY_SOFTWARE: "1"
532+
NEMOCLAW_SANDBOX_NAME: "e2e-overlayfs"
533+
GITHUB_TOKEN: ${{ github.token }}
534+
run: bash test/e2e/test-overlayfs-autofix.sh
535+
536+
- name: Upload onboard logs on failure
537+
if: failure()
538+
uses: actions/upload-artifact@v4
539+
with:
540+
name: overlayfs-autofix-logs
541+
path: |
542+
/tmp/nemoclaw-e2e-install.log
543+
/tmp/nemoclaw-e2e-onboard-positive.log
544+
/tmp/nemoclaw-e2e-onboard-negative.log
545+
if-no-files-found: ignore
546+
514547
# ── GPU E2E (Ollama local inference) ──────────────────────────
515548
# Enable by setting repository variable GPU_E2E_ENABLED=true
516549
# (Settings → Secrets and variables → Actions → Variables)
@@ -581,6 +614,7 @@ jobs:
581614
rebuild-openclaw-e2e,
582615
upgrade-stale-sandbox-e2e,
583616
rebuild-hermes-e2e,
617+
overlayfs-autofix-e2e,
584618
gpu-e2e,
585619
]
586620
if: ${{ always() && (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled')) }}

‎docs/get-started/prerequisites.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,12 @@ For NemoClaw-managed environments, use `nemoclaw onboard` when you need to creat
4747
Avoid `openshell self-update`, `npm update -g openshell`, `openshell gateway start --recreate`, or `openshell sandbox create` directly unless you intend to manage OpenShell separately and then rerun `nemoclaw onboard`.
4848
:::
4949

50+
:::{note} Docker storage driver
51+
On Linux hosts running Docker 26 or later with the [containerd image store](https://docs.docker.com/engine/storage/containerd/) enabled (the install-time default for fresh `docker-ce` installations on Ubuntu 24.04 and similar distros), `nemoclaw onboard` transparently builds a `fuse-overlayfs`-enabled cluster image to bypass a kernel-level nested-overlay limitation in k3s.
52+
No manual setup is required.
53+
See the [troubleshooting guide](../reference/troubleshooting.md) for the override knobs and a manual `daemon.json` alternative.
54+
:::
55+
5056
## Platforms
5157

5258
The following table lists tested platform and runtime combinations.

‎docs/reference/troubleshooting.md‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,50 @@ $ nemoclaw onboard
229229
Podman is not a tested runtime.
230230
If onboarding or sandbox lifecycle fails, switch to a tested runtime (Docker Desktop, Colima, or Docker Engine) and rerun onboarding.
231231

232+
### Cluster fails with `overlayfs snapshotter cannot be enabled` on Docker 26+
233+
234+
Docker Engine 26 and later default fresh installations to the [containerd image store](https://docs.docker.com/engine/storage/containerd/), which exposes its layers via the `overlayfs` snapshotter rather than the legacy `overlay2` graph driver.
235+
The k3s server inside the OpenShell cluster image needs to mount its own overlay filesystem on top, and the kernel rejects nesting two non-trivial overlay mounts.
236+
The cluster container then loops with:
237+
238+
```text
239+
"overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd",
240+
try using "fuse-overlayfs" or "native":
241+
failed to mount overlay: ... err: invalid argument
242+
```
243+
244+
This is a Docker default-driver change, not a NemoClaw or OpenShell regression.
245+
The same hardware running Docker 25 or earlier — or any Docker version with the containerd image store disabled — uses the legacy `overlay2` driver and is unaffected.
246+
247+
NemoClaw detects the Docker 26+ containerd-snapshotter overlayfs configuration during onboarding and transparently builds a small drop-in replacement for the cluster image on the local Docker engine.
248+
The patched image installs `fuse-overlayfs` and selects it as the k3s snapshotter, bypassing the kernel-level nested-overlay limitation.
249+
No host configuration changes, sudo, or Docker restart required.
250+
251+
The auto-fix runs once per OpenShell version on the affected host.
252+
Subsequent onboarding runs reuse the cached patched image.
253+
Hosts without the conflict (`Driver: overlay2` in `docker info`, macOS Docker Desktop, or Linux installations that disable the containerd image store) see no change in behavior.
254+
255+
Override knobs:
256+
257+
- `NEMOCLAW_DISABLE_OVERLAY_FIX=1` — skip the auto-fix and run against the unmodified upstream cluster image.
258+
Useful for diagnosis or when you have already applied the manual workaround below.
259+
- `NEMOCLAW_OVERLAY_SNAPSHOTTER=native` — build the patched image with k3s's `native` snapshotter instead of `fuse-overlayfs`.
260+
The `native` snapshotter copies image layers instead of overlaying them, so it uses more disk but does not depend on FUSE.
261+
Default is `fuse-overlayfs`.
262+
263+
If you prefer to disable the new Docker storage driver instead of running the patched image, edit `/etc/docker/daemon.json`:
264+
265+
```json
266+
{
267+
"storage-driver": "overlay2",
268+
"features": { "containerd-snapshotter": false }
269+
}
270+
```
271+
272+
Then restart Docker (`sudo systemctl restart docker`) and re-run `nemoclaw onboard`.
273+
This restores the legacy `overlay2` driver host-wide, which kills any other running containers — prefer the auto-fix unless you need the change for unrelated reasons.
274+
Switching storage drivers also rebuilds the entire local image graph: previously-pulled images become unusable and Docker re-pulls them on first reference, so expect a cold cache and additional disk usage right after the restart.
275+
232276
### OpenShell version above maximum
233277

234278
Each NemoClaw release validates against a range of tested OpenShell versions.

0 commit comments

Comments
 (0)