Run containers as the image's USER and WORKDIR, with layer file owners kept by aramase · Pull Request #2392 · agent-substrate/substrate

Anish Ramasekar (aramase) · 2026-10-09T04:44:50Z

Containers ignored the image's USER and WORKDIR and ran as root in /, and atelet's unpack dropped the file owners image layers record. So an image that hands a directory to its user, such as jupyter's /home/jovyan, didn't work as built.

atelet applies each tar entry's uid/gid and keeps setuid, setgid and sticky bits. It adds CHOWN, FOWNER and FSETID to its DAC_OVERRIDE. The image cache layout moves to version 3.
The process runs as the image's USER, resolved as Docker does against the image's own /etc/passwd and /etc/group, and starts in its WORKDIR (default /).
The jupyter demo runs as jovyan and drops DAC_OVERRIDE and --allow-root.

The USER and WORKDIR commits are Davanum Srinivas (@dims)'s from #1918, kept as written. This PR supersedes #1918.

Fixes #2036

Breaking change

Please add the breaking-change label.

After an upgrade or rollback, atelet refuses the existing image cache. Delete /var/lib/ate/image-cache on each node once no actor runs on it, as atelet's error says. After an upgrade, a node reboot also works.
A custom atelet securityContext needs CHOWN, DAC_OVERRIDE, FOWNER and FSETID.
Images with a USER stop running as root, and images with a WORKDIR stop starting in /. A non-root USER is expected to have no effective capabilities, so securityContext.capabilities.add no longer helps it. This is checked under runc and in the gVisor source, not yet inside a gVisor or microVM sandbox.
gVisor snapshots of such images don't restore, because runsc restore checks each container's user and working directory. Recreate those templates. MicroVM snapshots resume as they were.
Files an earlier run wrote as root into a durable dir stay root-owned.

Tested: go test -race on atelet, imagecache, ocispec, tarutil, kata and the probe fixture, on macOS and on Linux as non-root, and imagecache with -count=3 -shuffle=on. The root-only imagecache and tarutil tests pass as root on a privileged Linux container. hack/verify-all.sh passes. The jupyter image serves /api under Docker as 1000:100 with every capability dropped. Davanum Srinivas (@dims) ran the jupyter demo and a non-root node image as their users on kind, on both runtimes, before #1904 merged; on microVM uid 1000 couldn't bind :80 then. A new imagefs e2e starts an actor from a probe image with USER nonroot:nonroot and WORKDIR /home/nonroot. It checks uid and gid 65532, no supplementary group besides 65532, no effective capabilities, the working directory, and that the user can create a file there. CI runs it on gVisor and on microVM. At this head nothing ran in a gVisor or microVM sandbox yet: not the jupyter demo and not the imagefs e2e.

Tests pass
Docs updated

…patch

Anish Ramasekar (aramase) · 2026-10-09T18:52:23Z

cc Benjamin Elder (@BenTheElder)

Benjamin Elder (BenTheElder)

mostly LGTM, needs a rebase, many lingering ateom refs to cleanup and the import path changes / vanity URL

Anish Ramasekar (aramase) · 2026-10-16T22:22:35Z

Benjamin Elder (@BenTheElder) this is ready for another review!

Benjamin Elder (BenTheElder) · 2026-10-16T21:51:38Z

+	// the target is a regular file it could read and write, which
+	// CAP_DAC_OVERRIDE grants (a regular file another user owns never keeps
+	// setuid or setgid in atelet, see restoreSetIDBits). Other targets
+	// (symlinks, fifos) are given to root before the link is made.


🤖 nit 🟢 – fifos hit the default case below and fail the unpack, so this should probably only mention symlinks?

Right, fifos fail the unpack as before, so a symlink is the only non-regular hardlink target; the comment now says that and the check tests the symlink mode directly.

Benjamin Elder (BenTheElder) · 2026-10-16T21:51:39Z

-	dirs := make([]string, 0, len(dirModes))
-	for name := range dirModes {
-		dirs = append(dirs, name)
+	// Restore the image's intended directory modes, mtimes and owners now that


🤖 nit 🟢 – This comment is 22 lines, and the ones above dirHdrs and restoreSetIDBits are close to that. Can we trim these to the ordering constraints (deepest first, chmod and mtime before chown) and drop the rest?

Trimmed all three to deepest first and chmod/mtime before chown, plus one clause that chown keeps setgid and sticky bits; the protected_hardlinks note moved to the hardlink case.

Benjamin Elder (BenTheElder) · 2026-10-16T21:51:39Z

+// with the path of each, without following symlinks.
+func dirPaths(root *os.Root, dir string) (map[dirInode]string, error) {
+	paths := map[dirInode]string{}
+	err := fs.WalkDir(root.FS(), dir, func(name string, d fs.DirEntry, err error) error {


🤖 nit 🟢 – root.FS() rejects any path that isn't valid UTF-8, so this walk fails when the directory, or one below it, has a Latin-1 name, which tar allows. Replacing such a directory, or the fallback walk in the restore pass, then fails the whole unpack where root.RemoveAll used to cope. Walking with root.Open and ReadDir directly would avoid it.

Good catch: dirPaths now walks with root.Open and ReadDir, and TestUnpackLayer_NonUTF8DirectoryNames covers a Latin-1 directory on the replace path and the restore fallback walk; it fails before and passes after.

Benjamin Elder (BenTheElder) · 2026-10-16T21:51:39Z

+// permission bits plus setuid, setgid and sticky, as containerd and moby apply
+// them. Without the sticky bit /tmp would be 0777 instead of 1777 in the
+// actor, letting any user delete another's files there.
+func tarMode(hdr *tar.Header) os.FileMode {


🤖 nit 🟢 – This is the same mask as tarutil.restoredMode. Maybe move that into owner.go next to Lchown, which already builds everywhere, and use it here?

Moved to tarutil.Mode and tarutil.ModeBits in owner.go; snapshot restore and image unpack both use them.

Benjamin Elder (BenTheElder) · 2026-10-17T02:19:27Z

Should atelet get FOWNER and FSETID? Without them, Debian-style setgid binaries such as unix_chkpwd, chage, crontab and wall lose the bit. restoreSetIDBits also logs a warning per file on every pull. Adding them would also remove the ordering workarounds. DAC_OVERRIDE already lets atelet rewrite any file, so they wouldn't widen much.

Benjamin Elder (BenTheElder) · 2026-10-17T02:20:32Z

though x-ref #2428

Anish Ramasekar (aramase) · 2026-10-17T17:56:28Z

Yes, added both to atelet in 77e2e6f. With them the unpack keeps the set-id bits (checked on unix_chkpwd, chage and expiry), and the two chowns that stood in for FOWNER are gone. The chmod after the chown stays, since chown clears setuid and setgid even for root.

Re #2428: it changes the worker cap sets, not atelet's, but both PRs edit the same atelet/worker paragraph in internal/imagecache/README.md and the imagecache.go package comment, so whichever lands second needs to reconcile that text.

@BenTheElder

…gent-substrate#2979) Follow-up to agent-substrate#2279. @BenTheElder's last review asked for these, and they were pushed to agent-substrate#2279 before it merged, but main got the earlier commit. - Layout version moves to 2. Layers in a version 1 pool may not have been hashed against their diffID, and eviction never retires a layer that images keep using, so atelet refuses the old pool. - `pullWith` logs a diffID mismatch at WARN with ref, digest, and diffID. The RPC error was its only trace. ## Breaking change After an upgrade or rollback, atelet refuses the existing image cache. Before 1.0 a minor upgrade is a reinstall, so the upgrade runbook does not change. agent-substrate#2392 also moves the layout version to 2. Whichever lands second takes 3. Written with Claude Code.

Benjamin Elder (BenTheElder)

Lot of commits, did a review pass and tested some new review assistance agents on this one ...

I think this is something we clearly need to do but we have to be careful and I have a few concerns.

Some of these code comments could do with humanizing (not a new problem ... plenty of my earlier PRs especially too).

For sequencing we'll want to add correct user quickly after this (#1918). Eitan Yarmush (@EItanya) Tim Hockin (@thockin) Dmitry Berkovich (@dberkov) this would've been nice to land pre-GA, IMHO this is in the security direction, but we also probably need an API addition to allow overriding the image user + workdir on top of both of these ...

atelet unpacked image layers as root and dropped each entry's owner and set-id and sticky bits. The unpack now applies both, and atelet gets CHOWN, FOWNER and FSETID to do so. The worker restores a directory's owner after its mode so gVisor keeps setgid. atelet refuses a version 2 cache (layout version 3), whose layers are root-owned. Signed-off-by: Anish Ramasekar <[email protected]>

rootfs and upper were 0700. On gVisor the container's / takes that mode, so a non-root process could not search its own root. 0755 is what containerd and every image root use; the 0700 actor directory above keeps other actors out. The micro-VM guest already sees 0755. Signed-off-by: Davanum Srinivas <[email protected]> Signed-off-by: Anish Ramasekar <[email protected]>

atelet writes the OCI spec before the worker mounts the rootfs, so it cannot read the image's /etc/passwd from a mount. Image.ReadFile reads a file from the cached layers top-down, honoring recorded whiteouts and opaque dirs. Capped at 1 MiB. Signed-off-by: Davanum Srinivas <[email protected]> Signed-off-by: Anish Ramasekar <[email protected]>

Resolve USER as Docker does, against the image's own /etc/passwd and /etc/group: uid:gid as is, a bare uid with its login group (gid 0 without an entry), names looked up or Run/Restore fails, group memberships as supplementary groups. No USER means root. atelet resolves it into the worker's ContainerSpec. runsc restore checks Process.User, so a gVisor snapshot taken before this of an image that sets USER needs re-taking. Signed-off-by: Davanum Srinivas <[email protected]> Signed-off-by: Anish Ramasekar <[email protected]>

Start the process in the image's WORKDIR instead of "/", and create the directory in the actor's upper when the image lacks it. atelet resolves it into the worker's ContainerSpec. runsc restore checks Process.Cwd as well as Process.User, so gVisor snapshots from before this need re-taking. Signed-off-by: Davanum Srinivas <[email protected]> Signed-off-by: Anish Ramasekar <[email protected]>

Substrate makes no compatibility promise before v1.0.0, so a note about one release's snapshots is noise in the upgrade procedure. The API guide describes how the image's USER and WORKDIR apply. Signed-off-by: Davanum Srinivas <[email protected]> Signed-off-by: Anish Ramasekar <[email protected]>

The actor images built from this repository write at the root of their filesystem and into the durable-dir volumes atelet creates as root. With the default base now nonroot and a container running as its image USER, they would start as uid 65532 and lose those writes, so the e2e demos break. These images stay on the root variant of the same distroless build. Signed-off-by: Davanum Srinivas <[email protected]> Signed-off-by: Anish Ramasekar <[email protected]>

Signed-off-by: Anish Ramasekar <[email protected]>

Tim Hockin (thockin)

I am OK with the state of this - ping me when you merge 1918 into it so I can look at the rest

The imagefs suite starts an actor from a probe built with USER nonroot:nonroot on the distroless nonroot base. It checks the uid, gid, supplementary groups, effective capabilities and working directory, and that the user can create a file in its WORKDIR. Signed-off-by: Anish Ramasekar <[email protected]>

Anish Ramasekar (aramase) · 2026-10-29T22:10:01Z

I am OK with the state of this - ping me when you merge 1918 into it so I can look at the rest

Tim Hockin (@thockin) #1918 is merged into this PR now. PTAL!

cc Davanum Srinivas (@dims)

Tim Hockin (thockin)

I tried but all I could find were inconsequential nits which were not worth sending comments on

Davanum Srinivas (dims) · 2026-10-29T22:28:17Z

I tried but all I could find were inconsequential nits which were not worth sending comments on

🙃 thanks!

…nd WORKDIR, with layer file owners kept (#3473) Cherry-pick of #2392 onto `release-1.0` (`git cherry-pick -x` of the squash commit), it applied cleanly with no changes. Containers run as the image's `USER` and start in its `WORKDIR`, and atelet keeps the file owners recorded in image layers. Fixes #2036 for 1.0. AI use: AI-assisted. Clean pick. This carries the same breaking changes as #2392: - After an upgrade or rollback, atelet refuses the existing image cache. Delete `/var/lib/ate/image-cache` on each node once no actor runs on it, as atelet's error says. After an upgrade, a node reboot also works. - A custom atelet securityContext needs `CHOWN`, `DAC_OVERRIDE`, `FOWNER` and `FSETID`. - Images with a `USER` stop running as root, and images with a `WORKDIR` stop starting in `/`. A non-root `USER` has no effective capabilities, so `securityContext.capabilities.add` no longer helps it. - gVisor snapshots of such images don't restore, because `runsc restore` checks each container's user and working directory. Recreate those templates. MicroVM snapshots resume as they were. - Files an earlier run wrote as root into a durable dir stay root-owned. - [x] Tests pass - [x] Docs updated Signed-off-by: Anish Ramasekar <[email protected]> Signed-off-by: Davanum Srinivas <[email protected]> Signed-off-by: Benjamin Elder <[email protected]> Co-authored-by: Anish Ramasekar <[email protected]> Co-authored-by: Davanum Srinivas <[email protected]>

Anish Ramasekar (aramase) marked this pull request as ready for review October 9, 2026 06:28

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 9, 2026

docs(fork): name agent-substrate/substrate#2392 as the exit of the layer-owner patch giantswarm/substrate#221

Merged

Quentin Bisson (QuentinBisson) added a commit to giantswarm/substrate that referenced this pull request Oct 9, 2026

docs(fork): name agent-substrate#2392 as the exit of the layer-owner …

b63ff86

…patch

heraldbot Bot mentioned this pull request Oct 12, 2026

sync: re-pin giantswarm onto v0.5.0-alpha2 giantswarm/substrate#346

Merged

1 of 4 tasks

Davanum Srinivas (dims) mentioned this pull request Oct 15, 2026

Honor a container image's USER and WORKDIR #1918

Closed

2 tasks done

Benjamin Elder (BenTheElder) added kind/bug Something isn't working / bugfixes area/node breaking-change PRs with breaking changes labels Oct 16, 2026

Benjamin Elder (BenTheElder) self-requested a review October 16, 2026 05:27

Benjamin Elder (BenTheElder) reviewed Oct 16, 2026

View reviewed changes

Comment thread demos/jupyter/jupyter-template.yaml.tmpl Resolved

Benjamin Elder (BenTheElder) reviewed Oct 16, 2026

View reviewed changes

Comment thread internal/imagecache/unpack.go Resolved

Comment thread internal/imagecache/unpack.go Resolved

Comment thread internal/imagecache/imagecache.go Resolved

Benjamin Elder (BenTheElder) reviewed Oct 16, 2026

View reviewed changes

Anish Ramasekar (aramase) force-pushed the aramase/i/imagecache_tar_ownership branch from a147a98 to 73db1c9 Compare October 16, 2026 17:32

Benjamin Elder (BenTheElder) reviewed Oct 16, 2026

View reviewed changes

Anish Ramasekar (aramase) force-pushed the aramase/i/imagecache_tar_ownership branch from 73db1c9 to aab7662 Compare October 17, 2026 00:18

Anish Ramasekar (aramase) requested a review from Benjamin Elder (BenTheElder) October 17, 2026 00:19

Benjamin Elder (BenTheElder) mentioned this pull request Oct 20, 2026

imagecache: Verify layer content against its diffID before it enters the pool #2279

Merged

igooch mentioned this pull request Oct 20, 2026

imagecache: Refuse a layout version 1 pool and log diffID mismatches #2979

Merged

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 20, 2026

docs(fork): record the compat-migrations theme decision in its ledger headers giantswarm/substrate#516

Merged

Anish Ramasekar (aramase) force-pushed the aramase/i/imagecache_tar_ownership branch from 77e2e6f to a43f209 Compare October 20, 2026 21:36

igooch mentioned this pull request Oct 21, 2026

atelet: move to a new image cache directory when the layout version changes #3075

Open

Benjamin Elder (BenTheElder) added the area/security Security related issue/pr label Oct 25, 2026

Benjamin Elder (BenTheElder) reviewed Oct 25, 2026

View reviewed changes

Anish Ramasekar (aramase) force-pushed the aramase/i/imagecache_tar_ownership branch from a43f209 to 41ee18f Compare October 26, 2026 20:47

Tim Hockin (thockin) self-assigned this Oct 29, 2026

Dmitry Berkovich (dberkov) assigned Benjamin Elder (BenTheElder) Oct 29, 2026

Anish Ramasekar (aramase) force-pushed the aramase/i/imagecache_tar_ownership branch from 4b6a50d to 97bd5d9 Compare October 29, 2026 18:47

Davanum Srinivas (dims) and others added 7 commits October 29, 2026 12:37

fix: drop root-only workarounds now that containers run as their USER

5a15646

Signed-off-by: Anish Ramasekar <[email protected]>

Tim Hockin (thockin) reviewed Oct 29, 2026

View reviewed changes

Comment thread internal/tarutil/owner.go Resolved

Anish Ramasekar (aramase) changed the title ~~Keep the file owners an image's layers record inside the actor~~ Run containers as the image's USER and WORKDIR, with layer file owners kept Oct 29, 2026

Tim Hockin (thockin) approved these changes Oct 29, 2026

View reviewed changes

Tim Hockin (thockin) added this pull request to the merge queue Oct 29, 2026

Merged via the queue into agent-substrate:main with commit 5d226a5 Oct 29, 2026
7 checks passed

Anish Ramasekar (aramase) deleted the aramase/i/imagecache_tar_ownership branch October 29, 2026 23:01

Benjamin Elder (BenTheElder) mentioned this pull request Oct 30, 2026

[release-1.0] Cherry-pick #2392: Run containers as the image's USER and WORKDIR, with layer file owners kept #3473

Merged

2 tasks done

Benjamin Elder (BenTheElder) mentioned this pull request Oct 30, 2026

[bug-bash] Track gVisor / microVM parity gaps for 1.0 #3226

Open

16 of 23 tasks

Haven Xia (HavenXia) mentioned this pull request Oct 30, 2026

CI: Add the upgrade E2E test for each release #2712

Merged

2 tasks done

Eric Curtin (ericcurtin) mentioned this pull request Nov 2, 2026

Non-root actor workloads can traverse the root filesystem #2430

Closed

2 tasks done

Greg Castle (destijl) mentioned this pull request Nov 3, 2026

Restrict actor capabilities.add and make mounts nodev #3580

Merged

Haven Xia (HavenXia) mentioned this pull request Nov 5, 2026

Extend CONTRIBUTING.md to discuss compatibility #3646

Merged

This was referenced Nov 10, 2026

Sync patched with upstream main oldsj/substrate#5

Closed

devenv-sample: Substrate 1.0 runs actors as the image user oldsj/mainloop#354

Merged

Conversation

Anish Ramasekar (aramase) commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Breaking change

Uh oh!

Anish Ramasekar (aramase) commented Oct 9, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Anish Ramasekar (aramase) commented Oct 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Benjamin Elder (BenTheElder) commented Oct 17, 2026

Uh oh!

Benjamin Elder (BenTheElder) commented Oct 17, 2026

Uh oh!

Anish Ramasekar (aramase) commented Oct 17, 2026

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Anish Ramasekar (aramase) commented Oct 29, 2026

Uh oh!

Tim Hockin (thockin) left a comment

Choose a reason for hiding this comment

Uh oh!

Davanum Srinivas (dims) commented Oct 29, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Anish Ramasekar (aramase) commented Oct 9, 2026 •

edited

Loading