ocispec: mount tmpfs at /dev/shm, and /tmp on both runtimes by nybidari · Pull Request #3533 · agent-substrate/substrate

nybidari · 2026-11-01T16:02:41Z

Part of #3226.

gvisor and microvm previously mounted different filesystems at /dev/shm, and /tmp:

/dev/shm on gVisor was a 0755 directory on runsc's internal dev filesystem, blocking non-root shared memory writes.
/tmp was part of the host-backed rootfs upper layer on microVM and a conditional tmpfs on gVisor when the image's /tmp was empty.
Dropped the /run tmpfs from both runtimes, so /run comes from the image's rootfs, as on Kubernetes.

Move the standard OCI system mounts (/proc, /dev, /dev/pts, /dev/shm, /dev/mqueue, /sys) into ocispec.Build and add an explicit mode=1777 tmpfs mount at /tmp. If a container mounts a volume at a system mount destination such as /tmp, ocispec.Build replaces the default tmpfs entry there. ShapeMicroVM now rewrites bind mounts in place.

AI use:
AI-assisted, I reviewed and tested it.

Breaking change:

Old gVisor MEMORY snapshots, including goldens, don't restore. Recut golden snapshots after upgrading.
MicroVM MEMORY snapshots and goldens still resume, but keep the old mounts until they are recut.
MicroVM /tmp: now a tmpfs in guest memory instead of the host-backed rootfs.
- Its files count against the actor's memory limit.
- It is capped at half of the guest's memory.
- ROOTFS snapshots no longer keep it, and files older ROOTFS snapshots saved there are hidden.
- Files the image ships under /tmp are hidden.
gVisor /dev/shm: now a 64 MiB tmpfs (mode=1777) instead of a root-only directory. Non-root processes can now use it, but it is capped at 64 MiB.
gVisor /tmp: now always a tmpfs. runsc used to mount one only when the image's /tmp was empty, so files the image ships under /tmp are now hidden.

If an actor needs more space in /dev/shm or /tmp, or needs files there kept in ROOTFS snapshots, mount a durableDir volume at that path. It replaces the default tmpfs. Images that ship files under /tmp should create them at startup instead.

Tests pass
Docs updated

Signed-off-by: Nayana Bidari <[email protected]>

Benjamin Elder (BenTheElder)

Thanks.

At a high level: I traced back and /run got in here from ctr instead of any real standard and just slipped through from the very early uVM POCs. We should drop it from both.

Benjamin Elder (BenTheElder) · 2026-11-02T19:01:03Z

+		{Destination: "/dev/pts", Type: "devpts", Source: "devpts", Options: []string{"nosuid", "noexec", "newinstance", "ptmxmode=0666", "mode=0620", "gid=5"}},
+		{Destination: "/dev/shm", Type: "tmpfs", Source: "shm", Options: []string{"nosuid", "noexec", "nodev", "mode=1777", "size=65536k"}},
+		{Destination: "/sys", Type: "sysfs", Source: "sysfs", Options: []string{"nosuid", "noexec", "nodev", "ro"}},
+		{Destination: "/run", Type: "tmpfs", Source: "tmpfs", Options: []string{"nosuid", "strictatime", "mode=755", "size=65536k"}},


non-blocking: containerd's CRI plugin strips this /run mount from its default spec, so on Kubernetes the container sees the image's /run:
https://github.com/containerd/containerd/blob/b7a25261e029d68978bbf290df0a62fc819b3c51/internal/cri/server/container_create.go#L693
https://github.com/containerd/containerd/blob/b7a25261e029d68978bbf290df0a62fc819b3c51/pkg/oci/spec_opts.go#L1721-L1723

So gVisor already matched Kubernetes here. Should we drop /run from both instead? Otherwise an image that ships a /run/<app> (or /var/run/<app>) directory owned by its non-root USER works on Kubernetes and fails here, the tmpfs is root-owned 0755 so the process can't recreate it.

If we keep it, sandbox-classes.md should say /run hides the image's files too, it only mentions /tmp.

Dropped /run for both runtimes.

Benjamin Elder (BenTheElder) · 2026-11-02T19:01:03Z

+		{Destination: "/dev/shm", Type: "tmpfs", Source: "shm", Options: []string{"nosuid", "noexec", "nodev", "mode=1777", "size=65536k"}},
+		{Destination: "/sys", Type: "sysfs", Source: "sysfs", Options: []string{"nosuid", "noexec", "nodev", "ro"}},
+		{Destination: "/run", Type: "tmpfs", Source: "tmpfs", Options: []string{"nosuid", "strictatime", "mode=755", "size=65536k"}},
+		{Destination: "/tmp", Type: "tmpfs", Source: "tmpfs", Options: []string{"nosuid", "nodev", "mode=1777"}},


non-blocking: with no size= this is half of guest RAM on microVM, but on gVisor it's half of the node's memory, runsc sets the tmpfs default from the host's /proc/meminfo:
https://github.com/google/gvisor/blob/e8a2a28891ba41f3fb737dee8bb7120745d75826/pkg/sentry/fsimpl/tmpfs/tmpfs.go#L204-L218
https://github.com/google/gvisor/blob/e8a2a28891ba41f3fb737dee8bb7120745d75826/runsc/boot/loader.go#L856-L858

So a 512 MiB actor filling /tmp gets ENOSPC at ~256 MiB on microVM, and on gVisor it runs past its memory limit instead. Should we set size= from the actor's memory limit so both match? Otherwise the docs should say the cap differs, and that /tmp counts against the memory limit on both.

Done. ShapeGVisor now sets size= to half the memory limit, which matches the microVM guest default. The docs say the cap and that /tmp counts against the memory limit on both. A gVisor actor with no memory limit still gets runsc's default, and the docs call that out.

Benjamin Elder (BenTheElder) · 2026-11-02T19:01:03Z

 		return field.ErrorList{field.Invalid(fldPath, p, "must be a clean absolute Unix path: must start with '/', not be '/', and contain no ':', '..', '.', '//', trailing '/', or control characters")}
 	}
+	switch p {
+	case "/dev", "/dev/pts", "/proc", "/sys":


non-blocking: microVM fails container start on any bind under /proc (kata-agent only allows a few lxcfs files), and runc does the same:
https://github.com/kata-containers/kata-containers/blob/cf82bb35c80320178bf7570252fe75d6fb263209/src/agent/rustjail/src/mount.rs#L356-L361
https://github.com/opencontainers/runc/blob/7d36b05953ebb377900a0a565d8b0754e464be0b/libcontainer/rootfs_linux.go#L871-L895

Can we reject the whole /proc tree here, so a durableDir at /proc/foo fails at admission instead of when the actor starts? Probably worth the same for /sys/... and /dev/pts/....

Benjamin Elder (BenTheElder) · 2026-11-02T19:01:03Z

+	// Only the guest gets /dev/mqueue; see systemMounts. A volume there
+	// replaces it. kata mounts in order, so it goes right after /dev.
+	if !slices.ContainsFunc(spec.Mounts, func(m specs.Mount) bool { return m.Destination == "/dev/mqueue" }) {
+		dev := slices.IndexFunc(spec.Mounts, func(m specs.Mount) bool { return m.Destination == "/dev" })
+		spec.Mounts = slices.Insert(spec.Mounts, dev+1, specs.Mount{
+			Destination: "/dev/mqueue",
+			Type:        "mqueue",
+			Source:      "mqueue",
+			Options:     []string{"nosuid", "noexec", "nodev"},
+		})
+	}


non-blocking: can we put /dev/mqueue in systemMounts after /dev/pts and have ShapeGVisor drop it along with /dev and /dev/pts? runsc ignores the mqueue type anyway. Then Build's in-place replace covers both the order and a volume there, and this block and most of TestShapeMicroVM_Mqueue go away (this also inserts at index 0 if there's ever no /dev mount).

Makes sense, done.

Benjamin Elder (BenTheElder) · 2026-11-02T19:01:03Z

  // +k8s:required
  // +k8s:maxLength=4096
-  // +k8s:customValidation # clean-absolute-path shape; no regex/pattern tag exists
+  // +k8s:customValidation # clean absolute path, not /dev, /dev/pts, /proc or /sys


nit: the mount_path comment above didn't get the "must not be /dev, /dev/pts, /proc, or /sys" sentence the ateapi and atelet protos got.

Benjamin Elder (BenTheElder)

nothing blocking, docs fixup can be a tiny follow-up if we get this in before you see it.

need to get a +2 from another freeze approver, pinging

Benjamin Elder (BenTheElder) · 2026-11-03T01:24:10Z

@@ -64,8 +61,11 @@ See [Container Capabilities](api-guide.md#container-capabilities-securitycontext

 ## System Mounts (`/dev/shm`, `/run`, `/tmp`)


nit: /run isn't a system mount anymore, drop it from the heading?

Benjamin Elder (BenTheElder) · 2026-11-03T01:24:10Z

+// runsc replaces /dev and /dev/pts with its own mounts and ignores mqueue
+// mounts and the nodev and strictatime options, so ShapeGVisor drops them.
+// /tmp is capped at half the memory limit when tmpfs.
+func TestShapeGVisor_SystemMounts(t *testing.T) {


nit: the rows only differ in /tmp (and /dev/mqueue in the last one), can we check just the mounts each row is about, and keep one row with the full list? it's hard to spot what each case is testing.

Kubernetes shows the image's /run, since containerd's CRI plugin drops the /run tmpfs from its default spec. Drop it on both runtimes, so an image with a non-root /run directory works here too. runsc sizes a tmpfs from the node's memory, so cap /tmp at half the memory limit (at least 1 byte), as the microVM guest kernel does. Put /dev/mqueue in systemMounts and drop it on gVisor, which ignores it. Reject /dev and trees under /dev/pts, /proc, and /sys at admission, since kata-agent refuses a bind under /proc and runsc replaces /dev and /dev/pts. Signed-off-by: Nayana Bidari <[email protected]>

…bstrate#3533) Part of agent-substrate#3226. `gvisor` and `microvm` previously mounted different filesystems at `/dev/shm`, and `/tmp`: - `/dev/shm` on gVisor was a `0755` directory on `runsc`'s internal `dev` filesystem, blocking non-root shared memory writes. - `/tmp` was part of the host-backed rootfs upper layer on microVM and a conditional `tmpfs` on gVisor when the image's `/tmp` was empty. - Dropped the `/run` tmpfs from both runtimes, so `/run` comes from the image's rootfs, as on Kubernetes. Move the standard OCI system mounts (`/proc`, `/dev`, `/dev/pts`, `/dev/shm`, `/dev/mqueue`, `/sys`) into `ocispec.Build` and add an explicit `mode=1777` `tmpfs` mount at `/tmp`. If a container mounts a volume at a system mount destination such as `/tmp`, `ocispec.Build` replaces the default `tmpfs` entry there. `ShapeMicroVM` now rewrites `bind` mounts in place. AI use: AI-assisted, I reviewed and tested it. Breaking change: - Old gVisor MEMORY snapshots, including goldens, don't restore. Recut golden snapshots after upgrading. - MicroVM MEMORY snapshots and goldens still resume, but keep the old mounts until they are recut. - MicroVM `/tmp`: now a tmpfs in guest memory instead of the host-backed rootfs. - Its files count against the actor's memory limit. - It is capped at half of the guest's memory. - ROOTFS snapshots no longer keep it, and files older ROOTFS snapshots saved there are hidden. - Files the image ships under `/tmp` are hidden. - gVisor `/dev/shm`: now a 64 MiB tmpfs (`mode=1777`) instead of a root-only directory. Non-root processes can now use it, but it is capped at 64 MiB. - gVisor `/tmp`: now always a tmpfs. runsc used to mount one only when the image's `/tmp` was empty, so files the image ships under `/tmp` are now hidden. If an actor needs more space in `/dev/shm` or `/tmp`, or needs files there kept in ROOTFS snapshots, mount a `durableDir` volume at that path. It replaces the default tmpfs. Images that ship files under `/tmp` should create them at startup instead. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Nayana Bidari <[email protected]> (cherry picked from commit fd576d3) Signed-off-by: Eitan Yarmush <[email protected]>

Bring `release-1.0` up to `main` at 5a5b192 with the 19 changes merged since the previous batch in #3536. The final tree matches that commit exactly. **Please use Rebase and merge to preserve the individual commits and author attribution.** All 19 applied cleanly in upstream order. Original authors, author dates, co-author and sign-off trailers, and `-x` source references are retained. Upgrade notes: - #3533: old gVisor MEMORY snapshots, including goldens, no longer restore. Recreate them after upgrading. MicroVM MEMORY snapshots still resume with their old mounts until recreated. `/tmp` is now always tmpfs; ROOTFS snapshots no longer preserve its contents, and image-provided files there are hidden. MicroVM `/tmp` counts against actor memory and is capped at half the guest's memory. gVisor `/dev/shm` is now a 64 MiB tmpfs. Mount a `durableDir` at either path when persistence or more space is needed. `/run` now comes from the image rootfs. - #3447: remove `--actor-jwt-jwks-file` from custom credential-exchange sidecar configs. Set `--actor-jwt-issuer-ca-file` only when the issuer uses a private CA other than the service DNS CA. - #3450: do not reuse a deleted atespace name. See the updated atespace name-reuse guidance. Includes #3505, which overlaps the open backport in #3547. GKE provisioning and teardown now live in [substrate-gke](https://github.com/ai-on-gke/substrate-gke) rather than this repository. <details> <summary>Included PRs, in cherry-pick order</summary> 1. #3500: demos/{sandbox,jupyter,multi-template}: fix tutorial prerequisites, walkthroughs, and counter volume 2. #3498: demos/counter: inline gVisor walkthrough and fix setup commands 3. #3511: e2e tests: Extend the existing tests to cover tag actor. 4. #2775: Generate API reference documentation and publish it on Substrate website 5. #3119: Document how people, pods and CI jobs should authenticate to ate-api 6. #3505: tools: remove setup-gcp and hack/teardown.sh 7. #3147: e2e: resume fidelity after SandboxConfig version edits 8. #3370: ate-setup: reconcile ate-api-server ConfigMaps and roll out on config changes 9. #1607: hack: reject breaking proto changes across each proto package 10. #3415: docs: update and add security diagrams 11. #3485: Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk 12. #3447: Verify actor JWTs in the credential exchange through OIDC discovery 13. #3533: ocispec: mount tmpfs at /dev/shm, and /tmp on both runtimes 14. #3450: docs: Document that atespace names must not be reused 15. #3410: Document using more than one egress credential provider 16. #3486: atelet: register the device plugin on a new socket each time 17. #3553: ate-setup: wait for the demo namespace in delete demo 18. #3560: Fix multiple issues in claude-code-multiplex demo 19. #3517: docs: remove inaccurate early-stage architecture documentation </details> Validation: every intermediate tree matches its upstream source commit, all 19 authorship records and original attribution trailers are preserved, and `git diff --check` passes. The original 18 backport commits are unchanged. AI use: AI-assisted cherry-picking, attribution checks, and review. - [ ] Tests pass - [x] Docs updated --------- Signed-off-by: Mofi Rahman <[email protected]> Signed-off-by: Eitan Yarmush <[email protected]> Signed-off-by: Nayana Bidari <[email protected]> Signed-off-by: Julian Gutierrez Oschmann <[email protected]> Signed-off-by: Max Thompson <[email protected]> Signed-off-by: Steven Shriver <[email protected]> Signed-off-by: Grant McCloskey <[email protected]> Signed-off-by: Shrutiya <[email protected]> Signed-off-by: Haiyan Meng <[email protected]> Signed-off-by: Greg Castle <[email protected]> Signed-off-by: Benjamin Elder <[email protected]> Signed-off-by: Anish Ramasekar <[email protected]> Signed-off-by: Shruti Nair <[email protected]> Signed-off-by: Haven Xia <[email protected]> Co-authored-by: Mofi Rahman <[email protected]> Co-authored-by: nybidari <[email protected]> Co-authored-by: Julian Gutierrez Oschmann <[email protected]> Co-authored-by: Max Thompson <[email protected]> Co-authored-by: Grant McCloskey <[email protected]> Co-authored-by: Steven Shriver <[email protected]> Co-authored-by: shrutiyam-glitch <[email protected]> Co-authored-by: haiyanmeng <[email protected]> Co-authored-by: Greg Castle <[email protected]> Co-authored-by: Benjamin Elder <[email protected]> Co-authored-by: Anish Ramasekar <[email protected]> Co-authored-by: Shruti Nair <[email protected]> Co-authored-by: Haven Xia <[email protected]> Co-authored-by: Adam Wathieu <[email protected]>

Part of #3226. `gvisor` and `microvm` previously mounted different filesystems at `/dev/shm`, and `/tmp`: - `/dev/shm` on gVisor was a `0755` directory on `runsc`'s internal `dev` filesystem, blocking non-root shared memory writes. - `/tmp` was part of the host-backed rootfs upper layer on microVM and a conditional `tmpfs` on gVisor when the image's `/tmp` was empty. - Dropped the `/run` tmpfs from both runtimes, so `/run` comes from the image's rootfs, as on Kubernetes. Move the standard OCI system mounts (`/proc`, `/dev`, `/dev/pts`, `/dev/shm`, `/dev/mqueue`, `/sys`) into `ocispec.Build` and add an explicit `mode=1777` `tmpfs` mount at `/tmp`. If a container mounts a volume at a system mount destination such as `/tmp`, `ocispec.Build` replaces the default `tmpfs` entry there. `ShapeMicroVM` now rewrites `bind` mounts in place. AI use: AI-assisted, I reviewed and tested it. Breaking change: - Old gVisor MEMORY snapshots, including goldens, don't restore. Recut golden snapshots after upgrading. - MicroVM MEMORY snapshots and goldens still resume, but keep the old mounts until they are recut. - MicroVM `/tmp`: now a tmpfs in guest memory instead of the host-backed rootfs. - Its files count against the actor's memory limit. - It is capped at half of the guest's memory. - ROOTFS snapshots no longer keep it, and files older ROOTFS snapshots saved there are hidden. - Files the image ships under `/tmp` are hidden. - gVisor `/dev/shm`: now a 64 MiB tmpfs (`mode=1777`) instead of a root-only directory. Non-root processes can now use it, but it is capped at 64 MiB. - gVisor `/tmp`: now always a tmpfs. runsc used to mount one only when the image's `/tmp` was empty, so files the image ships under `/tmp` are now hidden. If an actor needs more space in `/dev/shm` or `/tmp`, or needs files there kept in ROOTFS snapshots, mount a `durableDir` volume at that path. It replaces the default tmpfs. Images that ship files under `/tmp` should create them at startup instead. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Nayana Bidari <[email protected]> (cherry picked from commit fd576d3) Signed-off-by: Eitan Yarmush <[email protected]>

@fidencio

…e#3580) Fixes [agent-substrate#3579](agent-substrate#3579) ### Summary Restricts `securityContext.capabilities.add` to a hardened allowlist of 12 Linux capabilities and blocks dangerous capabilities such as `MKNOD`, `SYS_ADMIN`, and `SYS_RAWIO` before 1.0. In microVM sandboxes, granting `MKNOD` to a container running as root allows it to create block device nodes matching the guest's `virtio-pmem` root disk and directly write to the guest filesystem. Other capabilities like `SYS_ADMIN` and `SYS_RAWIO` expose dangerous guest kernel attack surface. ### Changes * **Capability Allowlist:** `securityContext.capabilities.add` now only permits 12 capabilities required for package installs, non-root user switches, and standard container workflows: * `AUDIT_WRITE` * `CHOWN` * `DAC_OVERRIDE` * `FOWNER` * `FSETID` * `KILL` * `NET_BIND_SERVICE` * `SETFCAP` * `SETGID` * `SETPCAP` * `SETUID` * `SYS_CHROOT` * **Admission Enforcement:** `ateapi` and `atelet` reject actor template creations and updates specifying capabilities outside this allowlist. * **Drop Unrestricted:** `securityContext.capabilities.drop` continues to accept any capability name. * **MicroVM Hardening:** * Volume bind mounts are now mounted with `nodev`, and the microVM guest mounts the virtio-fs share with `nodev` (covering automounted submounts and rootfs). `ShapeGVisor` strips `nodev`, keeping gVisor specs identical and preserving checkpoint/restore compatibility. * MicroVM guest sysctl `net.ipv4.ping_group_range = "0 2147483647"` is configured so standard unprivileged `ping` continues to work without requiring `NET_RAW`. * **Documentation & Schemas:** Updated `api-guide.md`, `sandbox-classes.md`, and regenerated protobuf bindings (`ateapi.pb.go`, `atelet.pb.go`). ### Breaking change: gVisor `MEMORY` snapshots taken before this release won't restore gVisor now gets `nodev` on volume and system mounts (`/proc`, `/dev/shm`, `/sys`, `/tmp`), matching microVM. runsc applies it to the host bind mount. `runsc restore` rejects a checkpoint whose mounts differ from the new spec (`"Mounts" does not match across checkpoint restore`), so gVisor `MEMORY` snapshots taken before the upgrade can't be restored. rc.2 already breaks these snapshots with [agent-substrate#3533](agent-substrate#3533) (tmpfs `/dev/shm` and `/tmp`), so this adds no new upgrade step on top of that. MicroVM actors are unaffected. **What breaks:** - Suspended gVisor actors whose latest snapshot is `MEMORY` fidelity (the default) move to `ACTOR_STATE_CRASHED` on `ResumeActor`. - New actors from gVisor templates created before the upgrade crash on first resume, because the template's golden snapshot is an old `MEMORY` checkpoint. **Upgrade steps:** 1. **Before upgrading,** resume any suspended gVisor actor whose in-memory state you need, and save that state to durable storage or let its work finish. 2. **Re-create gVisor templates.** `ActorTemplate` is immutable, so create a new template (e.g. `my-template-v2`), or delete and re-create it, to build a new golden snapshot. Point clients at the new template. 3. **Replace crashed actors.** `RevertActor` won't help: it reverts to the same pre-upgrade checkpoint, which crashes again. `UpdateActor --template` won't either, because resume restores the actor's own snapshot. Delete the actor and create it again from the new template. Its in-memory state and its snapshot's rootfs and volume contents are lost. ### Testing * **Unit Tests:** Updated resource validation tests, parity tests, and microVM spec tests (`go test ./...` passes in container). * **Live GKE Testing (`sgke-rc1`):** * Verified template rejection when adding `MKNOD` or `SYS_ADMIN`. * Verified unprivileged ICMP `ping` works in microVM guests without `NET_RAW`. * Verified that pre-existing gVisor actor snapshots restore cleanly. * **e2e Probe Coverage:** * Verified inside the guest that `mknod` fails with `EPERM`. * Verified that unprivileged users can open `SOCK_DGRAM`/`IPPROTO_ICMP` sockets across both runtimes. Thanks to [Fabiano Fidêncio (@fidencio)](https://github.com/fidencio) for the report on MKNOD that led us to make this change. --------- Signed-off-by: Greg Castle <[email protected]> (cherry picked from commit 17fa858) Signed-off-by: dberkov <[email protected]>

@fidencio

Fixes [#3579](#3579) ### Summary Restricts `securityContext.capabilities.add` to a hardened allowlist of 12 Linux capabilities and blocks dangerous capabilities such as `MKNOD`, `SYS_ADMIN`, and `SYS_RAWIO` before 1.0. In microVM sandboxes, granting `MKNOD` to a container running as root allows it to create block device nodes matching the guest's `virtio-pmem` root disk and directly write to the guest filesystem. Other capabilities like `SYS_ADMIN` and `SYS_RAWIO` expose dangerous guest kernel attack surface. ### Changes * **Capability Allowlist:** `securityContext.capabilities.add` now only permits 12 capabilities required for package installs, non-root user switches, and standard container workflows: * `AUDIT_WRITE` * `CHOWN` * `DAC_OVERRIDE` * `FOWNER` * `FSETID` * `KILL` * `NET_BIND_SERVICE` * `SETFCAP` * `SETGID` * `SETPCAP` * `SETUID` * `SYS_CHROOT` * **Admission Enforcement:** `ateapi` and `atelet` reject actor template creations and updates specifying capabilities outside this allowlist. * **Drop Unrestricted:** `securityContext.capabilities.drop` continues to accept any capability name. * **MicroVM Hardening:** * Volume bind mounts are now mounted with `nodev`, and the microVM guest mounts the virtio-fs share with `nodev` (covering automounted submounts and rootfs). `ShapeGVisor` strips `nodev`, keeping gVisor specs identical and preserving checkpoint/restore compatibility. * MicroVM guest sysctl `net.ipv4.ping_group_range = "0 2147483647"` is configured so standard unprivileged `ping` continues to work without requiring `NET_RAW`. * **Documentation & Schemas:** Updated `api-guide.md`, `sandbox-classes.md`, and regenerated protobuf bindings (`ateapi.pb.go`, `atelet.pb.go`). ### Breaking change: gVisor `MEMORY` snapshots taken before this release won't restore gVisor now gets `nodev` on volume and system mounts (`/proc`, `/dev/shm`, `/sys`, `/tmp`), matching microVM. runsc applies it to the host bind mount. `runsc restore` rejects a checkpoint whose mounts differ from the new spec (`"Mounts" does not match across checkpoint restore`), so gVisor `MEMORY` snapshots taken before the upgrade can't be restored. rc.2 already breaks these snapshots with [#3533](#3533) (tmpfs `/dev/shm` and `/tmp`), so this adds no new upgrade step on top of that. MicroVM actors are unaffected. **What breaks:** - Suspended gVisor actors whose latest snapshot is `MEMORY` fidelity (the default) move to `ACTOR_STATE_CRASHED` on `ResumeActor`. - New actors from gVisor templates created before the upgrade crash on first resume, because the template's golden snapshot is an old `MEMORY` checkpoint. **Upgrade steps:** 1. **Before upgrading,** resume any suspended gVisor actor whose in-memory state you need, and save that state to durable storage or let its work finish. 2. **Re-create gVisor templates.** `ActorTemplate` is immutable, so create a new template (e.g. `my-template-v2`), or delete and re-create it, to build a new golden snapshot. Point clients at the new template. 3. **Replace crashed actors.** `RevertActor` won't help: it reverts to the same pre-upgrade checkpoint, which crashes again. `UpdateActor --template` won't either, because resume restores the actor's own snapshot. Delete the actor and create it again from the new template. Its in-memory state and its snapshot's rootfs and volume contents are lost. ### Testing * **Unit Tests:** Updated resource validation tests, parity tests, and microVM spec tests (`go test ./...` passes in container). * **Live GKE Testing (`sgke-rc1`):** * Verified template rejection when adding `MKNOD` or `SYS_ADMIN`. * Verified unprivileged ICMP `ping` works in microVM guests without `NET_RAW`. * Verified that pre-existing gVisor actor snapshots restore cleanly. * **e2e Probe Coverage:** * Verified inside the guest that `mknod` fails with `EPERM`. * Verified that unprivileged users can open `SOCK_DGRAM`/`IPPROTO_ICMP` sockets across both runtimes. Thanks to [Fabiano Fidêncio (@fidencio)](https://github.com/fidencio) for the report on MKNOD that led us to make this change. --------- Signed-off-by: Greg Castle <[email protected]> (cherry picked from commit 17fa858) Signed-off-by: dberkov <[email protected]>

@fidencio

…e#3580) Fixes [agent-substrate#3579](agent-substrate#3579) ### Summary Restricts `securityContext.capabilities.add` to a hardened allowlist of 12 Linux capabilities and blocks dangerous capabilities such as `MKNOD`, `SYS_ADMIN`, and `SYS_RAWIO` before 1.0. In microVM sandboxes, granting `MKNOD` to a container running as root allows it to create block device nodes matching the guest's `virtio-pmem` root disk and directly write to the guest filesystem. Other capabilities like `SYS_ADMIN` and `SYS_RAWIO` expose dangerous guest kernel attack surface. ### Changes * **Capability Allowlist:** `securityContext.capabilities.add` now only permits 12 capabilities required for package installs, non-root user switches, and standard container workflows: * `AUDIT_WRITE` * `CHOWN` * `DAC_OVERRIDE` * `FOWNER` * `FSETID` * `KILL` * `NET_BIND_SERVICE` * `SETFCAP` * `SETGID` * `SETPCAP` * `SETUID` * `SYS_CHROOT` * **Admission Enforcement:** `ateapi` and `atelet` reject actor template creations and updates specifying capabilities outside this allowlist. * **Drop Unrestricted:** `securityContext.capabilities.drop` continues to accept any capability name. * **MicroVM Hardening:** * Volume bind mounts are now mounted with `nodev`, and the microVM guest mounts the virtio-fs share with `nodev` (covering automounted submounts and rootfs). `ShapeGVisor` strips `nodev`, keeping gVisor specs identical and preserving checkpoint/restore compatibility. * MicroVM guest sysctl `net.ipv4.ping_group_range = "0 2147483647"` is configured so standard unprivileged `ping` continues to work without requiring `NET_RAW`. * **Documentation & Schemas:** Updated `api-guide.md`, `sandbox-classes.md`, and regenerated protobuf bindings (`ateapi.pb.go`, `atelet.pb.go`). ### Breaking change: gVisor `MEMORY` snapshots taken before this release won't restore gVisor now gets `nodev` on volume and system mounts (`/proc`, `/dev/shm`, `/sys`, `/tmp`), matching microVM. runsc applies it to the host bind mount. `runsc restore` rejects a checkpoint whose mounts differ from the new spec (`"Mounts" does not match across checkpoint restore`), so gVisor `MEMORY` snapshots taken before the upgrade can't be restored. rc.2 already breaks these snapshots with [agent-substrate#3533](agent-substrate#3533) (tmpfs `/dev/shm` and `/tmp`), so this adds no new upgrade step on top of that. MicroVM actors are unaffected. **What breaks:** - Suspended gVisor actors whose latest snapshot is `MEMORY` fidelity (the default) move to `ACTOR_STATE_CRASHED` on `ResumeActor`. - New actors from gVisor templates created before the upgrade crash on first resume, because the template's golden snapshot is an old `MEMORY` checkpoint. **Upgrade steps:** 1. **Before upgrading,** resume any suspended gVisor actor whose in-memory state you need, and save that state to durable storage or let its work finish. 2. **Re-create gVisor templates.** `ActorTemplate` is immutable, so create a new template (e.g. `my-template-v2`), or delete and re-create it, to build a new golden snapshot. Point clients at the new template. 3. **Replace crashed actors.** `RevertActor` won't help: it reverts to the same pre-upgrade checkpoint, which crashes again. `UpdateActor --template` won't either, because resume restores the actor's own snapshot. Delete the actor and create it again from the new template. Its in-memory state and its snapshot's rootfs and volume contents are lost. ### Testing * **Unit Tests:** Updated resource validation tests, parity tests, and microVM spec tests (`go test ./...` passes in container). * **Live GKE Testing (`sgke-rc1`):** * Verified template rejection when adding `MKNOD` or `SYS_ADMIN`. * Verified unprivileged ICMP `ping` works in microVM guests without `NET_RAW`. * Verified that pre-existing gVisor actor snapshots restore cleanly. * **e2e Probe Coverage:** * Verified inside the guest that `mknod` fails with `EPERM`. * Verified that unprivileged users can open `SOCK_DGRAM`/`IPPROTO_ICMP` sockets across both runtimes. Thanks to [Fabiano Fidêncio (@fidencio)](https://github.com/fidencio) for the report on MKNOD that led us to make this change. --------- Signed-off-by: Greg Castle <[email protected]>

…#3647) The `glutton` template stored `WriteDisk` files under `/tmp`, which is a `tmpfs` on both sandbox classes (agent-substrate#3533), so they counted as memory and no benchmark wrote to the rootfs upper layer. Use `/var/lib/glutton` with no volume mounted instead, so `agentsession`'s files land in the rootfs upper layer and in its snapshot. On gVisor, `agentsession` now writes to the rootfs for the first time, so its numbers will change. On microVM, its files go back to the host-backed rootfs upper layer, as before agent-substrate#3533. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated Signed-off-by: Chenyi Wang <[email protected]>

nybidari force-pushed the align-system-mounts branch 2 times, most recently from 5ec1969 to f6d216d Compare November 1, 2026 17:22

nybidari requested review from Benjamin Elder (BenTheElder), Dmitry Berkovich (dberkov) and Fabricio Voznika (fvoznika) November 1, 2026 17:23

Benjamin Elder (BenTheElder) added prio/P0 Highest priority / required for next milestone area/gvisor area/microVM milestone-ack labels Nov 2, 2026

Benjamin Elder (BenTheElder) added this to the M3 milestone Nov 2, 2026

ocispec: mount tmpfs at /dev/shm, /run, and /tmp on both runtimes

c49333a

Signed-off-by: Nayana Bidari <[email protected]>

nybidari force-pushed the align-system-mounts branch from f6d216d to c49333a Compare November 2, 2026 17:58

Benjamin Elder (BenTheElder) reviewed Nov 2, 2026

View reviewed changes

nybidari changed the title ~~ocispec: mount tmpfs at /dev/shm, /run, and /tmp on both runtimes~~ ocispec: mount tmpfs at /dev/shm, and /tmp on both runtimes Nov 3, 2026

nybidari force-pushed the align-system-mounts branch from 37cfeb8 to de057d2 Compare November 3, 2026 00:59

nybidari requested a review from Benjamin Elder (BenTheElder) November 3, 2026 01:08

Benjamin Elder (BenTheElder) previously approved these changes Nov 3, 2026

View reviewed changes

Benjamin Elder (BenTheElder) added the breaking-change PRs with breaking changes label Nov 3, 2026

Benjamin Elder (BenTheElder) mentioned this pull request Nov 3, 2026

[bug-bash] Track gVisor / microVM parity gaps for 1.0 #3226

Open

16 of 23 tasks

nybidari dismissed Benjamin Elder (BenTheElder)’s stale review via 4051e6c November 3, 2026 02:49

nybidari force-pushed the align-system-mounts branch from de057d2 to 4051e6c Compare November 3, 2026 02:49

Tim Hockin (thockin) assigned Benjamin Elder (BenTheElder) and Tim Hockin (thockin) Nov 3, 2026

Tim Hockin (thockin) approved these changes Nov 3, 2026

View reviewed changes

Tim Hockin (thockin) added this pull request to the merge queue Nov 3, 2026

Merged via the queue into agent-substrate:main with commit fd576d3 Nov 3, 2026
7 checks passed

Eitan Yarmush (EItanya) mentioned this pull request Nov 3, 2026

[release-1.0] Cherry-pick 19 main changes through 5a5b19219 #3571

Merged

1 of 2 tasks

Eitan Yarmush (EItanya) mentioned this pull request Nov 3, 2026

[release-1.0] Cherry-pick 21 main changes through fc931c4e2 #3596

Merged

1 of 2 tasks

Greg Castle (destijl) mentioned this pull request Nov 3, 2026

Restrict actor capabilities.add and make mounts nodev #3580

Merged

Dmitry Berkovich (dberkov) mentioned this pull request Nov 4, 2026

[release-1.0] Cherry-pick #3580: Restrict actor capabilities.add and make mounts nodev #3611

Merged

2 tasks done

Chenyi Wang (chw120) mentioned this pull request Nov 5, 2026

benchmarking: put the glutton data dir on the rootfs #3647

Merged

2 tasks done

noobHappylife mentioned this pull request Nov 10, 2026

Support configurable /dev/shm tmpfs sizing #3760

Open

		@@ -64,8 +61,11 @@ See [Container Capabilities](api-guide.md#container-capabilities-securitycontext

		## System Mounts (`/dev/shm`, `/run`, `/tmp`)

Conversation

nybidari commented Nov 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nybidari commented Nov 1, 2026 •

edited

Loading