Skip to content

Commit fd576d3

Browse files
authored
ocispec: mount tmpfs at /dev/shm, and /tmp on both runtimes (#3533)
Part of #3226. `gvisor` and `microvm` previously mounted different filesystems at `/dev/shm`, and `/tmp`: - `/dev/shm` on gVisor was a `0755` directory on `runsc`'s internal `dev` filesystem, blocking non-root shared memory writes. - `/tmp` was part of the host-backed rootfs upper layer on microVM and a conditional `tmpfs` on gVisor when the image's `/tmp` was empty. - Dropped the `/run` tmpfs from both runtimes, so `/run` comes from the image's rootfs, as on Kubernetes. Move the standard OCI system mounts (`/proc`, `/dev`, `/dev/pts`, `/dev/shm`, `/dev/mqueue`, `/sys`) into `ocispec.Build` and add an explicit `mode=1777` `tmpfs` mount at `/tmp`. If a container mounts a volume at a system mount destination such as `/tmp`, `ocispec.Build` replaces the default `tmpfs` entry there. `ShapeMicroVM` now rewrites `bind` mounts in place. AI use: AI-assisted, I reviewed and tested it. Breaking change: - Old gVisor MEMORY snapshots, including goldens, don't restore. Recut golden snapshots after upgrading. - MicroVM MEMORY snapshots and goldens still resume, but keep the old mounts until they are recut. - MicroVM `/tmp`: now a tmpfs in guest memory instead of the host-backed rootfs. - Its files count against the actor's memory limit. - It is capped at half of the guest's memory. - ROOTFS snapshots no longer keep it, and files older ROOTFS snapshots saved there are hidden. - Files the image ships under `/tmp` are hidden. - gVisor `/dev/shm`: now a 64 MiB tmpfs (`mode=1777`) instead of a root-only directory. Non-root processes can now use it, but it is capped at 64 MiB. - gVisor `/tmp`: now always a tmpfs. runsc used to mount one only when the image's `/tmp` was empty, so files the image ships under `/tmp` are now hidden. If an actor needs more space in `/dev/shm` or `/tmp`, or needs files there kept in ROOTFS snapshots, mount a `durableDir` volume at that path. It replaces the default tmpfs. Images that ship files under `/tmp` should create them at startup instead. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Nayana Bidari <[email protected]>
1 parent 92040fe commit fd576d3

18 files changed

Lines changed: 333 additions & 125 deletions

File tree

‎api/ateapi/v1/ateapi.pb.go‎

Lines changed: 3 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/ateapi/v1/ateapi.proto‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1670,11 +1670,12 @@ message VolumeMount {
16701670

16711671
// mount_path within the container. Must be a clean absolute Unix path:
16721672
// must start with '/', not be '/', and contain no ':', '..', '.', '//',
1673-
// trailing '/', or control characters.
1673+
// trailing '/', or control characters. Must not be /dev, and must not be
1674+
// /dev/pts, /proc, /sys, or a path under them.
16741675
//
16751676
// +k8s:required
16761677
// +k8s:maxLength=4096
1677-
// +k8s:customValidation # clean-absolute-path shape; no regex/pattern tag exists
1678+
// +k8s:customValidation # clean absolute path, not /dev, outside /dev/pts, /proc and /sys
16781679
string mount_path = 2;
16791680
}
16801681

‎docs/reference/sandbox-classes.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ title: "Sandbox Classes"
33
linkTitle: "Sandbox Classes"
44
weight: 6
55
description: >
6-
How the gvisor and microvm sandbox classes differ in sizing, capabilities, mounts, and snapshots.
6+
How the gvisor and microvm sandbox classes differ in sizing, capabilities, and snapshots.
77
---
88

99
# Sandbox Classes
@@ -23,9 +23,6 @@ An [`ActorTemplate`](api-guide.md#2-actortemplate-the-workload-blueprint) select
2323
| Minimum `limits.memoryBytes` | None enforced by worker | `128 MiB` (`134217728` bytes) |
2424
| Per-container limits (`containers[].resources.limits`) | Accepted in the API, ignored at runtime | Enforced via guest cgroups (`cpuMillicores` floored at 10) |
2525
| `NET_RAW` capability | Stripped (`runsc` default `--net-raw=false`) | Granted when listed in `securityContext.capabilities.add` |
26-
| `/dev/shm` | `0755` directory on the `runsc` `dev` filesystem | `64 MiB` `tmpfs` (`mode=1777`) |
27-
| `/run` | Part of the container rootfs overlay | `64 MiB` `tmpfs` (`mode=0755`) |
28-
| `/tmp` | `tmpfs` when the image's `/tmp` is missing or empty, otherwise rootfs overlay | Part of the container rootfs overlay |
2926
| `VOLUMES` and `ROOTFS` snapshot format | `runsc` filesystem checkpoint bundles (`fs/`, `rootfs/`) | Tar archives of the host directories |
3027

3128
## CPU Count and Throttling
@@ -62,10 +59,13 @@ They differ on `NET_RAW`:
6259

6360
See [Container Capabilities](api-guide.md#container-capabilities-securitycontextcapabilities) in the API Guide.
6461

65-
## System Mounts (`/dev/shm`, `/run`, `/tmp`)
62+
## System Mounts (`/dev/shm` and `/tmp`)
6663

67-
- **gVisor (`gvisor`):** `runsc` mounts its internal `dev` filesystem at `/dev`, where `/dev/shm` is a `0755` directory, so only root can create files there. `/run` lives on the container's rootfs overlay. `runsc` mounts a `tmpfs` (`mode=01777`) at `/tmp` when the image's `/tmp` is missing or empty and no volume is mounted there. Files in `/dev/shm` and in that `/tmp` `tmpfs` live outside the rootfs overlay, so only `MEMORY` snapshots keep them.
68-
- **MicroVM (`microvm`):** `ateworker-microvm` mounts a `64 MiB` `tmpfs` at `/dev/shm` (`mode=1777`, `nosuid`, `nodev`, `noexec`) and at `/run` (`mode=0755`, `nosuid`, `strictatime`). `/tmp` lives on the container's host-backed rootfs overlay, so `ROOTFS` snapshots keep its files. Files in `/dev/shm` and `/run` live in guest memory, so only `MEMORY` snapshots keep them.
64+
Each container gets its own `tmpfs` at `/dev/shm` (`64 MiB`, mode `1777`, `noexec`) and `/tmp` (mode `1777`). `/tmp` holds at most half of `limits.memoryBytes` on `gvisor`, and half of usable guest RAM (`MemTotal`) on `microvm`. The `/tmp` mount hides any files the image has in `/tmp`. A volume at either path replaces the `tmpfs`. Files in these `tmpfs` mounts count against the actor's memory limit. They live outside the container's rootfs overlay, so `MEMORY` snapshots keep them and `ROOTFS` snapshots start with them empty.
65+
66+
Neither sandbox class mounts anything at `/run`. As with containerd, `/run` is part of the container's rootfs overlay, so the image's files there stay visible and `ROOTFS` snapshots keep them.
67+
68+
Volumes cannot mount at `/dev`, or at or under `/dev/pts`, `/proc`, or `/sys`.
6969

7070
## Snapshots (`VOLUMES`, `ROOTFS`, and `MEMORY`)
7171

‎internal/e2e/fixtures/probe/nonroot/main.go‎

Lines changed: 10 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,8 @@ type identityResponse struct {
3737
Cwd string `json:"cwd"`
3838
// CapEff is the CapEff mask from /proc/self/status, in hex.
3939
CapEff string `json:"capEff"`
40-
// WriteError is why creating a file in the working directory failed.
40+
// WriteError is why creating a file in the working directory, /tmp, or
41+
// /dev/shm failed.
4142
WriteError string `json:"writeError,omitempty"`
4243
Error string `json:"error,omitempty"`
4344
}
@@ -63,13 +64,15 @@ func identity(w http.ResponseWriter, _ *http.Request) {
6364
} else {
6465
resp.Error += "reading /proc/self/status: " + err.Error() + "; "
6566
}
66-
if f, err := os.CreateTemp(".", "write-check-"); err == nil {
67-
_ = f.Close()
68-
if err := os.Remove(f.Name()); err != nil {
69-
resp.WriteError = err.Error()
67+
for _, dir := range []string{".", "/tmp", "/dev/shm"} {
68+
f, err := os.CreateTemp(dir, "write-check-")
69+
if err == nil {
70+
_ = f.Close()
71+
err = os.Remove(f.Name())
72+
}
73+
if err != nil {
74+
resp.WriteError += err.Error() + "; "
7075
}
71-
} else {
72-
resp.WriteError = err.Error()
7376
}
7477

7578
w.Header().Set("Content-Type", "application/json")

‎internal/e2e/suites/imagefs/imagefs_test.go‎

Lines changed: 31 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -43,20 +43,28 @@ type statResponse struct {
4343
// .ko.yaml builds the probe on the root distroless static base, whose base
4444
// layer has /home/nonroot as drwx------ owned by 65532:65532 (distroless's
4545
// nonroot user), /var/tmp as root-owned 1777, and root-owned system files.
46-
// /var/tmp, not /tmp, checks the sticky bit: gVisor mounts its own 1777 tmpfs
47-
// over an empty /tmp.
46+
// /var/tmp, not /tmp, checks the sticky bit: both runtimes mount a tmpfs over
47+
// /tmp.
4848
var layerOwners = []statResponse{
4949
{Path: "/home/nonroot", UID: 65532, GID: 65532, Mode: "drwx------"},
5050
{Path: "/var/tmp", UID: 0, GID: 0, Mode: "dtrwxrwxrwx"},
5151
{Path: "/etc/passwd", UID: 0, GID: 0, Mode: "-rw-r--r--"},
5252
{Path: "/home", UID: 0, GID: 0, Mode: "drwxr-xr-x"},
5353
}
5454

55+
// systemMountOwners are the owner and mode of the tmpfs mounts both runtimes
56+
// put at /tmp and /dev/shm.
57+
var systemMountOwners = []statResponse{
58+
{Path: "/tmp", UID: 0, GID: 0, Mode: "dtrwxrwxrwx"},
59+
{Path: "/dev/shm", UID: 0, GID: 0, Mode: "dtrwxrwxrwx"},
60+
}
61+
5562
// TestImageLayerOwnership asserts that files keep the owners the image's
5663
// layer tars record, as they do under containerd: a private home directory
5764
// given to a non-root user must not show up as root-owned inside the actor.
58-
// It checks once on the first start and again after a suspend and resume,
59-
// which restores the actor over the same cached layers.
65+
// It also checks the tmpfs mounts in systemMountOwners. It checks once on the
66+
// first start and again after a suspend and resume, which restores the actor
67+
// over the same cached layers.
6068
//
6169
// It runs against whichever sandbox class E2E_SANDBOX_CLASS selects, so CI
6270
// covers both gVisor and microVM from one suite.
@@ -85,10 +93,14 @@ func TestImageLayerOwnership(t *testing.T) {
8593
}
8694
defer rc.Close()
8795

96+
check := func(t *testing.T) {
97+
checkOwners(t, ctx, rc, ref, layerOwners, "the image's layer")
98+
checkOwners(t, ctx, rc, ref, systemMountOwners, "the system mount")
99+
}
88100
if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapi.ResumeActorRequest{Actor: ref}); err != nil {
89101
t.Fatalf("ResumeActor %q: %v", ref.Name, err)
90102
}
91-
t.Run("first start", func(t *testing.T) { checkOwners(t, ctx, rc, ref) })
103+
t.Run("first start", check)
92104

93105
suspended, err := clients.SubstrateAPI.SuspendActor(ctx, &ateapi.SuspendActorRequest{Actor: ref})
94106
if err != nil {
@@ -100,7 +112,7 @@ func TestImageLayerOwnership(t *testing.T) {
100112
if _, err := e2e.ResumeActorAwaitCapacity(t, ctx, clients, &ateapi.ResumeActorRequest{Actor: ref}); err != nil {
101113
t.Fatalf("ResumeActor %q after suspend: %v", ref.Name, err)
102114
}
103-
t.Run("after suspend and resume", func(t *testing.T) { checkOwners(t, ctx, rc, ref) })
115+
t.Run("after suspend and resume", check)
104116
}
105117

106118
// identityResponse mirrors the nonroot probe's /identity payload.
@@ -126,8 +138,9 @@ const (
126138
// TestImageUserAndWorkdir asserts that the actor's process runs as its image's
127139
// USER, resolved by name against the image's own /etc/passwd and /etc/group,
128140
// with no effective capabilities, and starts in the image's WORKDIR, where
129-
// that user can create files. The WORKDIR is private to the user, so the
130-
// write also needs the layer owners TestImageLayerOwnership checks.
141+
// that user can create files alongside /tmp and /dev/shm. The WORKDIR is
142+
// private to the user, so the write also needs the layer owners
143+
// TestImageLayerOwnership checks.
131144
//
132145
// The golden bake runs the probe as that user and binds :80, and the actor
133146
// resumes from that golden snapshot. It runs against whichever sandbox class
@@ -187,7 +200,7 @@ func TestImageUserAndWorkdir(t *testing.T) {
187200
t.Errorf("working directory = %q, want the image's WORKDIR %q", got.Cwd, nonrootWorkdir)
188201
}
189202
if got.WriteError != "" {
190-
t.Errorf("creating a file in the WORKDIR as the image's USER: %s", got.WriteError)
203+
t.Errorf("creating a file in the WORKDIR, /tmp, or /dev/shm as the image's USER: %s", got.WriteError)
191204
}
192205
}
193206

@@ -209,17 +222,19 @@ func probeIdentity(t *testing.T, ctx context.Context, rc *e2e.RouterClient, ref
209222
return out
210223
}
211224

212-
func checkOwners(t *testing.T, ctx context.Context, rc *e2e.RouterClient, ref *ateapi.ObjectRef) {
225+
// checkOwners checks the owner and mode of each path in want, naming source on
226+
// mismatch.
227+
func checkOwners(t *testing.T, ctx context.Context, rc *e2e.RouterClient, ref *ateapi.ObjectRef, want []statResponse, source string) {
213228
t.Helper()
214-
for _, want := range layerOwners {
215-
got := probeStat(t, ctx, rc, ref, want.Path)
229+
for _, w := range want {
230+
got := probeStat(t, ctx, rc, ref, w.Path)
216231
if got.Error != "" {
217-
t.Errorf("stat %s inside the actor: %s", want.Path, got.Error)
232+
t.Errorf("stat %s inside the actor: %s", w.Path, got.Error)
218233
continue
219234
}
220-
if got.UID != want.UID || got.GID != want.GID || got.Mode != want.Mode {
221-
t.Errorf("%s inside the actor is %s %d:%d, want %s %d:%d as the image's layer records it",
222-
want.Path, got.Mode, got.UID, got.GID, want.Mode, want.UID, want.GID)
235+
if got.UID != w.UID || got.GID != w.GID || got.Mode != w.Mode {
236+
t.Errorf("%s inside the actor is %s %d:%d, want %s %d:%d from %s",
237+
w.Path, got.Mode, got.UID, got.GID, w.Mode, w.UID, w.GID, source)
223238
}
224239
}
225240
}

‎internal/ocispec/gvisor.go‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,10 @@
1515
package ocispec
1616

1717
import (
18+
"fmt"
1819
"path/filepath"
20+
"slices"
21+
"strings"
1922

2023
"github.com/opencontainers/runtime-spec/specs-go"
2124
"go.ate.dev/substrate/internal/proto/ateworkerpb"
@@ -35,8 +38,9 @@ type GVisorOptions struct {
3538
ContainerName string
3639
// DurableVolumes are declared on the sandbox spec only.
3740
DurableVolumes []string
38-
// Size sizes the container's cgroup leaf. Only gVisor applies it; a micro-VM
39-
// container's limits come from its own declared resources (see sizing).
41+
// Size sizes the container's cgroup leaf and caps its /tmp. Only gVisor
42+
// applies it; a microVM container's limits come from its own declared
43+
// resources (see sizing).
4044
Size sizing.SandboxSize
4145
}
4246

@@ -46,7 +50,8 @@ func GVisorCgroupLeaf(actorUID, containerName string) string {
4650
}
4751

4852
// ShapeGVisor adds runsc CRI annotations, durable-dir mount hints, and
49-
// per-container cgroups to the spec. It is idempotent.
53+
// per-container cgroups to the spec, caps /tmp at half the memory limit, and
54+
// drops mounts and mount options runsc supplies or ignores. It is idempotent.
5055
func ShapeGVisor(spec *specs.Spec, o GVisorOptions) {
5156
if spec.Annotations == nil {
5257
spec.Annotations = make(map[string]string)
@@ -64,6 +69,26 @@ func ShapeGVisor(spec *specs.Spec, o GVisorOptions) {
6469
spec.Annotations["io.kubernetes.cri.sandbox-id"] = SandboxID
6570
}
6671

72+
// runsc replaces /dev and /dev/pts with its own mounts and ignores mqueue
73+
// mounts and the nodev and strictatime options, logging a warning for each.
74+
spec.Mounts = slices.DeleteFunc(spec.Mounts, func(m specs.Mount) bool {
75+
return m.Destination == "/dev" || m.Destination == "/dev/pts" || m.Type == "mqueue"
76+
})
77+
for i := range spec.Mounts {
78+
spec.Mounts[i].Options = slices.DeleteFunc(slices.Clone(spec.Mounts[i].Options), func(opt string) bool {
79+
return opt == "nodev" || opt == "strictatime"
80+
})
81+
}
82+
// runsc's tmpfs defaults to half of the node's memory. Cap /tmp at half the
83+
// actor's memory limit, matching the guest kernel's default on microVM.
84+
// runsc requires unchanged mount options across MEMORY restores.
85+
if mem := o.Size.MemoryBytes; mem > 0 {
86+
if i := slices.IndexFunc(spec.Mounts, func(m specs.Mount) bool { return m.Destination == "/tmp" && m.Type == "tmpfs" }); i >= 0 {
87+
opts := slices.DeleteFunc(spec.Mounts[i].Options, func(opt string) bool { return strings.HasPrefix(opt, "size=") })
88+
spec.Mounts[i].Options = append(opts, fmt.Sprintf("size=%d", max(mem/2, 1)))
89+
}
90+
}
91+
6792
if spec.Linux == nil {
6893
spec.Linux = &specs.Linux{}
6994
}

‎internal/ocispec/gvisor_test.go‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,10 +15,13 @@
1515
package ocispec
1616

1717
import (
18+
"reflect"
1819
"strings"
1920
"testing"
2021

2122
specs "github.com/opencontainers/runtime-spec/specs-go"
23+
"go.ate.dev/substrate/internal/proto/ateworkerpb"
24+
"go.ate.dev/substrate/internal/sizing"
2225
)
2326

2427
func TestShapeGVisorGivesEachActorItsOwnCgroupLeaf(t *testing.T) {
@@ -53,3 +56,74 @@ func TestGVisorCgroupLeafMatchesTheShapedPath(t *testing.T) {
5356
t.Errorf("shaped cgroupsPath = %q, want %q", spec.Linux.CgroupsPath, want)
5457
}
5558
}
59+
60+
// runsc replaces /dev and /dev/pts with its own mounts and ignores mqueue
61+
// mounts and the nodev and strictatime options, so ShapeGVisor drops them.
62+
// /tmp is capped at half the memory limit when tmpfs.
63+
func TestShapeGVisor_SystemMounts(t *testing.T) {
64+
const durableDir = "/node/actors/a/durable-dir"
65+
for _, tc := range []struct {
66+
name string
67+
memory int64
68+
mounts []*ateworkerpb.VolumeMount
69+
wantAll []specs.Mount
70+
wantEach []specs.Mount
71+
}{
72+
{
73+
name: "memory limit",
74+
memory: 512 << 20,
75+
wantAll: []specs.Mount{
76+
{Destination: "/proc", Type: "proc", Source: "proc", Options: []string{"nosuid", "noexec"}},
77+
{Destination: "/dev/shm", Type: "tmpfs", Source: "shm", Options: []string{"nosuid", "noexec", "mode=1777", "size=65536k"}},
78+
{Destination: "/sys", Type: "sysfs", Source: "sysfs", Options: []string{"nosuid", "noexec", "ro"}},
79+
{Destination: "/tmp", Type: "tmpfs", Source: "tmpfs", Options: []string{"nosuid", "mode=1777", "size=268435456"}},
80+
},
81+
},
82+
{
83+
name: "minimum positive memory limit",
84+
memory: 1,
85+
wantEach: []specs.Mount{
86+
{Destination: "/tmp", Type: "tmpfs", Source: "tmpfs", Options: []string{"nosuid", "mode=1777", "size=1"}},
87+
},
88+
},
89+
{
90+
name: "no memory limit",
91+
memory: 0,
92+
wantEach: []specs.Mount{
93+
{Destination: "/tmp", Type: "tmpfs", Source: "tmpfs", Options: []string{"nosuid", "mode=1777"}},
94+
},
95+
},
96+
{
97+
name: "volumes at /dev/mqueue and /tmp",
98+
memory: 512 << 20,
99+
mounts: []*ateworkerpb.VolumeMount{
100+
{Name: "mq", MountPath: "/dev/mqueue"},
101+
{Name: "tmp", MountPath: "/tmp"},
102+
},
103+
wantEach: []specs.Mount{
104+
{Destination: "/dev/mqueue", Type: "bind", Source: durableDir + "/mq", Options: []string{"bind", "rw"}},
105+
{Destination: "/tmp", Type: "bind", Source: durableDir + "/tmp", Options: []string{"bind", "rw"}},
106+
},
107+
},
108+
} {
109+
t.Run(tc.name, func(t *testing.T) {
110+
actorDirs := &ateworkerpb.ActorDirs{DurableDirVolumeMountsDir: durableDir}
111+
spec := Build(Options{
112+
ActorDirs: actorDirs,
113+
Container: &ateworkerpb.Container{
114+
ContainerSpec: &ateworkerpb.ContainerSpec{Args: []string{"/app"}},
115+
DurableDirVolumeMounts: tc.mounts,
116+
},
117+
})
118+
ShapeGVisor(spec, GVisorOptions{ActorUID: "uid-a", ActorDirs: actorDirs, ContainerName: "app", Size: sizing.SandboxSize{MemoryBytes: tc.memory}})
119+
if tc.wantAll != nil && !reflect.DeepEqual(spec.Mounts, tc.wantAll) {
120+
t.Errorf("Mounts =\n%+v\nwant\n%+v", spec.Mounts, tc.wantAll)
121+
}
122+
for _, want := range tc.wantEach {
123+
if got := mountFor(t, spec, want.Destination); !reflect.DeepEqual(got, want) {
124+
t.Errorf("%s =\n%+v\nwant\n%+v", want.Destination, got, want)
125+
}
126+
}
127+
})
128+
}
129+
}

0 commit comments

Comments
 (0)