Skip to content

Commit 17fa858

Browse files
authored
Restrict actor capabilities.add and make mounts nodev (#3580)
Fixes [#3579](#3579) ### Summary Restricts `securityContext.capabilities.add` to a hardened allowlist of 12 Linux capabilities and blocks dangerous capabilities such as `MKNOD`, `SYS_ADMIN`, and `SYS_RAWIO` before 1.0. In microVM sandboxes, granting `MKNOD` to a container running as root allows it to create block device nodes matching the guest's `virtio-pmem` root disk and directly write to the guest filesystem. Other capabilities like `SYS_ADMIN` and `SYS_RAWIO` expose dangerous guest kernel attack surface. ### Changes * **Capability Allowlist:** `securityContext.capabilities.add` now only permits 12 capabilities required for package installs, non-root user switches, and standard container workflows: * `AUDIT_WRITE` * `CHOWN` * `DAC_OVERRIDE` * `FOWNER` * `FSETID` * `KILL` * `NET_BIND_SERVICE` * `SETFCAP` * `SETGID` * `SETPCAP` * `SETUID` * `SYS_CHROOT` * **Admission Enforcement:** `ateapi` and `atelet` reject actor template creations and updates specifying capabilities outside this allowlist. * **Drop Unrestricted:** `securityContext.capabilities.drop` continues to accept any capability name. * **MicroVM Hardening:** * Volume bind mounts are now mounted with `nodev`, and the microVM guest mounts the virtio-fs share with `nodev` (covering automounted submounts and rootfs). `ShapeGVisor` strips `nodev`, keeping gVisor specs identical and preserving checkpoint/restore compatibility. * MicroVM guest sysctl `net.ipv4.ping_group_range = "0 2147483647"` is configured so standard unprivileged `ping` continues to work without requiring `NET_RAW`. * **Documentation & Schemas:** Updated `api-guide.md`, `sandbox-classes.md`, and regenerated protobuf bindings (`ateapi.pb.go`, `atelet.pb.go`). ### Breaking change: gVisor `MEMORY` snapshots taken before this release won't restore gVisor now gets `nodev` on volume and system mounts (`/proc`, `/dev/shm`, `/sys`, `/tmp`), matching microVM. runsc applies it to the host bind mount. `runsc restore` rejects a checkpoint whose mounts differ from the new spec (`"Mounts" does not match across checkpoint restore`), so gVisor `MEMORY` snapshots taken before the upgrade can't be restored. rc.2 already breaks these snapshots with [#3533](#3533) (tmpfs `/dev/shm` and `/tmp`), so this adds no new upgrade step on top of that. MicroVM actors are unaffected. **What breaks:** - Suspended gVisor actors whose latest snapshot is `MEMORY` fidelity (the default) move to `ACTOR_STATE_CRASHED` on `ResumeActor`. - New actors from gVisor templates created before the upgrade crash on first resume, because the template's golden snapshot is an old `MEMORY` checkpoint. **Upgrade steps:** 1. **Before upgrading,** resume any suspended gVisor actor whose in-memory state you need, and save that state to durable storage or let its work finish. 2. **Re-create gVisor templates.** `ActorTemplate` is immutable, so create a new template (e.g. `my-template-v2`), or delete and re-create it, to build a new golden snapshot. Point clients at the new template. 3. **Replace crashed actors.** `RevertActor` won't help: it reverts to the same pre-upgrade checkpoint, which crashes again. `UpdateActor --template` won't either, because resume restores the actor's own snapshot. Delete the actor and create it again from the new template. Its in-memory state and its snapshot's rootfs and volume contents are lost. ### Testing * **Unit Tests:** Updated resource validation tests, parity tests, and microVM spec tests (`go test ./...` passes in container). * **Live GKE Testing (`sgke-rc1`):** * Verified template rejection when adding `MKNOD` or `SYS_ADMIN`. * Verified unprivileged ICMP `ping` works in microVM guests without `NET_RAW`. * Verified that pre-existing gVisor actor snapshots restore cleanly. * **e2e Probe Coverage:** * Verified inside the guest that `mknod` fails with `EPERM`. * Verified that unprivileged users can open `SOCK_DGRAM`/`IPPROTO_ICMP` sockets across both runtimes. Thanks to [Fabiano Fidêncio (@fidencio)](https://github.com/fidencio) for the report on MKNOD that led us to make this change. --------- Signed-off-by: Greg Castle <[email protected]>
1 parent 9922363 commit 17fa858

25 files changed

Lines changed: 211 additions & 92 deletions

File tree

‎api/ateapi/v1/ateapi.pb.go‎

Lines changed: 4 additions & 3 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/ateapi/v1/ateapi.proto‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1409,13 +1409,14 @@ message SecurityContext {
14091409
// granted. Capabilities are named without the "CAP_" prefix; "ALL" in drop
14101410
// removes the whole default set.
14111411
message Capabilities {
1412-
// add lists capabilities to grant on top of the default set. "ALL" is
1413-
// rejected: name the individual capabilities the container needs.
1412+
// add lists capabilities to grant on top of the default set. Only
1413+
// AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, NET_BIND_SERVICE,
1414+
// SETFCAP, SETGID, SETPCAP, SETUID and SYS_CHROOT are accepted.
14141415
//
14151416
// +k8s:optional
14161417
// +k8s:maxItems=64
14171418
// +k8s:listType=set
1418-
// +k8s:customValidation # a Linux capability name; "ALL" not accepted
1419+
// +k8s:customValidation # one of resources.AllowedAddCapabilities
14191420
repeated string add = 1;
14201421

14211422
// drop lists capabilities to remove from the default set. "ALL" drops the

‎cmd/ateapi/internal/apivalidation/actor_template.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -178,9 +178,9 @@ func ValidateCustom_EnvVar_Name(_ context.Context, _ operation.Operation, fldPat
178178
}
179179

180180
func ValidateCustom_Capabilities_Add(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []string) field.ErrorList {
181-
return resources.ValidateCapabilities(fldPath, value, false)
181+
return resources.ValidateAddCapabilities(fldPath, value)
182182
}
183183

184184
func ValidateCustom_Capabilities_Drop(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []string) field.ErrorList {
185-
return resources.ValidateCapabilities(fldPath, value, true)
185+
return resources.ValidateDropCapabilities(fldPath, value)
186186
}

‎cmd/ateapi/internal/apivalidation/actor_template_test.go‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -712,6 +712,12 @@ func TestValidateActorTemplate(t *testing.T) {
712712
tmpl.Containers[0].SecurityContext = &ateapi.SecurityContext{Capabilities: &ateapi.Capabilities{Add: []string{"NET_ADIMN"}}}
713713
},
714714
want: field.ErrorList{field.NotSupported[string](field.NewPath("containers").Index(0).Child("security_context", "capabilities", "add").Index(0), nil, nil)},
715+
}, {
716+
name: "capability outside the add allowlist",
717+
mutate: func(tmpl *ateapi.ActorTemplate) {
718+
tmpl.Containers[0].SecurityContext = &ateapi.SecurityContext{Capabilities: &ateapi.Capabilities{Add: []string{"MKNOD"}}}
719+
},
720+
want: field.ErrorList{field.NotSupported[string](field.NewPath("containers").Index(0).Child("security_context", "capabilities", "add").Index(0), nil, nil)},
715721
}, {
716722
name: "duplicate capability",
717723
mutate: func(tmpl *ateapi.ActorTemplate) {

‎cmd/ateapi/internal/controlapi/workload_spec_test.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -486,7 +486,7 @@ func TestWorkloadSpecFromActorTemplatePropagatesSecurityContext(t *testing.T) {
486486
Image: "main",
487487
SecurityContext: &ateapi.SecurityContext{
488488
Capabilities: &ateapi.Capabilities{
489-
Add: []string{"NET_ADMIN"},
489+
Add: []string{"SETUID"},
490490
Drop: []string{"ALL"},
491491
},
492492
},
@@ -515,7 +515,7 @@ func TestWorkloadSpecFromActorTemplatePropagatesSecurityContext(t *testing.T) {
515515
Image: "main",
516516
SecurityContext: &ateletpb.SecurityContext{
517517
Capabilities: &ateletpb.Capabilities{
518-
Add: []string{"NET_ADMIN"},
518+
Add: []string{"SETUID"},
519519
Drop: []string{"ALL"},
520520
},
521521
},

‎cmd/atelet/internal/apivalidation/validation_test.go‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1815,9 +1815,19 @@ func TestValidateContainer(t *testing.T) {
18151815
name: "capabilities: unknown name rejected",
18161816
obj: caps(nil, []string{"NET_ADIMN"}),
18171817
want: field.ErrorList{field.NotSupported[string](capsPath.Child("drop").Index(0), nil, nil)},
1818+
}, {
1819+
name: "capabilities: add outside the allowlist rejected",
1820+
obj: caps([]string{"MKNOD", "SYS_ADMIN"}, nil),
1821+
want: field.ErrorList{
1822+
field.NotSupported[string](capsPath.Child("add").Index(0), nil, nil),
1823+
field.NotSupported[string](capsPath.Child("add").Index(1), nil, nil),
1824+
},
1825+
}, {
1826+
name: "capabilities: drop outside the allowlist accepted",
1827+
obj: caps(nil, []string{"MKNOD", "SYS_ADMIN"}),
18181828
}, {
18191829
name: "capabilities: duplicate rejected by the set",
1820-
obj: caps([]string{"SYS_TIME", "SYS_TIME"}, nil),
1830+
obj: caps([]string{"SETUID", "SETUID"}, nil),
18211831
want: field.ErrorList{field.Duplicate(capsPath.Child("add").Index(1), nil)},
18221832
},
18231833

‎cmd/atelet/internal/apivalidation/workloadspec.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -78,11 +78,11 @@ func ValidateCustom_Container_VolumeMounts(_ context.Context, _ operation.Operat
7878
}
7979

8080
func ValidateCustom_Capabilities_Add(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []string) field.ErrorList {
81-
return resources.ValidateCapabilities(fldPath, value, false)
81+
return resources.ValidateAddCapabilities(fldPath, value)
8282
}
8383

8484
func ValidateCustom_Capabilities_Drop(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ []string) field.ErrorList {
85-
return resources.ValidateCapabilities(fldPath, value, true)
85+
return resources.ValidateDropCapabilities(fldPath, value)
8686
}
8787

8888
func ValidateCustom_HttpGetAction_Path(_ context.Context, _ operation.Operation, fldPath *field.Path, value, _ *string) field.ErrorList {

‎cmd/atelet/main_test.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -251,7 +251,7 @@ func TestPrepareOCIBundlesFillsContainerSpec(t *testing.T) {
251251
Args: []string{"serve"},
252252
Env: []*ateletpb.EnvVar{{Name: "GREETING", Value: "hello"}},
253253
SecurityContext: &ateletpb.SecurityContext{
254-
Capabilities: &ateletpb.Capabilities{Add: []string{"NET_ADMIN"}, Drop: []string{"KILL"}},
254+
Capabilities: &ateletpb.Capabilities{Add: []string{"SETUID"}, Drop: []string{"KILL"}},
255255
},
256256
Resources: &ateletpb.ResourceRequirements{Limits: &ateletpb.Resources{MemoryBytes: 1 << 30, CpuMillicores: 500}},
257257
}},
@@ -265,7 +265,7 @@ func TestPrepareOCIBundlesFillsContainerSpec(t *testing.T) {
265265
want := []*ateworkerpb.ContainerSpec{{
266266
Args: []string{"/bin/app", "serve"},
267267
Env: []string{"GREETING=hello", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"},
268-
Capabilities: []string{"CAP_AUDIT_WRITE", "CAP_NET_ADMIN", "CAP_NET_BIND_SERVICE"},
268+
Capabilities: []string{"CAP_AUDIT_WRITE", "CAP_NET_BIND_SERVICE", "CAP_SETUID"},
269269
Resources: &ateworkerpb.ResourceRequirements{Limits: &ateworkerpb.Resources{MemoryBytes: 1 << 30, CpuMillicores: 500}},
270270
Cwd: "/",
271271
}}

‎cmd/atelet/oci_test.go‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -266,16 +266,16 @@ func TestResolveCapabilities(t *testing.T) {
266266
want: withoutCaps(wantDefaultCapabilities, "CAP_NET_BIND_SERVICE", "CAP_AUDIT_WRITE"),
267267
}, {
268268
name: "add grants on top of the default set",
269-
caps: &ateletpb.Capabilities{Add: []string{"SYS_ADMIN"}},
270-
want: withCaps(wantDefaultCapabilities, "CAP_SYS_ADMIN"),
269+
caps: &ateletpb.Capabilities{Add: []string{"SYS_CHROOT"}},
270+
want: withCaps(wantDefaultCapabilities, "CAP_SYS_CHROOT"),
271271
}, {
272272
name: "drop ALL clears the default set",
273273
caps: &ateletpb.Capabilities{Drop: []string{"ALL"}},
274274
want: nil,
275275
}, {
276276
name: "drop ALL with add gives an exact set",
277-
caps: &ateletpb.Capabilities{Drop: []string{"ALL"}, Add: []string{"NET_ADMIN", "CHOWN"}},
278-
want: []string{"CAP_CHOWN", "CAP_NET_ADMIN"},
277+
caps: &ateletpb.Capabilities{Drop: []string{"ALL"}, Add: []string{"SETUID", "CHOWN"}},
278+
want: []string{"CAP_CHOWN", "CAP_SETUID"},
279279
}, {
280280
// Drop applies first, so naming a capability in both grants it.
281281
name: "add wins over drop",

‎cmd/ateworker-microvm/internal/kata/overlay_linux.go‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -431,9 +431,12 @@ type CreateSandboxOpts struct {
431431
// volumes every container runs on). Mirrors kata startSandbox.
432432
func (a *AgentClient) CreateSandboxForActor(ctx context.Context, opts CreateSandboxOpts) error {
433433
storages := []*agentpb.Storage{{
434-
Driver: virtioFSDriver,
435-
Source: FsTag,
436-
Fstype: typeVirtioFS,
434+
Driver: virtioFSDriver,
435+
Source: FsTag,
436+
Fstype: typeVirtioFS,
437+
// nodev on the share itself also covers its automounted submounts,
438+
// which a nodev remount of each container bind does not reach.
439+
Options: []string{"nodev"},
437440
MountPoint: guestSharedDir,
438441
}}
439442
return a.CreateSandbox(ctx, &agentpb.CreateSandboxRequest{

0 commit comments

Comments
 (0)