Add generated seccomp profiles for workers - #2750

Every worker drops ALL container-runtime default capabilities, adds back a
fixed runtime-specific set, and sets allowPrivilegeEscalation: false.

gVisor workers keep 9 capabilities: NET_ADMIN, SYS_ADMIN, SETUID, SETGID,
DAC_OVERRIDE, FOWNER, CHOWN, NET_RAW and SETFCAP. MKNOD, SETPCAP, SYS_CHROOT
and SYS_PTRACE are not granted: runsc's directfs user namespace supplies its
own SYS_CHROOT and SYS_PTRACE, and 0:0 overlay whiteouts have not needed
MKNOD since Linux 5.8. SETFCAP stays because runsc cannot start its gofer
without it (fork/exec /proc/self/exe: operation not permitted).

Micro-VM workers keep 12: virtiofsd's normal nine child capabilities plus
NET_ADMIN, SYS_ADMIN and SETPCAP. SETPCAP is kept so virtiofsd's serving
child can drop its bounding set with PR_CAPBSET_DROP, which libcap-ng only
issues while CAP_SETPCAP is effective; without it the child keeps the
worker's bounding set, and a virtiofsd built against unpatched libcap-ng
refuses to start. NET_RAW, SYS_CHROOT and SYS_PTRACE are not granted.

The e2e test now checks what the kernel enforces, not only the rendered
PodSpec: a testserver procstatus subcommand runs in an ephemeral container
sharing the worker's PID namespace and reports /proc/<pid>/status for the
worker and its sandbox children. The test asserts the worker's permitted,
effective and bounding sets equal the fixed mask with nothing ambient and
no_new_privs on, that the sandbox children run under their own seccomp
filters, and on micro-VM that virtiofsd's child holds its nine capabilities
with an empty bounding set.

TestVirtiofsdArgs only rejects arguments that add a capability back to
virtiofsd (--modcaps=+...); dropping one is allowed.

Fixes agent-substrate#2422. Worker seccomp is split out to agent-substrate#2749 (profiles in agent-substrate#2750).

AI-assisted.

Every worker drops ALL container-runtime default capabilities, adds back a
fixed runtime-specific set, and sets allowPrivilegeEscalation: false.

gVisor workers keep 9 capabilities: NET_ADMIN, SYS_ADMIN, SETUID, SETGID,
DAC_OVERRIDE, FOWNER, CHOWN, NET_RAW and SETFCAP. MKNOD, SETPCAP, SYS_CHROOT
and SYS_PTRACE are not granted: runsc's directfs user namespace supplies its
own SYS_CHROOT and SYS_PTRACE, and 0:0 overlay whiteouts have not needed
MKNOD since Linux 5.8. SETFCAP stays because runsc cannot start its gofer
without it (fork/exec /proc/self/exe: operation not permitted).

MicroVM workers keep 12: virtiofsd's normal nine child capabilities plus
NET_ADMIN, SYS_ADMIN and SETPCAP. SETPCAP is kept so virtiofsd's serving
child can drop its bounding set with PR_CAPBSET_DROP, which libcap-ng only
issues while CAP_SETPCAP is effective; without it the child keeps the
worker's bounding set, and a virtiofsd built against unpatched libcap-ng
refuses to start. NET_RAW, SYS_CHROOT and SYS_PTRACE are not granted.

The e2e test now checks what the kernel enforces, not only the rendered
PodSpec: a testserver procstatus subcommand runs in an ephemeral container
sharing the worker's PID namespace and reports /proc/<pid>/status for the
worker and its sandbox children. The test asserts the worker's permitted,
effective and bounding sets equal the fixed mask with nothing ambient and
no_new_privs on, that the sandbox children run under their own seccomp
filters, and on microVM that virtiofsd's child holds its nine capabilities
with an empty bounding set.

TestVirtiofsdArgs only rejects arguments that add a capability back to
virtiofsd (--modcaps=+...); dropping one is allowed.

Fixes agent-substrate#2422. Worker seccomp is split out to agent-substrate#2749 (profiles in agent-substrate#2750).

AI-assisted.

Every worker drops ALL container-runtime default capabilities, adds back a
fixed runtime-specific set, and sets allowPrivilegeEscalation: false.

gVisor workers keep 9 capabilities: NET_ADMIN, SYS_ADMIN, SETUID, SETGID,
DAC_OVERRIDE, FOWNER, CHOWN, NET_RAW and SETFCAP. MKNOD, SETPCAP, SYS_CHROOT
and SYS_PTRACE are not granted: runsc's directfs user namespace supplies its
own SYS_CHROOT and SYS_PTRACE, and 0:0 overlay whiteouts have not needed
MKNOD since Linux 5.8. SETFCAP stays because runsc cannot start its gofer
without it (fork/exec /proc/self/exe: operation not permitted).

MicroVM workers keep 12: virtiofsd's normal nine child capabilities plus
NET_ADMIN, SYS_ADMIN and SETPCAP. SETPCAP is kept so virtiofsd's serving
child can drop its bounding set with PR_CAPBSET_DROP, which libcap-ng only
issues while CAP_SETPCAP is effective; without it the child keeps the
worker's bounding set, and a virtiofsd built against unpatched libcap-ng
refuses to start. NET_RAW, SYS_CHROOT and SYS_PTRACE are not granted.

The e2e test now checks what the kernel enforces, not only the rendered
PodSpec: a testserver procstatus subcommand runs in an ephemeral container
sharing the worker's PID namespace and reports /proc/<pid>/status for the
worker and its sandbox children. The test asserts the worker's permitted,
effective and bounding sets equal the fixed mask with nothing ambient and
no_new_privs on, that the sandbox children run under their own seccomp
filters, and on microVM that virtiofsd's child holds its nine capabilities
with an empty bounding set.

TestVirtiofsdArgs only rejects arguments that add a capability back to
virtiofsd (--modcaps=+...); dropping one is allowed.

Fixes agent-substrate#2422. Worker seccomp is split out to agent-substrate#2749 (profiles in agent-substrate#2750).

AI-assisted.

Signed-off-by: Tom O'Rourke <[email protected]>

Every worker drops ALL container-runtime default capabilities, adds back a
fixed runtime-specific set, and sets allowPrivilegeEscalation: false.

gVisor workers keep 9 capabilities: NET_ADMIN, SYS_ADMIN, SETUID, SETGID,
DAC_OVERRIDE, FOWNER, CHOWN, NET_RAW and SETFCAP. MKNOD, SETPCAP, SYS_CHROOT
and SYS_PTRACE are not granted: runsc's directfs user namespace supplies its
own SYS_CHROOT and SYS_PTRACE, and 0:0 overlay whiteouts have not needed
MKNOD since Linux 5.8. SETFCAP stays because runsc cannot start its gofer
without it (fork/exec /proc/self/exe: operation not permitted).

MicroVM workers keep 12: virtiofsd's normal nine child capabilities plus
NET_ADMIN, SYS_ADMIN and SETPCAP. SETPCAP is kept so virtiofsd's serving
child can drop its bounding set with PR_CAPBSET_DROP, which libcap-ng only
issues while CAP_SETPCAP is effective; without it the child keeps the
worker's bounding set, and a virtiofsd built against unpatched libcap-ng
refuses to start. NET_RAW, SYS_CHROOT and SYS_PTRACE are not granted.

The e2e test now checks what the kernel enforces, not only the rendered
PodSpec: a testserver procstatus subcommand runs in an ephemeral container
sharing the worker's PID namespace and reports /proc/<pid>/status for the
worker and its sandbox children. The test asserts the worker's permitted,
effective and bounding sets equal the fixed mask with nothing ambient and
no_new_privs on, that the sandbox children run under their own seccomp
filters, and on microVM that virtiofsd's child holds its nine capabilities
with an empty bounding set.

TestVirtiofsdArgs only rejects arguments that add a capability back to
virtiofsd (--modcaps=+...); dropping one is allowed.

Fixes agent-substrate#2422. Worker seccomp is split out to agent-substrate#2749 (profiles in agent-substrate#2750).

AI-assisted.

Signed-off-by: Tom O'Rourke <[email protected]>

…trate#2428)

## Summary

Use fixed runtime-specific worker capability sets, following maintainer
feedback that Substrate should own the minimum permissions needed to
implement its sandboxes. Every worker drops ALL container-runtime
default capabilities, adds back its runtime-specific set, and sets
`allowPrivilegeEscalation: false`.

| Component | Previous set | New set | Removed |
| --- | --- | --- | --- |
| gVisor worker | 13 capabilities | 9 capabilities (`0x802030cb`) |
`MKNOD`, `SETPCAP`, `SYS_CHROOT`, `SYS_PTRACE` |
| microVM worker | 15 capabilities | 11 capabilities (`0x882011db`) |
`DAC_READ_SEARCH`, `NET_RAW`, `SYS_CHROOT`, `SYS_PTRACE` |
| virtiofsd serving child | 9 capabilities | Its 8 default capabilities
(`0x880000db`), empty bounding set | `DAC_READ_SEARCH` |

Both workers still run as root and retain `SYS_ADMIN` and `NET_ADMIN`.
This is incremental worker hardening; the existing
deployment-permissions requirements still apply.

Fixes agent-substrate#2422. That issue asked for these settings to be configurable per
pool; following review, the workers get fixed, reviewed defaults
instead. Worker seccomp is split out to agent-substrate#2749, with the profiles in
draft agent-substrate#2750.

## Changes

- Make the tested reduced capability sets fixed defaults rather than
exposing capability drops per pool.
- Remove the proposed WorkerPool security API, its validation and
generated schema changes, and the example worker seccomp profiles and
generator, which move to agent-substrate#2750. Outer worker seccomp and AppArmor remain
Unconfined.
- Keep `SETPCAP` on microVM workers so virtiofsd's serving child can
drop its bounding set with `PR_CAPBSET_DROP`, which libcap-ng only
issues while `CAP_SETPCAP` is effective. Keep `SETFCAP` on gVisor
workers: runsc cannot start its gofer without it.
- Run virtiofsd with `--inode-file-handles=never` and stop granting
`DAC_READ_SEARCH`, which virtiofsd only needs to open file handles. The
shared directory is on the worker's overlayfs rootfs, which cannot
produce decodable file handles, so virtiofsd already used `O_PATH` fds.
virtiofsd also explicitly selects its namespace sandbox and seccomp
filter.
- Verify exact runtime defaults and reconciliation after Deployment
security settings drift.
- `TestWorkerSecurityFilesystem` checks what the kernel enforces, not
only the rendered PodSpec. A testserver `procstatus` subcommand runs in
an ephemeral container in the worker's PID namespace and reports
`/proc/<pid>/status` by process name. The test asserts the worker's
permitted, effective and bounding sets equal the fixed mask with nothing
ambient and `NoNewPrivs: 1`; runsc's gofer and sandbox run under
seccomp; on microVM, virtiofsd's serving child holds its eight
capabilities with an empty bounding set, and every cloud-hypervisor
thread except main runs under seccomp. It still covers non-root-owned
files through pause/resume and suspend/resume, and on microVM non-root
access, cross-UID denial and device-node creation.
- Update the API guide and security documentation, including when the
change rolls worker pools, and the image cache README and comments that
still described privileged workers needing `CAP_MKNOD`. Creating the 0:0
overlay whiteouts has not needed `CAP_MKNOD` since Linux 5.8.
- Rebased on upstream `main` at `0ddbf78d`.

Actor container capabilities remain configured separately through
`ActorTemplate.containers[].securityContext.capabilities`. There is no
new WorkerPool API or opt-in configuration.

## Upgrade

The new capability sets change the worker pod template, so every
WorkerPool rolls when `ate-controller` is upgraded, not only when its
worker image changes. Follow the [upgrade
runbook](docs/guides/upgrade.md) in its order:

1. Do steps 1 and 2 as usual.
2. In step 3, move every WorkerPool to the new worker image before you
upgrade `ate-controller`. A microVM pool left on an older image fails at
the next step: its virtiofsd exits with `can't apply the child
capabilities`, and actors on it cannot start or restore. Check that no
pool is left on the old image:
   ```bash
kubectl get workerpools -A -o
custom-columns='NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.sandboxClasses[0].name,IMAGE:.spec.workerImage'
   ```
3. In step 4b, `go run ./cmd/ate-setup deploy ate-controller --config
~/ate-upgrade/new.yaml`. Unlike the runbook says for other releases,
every pool rolls once here, with the same `SIGTERM` and 30 minutes per
actor as step 3. Watch it with `kubectl get deploy -A -l
ate.dev/worker-pool`, then check one worker of each class:
   ```bash
kubectl -n $NS get pod <worker> -o
jsonpath='{.spec.containers[0].securityContext.capabilities.add}'
   ```
gVisor lists 9 capabilities and microVM 11, without `DAC_READ_SEARCH`.
Start one microVM actor to confirm virtiofsd comes up.
4. Finish steps 4c to 4e as usual.

Rollback: roll back step 4 first, as the runbook says. The pools roll
again to the old capability set, which runs both the new and old worker
images. Then roll back step 3. Rolling back step 3 alone, with the new
`ate-controller` still running, puts old microVM images on the new
template and breaks them.

Timing: if this lands in 1.0.0 there is no skew issue, because upgrades
from 0.x are reinstalls. If it lands after 1.0.0, it changes the worker
pod template within a major, which the [version skew
policy](docs/reference/version-skew-policy.md) does not allow, and needs
the `breaking-change` label.

## Validation

On the current revision (`86e684c4`, on upstream `main` at `0ddbf78d`;
`86e684c4` only changes docs, so the checks below ran on `6714b09b`):

- `hack/verify-all.sh`, `go vet` and `go test -race` on the changed
packages: passed.
- Live on EKS (Kubernetes 1.37, AL2023, kernel 6.18), reinstalled from
this revision with the v1 CRDs through `ate-setup deploy`, gVisor on
`m6i.xlarge` and microVM on `i4i.metal`. These passed on both runtimes:
`TestWorkerSecurityFilesystem`, `TestGoldenSnapshotLifecycle`,
`TestActorSnapshotLifecycle`, `TestPauseResumeActor`,
`TestRevertActorToMostRecentDurableSnapshot`,
`TestGracefulWorkerTerminationSuspend`, `TestDurableDirLifecycle`,
`TestMultipleDurableDirLifecycle`, `TestImageLayerOwnership`,
`TestImageUserAndWorkdir`, `TestCombinedVolumes` (external cases
skipped), `TestActorEgress`, `TestActorEgressHTTPS` and
`TestActorIdentity_AfterRestore_IsOwnID_NotGolden`, plus
`TestMicroVMClockAfterDelayedRestore` on microVM.
- Process state read from `/proc` on the nodes:
- gVisor worker `0x802030cb`; runsc sandbox `0x8001f` and gofer
`0x4001f`, both with seccomp.
- microVM worker `0x882011db`; virtiofsd serving child `0x880000db` with
bounding set 0 and `Seccomp: 2`; cloud-hypervisor threads other than
main at `Seccomp: 2`.
  - Both workers have no ambient capabilities and `NoNewPrivs: 1`.
- Upgrade order: a WorkerPool on a microVM worker image from `main`
under this `ate-controller` fails with `can't apply the child
capabilities`, and recovers when moved back to this revision's image.
- A syscall probe in the gVisor worker container gets `EPERM` from
`userfaultfd`, which needs `CAP_SYS_PTRACE` while
`vm.unprivileged_userfaultfd` is 0. It reached the kernel with the
previous capability set.
- The kind e2e lanes in `pr-workflow` cover both sandbox classes; they
need a maintainer to approve the run on this fork PR.

[Test plan and findings for the earlier opt-in revision
(PDF)](https://github.com/tjorourke/substrate/blob/7e3040822b85ac0de2703bc8268e4e7f78f26e11/docs/validation/worker-security-test-report.pdf)

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

AI-assisted.

---------

Signed-off-by: Tom O'Rourke <[email protected]>

## Summary

Use fixed runtime-specific worker capability sets, following maintainer
feedback that Substrate should own the minimum permissions needed to
implement its sandboxes. Every worker drops ALL container-runtime
default capabilities, adds back its runtime-specific set, and sets
`allowPrivilegeEscalation: false`.

| Component | Previous set | New set | Removed |
| --- | --- | --- | --- |
| gVisor worker | 13 capabilities | 9 capabilities (`0x802030cb`) |
`MKNOD`, `SETPCAP`, `SYS_CHROOT`, `SYS_PTRACE` |
| microVM worker | 15 capabilities | 11 capabilities (`0x882011db`) |
`DAC_READ_SEARCH`, `NET_RAW`, `SYS_CHROOT`, `SYS_PTRACE` |
| virtiofsd serving child | 9 capabilities | Its 8 default capabilities
(`0x880000db`), empty bounding set | `DAC_READ_SEARCH` |

Both workers still run as root and retain `SYS_ADMIN` and `NET_ADMIN`.
This is incremental worker hardening; the existing
deployment-permissions requirements still apply.

Fixes #2422. That issue asked for these settings to be configurable per
pool; following review, the workers get fixed, reviewed defaults
instead. Worker seccomp is split out to #2749, with the profiles in
draft #2750.

## Changes

- Make the tested reduced capability sets fixed defaults rather than
exposing capability drops per pool.
- Remove the proposed WorkerPool security API, its validation and
generated schema changes, and the example worker seccomp profiles and
generator, which move to #2750. Outer worker seccomp and AppArmor remain
Unconfined.
- Keep `SETPCAP` on microVM workers so virtiofsd's serving child can
drop its bounding set with `PR_CAPBSET_DROP`, which libcap-ng only
issues while `CAP_SETPCAP` is effective. Keep `SETFCAP` on gVisor
workers: runsc cannot start its gofer without it.
- Run virtiofsd with `--inode-file-handles=never` and stop granting
`DAC_READ_SEARCH`, which virtiofsd only needs to open file handles. The
shared directory is on the worker's overlayfs rootfs, which cannot
produce decodable file handles, so virtiofsd already used `O_PATH` fds.
virtiofsd also explicitly selects its namespace sandbox and seccomp
filter.
- Verify exact runtime defaults and reconciliation after Deployment
security settings drift.
- `TestWorkerSecurityFilesystem` checks what the kernel enforces, not
only the rendered PodSpec. A testserver `procstatus` subcommand runs in
an ephemeral container in the worker's PID namespace and reports
`/proc/<pid>/status` by process name. The test asserts the worker's
permitted, effective and bounding sets equal the fixed mask with nothing
ambient and `NoNewPrivs: 1`; runsc's gofer and sandbox run under
seccomp; on microVM, virtiofsd's serving child holds its eight
capabilities with an empty bounding set, and every cloud-hypervisor
thread except main runs under seccomp. It still covers non-root-owned
files through pause/resume and suspend/resume, and on microVM non-root
access, cross-UID denial and device-node creation.
- Update the API guide and security documentation, including when the
change rolls worker pools, and the image cache README and comments that
still described privileged workers needing `CAP_MKNOD`. Creating the 0:0
overlay whiteouts has not needed `CAP_MKNOD` since Linux 5.8.
- Rebased on upstream `main` at `0ddbf78d`.

Actor container capabilities remain configured separately through
`ActorTemplate.containers[].securityContext.capabilities`. There is no
new WorkerPool API or opt-in configuration.

## Upgrade

The new capability sets change the worker pod template, so every
WorkerPool rolls when `ate-controller` is upgraded, not only when its
worker image changes. Follow the [upgrade
runbook](docs/guides/upgrade.md) in its order:

1. Do steps 1 and 2 as usual.
2. In step 3, move every WorkerPool to the new worker image before you
upgrade `ate-controller`. A microVM pool left on an older image fails at
the next step: its virtiofsd exits with `can't apply the child
capabilities`, and actors on it cannot start or restore. Check that no
pool is left on the old image:
   ```bash
kubectl get workerpools -A -o
custom-columns='NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.sandboxClasses[0].name,IMAGE:.spec.workerImage'
   ```
3. In step 4b, `go run ./cmd/ate-setup deploy ate-controller --config
~/ate-upgrade/new.yaml`. Unlike the runbook says for other releases,
every pool rolls once here, with the same `SIGTERM` and 30 minutes per
actor as step 3. Watch it with `kubectl get deploy -A -l
ate.dev/worker-pool`, then check one worker of each class:
   ```bash
kubectl -n $NS get pod <worker> -o
jsonpath='{.spec.containers[0].securityContext.capabilities.add}'
   ```
gVisor lists 9 capabilities and microVM 11, without `DAC_READ_SEARCH`.
Start one microVM actor to confirm virtiofsd comes up.
4. Finish steps 4c to 4e as usual.

Rollback: roll back step 4 first, as the runbook says. The pools roll
again to the old capability set, which runs both the new and old worker
images. Then roll back step 3. Rolling back step 3 alone, with the new
`ate-controller` still running, puts old microVM images on the new
template and breaks them.

Timing: if this lands in 1.0.0 there is no skew issue, because upgrades
from 0.x are reinstalls. If it lands after 1.0.0, it changes the worker
pod template within a major, which the [version skew
policy](docs/reference/version-skew-policy.md) does not allow, and needs
the `breaking-change` label.

## Validation

On the current revision (`86e684c4`, on upstream `main` at `0ddbf78d`;
`86e684c4` only changes docs, so the checks below ran on `6714b09b`):

- `hack/verify-all.sh`, `go vet` and `go test -race` on the changed
packages: passed.
- Live on EKS (Kubernetes 1.37, AL2023, kernel 6.18), reinstalled from
this revision with the v1 CRDs through `ate-setup deploy`, gVisor on
`m6i.xlarge` and microVM on `i4i.metal`. These passed on both runtimes:
`TestWorkerSecurityFilesystem`, `TestGoldenSnapshotLifecycle`,
`TestActorSnapshotLifecycle`, `TestPauseResumeActor`,
`TestRevertActorToMostRecentDurableSnapshot`,
`TestGracefulWorkerTerminationSuspend`, `TestDurableDirLifecycle`,
`TestMultipleDurableDirLifecycle`, `TestImageLayerOwnership`,
`TestImageUserAndWorkdir`, `TestCombinedVolumes` (external cases
skipped), `TestActorEgress`, `TestActorEgressHTTPS` and
`TestActorIdentity_AfterRestore_IsOwnID_NotGolden`, plus
`TestMicroVMClockAfterDelayedRestore` on microVM.
- Process state read from `/proc` on the nodes:
- gVisor worker `0x802030cb`; runsc sandbox `0x8001f` and gofer
`0x4001f`, both with seccomp.
- microVM worker `0x882011db`; virtiofsd serving child `0x880000db` with
bounding set 0 and `Seccomp: 2`; cloud-hypervisor threads other than
main at `Seccomp: 2`.
  - Both workers have no ambient capabilities and `NoNewPrivs: 1`.
- Upgrade order: a WorkerPool on a microVM worker image from `main`
under this `ate-controller` fails with `can't apply the child
capabilities`, and recovers when moved back to this revision's image.
- A syscall probe in the gVisor worker container gets `EPERM` from
`userfaultfd`, which needs `CAP_SYS_PTRACE` while
`vm.unprivileged_userfaultfd` is 0. It reached the kernel with the
previous capability set.
- The kind e2e lanes in `pr-workflow` cover both sandbox classes; they
need a maintainer to approve the run on this fork PR.

[Test plan and findings for the earlier opt-in revision
(PDF)](https://github.com/tjorourke/substrate/blob/7e3040822b85ac0de2703bc8268e4e7f78f26e11/docs/validation/worker-security-test-report.pdf)

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

AI-assisted.

---------

Signed-off-by: Tom O'Rourke <[email protected]>
(cherry picked from commit bd1f188)
Signed-off-by: Eitan Yarmush <[email protected]>