Worker seccomp: replace Unconfined with a Substrate-owned profile #2749

Split out of #2428, where Benjamin Elder (@BenTheElder) asked for worker seccomp to be handled as a separate change.

Current state

Both worker classes run with seccompProfile: Unconfined. RuntimeDefault is not an option: it denies pivot_root(2) whatever capabilities the worker holds, and runsc's gofer and virtiofsd both call it while building their own sandboxes (#1368, #1369). AppArmor is Unconfined too, because the default AppArmor profile denies mount.

With #2428 each worker keeps a fixed capability set, 9 for gVisor and 11 for micro-VM, including SYS_ADMIN and NET_ADMIN, and runs with no_new_privs. Nothing limits which syscalls it can make. A probe started in a gVisor worker container on EKS with the #2428 defaults reached the kernel with bpf, perf_event_open, io_uring_setup, keyctl, add_key, request_key and open_by_handle_at, and opened AF_VSOCK and AF_ALG sockets. With SYS_ADMIN held, the worker can load BPF programs.

The inner sandboxes already filter themselves: runsc's gofer and sentry, virtiofsd (--seccomp=kill), and cloud-hypervisor's VMM, vCPU and device threads all install their own filters. The gap is the worker process itself, the cloud-hypervisor main thread, and everything before those filters are installed.

A profile that works

Draft #2750 adds generated amd64 and arm64 Localhost profiles: moby's default allow list (seccomp/v0.2.4) plus the namespace and mount syscalls the workers need (clone, clone3, unshare, setns, mount, umount2, pivot_root, chroot and the new mount API). Every other syscall returns EPERM. chroot stays even though the workers no longer hold SYS_CHROOT, because runsc's gofer calls it inside its own user namespace.

Tested on EKS (Kubernetes 1.37, AL2023, kernel 6.18, containerd 2.2.7) on top of #2428 rebased onto c4518a13, with the amd64 profile applied to every worker through a test-only controller change:

• TestWorkerSecurityFilesystem passed on gVisor (m6i.xlarge) and micro-VM (i4i.metal): golden creation, pause/resume and suspend/resume, with the ownership, mode and contents checks.
• On the nodes, the worker ran in seccomp mode 2, and runsc's gofer and boot processes and the virtiofsd serving child still added their own filters on top.
• The same probe got EPERM for bpf, perf_event_open, io_uring_setup, keyctl, add_key, request_key, userfaultfd, open_by_handle_at, init_module, kexec_load, sethostname, AF_VSOCK and AF_ALG. AF_PACKET and netlink sockets, unshare, mount, fsopen, pivot_root and setns still reached the kernel.

The arm64 profile is generated from the same lists but has not been run live.

Open question: getting the profile onto nodes

A Localhost profile has to be under the kubelet's seccomp directory on a node before a worker can start there. The options I can see:

1. atelet installs it. atelet already runs on every node that hosts workers. It writes the profile for its architecture to a versioned name under the kubelet's seccomp directory, and the controller sets Localhost with that name. This keeps the fixed-defaults approach from Tighten worker permissions with runtime-specific defaults #2428 and adds no API. Costs: atelet needs a hostPath into the kubelet directory, whose location varies by distribution and so has to be configurable; a worker cannot start on a node until atelet has written the file; and when the profile changes, atelet has to roll out before the controller, with older versions left in place for workers that still use them.
2. The worker applies the filter itself at startup, before it starts a sandbox, as runsc and virtiofsd do. The filter is inherited by runsc, virtiofsd and cloud-hypervisor. There are no files on nodes, it ships with the binaries it confines, and the pod spec stays Unconfined. Costs: Substrate needs a BPF builder (none is vendored, and libseccomp needs cgo), and processes started with kubectl exec in the worker container are not covered.
3. Security Profiles Operator distributes a SeccompProfile. Least code in Substrate, but every install needs another operator.

I'd go with option 1. It is the standard Kubernetes mechanism, keeps the profile visible in the pod spec, and atelet is already the per-node component. Happy to go another way.

Also to decide:

• One profile for both runtimes, as drafted, or one per runtime.
• How to keep the base list in step with moby. go generate ./hack/seccomp regenerates the profiles from the lists in hack/seccomp/syscalls.go.
• Whether AppArmor should follow the same route. A custom AppArmor profile has the same distribution problem, with the extra step of loading it into each node's kernel.