helm: default to the documented capability set instead of privileged: true #665

Since #649 a plain helm install gives a working UPF, with securityContext.privileged: true as the default and, in the README and the values comments, a finer-grained alternative:

securityContext:
  capabilities:
    add: [NET_ADMIN, SYS_ADMIN, SYS_RESOURCE, BPF, PERFMON]

I ran the chart with that set as the default and nothing else changed, on a three-node kind cluster with net.ipv4.ip_forward allowed on the kubelets: the pod starts, the entrypoint mounts bpffs and debugfs under CAP_SYS_ADMIN, the UPF serves on :8080 and :9090, and it comes back after scaling and after a node drain. The capability set is enough for what the image does today.

Why the smaller set should be the default

• privileged: true is not "these capabilities plus a bit": it is every capability (SYS_MODULE, SYS_RAWIO, SYS_PTRACE, DAC_READ_SEARCH, MKNOD and the rest), no device cgroup so every host device node is reachable, /proc and /sys unmasked and writable, and the runtime's seccomp and AppArmor profiles switched off. A compromised privileged container is root on the node. With the capability set, the container keeps the device cgroup, the masked paths, seccomp and AppArmor, and gets exactly what the data plane uses: NET_ADMIN for XDP and routes, BPF and PERFMON for loading and attaching programs, SYS_RESOURCE for RLIMIT_MEMLOCK, SYS_ADMIN for the two mounts.
• A default is what most installs run. Everyone who installs the chart as shipped gets the privileged container; only the few who read the comment get the smaller set. It should be the other way round: the smaller set by default, and the larger one for whoever needs it.
• Policy and review. Admission policies (Kyverno, Gatekeeper, the common CIS and NSA checks) and chart scanners flag privileged: true first, before anything else; a capability list is reviewable, a privileged flag is a blanket. Operators who have to justify what runs in their clusters can justify five named capabilities.
• The chart already supports it: the change is the two lines above in values.yaml and turning the comment around, with privileged: true as the documented fallback.

Where the capability set is not enough, and what to do there

• Nodes with AppArmor enforcing the runtime's default profile (docker-default, containerd's cri-containerd.apparmor.d): that profile denies mount inside the container, so the entrypoint's bpffs and debugfs mounts fail where they pass on a privileged container, which runs unconfined. The remedy is to set the container's AppArmor profile to unconfined (securityContext.appArmorProfile: {type: Unconfined} on Kubernetes 1.30+, the container.apparmor.security.beta.kubernetes.io/<name> annotation before that), which the chart could set alongside the capabilities; I have not tested that combination, kind nodes do not load AppArmor profiles.
• Custom seccomp profiles tighter than the runtime default, which block the mount or bpf syscalls: the privileged container runs unconfined, the capability set does not. Those environments need their profile adjusted or the fallback.
• Kernels before 5.8 do not have CAP_BPF and CAP_PERFMON; eBPF there needs CAP_SYS_ADMIN, which the set includes, but a runtime may refuse the unknown capability names. The fallback covers that.
• Sandboxed runtimes (gVisor, Kata) and namespaces under the Pod Security baseline or restricted profile reject both the capability set and the privileged container, so nothing changes for them.

Proposal: make the capability set the chart's default securityContext, keep privileged: true in the README as the fallback for the environments above, and note the AppArmor case next to it.