Repository navigation
Harden control plane manifests: securityContext, non-root images, narrower RBAC #2936
Description
We run Substrate on GKE and install it with Kustomize over manifests/ate-install/, reconciled by Flux. We hardened the control plane manifests before putting it in a shared cluster, and would like to send the changes upstream so every install method in the extensible install proposal inherits them.
We have run all of this on v0.5.0. I checked each item against main at 90de075, and each is still missing there. I'd send them as the three PRs below, each verified on kind first.
1. securityContext and non-root images
Several control plane containers set no securityContext: ate-api-server, ate-controller, both containers in atenet-router, and ext-proc in atenet-egress. ko builds every image on gcr.io/distroless/static-debian13:latest, which runs as root.
Proposed:
- Switch
defaultBaseImagein.ko.yamltostatic-debian13:nonroot, pinned by digest.ateletkeeps itsbase-debian13override. - Set
allowPrivilegeEscalation: falseandcapabilities.drop: [ALL]on every control plane container. - Set
runAsNonRoot: trueonate-api-server,ate-controller,atenet-router,podcertificate-controller,ate-idp-server, and theobjectstore-pluginsidecar inate-api-server. The nonroot base runs as UID 65532, so these need norunAsUser.
Exceptions:
ate-api-serverkeepsNET_BIND_SERVICEbecause it binds:443.podcertificate-controlleralready does the same.- The
envoycontainer inatenet-routeralso needsrunAsUser: 65532, because the Envoy base image sets no user.atenet-egressalready does this. ate-controllercan also takereadOnlyRootFilesystem: true. It writes nothing to disk and runs no webhook server.ateletand itsobjectstore-pluginsidecar stay root and get onlyallowPrivilegeEscalation: false.
The base image change and runAsNonRoot have to go in the same PR. With today's root base, runAsNonRoot fails every pod with CreateContainerConfigError. ate-idp-server is new since v0.5.0, so we have not run it non-root yet. I'll check it on kind before opening the PR.
2. ate-controller reads every Secret in the cluster
The marker in egressmitmtrust_controller.go generates a ClusterRole rule with get/list/watch on all Secrets. The controller reads exactly one Secret, egress-mitm-ca-pool, and managerOptions in main.go already field-selects the cache to it.
We moved the rule to the namespaced Role and added resourceNames: [egress-mitm-ca-pool]. list and watch still work with resourceNames here, because the apiserver treats an exact metadata.name field selector as the request name.
Upstream this would be a marker change (namespace= and resourceNames=) plus regenerating role.yaml.
3. podcertificate-controller runs as the default ServiceAccount
pod-certificate-controller.yaml sets serviceAccountName: default, and both podcert bindings name default. Any pod created in podcertificate-controller-system without a serviceAccountName gets sign and attest permissions for every Substrate signer.
Proposed: add a podcertificate-controller ServiceAccount, and use it on the Deployment and in both bindings.
Related: the controller has no readiness probe. If its RBAC is wrong, the pod stays Ready and other pods' podCertificate volumes never fill. I can open a separate issue for that if it's wanted.
Activity
stshrive commentedon Oct 20, 2026
At first glance, I don't see any issue with using nonroot versions of the images or limiting the secrets read for the egress-mitm-ca-pool, however I think there are some API-level considerations that need to be reviewed for some of the suggested changes.
FYI Julian Gutierrez Oschmann (@juli4n), Lior Lieberman (@LiorLieberman), Benjamin Elder (@BenTheElder), Taahir Ahmed (@ahmedtd), Greg Castle (@destijl)
mikedickey commentedon Oct 21, 2026
Thanks Steven Shriver (@stshrive) , I submitted PRs for the three changes (linked to this issue). I ran all the tests suites locally (kind) with good results, and we also have been using these in our GKE deployment of Substrate. All the changes are quite small and isolated.
BenTheElder commentedon Oct 22, 2026
See the mailinglist and slack, we are starting a GA code freeze.
seems OK but relatively unimportant. It's already a distroless image with Go binaries, I feel like our security folks would've had this tracked for M3 already if we were worried about this one
cc Taahir Ahmed (@ahmedtd) i think there's been discussion of this elsewhere.
mikedickey commentedon Oct 22, 2026
I also feel that running as root is the least concerning, given the distroless container images which are (presumably) running trusted code. However, I still think it's good defense in depth.
Over the past several weeks, every other substrate service moved from using default to its own service account. What is the rationale for leaving podcertificate-controller on the default SA?
Unconstrained cluster-wide secrets access is the most concerning to me.
I don't feel strongly if these belong in GA, but I wouldn't run substrate without them.
juli4n commentedon Oct 22, 2026
2 sounds like something important for GA (plus it's an easy fix).
Greg Castle (@destijl) thouhgts on 1 and 3?
BenTheElder commentedon Oct 22, 2026
Right, I'm currently putting #2428 first in that domain.
For the service accounts, I'm not sure if there was a rationale (Also that rendered 2 was written as a 3 but ... markdown) #2936 (comment)
destijl commentedon Oct 22, 2026
I think we should put all of these in before GA. I'd like any automated scanners looking at these pods to be like "that's a real tight ship" for anyone evaluating substrate.
#3028 : It's much easier if they are minimal from the start. I don't think we're gaining a ton of real security, but it's good hygiene and this will make it easier for substrate to run with folks that have very tight admission rules.
#3036 : yes, this is basic least privilege
#3059 : I doubt there's much concrete security value to doing this today, but it follows least privilege and avoids potential surprises in the future if other things run in the namespace. And same comment on tight admission rules.
atecontroller: Limit the Secrets read to the egress MITM CA pool (age…
manifests: Give podcertificate-controller its own ServiceAccount (age…
BenTheElder commentedon Oct 23, 2026
That makes sense. I've been trying to focus on things that seem like serious issues, but this should be ~small and I would prefer that as well.
manifests: Run control plane containers as non-root (agent-substrate#…
atecontroller: Limit the Secrets read to the egress MITM CA pool (age…