Skip to content

atenet: Router checks the routed worker's SPIFFE ID on atunnel's cert - #2678

Merged
Lior Lieberman (LiorLieberman) merged 1 commit into
agent-substrate:mainfrom
LiorLieberman:router-upstream-san
Oct 15, 2026
Merged

Lior Lieberman (LiorLieberman) merged 1 commit into
agent-substrate:mainfrom
LiorLieberman:router-upstream-san

Conversation

@LiorLieberman

@LiorLieberman Lior Lieberman (LiorLieberman) commented Oct 14, 2026 •

Copy link
Copy Markdown
Collaborator

Part of #2078 (Envoy dataplane, Eitan Yarmush (@EItanya) John Howard (@howardjohn) for agw)

The router accepted any podidentity cert in the trust domain as atunnel, so any pod in the cluster could pose as a worker. It now requires the exact SPIFFE ID of the worker each request is routed to.

  • atecontroller's syncer records the worker pod's SPIFFE ID on the Worker (new worker_spiffe_id, built from the pod's namespace and ServiceAccount), and ateapi copies it into the actor's worker assignment. ateapi rejects a value that isn't a valid SPIFFE ID.
  • ext_proc passes that ID to Envoy as is and refuses to route without it. Envoy requires that exact SAN, on both the :443 and CONNECT paths. --upstream-spiffe-prefix is removed; the cluster still checks the trust domain.
  • The Worker carries the SPIFFE ID rather than the ServiceAccount, so the router doesn't need to know how the ID is built. A different ID scheme only changes atecontroller.
  • WorkerPools that don't set serviceAccountName now run as an ateworker ServiceAccount the controller creates, instead of default, so the worker's ID isn't shared with every other pod in the namespace.

Remaining gap: another pod running as the same ServiceAccount in the same namespace can still pose as the worker. Closing it needs a signer that puts the pod in the worker's SPIFFE ID; atecontroller would then record that ID, with no router change.

Breaking change

  • The router rejects workers registered before this change until their pods are replaced. Pools on the default ServiceAccount roll automatically; pools that set spec.template.serviceAccountName need a restart. Upgrade atecontroller and ate-api-server before atenet-router.
  • Workers now run as an ateworker ServiceAccount by default instead of default. RBAC, Workload Identity bindings and imagePullSecrets attached to default no longer apply to them; set serviceAccountName: default or move those to ateworker.
  • atenet-router no longer has --upstream-spiffe-prefix.
  • atecontroller now needs create and patch on serviceaccounts (included in the generated role).

It's a good idea to open an issue first for discussion.

  • Tests pass
  • Appropriate changes to documentation are included in the PR

@EItanya Eitan Yarmush (EItanya) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall makes sense to me

Comment on lines +41 to +44
- serviceaccounts
verbs:
- create
- patch

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Greg Castle (@destijl) do you think these new permissions will be a security issue?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ate-controller already has CRUD on pods cluster-wide. These don't seem scarier than that.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good, just checking in

// applyWorkerServiceAccount creates the default ServiceAccount worker pods run
// as. The pools in a namespace share it, so none of them owns it.
func (r *WorkerPoolReconciler) applyWorkerServiceAccount(ctx context.Context, namespace string) error {
saAC := corev1ac.ServiceAccount(installdefaults.WorkerServiceAccount, namespace)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems like the serviceAccount should be something set on the WorkerPool CRD --- there's nothing that says it needs to be the same service account in each namespace or across all workerpools.

But that can be a followup.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You already can set the ServiceAccount, but the current meaning is an existing one, not create. So we'd just need to figure out the semantic is all

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure I follow.

This function is called only if you dont have a serviceaccount set see (line 322 above)

	if wp.Spec.Template == nil || wp.Spec.Template.ServiceAccountName == nil {
		if err := r.applyWorkerServiceAccount(ctx, wp.Namespace); err != nil {
			return err
		}
	}

So if you set up your service account - great, if you dont, we set it up for you.

Comment on lines +41 to +44
- serviceaccounts
verbs:
- create
- patch

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ate-controller already has CRUD on pods cluster-wide. These don't seem scarier than that.

@LiorLieberman
Lior Lieberman (LiorLieberman) force-pushed the router-upstream-san branch 2 times, most recently from 3be940d to 7c22343 Compare October 15, 2026 20:43
…ificate

export NODE_EXTRA_CA_CERTS=/etc/ssl/cert.pem
Merged via the queue into agent-substrate:main with commit 088381b Oct 15, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants