Add kubectl ate proxy actor command. by juli4n · Pull Request #3191 · agent-substrate/substrate

Julian Gutierrez Oschmann (juli4n) · 2026-10-23T23:37:23Z

Allow a user to land traffic on an actor from a dev machine.

The kubectl-ate CLI now authenticates to atenet by minting a certificate (via PodCertificateRequest) for spiffe://cluster.local/ns/ate-system/sa/atenet-client.

Ideally, atenet should support JWT based auth and there would be no need for a certificate minting. But that support is not in place yet.

The kubectl ate proxy actor command also improves local dev ergonomics as it attachjes the actor header so clients don't need to be "substrate-aware". For example, a web browser can just point to the local proxy.

Fixed #3194 and #3187

Taahir Ahmed (ahmedtd) · 2026-10-26T20:07:29Z

 	// IngressAllowedSPIFFEIDs are the SPIFFE IDs the ingress router lets
 	// reach actors over its TLS listeners (ATE_INGRESS_ALLOWED_SPIFFE_IDS).
-	// See RequireIngressAllowedSPIFFEIDs.
+	// Unset defaults to spiffe://cluster.local/ns/<namespace>/sa/atenet-client.


Personally, I feel that our demo setups should be adding this, rather than adding it to every install by default.

Is this a value that the user could also want to be empty and patched later?

I agree with taahir. Default should be empty. demo should patch it

I changed it so that the demo instructions explicitly add the atenet-client. I still think this is not well thought-out and that it's not consistent with how the control plane bootstraps the ownership, but let's discuss that as a follow up.

Lior Lieberman (LiorLieberman) · 2026-10-27T16:34:08Z

-func (c *Config) RequireIngressAllowedSPIFFEIDs() error {
-	if c.Router == RouterAgentgateway || len(c.IngressAllowedSPIFFEIDs) > 0 {
-		return nil


Taahir Ahmed (@ahmedtd) thinking about it more. I think we should allow setting this empty. it is a pretty common architecture that actors can pull for work vs get work. Resume is also called from different controllers and not necessarily through atenet

Empty now is allowed and means no egress.

Lior Lieberman (LiorLieberman)

Some high level notes:

can we mint the proxy cert outside ate-system? whoever gets PCR create there can also mint atelet for any node and get actor certs
orthogonal to this PR: This pr is relying on the fact that non-Kubelet clients are allowed to create arbitary PodCertificateRequests, even if the pod or service account they reference does not exist (#2389). Bowei Du (@bowei) had an idea of two podcertcontrollers - a demo one that issues certs for made-up pods and a prod one that refuses them.

Haven Xia (HavenXia)

Should the e2e router client use the new atenet-client identity instead of ate-e2e-router-client/router-client?

Lior Lieberman (LiorLieberman)

LGTM generally,
Since we now dont reject empty list for spiffee ids, - can you add a big note somewhere in the installation md files that "In order to allow ingress traffic to actors you should add the identities in this flag. If you have one frontend / gateway - it could be this identity, but if you have different callers to atenet-router - it should list them" (not this words verbatim but something similar)

in a follow up (really not now) we may want to look at whether we want to even deploy atenet-router if the list is empty.

Theoretically - we it does nothing so why deploying it BUT on the flip side, if/once we support better ingress auth we should not really require this "simple" flag and deprecate this authz mode.

Lior Lieberman (LiorLieberman) · 2026-10-27T22:47:42Z

+the client certificate for `spiffe://cluster.local/ns/ate-system/sa/atenet-client`, so the install must list that ID in
+`--ingress-allowed-spiffe-ids`.


should we have a kubectl command to validate --ingress-allowed-spiffe-ids was set?

Lior Lieberman (LiorLieberman) · 2026-10-27T23:00:33Z

Should the e2e router client use the new atenet-client identity instead of ate-e2e-router-client/router-client?

follow up please !! :)

atenet-router requires mTLS with an allowed SPIFFE client certificate, so plain kubectl port-forward can no longer reach actors from a workstation. - Add kubectl ate proxy actor <actor> [[local-port:]remote-port ...] (default 8000:80) to tunnel TCP connections to an actor over atenet-router's :8444 CONNECT listener. It presents a certificate for spiffe://cluster.local/ns/<namespace>/sa/atenet-client, which the install must list in ingressAllowedSPIFFEIDs. - Extract podidentity.RequestCertificate and ateclient.ConnectActorWithHeaders so e2e tests and ateclient share certificate minting and CONNECT tunneling. - Let ate-setup and atenet-router accept an empty ingressAllowedSPIFFEIDs. The Envoy router then serves no listener, and agentgateway denies every client. Document which identities to list. AI use: AI-assisted, I reviewed and tested it. Signed-off-by: Julian Gutierrez Oschmann <[email protected]>

- Replace kubectl port-forward svc/atenet-router 8000:80 with kubectl ate proxy actor across README.md, demo walkthroughs, and verification scripts. - Remove the unused jupyter-proxy NGINX deployment from the Jupyter demo. - Update demos/sandbox/client to dial ate-api-server via ateclient.NewClient. - Update demos/egress/test-egress.sh for the worker-for-actor SPIFFE URI. AI use: AI-assisted, I reviewed and tested it. Signed-off-by: Julian Gutierrez Oschmann <[email protected]>

Benjamin Elder (BenTheElder)

I think these can all be small follow-ups.

Benjamin Elder (BenTheElder) · 2026-10-27T23:39:15Z

+func (c *routerClientCerts) get(ctx context.Context) (*tls.Certificate, error) {
+	c.mu.Lock()
+	defer c.mu.Unlock()
+	now := time.Now()
+	if c.cert != nil && now.Before(c.refreshAt) {
+		return c.cert, nil
+	}
+	timeout := routerCertTimeout
+	if c.cert != nil && now.Before(c.cert.Leaf.NotAfter) {
+		timeout = routerCertRefreshTimeout
+	}
+	namespace := systemNamespace()
+	cert, err := podidentity.RequestCertificate(ctx, c.clientset, routerCertRequester, namespace, installdefaults.IngressClientServiceAccount, timeout)


c.mu is held across RequestCertificate, and this runs inside the TLS handshake. Once refreshAt passes, every new connection blocks for up to 15s behind the refresh even though the cached cert is still valid. After the cert expires with the signer unavailable, each queued handshake then makes its own 2 minute attempt in turn, since refreshAt isn't moved on that path.

Can we hand out the cached cert while one caller refreshes, or refresh in the background?

Benjamin Elder (BenTheElder) · 2026-10-27T23:39:15Z

+// certificate and presents a podidentity certificate for the ingress client
+// ServiceAccount, which the router allows by default.


stale, the allowlist defaults to empty now, so the router doesn't allow this client unless the install lists it

Benjamin Elder (BenTheElder) · 2026-10-27T23:39:15Z

+	// client-go tears down a whole port-forward when any one of its
+	// streams errors, so each tunnel gets its own. That also picks
+	// a ready router pod again after one goes away.
+	localPort, stop, err := portforward.ServicePortForward(ctx, d.config, d.clientset, systemNamespace(), installdefaults.RouterServiceName, routerConnectTLSPort)


non-blocking: every accepted connection does a Service GET and a Pod LIST (internal/portforward/portforward.go:66, :74) plus a port-forward upgrade before any bytes move, on a rest.Config left at client-go's default 5 QPS / burst 10. That's two apiserver round trips of latency per connection, and a browser opening a burst of connections (the Jupyter flow) gets throttled to ~2.5 new tunnels/s after the first few.

https://github.com/kubernetes/client-go/blob/28076445520055420e3be4255b4cd27fd19df1f9/rest/config.go#L48-L49

Could we resolve the router pod once and only re-resolve when a port-forward fails?

Benjamin Elder (BenTheElder) · 2026-10-27T23:39:15Z

 ```

-### 3. Port-Forward Services
+### 3. Proxy to the Actor


nit: demos/claude-code-multiplex/README.md:31 links #3-port-forward-services here for the api port-forward, that anchor and the api port-forward are both gone now

The client called ResumeActor before starting its REPL. A request through the router resumes a suspended actor, so the call is redundant. It was also the first thing to fail before agent-substrate#3191 gave the client credentials. The connection stays where it was: before the REPL, on the signal context. NewClient uses that context only while connecting, for the kube API calls, the token mint and the port-forward readiness wait; the tunnel and gRPC connection outlive it and close with the client. Signed-off-by: Ziy1-Tan <[email protected]>

Julian Gutierrez Oschmann (juli4n) force-pushed the atenet-client-proxy branch from b71cab9 to 95e8b98 Compare October 25, 2026 22:31

Alex Bulankou (AlexBulankou) mentioned this pull request Oct 26, 2026

Pull in upstream GCP Setup changes and Verify v.0.6.0 ai-on-gke/substrate-gke#68

Closed

Julian Gutierrez Oschmann (juli4n) force-pushed the atenet-client-proxy branch 3 times, most recently from 8f8975d to 928511f Compare October 26, 2026 15:58

Julian Gutierrez Oschmann (juli4n) changed the title ~~POC: Add kubectl ate proxy actor command.~~ Add kubectl ate proxy actor command. Oct 26, 2026

Max Thompson (thompsonmax) reviewed Oct 26, 2026

View reviewed changes

Comment thread cmd/kubectl-ate/internal/cmd/proxy.go Resolved

Julian Gutierrez Oschmann (juli4n) requested a review from Taahir Ahmed (ahmedtd) October 26, 2026 19:47

Julian Gutierrez Oschmann (juli4n) marked this pull request as ready for review October 26, 2026 19:57

Taahir Ahmed (ahmedtd) reviewed Oct 26, 2026

View reviewed changes

Comment thread cmd/kubectl-ate/internal/cmd/proxy.go Resolved

Taahir Ahmed (ahmedtd) previously approved these changes Oct 26, 2026

View reviewed changes

Comment thread demos/counter/README.md Resolved

Comment thread internal/podidentity/podidentity.go Resolved

Julian Gutierrez Oschmann (juli4n) dismissed Taahir Ahmed (ahmedtd)’s stale review via 9ad76c1 October 26, 2026 21:20

Julian Gutierrez Oschmann (juli4n) force-pushed the atenet-client-proxy branch from 928511f to 9ad76c1 Compare October 26, 2026 21:20

Julian Gutierrez Oschmann (juli4n) added this to the M3 milestone Oct 27, 2026

Julian Gutierrez Oschmann (juli4n) force-pushed the atenet-client-proxy branch 2 times, most recently from f1f0cba to 18cf3c2 Compare October 27, 2026 15:03

Bowei Du (bowei) reviewed Oct 27, 2026

View reviewed changes

Comment thread cmd/atenet/internal/router/README.md Resolved

Lior Lieberman (LiorLieberman) reviewed Oct 27, 2026

View reviewed changes

Benjamin Elder (BenTheElder) self-assigned this Oct 27, 2026

Tim Hockin (thockin) self-assigned this Oct 27, 2026

Tim Hockin (thockin) added prio/P0 Highest priority / required for next milestone milestone-ack labels Oct 27, 2026

Bowei Du (bowei) added area/cli area/network area/security Security related issue/pr kind/feature An enhancement / feature request or implementation labels Oct 27, 2026

This was referenced Oct 27, 2026

[bug-bash] README and demo walkthroughs port-forward atenet-router port 80 #3225

Closed

Quickstart: deploy ate-system hard-requires an ingress SPIFFE allowlist the Quickstart never sets #3194

Closed

This was referenced Oct 27, 2026

Fix quickstart setup instructions #3016

Closed

docs: use the mTLS router client in the kind quickstart and counter demo #3113

Closed

Julian Gutierrez Oschmann (juli4n) force-pushed the atenet-client-proxy branch 2 times, most recently from 7dcc7d8 to 7b8bd99 Compare October 27, 2026 22:19

Haven Xia (HavenXia) reviewed Oct 27, 2026

View reviewed changes

Haven Xia (HavenXia) mentioned this pull request Oct 27, 2026

Delete hack/install-ate.sh and hack/install-ate-kind.sh #3354

Merged

2 tasks done

Lior Lieberman (LiorLieberman) reviewed Oct 27, 2026

View reviewed changes

Julian Gutierrez Oschmann (juli4n) force-pushed the atenet-client-proxy branch 2 times, most recently from e37bcff to 7eb301f Compare October 27, 2026 23:24

Lior Lieberman (LiorLieberman) previously approved these changes Oct 27, 2026

View reviewed changes

Haven Xia (HavenXia) previously approved these changes Oct 27, 2026

View reviewed changes

Julian Gutierrez Oschmann (juli4n) added 2 commits October 27, 2026 20:06

Julian Gutierrez Oschmann (juli4n) dismissed stale reviews from Haven Xia (HavenXia) and Lior Lieberman (LiorLieberman) via 5dd9c4e October 28, 2026 00:07

Julian Gutierrez Oschmann (juli4n) force-pushed the atenet-client-proxy branch from 7eb301f to 5dd9c4e Compare October 28, 2026 00:07

Benjamin Elder (BenTheElder) approved these changes Oct 28, 2026

View reviewed changes

Benjamin Elder (BenTheElder) added this pull request to the merge queue Oct 28, 2026

Merged via the queue into agent-substrate:main with commit c093a40 Oct 28, 2026
5 checks passed

Chenyi Wang (chw120) mentioned this pull request Oct 28, 2026

[Bug]: kubectl ate proxy actor returns an empty response when the client half-closes (e.g. nc -N) #3416

Open

2 tasks done

Haven Xia (HavenXia) mentioned this pull request Nov 2, 2026

demos/jupyter configures NGINX jupyter-proxy to upstream plaintext HTTP on port 80 to atenet-router without mTLS, failing with 502/504 #3277

Closed

This was referenced Nov 4, 2026

demos/sandbox: drop the redundant resume from the sandbox client #2468

Open

demos/sandbox: client resumes the actor explicitly before the REPL #2636

Open

Julian Gutierrez Oschmann (juli4n) mentioned this pull request Nov 6, 2026

Add kubectl ate proxy actor for simple local connectivity #213

Closed

		the client certificate for `spiffe://cluster.local/ns/ate-system/sa/atenet-client`, so the install must list that ID in
		`--ingress-allowed-spiffe-ids`.

		// certificate and presents a podidentity certificate for the ingress client
		// ServiceAccount, which the router allows by default.

Conversation

Julian Gutierrez Oschmann (juli4n) commented Oct 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Haven Xia (HavenXia) left a comment

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Lior Lieberman (LiorLieberman) commented Oct 27, 2026

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

Julian Gutierrez Oschmann (juli4n) commented Oct 23, 2026 •

edited

Loading

Lior Lieberman (LiorLieberman) left a comment •

edited

Loading