kubectl ate: authenticate with ExecCredential plugins named by --credential-plugin by thompsonmax · Pull Request #2851 · agent-substrate/substrate

Max Thompson (thompsonmax) · 2026-10-19T01:25:12Z

Part of #1464

Support ExecCredential kubectl ate authentication.

--credential-plugin names a kubectl credential plugin's command, and each --credential-plugin-arg passes it one argument. kubectl ate passes the plugin its --kubeconfig and --context as $ATE_KUBECONFIG and $ATE_KUBE_CONTEXT.
internal/ateclient runs the plugin itself and reuses its token until a minute before it expires. The default ServiceAccount token now refreshes a minute before it expires, which fixes [P1] ateclient mints a single 1-hour bearer token and never refreshes it — long-lived clients go permanently Unauthenticated #644.
kubectl ate service-account-token mints a token for the ServiceAccount named with --service-account.

Here's how each authentication option looks after this change:

--token-file: you supply the token yourself (Already existed before this change)

kubectl ate sends whatever token the file holds. It re-reads the file on every RPC, so a rotated file is picked up automatically, but it never runs anything to get a new token.

# A token you minted yourself
kubectl create token alice -n ate-users --audience=api.ate-system.svc > alice.token
kubectl ate --token-file=alice.token get actors -a alice-space

--credential-plugin: kubectl ate runs a command that produces the credential (What we're introducing here)

The command must print an ExecCredential (v1) as JSON. kubectl ate runs the command when it first needs a token, and again a minute before the token's expirationTimestamp.

# Built-in plugin, acting as a per-user ServiceAccount
kubectl ate --credential-plugin=kubectl-ate --credential-plugin-arg=service-account-token \
  --credential-plugin-arg=--service-account=ate-users/alice get actors -a alice-space

# A standalone plugin binary
kubectl ate --credential-plugin=/usr/local/bin/my-auth-plugin get actors -a alice-space

# Corporate OIDC through kubelogin
kubectl ate --credential-plugin=kubectl \
  --credential-plugin-arg=oidc-login --credential-plugin-arg=get-token \
  --credential-plugin-arg=--oidc-issuer-url=https://idp.example.com --credential-plugin-arg=--oidc-client-id=substrate \
  get actors -a alice-space

Rules that apply to both

Precedence: --token-file comes first, then --credential-plugin, then the shared ate-client token, which prints a warning.
Both flags: passing --token-file and --credential-plugin together is an error.
Plugin arguments: --credential-plugin takes only the command. Repeat --credential-plugin-arg for each argument, which kubectl ate passes through unchanged.

Reviewer notes: the commits split the PR by feature (the #644 refresh, --credential-plugin, kubectl ate service-account-token, docs) and can be reviewed in order. kubectl ate whoami moved to the follow-up #3261. Plugins must print a client.authentication.k8s.io/v1 ExecCredential, so plugins that only answer v1beta1 fail.

Testing: unit tests; e2e checks that a client with a credential plugin (cat of an ExecCredential) is allowed and denied as the plugin's ServiceAccount.

AI use: AI-assisted (Claude Code) for code, tests, and docs. I reviewed and tested it.

Fixes #644

Tests pass
Appropriate changes to documentation are included in the PR

Benjamin Elder (BenTheElder)

AI-assisted first pass, inline comments are mostly small. Two bigger comments:

Can we split this up? WhoAmI and kubectl ate credential / wrap could each be their own PR, this is a lot to review at once. Or at least one PR with more commits, they don't even all have to build, since the merge queue only squash merges. The latest AGENTS.md has some guidance for this, if you're generating this you can ask your agent something like "please pull the latest upstream AGENTS.md and apply it to the PR".
I'm not sure we need ATE_REFRESH_CREDENTIAL or the client cert path yet, can we start with the ServiceAccount token default (shared or per-user via --service-account) + client-go's expiry handling, and add the rest when we have a case for it?

We should try hard to not require non-portable flags / dedicated environment variables to get a working client.

nit: please note AI use in the PR description per the PR template
(either way, so far this has been a useful quality signal WRT agents that are not respecting AGENTS.md / docs or humans that are ignoring the PR template ...)

Benjamin Elder (BenTheElder) · 2026-10-25T00:43:21Z

+}
+
+// PrincipalKind is the authentication method that established a principal.
+enum PrincipalKind {


cc Julian Gutierrez Oschmann (@juli4n) Tim Hockin (@thockin) , we were discussing only using strings instead of proto enums for compat reasons. We should decide on that ASAP #1378

Let me know if/when there's a decision there and I'll change this ASAP.

That's bonkers. Why?

Max Thompson (thompsonmax) · 2026-10-26T16:51:46Z

AI-assisted first pass, inline comments are mostly small. Two bigger comments:

Can we split this up? WhoAmI and kubectl ate credential / wrap could each be their own PR, this is a lot to review at once. Or at least one PR with more commits, they don't even all have to build, since the merge queue only squash merges. The latest AGENTS.md has some guidance for this, if you're generating this you can ask your agent something like "please pull the latest upstream AGENTS.md and apply it to the PR".

I'm not sure we need ATE_REFRESH_CREDENTIAL or the client cert path yet, can we start with the ServiceAccount token default (shared or per-user via --service-account) + client-go's expiry handling, and add the rest when we have a case for it?

We should try hard to not require non-portable flags / dedicated environment variables to get a working client.

nit: please note AI use in the PR description per the PR template (either way, so far this has been a useful quality signal WRT agents that are not respecting AGENTS.md / docs or humans that are ignoring the PR template ...)

I had initially planned on splitting this up after prototyping, but since it seems we want to get this in fairly soon, it seemed better to keep it as a single PR. Separating out the commits for each piece sounds reasonable though. I'll split it up that way.
Sure, happy to trim this down. Cut out the ATE_REFRESH_CREDENTIAL/ATE_CREDENTIAL_PLUGIN flags, client cert support, and disk cache for credentials on the client-side for now.
Updated the PR description to include the AI acknowledgement :) (think I drafted this before that requirement was added to the template)

Taahir Ahmed (ahmedtd) · 2026-10-26T23:04:21Z

We should try hard to not require non-portable flags / dedicated environment variables to get a working client.

I'm not sure we want the default experience to require connecting to the k8s cluster and minting a random service account token that erases your human identity.

I (sort of) think that we should always make people pass an --exec-credential-plugin flag to kubectl ate for now.

Taahir Ahmed (ahmedtd) · 2026-10-26T23:06:28Z

+  string id = 1;
+
+  // How the caller authenticated.
+  PrincipalKind kind = 2;


Let's model this the way kubernetes does --- a single string credential ID. For JWTs, it would be a string like JTI=${jti-claim-from-jwt}. For certificates it would be a string like X509SHA256=${sha-256-of-the-certficate}

Or, if you want to include the issuer, have a oneof that wraps a JWTInfo and SPIFFECertificateInfo struct.

Moved WhoAmI to #3261. I think ideally we include the issuer, so we can model this like:

message WhoAmIResponse { // The principal ID that authorization policies name. string id = 1; // How the caller authenticated. Exactly one is set. // +k8s:optional // +k8s:unionMember JWTCredential jwt = 2; // +k8s:optional // +k8s:unionMember SPIFFECertificateCredential spiffe_certificate = 3; } message JWTCredential { string issuer = 1; } message SPIFFECertificateCredential {}

Taahir Ahmed (ahmedtd) · 2026-10-26T23:09:05Z

+}
+
+// PrincipalKind is the authentication method that established a principal.
+enum PrincipalKind {


That's bonkers. Why?

Taahir Ahmed (ahmedtd) · 2026-10-26T23:13:10Z

+)
+
+var credentialCmd = &cobra.Command{
+	Use:   "credential",


If I understand correctly, this command is a credential plugin that:

Authenticates to a k8s cluster (using your credentials configured in a .kubeconfig file)

Mints a kubernetes service account JWT for a named service account.

If so, it needs to be named something much more specific than "credential" --- I think, as it is, it will be confusing to users, who might think this is the one blessed way to get a credential for a substrate install.

What about kubectl ate service-account-token?

Went with that name, LMK if you're not a fan.

Taahir Ahmed (ahmedtd) · 2026-10-26T23:17:55Z

+	}
+
+	// An existing file runs with no arguments, so its path may hold spaces.
+	command, args := credentialPlugin, []string(nil)


I feel like instead of doing this parsing, we should take the information in a structured way, like:

kubectl ate foo --credential-plugin="binary" --credential-plugin-arg="--foo" --credential-plugin-arg="--bar"

Done, separated into structured --credential-plugin and --credential-plugin-arg flags.

We'll definitely want to clean this up post-GA.

Taahir Ahmed (ahmedtd) · 2026-10-26T23:26:15Z

+
+// execCreds runs a kubeconfig exec credential plugin through client-go's
+// authenticator, which caches the credential until its expirationTimestamp.
+// That authenticator is built for net/http, so execCreds drives it with


OK, there has to be a better way to do this. It would be better to fork the k8s exec credential code for now than to start up a magical local http server just for getting the credential.

Or just write our own code for invoking the exec plugin. The important thing we need from k8s is the protocol, so that people don't have to write new credential plugins for substrate. The actual code seems to be more trouble than it's worth

Done, ateclient now runs the plugin itself by setting KUBERNETES_EXEC_INFO and reads the v1 ExecCredential the plugin prints.

The client minted one hour-long token for the shared ate-client ServiceAccount when it connected and never replaced it, so long-running clients failed after an hour. It now mints on first use and again a minute before expiry, and keeps the current token until it expires if re-minting fails. NewClient now wraps Dial and DialOptions, which merge the direct and port-forward paths so the credential is set up in one place. Signed-off-by: Max Thompson <[email protected]>

--credential-plugin names the command of a kubectl credential plugin, and each --credential-plugin-arg passes it one argument. kubectl ate passes the plugin its --kubeconfig and --context as $ATE_KUBECONFIG and $ATE_KUBE_CONTEXT, and its stdin when stdin is a terminal. Passing both --token-file and --credential-plugin is an error. ateclient runs the plugin itself and reads the v1 ExecCredential it prints. It reuses the bearer token until a minute before the token's expirationTimestamp. A plugin that prints no token is an error. Signed-off-by: Max Thompson <[email protected]>

kubectl ate service-account-token mints a token for ate-api for the ServiceAccount named with --service-account and prints it as an ExecCredential for --credential-plugin, or only the token with -o token. With no --kubeconfig or --context it uses $ATE_KUBECONFIG and $ATE_KUBE_CONTEXT, so it mints through the same cluster as the kubectl ate command that ran it. Signed-off-by: Max Thompson <[email protected]>

Describe --credential-plugin and kubectl ate service-account-token, how to give a user a ServiceAccount of their own and the RBAC a non-admin needs, what deleting a ServiceAccount does not revoke, and in-cluster workloads. Signed-off-by: Max Thompson <[email protected]>

…gent-substrate#3119) Part of agent-substrate#1464. - Rewrites `docs/guides/authentication.md` around who is calling ate-api: a person running `kubectl ate`, a pod in the Substrate cluster or another cluster, or a CI job, so people stop leaning on the shared `ate-client` token. - Adds pod certificates as an option for client pods (the benchmarks use them), the `ate.dev/api-access` namespace label the ate-system NetworkPolicy requires, and the Kubernetes RBAC `kubectl ate` needs to reach ate-api. - Warns where an identity is weaker than it looks: another cluster's issuer can mint `ate-client`, GKE's Kubernetes Engine Developer and basic Editor roles can act as any ServiceAccount, ate-system port-forward rights reach loopback-only Envoy admin APIs that can stop the data plane, and GitHub and Google tokens authenticate anyone. - Moves the per-user ServiceAccount recipe and the signing-in and install-time-provider gaps out of `authorization.md`, and drops its claim that a Secret-based token works there (ate-api rejects its issuer). The recipe and RBAC use agent-substrate#2851's names (`ate-users`, `ate-api-client`). For reviewers: agent-substrate#2851 (`--credential-plugin`) rewrites the same sections, and its paragraph calling GitHub and Google tokens unsafe needs to match this one. Testing: checked each claim against the code on main (GitHub, GKE and Kubernetes behavior against their docs or source) and that the relative links resolve. None of the commands or manifests ran on a cluster. AI use: AI-assisted. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Max Thompson <[email protected]>

…3119) Part of #1464. - Rewrites `docs/guides/authentication.md` around who is calling ate-api: a person running `kubectl ate`, a pod in the Substrate cluster or another cluster, or a CI job, so people stop leaning on the shared `ate-client` token. - Adds pod certificates as an option for client pods (the benchmarks use them), the `ate.dev/api-access` namespace label the ate-system NetworkPolicy requires, and the Kubernetes RBAC `kubectl ate` needs to reach ate-api. - Warns where an identity is weaker than it looks: another cluster's issuer can mint `ate-client`, GKE's Kubernetes Engine Developer and basic Editor roles can act as any ServiceAccount, ate-system port-forward rights reach loopback-only Envoy admin APIs that can stop the data plane, and GitHub and Google tokens authenticate anyone. - Moves the per-user ServiceAccount recipe and the signing-in and install-time-provider gaps out of `authorization.md`, and drops its claim that a Secret-based token works there (ate-api rejects its issuer). The recipe and RBAC use #2851's names (`ate-users`, `ate-api-client`). For reviewers: #2851 (`--credential-plugin`) rewrites the same sections, and its paragraph calling GitHub and Google tokens unsafe needs to match this one. Testing: checked each claim against the code on main (GitHub, GKE and Kubernetes behavior against their docs or source) and that the relative links resolve. None of the commands or manifests ran on a cluster. AI use: AI-assisted. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Max Thompson <[email protected]> (cherry picked from commit f1973c1) Signed-off-by: Eitan Yarmush <[email protected]>

Bowei Du (bowei) added area/cli area/identity area/security Security related issue/pr kind/feature An enhancement / feature request or implementation labels Oct 19, 2026

Max Thompson (thompsonmax) force-pushed the ateconfig-exec-credential branch from f8db639 to cb06553 Compare October 20, 2026 02:26

Max Thompson (thompsonmax) changed the title ~~kubectl ate: authenticate with credential plugins configured in an AteConfig file~~ kubectl ate: authenticate with ExecCredential plugins named by --credential-plugin Oct 20, 2026

Max Thompson (thompsonmax) marked this pull request as ready for review October 20, 2026 02:41

Max Thompson (thompsonmax) requested review from Shruti Nair (SHRUTI6991) and Taahir Ahmed (ahmedtd) October 20, 2026 02:42

Max Thompson (thompsonmax) force-pushed the ateconfig-exec-credential branch from cb06553 to 56c6bf0 Compare October 20, 2026 02:45

This was referenced Oct 22, 2026

Document how people, pods and CI jobs should authenticate to ate-api #3119

Merged

Clean up client authentication story #1464

Closed

Max Thompson (thompsonmax) force-pushed the ateconfig-exec-credential branch 2 times, most recently from 9cb63fd to a2892f6 Compare October 24, 2026 15:32

Max Thompson (thompsonmax) requested a review from Dmitry Berkovich (dberkov) October 24, 2026 16:13

Dmitry Berkovich (dberkov) reviewed Oct 24, 2026

View reviewed changes

Comment thread internal/ateclient/credentials.go Resolved

Comment thread cmd/kubectl-ate/internal/cmd/credential.go Resolved

Comment thread docs/authentication.md Resolved

Benjamin Elder (BenTheElder) reviewed Oct 25, 2026

View reviewed changes

Max Thompson (thompsonmax) force-pushed the ateconfig-exec-credential branch from d762059 to de5dc63 Compare October 26, 2026 18:17

Max Thompson (thompsonmax) mentioned this pull request Oct 26, 2026

Add kubectl ate proxy actor command. #3191

Merged

Max Thompson (thompsonmax) force-pushed the ateconfig-exec-credential branch 3 times, most recently from 2decbb9 to 9a2eb79 Compare October 26, 2026 22:41

Max Thompson (thompsonmax) requested review from Benjamin Elder (BenTheElder) and Dmitry Berkovich (dberkov) October 26, 2026 22:43

Taahir Ahmed (ahmedtd) reviewed Oct 26, 2026

View reviewed changes

Max Thompson (thompsonmax) mentioned this pull request Oct 26, 2026

kubectl ate whoami: show which principal ate-api authenticated you as #3261

Open

1 of 2 tasks

Max Thompson (thompsonmax) force-pushed the ateconfig-exec-credential branch from 9a2eb79 to 3be4c10 Compare October 26, 2026 23:47

Max Thompson (thompsonmax) force-pushed the ateconfig-exec-credential branch from 3be4c10 to 5ea9878 Compare October 27, 2026 18:37

Max Thompson (thompsonmax) added 3 commits October 27, 2026 13:14

Max Thompson (thompsonmax) force-pushed the ateconfig-exec-credential branch from 5ea9878 to ac9d542 Compare October 27, 2026 20:16

Max Thompson (thompsonmax) requested a review from Taahir Ahmed (ahmedtd) October 27, 2026 20:31

Conversation

Max Thompson (thompsonmax) commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Max Thompson (thompsonmax) Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Max Thompson (thompsonmax) commented Oct 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Taahir Ahmed (ahmedtd) commented Oct 26, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Max Thompson (thompsonmax) Oct 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Max Thompson (thompsonmax) commented Oct 19, 2026 •

edited

Loading

Max Thompson (thompsonmax) Oct 26, 2026 •

edited

Loading

Max Thompson (thompsonmax) commented Oct 26, 2026 •

edited

Loading

Max Thompson (thompsonmax) Oct 27, 2026 •

edited

Loading