Skip to content

Add K8s support with SA Trust architecture#12434

Closed
NTurakulov wants to merge 4 commits into
openclaw:mainfrom
NTurakulov:feature/k8s-autoscale
Closed

Add K8s support with SA Trust architecture#12434
NTurakulov wants to merge 4 commits into
openclaw:mainfrom
NTurakulov:feature/k8s-autoscale

Conversation

@NTurakulov

@NTurakulov NTurakulov commented Feb 9, 2026

Copy link
Copy Markdown

Summary

Adds a Kubernetes Service Account Trust authentication method that enables workers to authenticate with the gateway using their pod SA token. This allows zero-touch scaling of worker nodes in Kubernetes — new pods authenticate automatically via the K8s TokenReview API, bypassing interactive device pairing.

Use Cases

  • Production K8s deployments: Run gateway and workers as separate pods with independent scaling.
  • Autoscaling: HPA or manual kubectl scale spawns workers that authenticate instantly.
  • Multi-tenant isolation: Deploy multiple isolated OpenClaw instances via Helm releases in separate namespaces.

Behavior Changes

  • New auth method k8s-sa-trust added to gateway authentication.
  • Workers with --k8s-trust flag read their pod SA token and present it on connect.
  • Gateway validates tokens via K8s TokenReview API when auth.k8sTrust is configured.
  • Workers authenticated via SA Trust skip device pairing (skipPairing).
  • No changes to existing auth flows — SA Trust is entirely opt-in.

Existing Functionality Check

  • Searched codebase for existing K8s/container auth — none found.
  • Searched GitHub issues for Kubernetes deployment requests.
  • Reviewed docs/install/docker.md — Docker guide exists but no K8s equivalent.

Changes

Source code (10 files):

File Change
src/gateway/k8s-auth.ts New K8sAuthenticator class — validates SA tokens via TokenReview API, namespace/SA allowlisting
src/gateway/k8s-auth.test.ts 13 unit tests covering validation, allowlisting, error handling, singleton
src/config/types.gateway.ts k8sTrust type definition on GatewayAuthConfig
src/config/zod-schema.ts Zod schema for k8sTrust config validation
src/gateway/auth.ts "k8s-sa-trust" added to auth method union type
src/gateway/protocol/schema/frames.ts k8sToken field in ConnectParams
src/gateway/client.ts k8sServiceAccountToken option in auth payload
src/gateway/server/ws-connection/message-handler.ts SA Trust validation + skipPairing for k8s-sa-trust
src/node-host/runner.ts k8sTrust option, reads SA token from pod volume mount
src/cli/node-cli/register.ts --k8s-trust CLI flag

Deployment files (22 files):

Path Contents
deploy/k8s/ Raw manifests: namespace, rbac, gateway-config, gateway, worker, networkpolicy, README
deploy/helm/openclaw/ Full Helm chart: Chart.yaml, values.yaml, values-local.yaml, 12 templates (statefulset, deployment, HPA, RBAC, configmap, secret, service, ingress, networkpolicy, service accounts, helpers, NOTES)

Documentation (1 file + Mintlify nav):

Path Contents
docs/gateway/kubernetes.md Architecture doc with Mintlify frontmatter: SA Trust design, threat model, token flow, failure modes, scaling behavior
docs/docs.json Added gateway/kubernetes to navigation + /kubernetes and /k8s redirects

Tests

  • 13 unit tests in src/gateway/k8s-auth.test.ts — all passing.
  • Tests cover: token validation (valid, invalid, expired), namespace allowlisting, service account allowlisting, combined restrictions, error handling (API errors, network failures, malformed responses), singleton pattern.
  • pnpm test — full suite passes.

Manual Testing

Prerequisites

  • K3s (or any K8s cluster) with kubectl access.
  • Container image built and available to the cluster.

Steps

  1. Deploy gateway with SA Trust config enabled.
  2. Deploy worker pods with --k8s-trust flag.
  3. Verify workers authenticate via SA Trust in gateway logs.
  4. Scale workers: kubectl scale deployment openclaw-worker --replicas=5.
  5. Verify new pods authenticate automatically (~10 seconds).

Evidence

  • K3s cluster: v1.34.3+k3s1, single-node, 3 namespaces deployed simultaneously.
  • Dynamic scaling: 2 → 5 workers, all authenticated via SA Trust.
  • Helm deployments: oc-test-1 and oc-test-2 namespaces via helm install, both operational.
  • Unit tests: 13/13 passing.

Sign-Off

  • Models used: Claude Opus 4
  • Submitter effort: multi-session (architecture design, implementation, deployment testing, Helm chart, debugging 6 deployment issues)

AI Disclosure

This PR was prepared with assistance from Claude Opus 4. The architecture design, implementation, K8s deployment testing, Helm chart creation, and debugging were developed collaboratively. I understand the code and the changes. Tested via automated tests (855/856 pass) and manual K3s deployment (3 namespaces, dynamic scaling 2→5 workers).

lobster-biscuit


Pre-PR Checklist

  • pnpm build — passes
  • pnpm test — 855/856 pass (1 pre-existing EACCES failure unrelated to this PR)
  • pnpm check — passes (tsgo + lint + format)
  • Relocate files from .nozim/ to final locations
  • Add Mintlify frontmatter to docs
  • Update docs/docs.json navigation + redirects
  • Add changelog entry
  • Rebase onto latest main

Greptile Overview

Greptile Summary

This PR adds an opt-in Kubernetes ServiceAccount Trust auth path for gateway↔worker connections. Workers can send their projected pod ServiceAccount JWT (k8sToken) during the connect handshake, and the gateway validates it via the Kubernetes TokenReview API when gateway.auth.k8sTrust is enabled. On successful validation/allowlisting, the connection is marked as k8s-sa-trust and device pairing is skipped. The PR also adds supporting CLI/runtime flags for node-host, config/zod schema updates, and Kubernetes deployment artifacts (raw manifests + Helm chart) plus documentation updates.

Confidence Score: 3/5

  • This PR is mergeable after addressing a couple of concrete security/operational issues in the new K8s TokenReview authentication path.
  • Core flow (worker sends SA token, gateway calls TokenReview, allowlists identity) is reasonable and well-tested, but the gateway’s TokenReview client currently ignores the configured CA cert (making in-cluster TLS likely to fail without extra env), and the allowlist defaults to allow-all when enabled, which is an unsafe default for an authentication feature that also bypasses device pairing.
  • src/gateway/k8s-auth.ts; src/gateway/server/ws-connection/message-handler.ts

(2/5) Greptile learns from your feedback when you react with thumbs up/down!

New auth flow for k8s SA-trust
Can run single Gateway node with autoscaled number of securely connected worker nodes
Helm chart for gateway + worker deployment
@openclaw-barnacle openclaw-barnacle Bot added docs Improvements or additions to documentation app: web-ui App: web-ui gateway Gateway runtime cli CLI command changes labels Feb 9, 2026

@greptile-apps greptile-apps Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 files reviewed, 3 comments

Edit Code Review Agent Settings | Greptile

Comment thread src/gateway/k8s-auth.ts
Comment on lines +1 to +3
import { readFileSync, existsSync } from "node:fs";

/**

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CA cert ignored

validateToken() reads the cluster CA (const _caCert = ...) but never uses it, while fetch() will still enforce TLS validation. In-cluster, this will fail against clusters where the API server cert isn’t trusted by Node’s default CA store unless the process is configured with NODE_EXTRA_CA_CERTS (not enforced anywhere in this PR). This makes k8s-sa-trust non-functional by default in many Kubernetes environments.

Consider wiring the CA into the HTTP client (e.g., Node fetch dispatcher/agent) or explicitly documenting/enforcing the required env/config so the gateway can reliably call TokenReview.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/k8s-auth.ts
Line: 1:3

Comment:
**CA cert ignored**

`validateToken()` reads the cluster CA (`const _caCert = ...`) but never uses it, while `fetch()` will still enforce TLS validation. In-cluster, this will fail against clusters where the API server cert isn’t trusted by Node’s default CA store unless the process is configured with `NODE_EXTRA_CA_CERTS` (not enforced anywhere in this PR). This makes `k8s-sa-trust` non-functional by default in many Kubernetes environments.

Consider wiring the CA into the HTTP client (e.g., Node fetch dispatcher/agent) or explicitly documenting/enforcing the required env/config so the gateway can reliably call TokenReview.

How can I resolve this? If you propose a fix, please make it concise.

Comment thread src/gateway/k8s-auth.ts
Comment on lines +150 to +152
isIdentityAllowed(namespace: string, serviceAccount: string, requestedRole: string): boolean {
const identities = this.config.allowedIdentities;
if (!identities || identities.length === 0) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Allowlist defaults to allow-all

isIdentityAllowed() returns true when allowedIdentities is missing/empty. That means enabling gateway.auth.k8sTrust.enabled without an allowlist will accept any authenticated ServiceAccount in the cluster (including from other namespaces), and skipPairing will then be granted. If this is meant to be secure-by-default, this should deny-by-default (or require an explicit wildcard config).

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/k8s-auth.ts
Line: 150:152

Comment:
**Allowlist defaults to allow-all**

`isIdentityAllowed()` returns `true` when `allowedIdentities` is missing/empty. That means enabling `gateway.auth.k8sTrust.enabled` without an allowlist will accept *any* authenticated ServiceAccount in the cluster (including from other namespaces), and `skipPairing` will then be granted. If this is meant to be secure-by-default, this should deny-by-default (or require an explicit wildcard config).

How can I resolve this? If you propose a fix, please make it concise.

Comment on lines 670 to 672
authMethod = "device-token";
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

K8s auth enables pairing bypass

skipPairing is set whenever authMethod === "k8s-sa-trust", regardless of k8sTrust.ephemeralSessions (which is documented as controlling persistence/pairing behavior). As written, enabling k8s-sa-trust always bypasses pairing even if an operator wanted TokenReview as an additional auth factor but still require device pairing. Either honor ephemeralSessions here or rename/remove the config knob to avoid a config that looks like it’s enforced but isn’t.

Prompt To Fix With AI
This is a comment left during a code review.
Path: src/gateway/server/ws-connection/message-handler.ts
Line: 670:672

Comment:
**K8s auth enables pairing bypass**

`skipPairing` is set whenever `authMethod === "k8s-sa-trust"`, regardless of `k8sTrust.ephemeralSessions` (which is documented as controlling persistence/pairing behavior). As written, enabling `k8s-sa-trust` always bypasses pairing even if an operator wanted TokenReview as an *additional* auth factor but still require device pairing. Either honor `ephemeralSessions` here or rename/remove the config knob to avoid a config that looks like it’s enforced but isn’t.

How can I resolve this? If you propose a fix, please make it concise.

@NTurakulov

Copy link
Copy Markdown
Author

Got the points, will get back with fixes

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: XL labels Feb 17, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Mar 9, 2026
@sallyom

sallyom commented Mar 12, 2026

Copy link
Copy Markdown
Contributor

Thanks for the work on this.

I’m closing this in favor of #34492 for the upstream Kubernetes install/docs path. There is some overlap here in the Kubernetes docs and deployment artifacts, but this PR is really centered on SA-trust auth and broader gateway/worker architecture changes. We'll keep the initial Kubernetes path simpler and avoid coupling it to new auth and scaling behavior.

The SA-trust work here is valuable, but I think it would be better proposed separately as a new focused PR on top of the simpler Kubernetes baseline.

Thanks again for the work.

@sallyom sallyom closed this Mar 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling app: web-ui App: web-ui cli CLI command changes docs Improvements or additions to documentation gateway Gateway runtime size: XL stale Marked as stale due to inactivity

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants