Skip to content

fix(kubernetes): shorten the HCL workspace and add an OIDC example - #14427

Merged
cv merged 2 commits into
v1from
fix/kubernetes-examples-p1
Nov 13, 2026
Merged

cv merged 2 commits into
v1from
fix/kubernetes-examples-p1

Conversation

@gaveezy

@gaveezy gaveezy commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

For Kubernetes users: examples/kubernetes/hosted-nvidia.tf now applies with its default workspace, the Kubernetes guide lists the NetworkPolicy enforcement that OpenShell requires, and the new examples/kubernetes/managed-oidc.yaml is a complete document for a managed gateway with the oidc profile and a registry pull Secret.

Problem

  • hosted-nvidia.tf, which examples/gtc-dc/kubernetes-amd64.tf links to, defaults workspace to gtc-dc-kubernetes-cloud, 23 characters. The pinned OpenShell refuses a workspace name over 19 characters (MAX_ROUTABLE_NAME_LEN), and only when apply creates the workspace, after the gateway is installed. The run recorded on feat(examples): manage GTC DC Kubernetes gateways #14315 hit this. The HCL plan scenario passed, because plan doesn't create the workspace.
  • The guide's prerequisites didn't say that OpenShell requires a network plugin that enforces NetworkPolicy. OpenShell's NetworkPolicies are what keep a sandbox's workload pod off the network. Kubernetes accepts them without such a plugin, so nothing reports one that's missing.
  • No example used the oidc profile on its own, or imagePullSecrets. The only OIDC example, gpu-autoscaling/two-runtimes.yaml, depends on the GPU chart. The guide also didn't say:
    • that the gateway token needs the admin role;
    • how to create the caConfigMap;
    • how the OpenShell CLI signs in where no browser can open;
    • which role lets the CLI reach the deployment's workspace.

Change

  • hosted-nvidia.tf:
    • Default workspace to gtc-dc-k8s-cloud.
    • Add a validation block with OpenShell's name rule: at most 19 characters, a DNS label, and no --.
    • gtc-dc-kubernetes-amd64-hcl.tftest.hcl plans the old default and expects that validation to fail.
    • examples/gtc-dc/README.md states the limit.
  • Prerequisites (kubernetes.md): add a network plugin that enforces ingress and egress NetworkPolicy, linked to OpenShell's requirement. Say that nothing reports a plugin that ignores NetworkPolicy. Add it to the threat model's "Relies on" column.
  • examples/kubernetes/managed-oidc.yaml:
    • An OpenClaw and a Hermes agent on the isolated network tier, sharing hosted NVIDIA inference.
    • The oidc profile with caConfigMap and a token, and imagePullSecrets.
    • The OIDC and registry sections of the guide link it.
  • OIDC section:
    • Add the kubectl create configmap command for caConfigMap.
    • The token needs the role admin_role names; without it, plan fails with observation permission denied.
    • When scopes_claim is set, the token also needs the same scopes as an external gateway's token.
  • "Reach the Gateway from the OpenShell CLI":

Left out:

Beyond CI

  • managed-oidc.yaml on k3s: applied on single-node k3s v1.37.1+k3s1 with Agent Sandbox 1.0.3, the local-path StorageClass, and k3s's NetworkPolicy controller.
    • Setup:
      • Images built from c5bf74d and pushed to a loopback registry.
      • Hosted NVIDIA inference.
      • The e2e test issuer in another namespace, with its CA in the issuer-ca ConfigMap.
      • The example with only its UID, context, namespace, issuer settings and image digests changed.
    • Results:
      • Plan and apply succeeded, and both agents replied through the gateway.
      • registry-credentials was on the gateway, supervisor and sandbox pods.
      • A second plan found no changes.
      • Destroy succeeded and kept the namespace, the gateway's volume and its Secrets.
  • Token without the admin role: on the same deployment, a token re-signed with only the user role made plan fail with observation permission denied.
  • hosted-nvidia.tf on k3s, with its new default workspace: the same cluster and images, with OpenTofu and the providers from a bundle built at c5bf74d.
    • Setup: the two claims prepared and labelled as examples/gtc-dc/README.md says. Only the provider versions, UID, context, namespace and images were set; workspace kept its default.
    • Init, validate, plan and apply succeeded, and OpenShell created the workspace gtc-dc-k8s-cloud.
    • OpenClaw and Hermes each replied READY through fabric-agent-ctl invoke, and each had its claim's workspace/ and artifacts/ mounted and writable.
    • A second plan found no changes.
    • The README's targeted destroy removed the gateway and both agents, and kept the storage resource and both claims.
  • OpenShell CLI sign-in on a three-node cluster: k3d v5.9.0 with k3s v1.37.1+k3s1 and Agent Sandbox 1.0.3, with Keycloak 26.4 in the cluster as the issuer and three confidential clients. The gateway was docs(examples): add Kubernetes placement and NodePort examples with an index #14428's published-nodeport.yaml, which uses the same oidc profile; NemoClaw's own token came from one of the clients.
    • The guide's steps with the pinned CLI 0.1.7: the client certificate copied from <release>-client-tls, SSL_CERT_FILE set to the issuer's CA, then openshell gateway add with OPENSHELL_OIDC_CLIENT_SECRET. The CLI printed Authenticated via client credentials, and openshell status reported Connected and Authenticated (OIDC).
    • This worked both through the port forward to the gateway pod and at the published endpoint.
    • With the admin role, openshell sandbox list listed the agent, and openshell sandbox exec … fabric-agent-ctl invoke got READY.
    • With only the user role, openshell status reported Authenticated (OIDC; authorization denied), and sandbox commands failed with not a member of workspace. After openshell workspace member add … --role user, the same client listed and invoked the agent, and a NemoClaw plan still found no changes.
  • Not run:

🤖 Generated with Claude Code

- hosted-nvidia.tf defaulted its workspace to a 23-character name, which
  OpenShell refuses at apply after installing the gateway. Default to
  gtc-dc-k8s-cloud and check OpenShell's name rule at plan.
- List a network plugin that enforces NetworkPolicy as a Kubernetes
  prerequisite, as OpenShell requires.
- Add examples/kubernetes/managed-oidc.yaml: an OpenClaw and a Hermes agent
  on a managed gateway with the oidc profile and a registry pull Secret.
- Document the ConfigMap command for caConfigMap, the admin role the
  gateway token needs, and the OpenShell CLI's client-credentials sign-in.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Hai Nguyen <[email protected]>
@gaveezy

gaveezy commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author
Authoring session — opening
Field Status Value Evidence or limit
Harness recorded Claude Code desktop app (Code tab), CLI 2.1.321 Version and entrypoint fields of the session record, to the cutoff 2026-11-12T03:10:01Z
Skills recorded workflow-authoring (injected); nemoclaw-contributor-create-pr (complete read) Skill injection and file reads in the session record; verify-yaml-deployment was read only in part, so it isn't counted
Instruction files recorded AGENTS.md (injected); WRITING.md (complete read); .github/PULL_REQUEST_TEMPLATE.md (complete read); .github/SESSION_REPORT_TEMPLATE.md (complete read); user memory index (injected) Injections and reads in the session record, to the cutoff
Tool calls recorded Main loop: Bash 129; Edit 11; update_plan 7; Write 6; Read 3; wrap_up 2; Workflow 1. Nested, in that one workflow's 42 subagents: Bash 2063; StructuredOutput 42; Write 5; Read 4; update_plan 3 Attempted calls, including failed ones, in the main record and 43 subagent records, to the cutoff. The whole session also reviewed the Kubernetes examples, which the workflow did, and wrote a second PR's branch
Tokens recorded input: 278475917; cached input: 273270912; output: 1444340 Final message.usage per unique API response to the cutoff: main loop 144 responses (input 34945751; cached 34493657; output 112637) plus 43 subagent records, 2010 responses (input 243530166; cached 238777255; output 1331703)
Session window recorded 2026-11-11T09:16:57Z to 2026-11-12T03:10:01Z; 17 h 53 min First event to cutoff, including idle time. The review ran from 09:16Z to about 10:20Z on 2026-11-11; this PR's task ran from 2026-11-12T02:44:21Z to the cutoff, 26 min
Compactions recorded 0 Complete record to cutoff
Models recorded claude-opus-5-5 Every main-loop and subagent turn
Reasoning levels recorded xhigh Effort field on every main-loop and subagent turn; no changes

@github-actions

Copy link
Copy Markdown
Contributor

@gaveezy gaveezy self-assigned this Nov 12, 2026
@wscurran wscurran added area: docs Documentation, examples, guides, or docs build area: security Security controls, permissions, secrets, or hardening chore Build, CI, dependency, or tooling maintenance platform: k8s Affects Kubernetes deployments v1 NemoClaw v1 branch labels Nov 12, 2026
@gaveezy
gaveezy requested review from cv and sandl99 November 13, 2026 00:39
A live run of the client-credentials sign-in showed that a token with only
the user role gets "not a member of workspace" until a platform admin adds
its subject, and that `openshell status` then reports "authorization denied".

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Signed-off-by: Hai Nguyen <[email protected]>
@gaveezy

gaveezy commented Nov 13, 2026

Copy link
Copy Markdown
Contributor Author

@sandl99 this PR fix a bug in the last example, can you review it

@cv
cv merged commit 96d15ff into v1 Nov 13, 2026
70 checks passed
@cv
cv deleted the fix/kubernetes-examples-p1 branch November 13, 2026 23:42
cv pushed a commit that referenced this pull request Nov 13, 2026
…n index (#14428)

For Kubernetes users: two new examples and an index for
`examples/kubernetes`. `sandbox-placement.yaml` places an agent's pod
and sizes its resources and workspace claim. `published-nodeport.yaml`
publishes a managed `oidc` gateway through a NodePort Service.
`README.md` says which example shows what and what CI tests.

## Problem

- **No index:** `examples/kubernetes` had nine files and no README, and
no guide linked `hosted-nvidia.*` or `gpu-autoscaling/`; only the
release notes link the latter.
- **No placement or sizing example:** none showed
`driverConfig.kubernetes` pod placement or resources, or the workspace
StorageClass and size settings, though the guide mentions
`workspace_storage_class` for large fleets.
- **No published-gateway example:** #14324 added
`gateway.kubernetes.expose`, but the guide shows only a fragment of it.

## Change

- **`examples/kubernetes/sandbox-placement.yaml`:**
- A managed `development` gateway with `allow_driver_config`,
`workspace_storage_class` and `workspace_default_storage_size`.
- One OpenClaw sandbox whose `driverConfig.kubernetes` sets a node
selector, a toleration, and CPU and memory requests and limits.
- The comments say that these settings reach the agent's pod, not the
supervisor's, and that OpenShell refuses extended resources other than
`nvidia.com/gpu`.
- **`examples/kubernetes/published-nodeport.yaml`:**
- A managed gateway with the `oidc` profile and `expose.service.type:
NodePort`, and one OpenClaw sandbox.
- It uses `oidc` because only that profile lets a client other than
NemoClaw sign in at the published endpoint.
- The comments cover which endpoint host and port to choose, that both
are fixed at the first plan, that every node listens on the port, and
that NemoClaw itself still forwards a port.
- **`examples/kubernetes/README.md`:**
- A table of the examples: gateway, agents and inference, and what each
shows.
- What CI tests: schema and compile for every YAML, and integration
scenarios for `hosted-nvidia.*` and `external-gateway.yaml`. No CI job
deploys these files.
- **`docs/build-deploy/kubernetes.md`** links the index and both
examples.
- **`deployment_journey.rs`:** the authoring test that offers each
single-sandbox example's values now counts the two new examples.

Left out:
- `runtime_class_name` and `priority_class_name`: OpenShell admits those
classes only with its attachable labels, which this change didn't try.
- A `LoadBalancer` variant: the pinned chart can't take one yet
(#14421).
- A row for `managed-oidc.yaml`, which #14427 adds; whichever of the two
merges second adds the row.
- Correcting the guide where it says a published gateway still requires
its client certificate. The pinned gateway asks for one but doesn't
require it: in the k3d run below, a call with only an OIDC token
succeeded. That fix touches text outside this change.

## Beyond CI

Both examples ran on two clusters:
- single-node k3s v1.37.1+k3s1;
- three nodes from k3d v5.9.0 with the same k3s, one of them tainted and
labelled.

Both clusters had Agent Sandbox 1.0.3 and the `local-path` StorageClass.
Both runs used hosted NVIDIA inference and images built from c5bf74d
in a loopback registry.
Each run changed only the values its header says to replace.
For each one: plan and apply succeeded, the agent replied, a second plan
found no changes, and destroy succeeded.

- **`sandbox-placement.yaml`:**
- On k3s, the node label was the node's `kubernetes.io/hostname`, and
the node had no taint.
- On k3d, the example kept its own label and toleration. One node had
the label `nodepool.example.com/name=agents` and the taint
`dedicated=agents:NoSchedule`, and a plain pod selecting that label
stayed Pending because of the taint.
- The agent's pod ran on that node, with the node selector, the
toleration, and the requests and limits. The supervisor's and the
gateway's pods ran on an untainted node, and the supervisor's had none
of these settings.
- The workspace claim was 10Gi from `local-path`, bound on the agent's
node, and writable by the agent.
- **`published-nodeport.yaml`:**
- On k3s, the endpoint was the node's address on port 30080, with the
e2e test issuer for OIDC.
- On k3d, the endpoint was the control-plane node's address on port
30080, with Keycloak 26.4 in the cluster as the issuer. The example kept
its `roles_claim`, `admin_role` and `user_role`.
- The gateway's Service was a NodePort on 30080, and its certificate
named the endpoint's address.
- The agent replied to a client that dialed the endpoint, without a port
forward.
- On k3d, OpenShell's `GetCurrentUser` succeeded through each of the
three nodes' port 30080, checking the certificate against the endpoint's
address. Two of those nodes ran no gateway pod.
- On k3d, the pinned OpenShell CLI 0.1.7 signed in at the endpoint with
a confidential client, as the guide describes. It invoked the agent with
`openshell sandbox exec` and got `READY`.
- **Not run:** nodes on separate hosts. The k3d nodes were containers on
one host.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Signed-off-by: Hai Nguyen <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: docs Documentation, examples, guides, or docs build area: security Security controls, permissions, secrets, or hardening chore Build, CI, dependency, or tooling maintenance platform: k8s Affects Kubernetes deployments v1 NemoClaw v1 branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants