Repository navigation
fix(kubernetes): shorten the HCL workspace and add an OIDC example - #14427
Merged
Merged
Conversation
- hosted-nvidia.tf defaulted its workspace to a 23-character name, which OpenShell refuses at apply after installing the gateway. Default to gtc-dc-k8s-cloud and check OpenShell's name rule at plan. - List a network plugin that enforces NetworkPolicy as a Kubernetes prerequisite, as OpenShell requires. - Add examples/kubernetes/managed-oidc.yaml: an OpenClaw and a Hermes agent on a managed gateway with the oidc profile and a registry pull Secret. - Document the ConfigMap command for caConfigMap, the admin role the gateway token needs, and the OpenShell CLI's client-credentials sign-in. Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Hai Nguyen <[email protected]>
Contributor
Author
Authoring session — opening
|
Contributor
A live run of the client-credentials sign-in showed that a token with only the user role gets "not a member of workspace" until a platform admin adds its subject, and that `openshell status` then reports "authorization denied". Co-Authored-By: Claude Opus 5.5 <[email protected]> Signed-off-by: Hai Nguyen <[email protected]>
Contributor
Author
|
@sandl99 this PR fix a bug in the last example, can you review it |
cv
pushed a commit
that referenced
this pull request
Nov 13, 2026
…n index (#14428) For Kubernetes users: two new examples and an index for `examples/kubernetes`. `sandbox-placement.yaml` places an agent's pod and sizes its resources and workspace claim. `published-nodeport.yaml` publishes a managed `oidc` gateway through a NodePort Service. `README.md` says which example shows what and what CI tests. ## Problem - **No index:** `examples/kubernetes` had nine files and no README, and no guide linked `hosted-nvidia.*` or `gpu-autoscaling/`; only the release notes link the latter. - **No placement or sizing example:** none showed `driverConfig.kubernetes` pod placement or resources, or the workspace StorageClass and size settings, though the guide mentions `workspace_storage_class` for large fleets. - **No published-gateway example:** #14324 added `gateway.kubernetes.expose`, but the guide shows only a fragment of it. ## Change - **`examples/kubernetes/sandbox-placement.yaml`:** - A managed `development` gateway with `allow_driver_config`, `workspace_storage_class` and `workspace_default_storage_size`. - One OpenClaw sandbox whose `driverConfig.kubernetes` sets a node selector, a toleration, and CPU and memory requests and limits. - The comments say that these settings reach the agent's pod, not the supervisor's, and that OpenShell refuses extended resources other than `nvidia.com/gpu`. - **`examples/kubernetes/published-nodeport.yaml`:** - A managed gateway with the `oidc` profile and `expose.service.type: NodePort`, and one OpenClaw sandbox. - It uses `oidc` because only that profile lets a client other than NemoClaw sign in at the published endpoint. - The comments cover which endpoint host and port to choose, that both are fixed at the first plan, that every node listens on the port, and that NemoClaw itself still forwards a port. - **`examples/kubernetes/README.md`:** - A table of the examples: gateway, agents and inference, and what each shows. - What CI tests: schema and compile for every YAML, and integration scenarios for `hosted-nvidia.*` and `external-gateway.yaml`. No CI job deploys these files. - **`docs/build-deploy/kubernetes.md`** links the index and both examples. - **`deployment_journey.rs`:** the authoring test that offers each single-sandbox example's values now counts the two new examples. Left out: - `runtime_class_name` and `priority_class_name`: OpenShell admits those classes only with its attachable labels, which this change didn't try. - A `LoadBalancer` variant: the pinned chart can't take one yet (#14421). - A row for `managed-oidc.yaml`, which #14427 adds; whichever of the two merges second adds the row. - Correcting the guide where it says a published gateway still requires its client certificate. The pinned gateway asks for one but doesn't require it: in the k3d run below, a call with only an OIDC token succeeded. That fix touches text outside this change. ## Beyond CI Both examples ran on two clusters: - single-node k3s v1.37.1+k3s1; - three nodes from k3d v5.9.0 with the same k3s, one of them tainted and labelled. Both clusters had Agent Sandbox 1.0.3 and the `local-path` StorageClass. Both runs used hosted NVIDIA inference and images built from c5bf74d in a loopback registry. Each run changed only the values its header says to replace. For each one: plan and apply succeeded, the agent replied, a second plan found no changes, and destroy succeeded. - **`sandbox-placement.yaml`:** - On k3s, the node label was the node's `kubernetes.io/hostname`, and the node had no taint. - On k3d, the example kept its own label and toleration. One node had the label `nodepool.example.com/name=agents` and the taint `dedicated=agents:NoSchedule`, and a plain pod selecting that label stayed Pending because of the taint. - The agent's pod ran on that node, with the node selector, the toleration, and the requests and limits. The supervisor's and the gateway's pods ran on an untainted node, and the supervisor's had none of these settings. - The workspace claim was 10Gi from `local-path`, bound on the agent's node, and writable by the agent. - **`published-nodeport.yaml`:** - On k3s, the endpoint was the node's address on port 30080, with the e2e test issuer for OIDC. - On k3d, the endpoint was the control-plane node's address on port 30080, with Keycloak 26.4 in the cluster as the issuer. The example kept its `roles_claim`, `admin_role` and `user_role`. - The gateway's Service was a NodePort on 30080, and its certificate named the endpoint's address. - The agent replied to a client that dialed the endpoint, without a port forward. - On k3d, OpenShell's `GetCurrentUser` succeeded through each of the three nodes' port 30080, checking the certificate against the endpoint's address. Two of those nodes ran no gateway pod. - On k3d, the pinned OpenShell CLI 0.1.7 signed in at the endpoint with a confidential client, as the guide describes. It invoked the agent with `openshell sandbox exec` and got `READY`. - **Not run:** nodes on separate hosts. The k3d nodes were containers on one host. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Hai Nguyen <[email protected]> Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For Kubernetes users:
examples/kubernetes/hosted-nvidia.tfnow applies with its default workspace, the Kubernetes guide lists the NetworkPolicy enforcement that OpenShell requires, and the newexamples/kubernetes/managed-oidc.yamlis a complete document for a managed gateway with theoidcprofile and a registry pull Secret.Problem
hosted-nvidia.tf, whichexamples/gtc-dc/kubernetes-amd64.tflinks to, defaultsworkspacetogtc-dc-kubernetes-cloud, 23 characters. The pinned OpenShell refuses a workspace name over 19 characters (MAX_ROUTABLE_NAME_LEN), and only when apply creates the workspace, after the gateway is installed. The run recorded on feat(examples): manage GTC DC Kubernetes gateways #14315 hit this. The HCL plan scenario passed, because plan doesn't create the workspace.oidcprofile on its own, orimagePullSecrets. The only OIDC example,gpu-autoscaling/two-runtimes.yaml, depends on the GPU chart. The guide also didn't say:caConfigMap;Change
hosted-nvidia.tf:workspacetogtc-dc-k8s-cloud.validationblock with OpenShell's name rule: at most 19 characters, a DNS label, and no--.gtc-dc-kubernetes-amd64-hcl.tftest.hclplans the old default and expects that validation to fail.examples/gtc-dc/README.mdstates the limit.kubernetes.md): add a network plugin that enforces ingress and egress NetworkPolicy, linked to OpenShell's requirement. Say that nothing reports a plugin that ignores NetworkPolicy. Add it to the threat model's "Relies on" column.examples/kubernetes/managed-oidc.yaml:oidcprofile withcaConfigMapand a token, andimagePullSecrets.kubectl create configmapcommand forcaConfigMap.admin_rolenames; without it, plan fails withobservation permission denied.scopes_claimis set, the token also needs the same scopes as an external gateway's token.OPENSHELL_OIDC_CLIENT_SECRET), withopenshell gateway loginto sign in again, because the pinned CLI doesn't renew that token (bug(cli): client-credentials gateway login stops working when the access token expires OpenShell#4287).admin_rolenames reaches the deployment's workspace. A token with only the roleuser_rolenames needs a platform admin to add its subject to the workspace first, and with only that roleopenshell statusreportsauthorization denied.Left out:
managed-oidc.yamlrow inexamples/kubernetes/README.md, which docs(examples): add Kubernetes placement and NodePort examples with an index #14428 adds; whichever of the two merges second adds the row.mtlsdirectory. That fix touches text outside this change.Beyond CI
managed-oidc.yamlon k3s: applied on single-node k3s v1.37.1+k3s1 with Agent Sandbox 1.0.3, thelocal-pathStorageClass, and k3s's NetworkPolicy controller.issuer-caConfigMap.registry-credentialswas on the gateway, supervisor and sandbox pods.observation permission denied.hosted-nvidia.tfon k3s, with its new default workspace: the same cluster and images, with OpenTofu and the providers from a bundle built at c5bf74d.examples/gtc-dc/README.mdsays. Only the provider versions, UID, context, namespace and images were set;workspacekept its default.gtc-dc-k8s-cloud.READYthroughfabric-agent-ctl invoke, and each had its claim'sworkspace/andartifacts/mounted and writable.published-nodeport.yaml, which uses the sameoidcprofile; NemoClaw's own token came from one of the clients.<release>-client-tls,SSL_CERT_FILEset to the issuer's CA, thenopenshell gateway addwithOPENSHELL_OIDC_CLIENT_SECRET. The CLI printedAuthenticated via client credentials, andopenshell statusreportedConnectedandAuthenticated (OIDC).openshell sandbox listlisted the agent, andopenshell sandbox exec … fabric-agent-ctl invokegotREADY.openshell statusreportedAuthenticated (OIDC; authorization denied), and sandbox commands failed withnot a member of workspace. Afteropenshell workspace member add … --role user, the same client listed and invoked the agent, and a NemoClaw plan still found no changes.🤖 Generated with Claude Code