You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 3baa673
Browse filesBrowse the repository at this point in the historyBrowse files
authored
feat(kubernetes): publish a managed gateway through a NodePort Service (#14324)
A managed Kubernetes or OpenShift gateway can be published at
`spec.gateway.endpoint` through the OpenShell chart's own NodePort
Service, with `spec.gateway.kubernetes.expose`, so clients such as the
OpenShell CLI reach it there, as at a Docker gateway's published port.
NemoClaw's own commands keep their port-forward.
## Problem
A managed Kubernetes gateway has no endpoint to dial. The schema
requires `https://127.0.0.1:PORT`, nothing listens there, and each
provider port-forwards to the gateway pod while OpenTofu runs. People
reach the gateway with `kubectl port-forward` and three copied
certificate files.
Part of #14313. This follows the decisions recorded on #14313 after this
PR's design reviews:
- The chart's own Service publishes the gateway, so NemoClaw bundles no
Kubernetes provider.
- LoadBalancer waits for chart inputs, which NVIDIA/OpenShell#5500
requests (#14421).
- Kubernetes, not NemoClaw, checks the node-port range, and only the
Service type is fixed after the first plan.
- Other clients dial the published endpoint; NemoClaw's own plan, apply,
and destroy keep their port-forward, so they still need
`pods/portforward`. #14422 tracks an opt-in direct dial.
Left for #14313: runs on clusters other than kind, including OpenShift.
Sandbox service URLs on a published gateway still name
`openshell.localhost` and the gateway pod's port, so they don't open
from other machines. NemoClaw leaves them to OpenShell instead of
working around it: #14316 waits on OpenShell to publish them at a
configured origin (NVIDIA/OpenShell#4953).
## Change
- `gateway.kubernetes.expose.service.type` takes `NodePort`. With
`expose`, the endpoint is `https://`, a lowercase DNS name or IPv4
address, and a port; Kubernetes checks the port against the cluster's
node-port range. Without `expose`, the `127.0.0.1` rule stays. The rule
moves to the document schema, because the platform resources rebuild the
Gateway definition from attributes that don't carry `expose`.
- The Helm values set the chart's Service to `type: NodePort` with
`nodePort` at the endpoint's port, which forwards to the gateway's TLS
listener, so TLS and client certificates pass through.
`service.metricsPort: 0` keeps the plaintext metrics port off the node:
at 0 the pinned chart publishes no metrics port and the gateway serves
no metrics (`templates/service.yaml`, `_toml.tpl`). Helm's wait and the
release's existing binding cover the Service, so NemoClaw declares no
Service of its own.
- The chart's `pkiInitJob` names the endpoint host in the gateway
certificate.
- The providers' connection forwards a free loopback port to the gateway
pod and names `127.0.0.1`, which the chart's certificate always covers,
also when the endpoint names the published host.
- Changing `expose` or its Service type after the first plan is refused,
like changing the endpoint.
- The intent record moves to version 21, so this release refuses state
that an earlier build wrote.
- LoadBalancer is left out: the pinned chart's Service takes no
annotations, `loadBalancerSourceRanges` or `loadBalancerClass`, and its
port is the gateway's own listening port, 8080. A limits row records it,
#14421 tracks it, and NVIDIA/OpenShell#5500 asks for the chart inputs.
- A chart contract scenario renders the pinned chart with an exposed
gateway's values: the release's Service publishes only the gateway's
port at the endpoint's node port, the gateway configuration has no
metrics listener, and the certificate job names the host.
- The `kind-kubernetes-oidc` live case publishes its gateway through the
NodePort at the kind node's address, instead of a third case. Its CLI
goes through the port-forward, and its probe dials the node's address
with the gateway's client certificate and the token, as another client
would; each agent must reply there. Live / Kind keeps its 300 s wall
budget: in the last CI run before this change, three cases took 342.9 s
on linux_arm64 and 368.4 s on linux_amd64, the NodePort case 94.5 s and
97.6 s. `cargo ci live-kind` replaces a published case's endpoint host
with the node's address.
- The Kubernetes guide, the limits, the release notes, and the test
guides describe the change.
Deliberately left out: an OpenShift Route, and a ClusterIP dial from a
pod.
## Beyond CI
- `cargo ci live-kind` ran both kind cases on linux_arm64 within the 300
s budget: 198.6 s wall, the OIDC case 77.9 s. With the chart's Service
values removed, the same run's apply still passed through the
port-forward and both replies through the node's address failed.
- Unverified: a published gateway on a cluster other than kind,
including OpenShift. That the gateway serves no metrics at `metricsPort:
0` is checked only in the rendered chart.
---------
Signed-off-by: Hai Nguyen <[email protected]>
Signed-off-by: Carlos Villela <[email protected]>
Co-authored-by: Claude Opus 5.5 <[email protected]>
Co-authored-by: Carlos Villela <[email protected]>
/// How clients outside the cluster reach the gateway. With it, `endpoint` is the address they dial and the gateway certificate names its host. NemoClaw's own commands still forward a free local port to the gateway pod. Its Service type is fixed after the first successful plan.
/// The gateway chart's own Service, which publishes the endpoint's port and forwards it to the gateway pod. TLS and client certificates pass through to the gateway unchanged.
51
+
pubservice:ExposureService,
52
+
}
53
+
54
+
/// The Service that publishes a managed Kubernetes gateway.
json!({"x-nemoclaw-default-rule":if field == "endpoint"{
359
-
"Without kubernetes, omitted or empty selects the local HTTP endpoint. Kubernetes requires an explicit HTTPS loopback endpoint and port."
359
+
"Without kubernetes, omitted or empty selects the local HTTP endpoint. Kubernetes requires an explicit HTTPS endpoint and port: 127.0.0.1 without kubernetes.expose, or the address clients dial with it."
360
360
} else if field == "networkCIDR"{
361
361
"Without kubernetes, omitted or empty selects 172.30.N.0/24, where N is the first byte of SHA-256(metadata.uid). Excluded by kubernetes."
"x-nemoclaw-error":"managed Kubernetes requires an explicit HTTPS 127.0.0.1 endpoint with a nonzero port and excludes local engine settings"
375
+
"x-nemoclaw-error":"managed Kubernetes requires an explicit HTTPS endpoint with a lowercase host and a nonzero port and excludes local engine settings"
373
376
});
374
377
gateway["else"] = local;
375
378
// Podman's socket depends on the host user, so it has no default.
"Document::parse rejects YAML aliases, anchors, merge keys, all explicit tags (including core tags such as !!binary), duplicate keys, multiple documents, and input larger than 1 MiB. It applies the compiled input schema before defaulting; Document::validate applies the normalized schema and semantic checks, including for directly constructed Rust values.",
464
477
"The parser checks the shape of every endpoint URL and the gateway endpoint's transport and address policy; OpenShell checks inference destinations when sandboxes connect. It also checks managed gateway port bounds, canonical private IPv4 /24 networks, local engine socket syntax, and publication address/port/network agreement.",
465
-
"Managed Kubernetes requires a namespace and an authentication profile, and a kubeconfig environment reference and context together; omitting both selects the service account of the pod that runs NemoClaw. Agent Sandbox and a default StorageClass are platform prerequisites that are not checked. Its HTTPS endpoint is exactly 127.0.0.1 with an explicit port from 1 through 65535 and no path. Local engine fields and managed inference services are excluded; gateway.runtime.provider is kubernetes or openshift. OpenShift uses the upstream Kubernetes driver and requires platform-owned OpenShift security prerequisites. Namespace identity, ownership, and credential files are checked during operations.",
478
+
"Managed Kubernetes requires a namespace and an authentication profile, and a kubeconfig environment reference and context together; omitting both selects the service account of the pod that runs NemoClaw. Agent Sandbox and a default StorageClass are platform prerequisites that are not checked. Without kubernetes.expose, its HTTPS endpoint is exactly 127.0.0.1 with an explicit port from 1 through 65535 and no path; with it, the endpoint names a lowercase DNS name or IPv4 address and an explicit port, which the chart's NodePort Service publishes; Kubernetes checks that the port is in the cluster's node-port range. Local engine fields and managed inference services are excluded; gateway.runtime.provider is kubernetes or openshift. OpenShift uses the upstream Kubernetes driver and requires platform-owned OpenShift security prerequisites. Namespace identity, ownership, and credential files are checked during operations.",
466
479
"Explicit sandbox policies are also checked by the pinned OpenShell policy parser and validator, including protocol-specific rule semantics, process identities, filesystem paths, and destination address restrictions.",
467
480
"Explicit filesystem grants must permit reads of the paths the selected image's runtime requires, such as Fabric's /opt/fabric; parent and read-write grants count. Planning checks them against the image's metadata; it does not resolve symlinks or establish runtime permissions.",
468
481
"The schema requires an explicit default for multiple model choices. Rust checks unique route names, that the default names a route, and that native model and tool fields have valid structural shapes. Fabric validates adapter-specific combinations.",
0 commit comments