Skip to content

Commit 64f9a35

Browse files
committed
fix(helm): improve GatewayClass overrides
Give chart-created Gateways a distinct default name and allow GRPCRoutes to select a listener on an existing Gateway. Fixes: #2469 Signed-off-by: Daneyon Hansen <[email protected]>
1 parent 0e981be commit 64f9a35

10 files changed

Lines changed: 200 additions & 26 deletions

File tree

‎.agents/skills/helm-dev-environment/SKILL.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -236,8 +236,9 @@ cd ../../..
236236
KUBECONFIG=kubeconfig mise run helm:gateway:apply
237237
```
238238

239-
`values-gateway.yaml` creates a `Gateway` (listener on port 80, class `eg`) and
240-
`GRPCRoute` in the `openshell` namespace. The `high-availability` profile
239+
`values-gateway.yaml` creates the `openshell-ingress` `Gateway` (listener on
240+
port 80, class `eg`) and an `openshell` `GRPCRoute` in the `openshell`
241+
namespace. The `high-availability` profile
241242
installs the Envoy Gateway Helm chart and layers both
242243
`values-high-availability.yaml` and `values-gateway.yaml` onto the OpenShell
243244
release.
@@ -257,7 +258,7 @@ as a real reverse proxy while gateway pods rotate behind it:
257258

258259
```bash
259260
KUBECONFIG=kubeconfig kubectl get svc -A \
260-
-l gateway.envoyproxy.io/owning-gateway-name=openshell
261+
-l gateway.envoyproxy.io/owning-gateway-name=openshell-ingress
261262
KUBECONFIG=kubeconfig kubectl -n <envoy-service-namespace> port-forward \
262263
svc/<envoy-service-name> 8080:80
263264
openshell gateway add http://127.0.0.1:8080 --name openshell --local

‎deploy/helm/openshell/README.md‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -366,14 +366,15 @@ discovery endpoint or its TLS CA.
366366
| grpcRoute.backendTLSPolicy.enabled | bool | `false` | Create a BackendTLSPolicy resource for end-to-end TLS between the Gateway proxy and the OpenShell gateway pod. The traffic flow is: client → HTTPS → Gateway (terminate) → TLS (re-encrypt) → gateway pod. Requires server.disableTls=false and server.tls.enableMtls=false. The certgen hook auto-creates the backend CA ConfigMap. |
367367
| grpcRoute.backendTLSPolicy.hostname | string | `""` | Hostname the Gateway proxy validates against the backend's TLS certificate SAN. Defaults to the service FQDN (`<fullname>.<namespace>.svc.cluster.local`) when empty, which matches the SAN included by both cert-manager and the pkiInitJob. |
368368
| grpcRoute.enabled | bool | `false` | Create a Gateway API GRPCRoute for the gateway service. |
369-
| grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". |
370-
| grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. |
369+
| grpcRoute.gateway.className | string | `"eg"` | GatewayClass for a chart-created Gateway. Set this to the name exposed by your Gateway API implementation. Envoy Gateway uses "eg" in the OpenShell examples. Ignored when gateway.create is false. |
370+
| grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. Use sectionName to select a specific listener when needed. |
371371
| grpcRoute.gateway.listener.allowedRoutes | string | `"Same"` | "Same" restricts attached routes to the release namespace; "All" allows any namespace. |
372372
| grpcRoute.gateway.listener.port | int | `80` | Listener port for the generated Gateway resource. Use 443 with protocol HTTPS. |
373-
| grpcRoute.gateway.listener.protocol | string | `"HTTP"` | Listener protocol for the generated Gateway resource: HTTP or HTTPS. HTTPS terminates TLS at the Envoy Gateway listener; pair it with server.disableTls=true so Envoy forwards plaintext to the gateway pod, and use OIDC for client identity (the gateway never sees the client cert). |
373+
| grpcRoute.gateway.listener.protocol | string | `"HTTP"` | Listener protocol for the generated Gateway resource: HTTP or HTTPS. HTTPS terminates TLS at the Gateway listener; pair it with server.disableTls=true so the proxy forwards plaintext to the gateway pod, and use OIDC for client identity (the gateway never sees the client certificate). |
374374
| grpcRoute.gateway.listener.tls.certificateRefs | list | `[]` | certificateRefs for the HTTPS listener. Required when protocol is HTTPS. Each entry needs a `name` pointing at a kubernetes.io/tls Secret in the Gateway's namespace. May reference a cert-manager-issued Secret or the existing openshell-server-tls Secret (its SANs must include the external hostname). |
375-
| grpcRoute.gateway.name | string | `""` | Name of the Gateway resource. Defaults to the chart fullname. |
376-
| grpcRoute.gateway.namespace | string | `""` | Namespace of the Gateway referenced by the GRPCRoute parentRef. Defaults to the release namespace. |
375+
| grpcRoute.gateway.name | string | `""` | Name of the Gateway resource. Defaults to `<fullname>-ingress`, which stays distinct from the OpenShell Service name. When gateway.create is false, set this to the name of the existing Gateway. |
376+
| grpcRoute.gateway.namespace | string | `""` | Namespace of the Gateway referenced by the GRPCRoute parentRef. Defaults to the release namespace. A chart-created Gateway always resides in the release namespace. |
377+
| grpcRoute.gateway.sectionName | string | `""` | Listener name on an existing Gateway. Leave empty to let Gateway API listener matching select compatible listeners. |
377378
| grpcRoute.hostnames | list | `[]` | Hostnames the GRPCRoute matches on. Leave empty to match all hosts. |
378379
| grpcRoute.replicaRouting.enabled | bool | `false` | Route requests carrying an `x-openshell-replica` header straight to that gateway replica. The CLI sets the header on long-lived sandbox connections (SSH, port forwards, exec) using the owner the gateway reports, so they skip the relay through a peer replica. Renders one Service per replica. Requires workload.kind=statefulset, workload.allowMultiReplicaStatefulSet=true, and at most 15 replicas. |
379380
| imagePullSecrets | list | `[]` | Image pull secrets attached to gateway and helper pods. |

‎deploy/helm/openshell/ci/values-gateway.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@
1111
# Service (named envoy-<namespace>-<gateway-name>-*) in the openshell namespace.
1212
#
1313
# To reach the gateway from outside a k3d cluster, port-forward to that service:
14-
# kubectl -n openshell get svc -l gateway.envoyproxy.io/owning-gateway-name=openshell
14+
# kubectl -n openshell get svc -l gateway.envoyproxy.io/owning-gateway-name=openshell-ingress
1515
# kubectl -n openshell port-forward svc/<envoy-service> 8080:80
1616
# # then: grpcurl -plaintext localhost:8080 ...
1717

‎deploy/helm/openshell/templates/_helpers.tpl‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -24,6 +24,16 @@ Create a default fully qualified app name.
2424
{{- end }}
2525
{{- end }}
2626

27+
{{/*
28+
Name of the Gateway API Gateway referenced by the GRPCRoute. Keep the default
29+
distinct from the OpenShell Service because some Gateway implementations name
30+
their generated proxy Service after the Gateway.
31+
*/}}
32+
{{- define "openshell.grpcRouteGatewayName" -}}
33+
{{- $fullname := include "openshell.fullname" . | trunc 55 | trimSuffix "-" -}}
34+
{{- default (printf "%s-ingress" $fullname) .Values.grpcRoute.gateway.name | trunc 63 | trimSuffix "-" -}}
35+
{{- end }}
36+
2737
{{/*
2838
Create chart name and version as used by the chart label.
2939
*/}}

‎deploy/helm/openshell/templates/gateway.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
apiVersion: gateway.networking.k8s.io/v1
55
kind: Gateway
66
metadata:
7-
name: {{ default (include "openshell.fullname" .) .Values.grpcRoute.gateway.name }}
7+
name: {{ include "openshell.grpcRouteGatewayName" . }}
88
namespace: {{ .Release.Namespace }}
99
labels:
1010
{{- include "openshell.labels" . | nindent 4 }}

‎deploy/helm/openshell/templates/grpcroute.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,8 +11,11 @@ metadata:
1111
{{- include "openshell.labels" . | nindent 4 }}
1212
spec:
1313
parentRefs:
14-
- name: {{ default $fullname .Values.grpcRoute.gateway.name }}
15-
namespace: {{ default .Release.Namespace .Values.grpcRoute.gateway.namespace }}
14+
- name: {{ include "openshell.grpcRouteGatewayName" . }}
15+
namespace: {{ ternary .Release.Namespace (default .Release.Namespace .Values.grpcRoute.gateway.namespace) .Values.grpcRoute.gateway.create }}
16+
{{- with .Values.grpcRoute.gateway.sectionName }}
17+
sectionName: {{ . | quote }}
18+
{{- end }}
1619
{{- if .Values.grpcRoute.hostnames }}
1720
hostnames:
1821
{{- toYaml .Values.grpcRoute.hostnames | nindent 4 }}
Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
suite: Gateway API resources
5+
templates:
6+
- templates/gateway.yaml
7+
- templates/grpcroute.yaml
8+
release:
9+
name: openshell
10+
namespace: openshell
11+
12+
tests:
13+
- it: gives a chart-created Gateway a non-conflicting default name
14+
template: templates/gateway.yaml
15+
set:
16+
grpcRoute.enabled: true
17+
grpcRoute.gateway.create: true
18+
asserts:
19+
- equal:
20+
path: metadata.name
21+
value: openshell-ingress
22+
- equal:
23+
path: spec.gatewayClassName
24+
value: eg
25+
26+
- it: references the same default Gateway from the GRPCRoute
27+
template: templates/grpcroute.yaml
28+
set:
29+
grpcRoute.enabled: true
30+
grpcRoute.gateway.create: true
31+
asserts:
32+
- equal:
33+
path: spec.parentRefs[0].name
34+
value: openshell-ingress
35+
- equal:
36+
path: spec.parentRefs[0].namespace
37+
value: openshell
38+
- notExists:
39+
path: spec.parentRefs[0].sectionName
40+
41+
- it: keeps a chart-created Gateway parent in the release namespace
42+
template: templates/grpcroute.yaml
43+
set:
44+
grpcRoute.enabled: true
45+
grpcRoute.gateway.create: true
46+
grpcRoute.gateway.namespace: another-namespace
47+
asserts:
48+
- equal:
49+
path: spec.parentRefs[0].namespace
50+
value: openshell
51+
52+
- it: renders an arbitrary GatewayClass and explicit Gateway name
53+
template: templates/gateway.yaml
54+
set:
55+
grpcRoute.enabled: true
56+
grpcRoute.gateway.create: true
57+
grpcRoute.gateway.className: custom-class
58+
grpcRoute.gateway.name: custom-ingress
59+
asserts:
60+
- equal:
61+
path: metadata.name
62+
value: custom-ingress
63+
- equal:
64+
path: spec.gatewayClassName
65+
value: custom-class
66+
67+
- it: preserves the ingress suffix for a long chart fullname
68+
template: templates/gateway.yaml
69+
set:
70+
fullnameOverride: abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijk
71+
grpcRoute.enabled: true
72+
grpcRoute.gateway.create: true
73+
asserts:
74+
- equal:
75+
path: metadata.name
76+
value: abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabc-ingress
77+
78+
- it: attaches to a listener on an existing Gateway
79+
template: templates/grpcroute.yaml
80+
set:
81+
grpcRoute.enabled: true
82+
grpcRoute.gateway.create: false
83+
grpcRoute.gateway.name: platform-gateway
84+
grpcRoute.gateway.namespace: gateway-system
85+
grpcRoute.gateway.sectionName: grpc
86+
asserts:
87+
- equal:
88+
path: spec.parentRefs[0].name
89+
value: platform-gateway
90+
- equal:
91+
path: spec.parentRefs[0].namespace
92+
value: gateway-system
93+
- equal:
94+
path: spec.parentRefs[0].sectionName
95+
value: grpc
96+
97+
- it: does not create a Gateway when attaching to an existing Gateway
98+
template: templates/gateway.yaml
99+
set:
100+
grpcRoute.enabled: true
101+
grpcRoute.gateway.create: false
102+
grpcRoute.gateway.name: platform-gateway
103+
asserts:
104+
- hasDocuments:
105+
count: 0

‎deploy/helm/openshell/values.yaml‎

Lines changed: 21 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -673,9 +673,10 @@ certManager:
673673
serverIpAddresses:
674674
- 127.0.0.1
675675

676-
# Kubernetes Gateway API - HTTPRoute and Gateway resources.
677-
# Requires a Gateway API controller in the cluster. Install Envoy Gateway via
678-
# the skaffold.yaml releases or independently:
676+
# Kubernetes Gateway API - GRPCRoute and Gateway resources.
677+
# Requires a Gateway API controller in the cluster. OpenShell tests the
678+
# examples with Envoy Gateway, which can be installed via skaffold.yaml or
679+
# independently:
679680
# helm install eg oci://docker.io/envoyproxy/gateway-helm \
680681
# --version v1.4.1 -n envoy-gateway-system --create-namespace
681682
grpcRoute:
@@ -685,23 +686,33 @@ grpcRoute:
685686
hostnames: []
686687
gateway:
687688
# -- When true, a Gateway resource is created in the release namespace.
688-
# Set to false and provide name/namespace to attach to a pre-existing Gateway.
689+
# Set to false and provide name/namespace to attach to a pre-existing
690+
# Gateway. Use sectionName to select a specific listener when needed.
689691
create: false
690-
# -- GatewayClass to reference. Envoy Gateway installs one named "eg".
692+
# -- GatewayClass for a chart-created Gateway. Set this to the name exposed
693+
# by your Gateway API implementation. Envoy Gateway uses "eg" in the
694+
# OpenShell examples. Ignored when gateway.create is false.
691695
className: "eg"
692-
# -- Name of the Gateway resource. Defaults to the chart fullname.
696+
# -- Name of the Gateway resource. Defaults to `<fullname>-ingress`, which
697+
# stays distinct from the OpenShell Service name. When gateway.create is
698+
# false, set this to the name of the existing Gateway.
693699
name: ""
694700
# -- Namespace of the Gateway referenced by the GRPCRoute parentRef.
695-
# Defaults to the release namespace.
701+
# Defaults to the release namespace. A chart-created Gateway always resides
702+
# in the release namespace.
696703
namespace: ""
704+
# -- Listener name on an existing Gateway. Leave empty to let Gateway API
705+
# listener matching select compatible listeners.
706+
sectionName: ""
697707
# Listener settings (only used when gateway.create is true).
698708
listener:
699709
# -- Listener port for the generated Gateway resource. Use 443 with protocol HTTPS.
700710
port: 80
701711
# -- Listener protocol for the generated Gateway resource: HTTP or HTTPS.
702-
# HTTPS terminates TLS at the Envoy Gateway listener; pair it with
703-
# server.disableTls=true so Envoy forwards plaintext to the gateway pod,
704-
# and use OIDC for client identity (the gateway never sees the client cert).
712+
# HTTPS terminates TLS at the Gateway listener; pair it with
713+
# server.disableTls=true so the proxy forwards plaintext to the gateway
714+
# pod, and use OIDC for client identity (the gateway never sees the client
715+
# certificate).
705716
protocol: HTTP
706717
# -- "Same" restricts attached routes to the release namespace; "All" allows any namespace.
707718
allowedRoutes: Same

‎docs/kubernetes/ingress.mdx‎

Lines changed: 45 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ position: 5
1010

1111
By default, the OpenShell gateway is only reachable inside the cluster. To let CLI clients connect without a `kubectl port-forward`, expose the gateway through an ingress.
1212

13-
OpenShell uses the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io) for ingress. The chart creates a `GRPCRoute` that routes inbound gRPC traffic to the gateway pod. You need a Gateway API implementation installed on your cluster to fulfill the `GRPCRoute`. This page uses [Envoy Gateway](https://gateway.envoyproxy.io), which the chart is tested with.
13+
OpenShell uses the [Kubernetes Gateway API](https://gateway-api.sigs.k8s.io) for ingress. The chart creates a `GRPCRoute` that routes inbound gRPC traffic to the gateway pod. You need a Gateway API implementation installed on your cluster to fulfill the `GRPCRoute`. The examples use [Envoy Gateway](https://gateway.envoyproxy.io), but you can select another implementation through `grpcRoute.gateway.className`.
1414

1515
## Install Envoy Gateway
1616

@@ -62,14 +62,56 @@ helm upgrade --install openshell \
6262
--set grpcRoute.gateway.className=eg
6363
```
6464

65+
The generated Gateway defaults to `<fullname>-ingress`, while the OpenShell Service keeps the `<fullname>` name. Keeping these names distinct avoids collisions with Gateway implementations that name their proxy Service after the Gateway.
66+
67+
<Note>
68+
Existing releases that created a Gateway under the previous `<fullname>` default can retain that resource name by setting `grpcRoute.gateway.name=<fullname>` before upgrading.
69+
</Note>
70+
71+
### Use another GatewayClass
72+
73+
Set `grpcRoute.gateway.className` to any GatewayClass installed by your Gateway API implementation:
74+
75+
```shell
76+
helm upgrade --install openshell \
77+
oci://ghcr.io/nvidia/openshell/helm-chart \
78+
--version <version> \
79+
--namespace openshell \
80+
--set grpcRoute.enabled=true \
81+
--set grpcRoute.gateway.create=true \
82+
--set grpcRoute.gateway.className=<gateway-class>
83+
```
84+
85+
For ingress, the OpenShell chart creates only the Gateway and GRPCRoute. Install and manage the selected controller, GatewayClass, and Gateway API CRDs separately.
86+
87+
### Use an existing Gateway
88+
89+
To attach OpenShell to a platform-managed Gateway, disable Gateway creation and set the parent Gateway name and namespace. Set `sectionName` when the route should attach to a specific listener:
90+
91+
```shell
92+
helm upgrade --install openshell \
93+
oci://ghcr.io/nvidia/openshell/helm-chart \
94+
--version <version> \
95+
--namespace openshell \
96+
--set grpcRoute.enabled=true \
97+
--set grpcRoute.gateway.create=false \
98+
--set grpcRoute.gateway.name=platform-gateway \
99+
--set grpcRoute.gateway.namespace=gateway-system \
100+
--set grpcRoute.gateway.sectionName=grpc
101+
```
102+
103+
The existing Gateway listener must allow routes from the OpenShell release namespace.
104+
65105
## Get the external address
66106

67-
After the Gateway is provisioned, Envoy Gateway creates a LoadBalancer service in the `openshell` namespace. Wait for it to get an external address:
107+
With the Envoy Gateway example above, the controller creates a LoadBalancer service in the `openshell` namespace. Wait for it to get an external address:
68108

69109
```shell
70-
kubectl -n openshell get svc -l gateway.envoyproxy.io/owning-gateway-name=openshell
110+
kubectl -n openshell get svc -l gateway.envoyproxy.io/owning-gateway-name=openshell-ingress
71111
```
72112

113+
For another Gateway API implementation or an existing Gateway, follow that implementation's instructions to find the external address.
114+
73115
After the `EXTERNAL-IP` is assigned, register the gateway with the CLI:
74116

75117
```shell

‎skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1036,7 +1036,8 @@ credential failures.
10361036
| Custom compute driver is unavailable | Driver process/socket missing, inaccessible, or selected name does not match its endpoint/config key | Socket ownership/mode, driver service logs, gateway `GetCapabilities` logs |
10371037
| Sandbox remains `Stopping` or `Starting` | Driver stop/start failed, retained resource is missing, or a fresh supervisor has not connected | Gateway and driver logs; `docker inspect`, `podman inspect`, Agent Sandbox status/PVC, or VM state marker and launcher process |
10381038
| Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials |
1039-
| Gateway API resources fail with `the server could not find the requested resource` | Optional Gateway API resources were applied without Envoy Gateway CRDs | Install Envoy Gateway and enable `grpcRoute` before applying the optional ingress resources |
1039+
| Gateway API resources fail with `the server could not find the requested resource` | Optional Gateway API resources were applied without Gateway API CRDs | Install the standard Gateway API CRDs and selected controller, then enable `grpcRoute` before applying the optional ingress resources |
1040+
| A custom GatewayClass does not program the chart-created Gateway, or its proxy Service conflicts with the OpenShell Service | The selected controller is unavailable, or an explicit `grpcRoute.gateway.name` reuses the OpenShell Service name | Check `kubectl get gatewayclass,gateway,grpcroute -A`; leave the Gateway name empty to use `<fullname>-ingress`, or set a distinct name. For an existing Gateway, set `gateway.create=false` plus `gateway.name`, `gateway.namespace`, and optional `gateway.sectionName` |
10401041
| HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://<host>` |
10411042
| With `grpcRoute.replicaRouting.enabled=true`, sandbox SSH, forward, or exec still relay through a peer replica | A `<release>-replica-<i>` Service has no endpoint, so Envoy returns `Unavailable` and the CLI retries unrouted | `kubectl -n openshell get endpoints <release>-replica-0 <release>-replica-1`; confirm `workload.kind=statefulset` and the GRPCRoute status is `Accepted`/`ResolvedRefs` |
10421043
| HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https://<host> --oidc-issuer <url>`, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster |

0 commit comments

Comments
 (0)