Skip to content

Commit d8bd7f9

Browse files
committed
feat(helm): add TLS termination for Envoy Gateway ingress
The chart's optional Gateway API ingress only rendered a plaintext HTTP listener, so the gateway could not be exposed over TLS. Add an HTTPS listener option that terminates TLS at the Envoy Gateway and forwards plaintext gRPC to the gateway pod. - gateway.yaml renders an HTTPS listener with `tls.mode: Terminate` and `certificateRefs` when `grpcRoute.gateway.listener.protocol=HTTPS`, keeping the default HTTP listener unchanged. Guards fail the render when `certificateRefs` is empty or `server.disableTls` is not true (the chart does not render a BackendTLSPolicy for re-encryption). - values.yaml adds `grpcRoute.gateway.listener.tls.certificateRefs`. - ci/values-gateway-tls.yaml exercises the HTTPS branch in lint/render. - docs/kubernetes/ingress.mdx documents HTTPS setup and clarifies that Envoy Gateway only terminates TLS (no OIDC SecurityPolicy); client identity uses OIDC bearer tokens, with the client-credentials grant for headless agents. - debug-openshell-cluster skill gains HTTPS-ingress troubleshooting rows. - Regenerated the chart README values table. Signed-off-by: Huabing (Robin) Zhao <[email protected]>
1 parent 3ace968 commit d8bd7f9

6 files changed

Lines changed: 123 additions & 5 deletions

File tree

‎.agents/skills/debug-openshell-cluster/SKILL.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -297,6 +297,8 @@ openshell logs <sandbox-name>
297297
| CLI TLS error | Local mTLS bundle does not match server cert/CA | Check `~/.config/openshell/gateways/<name>/mtls/` |
298298
| Image pull failure | Gateway or sandbox image cannot be pulled | Runtime events and image pull credentials |
299299
| `K8s namespace not ready` with `envoy-gateway-openshell.yaml: the server could not find the requested resource` | Optional Gateway API manifest was applied without Envoy Gateway CRDs, or k3s Helm controller startup exceeded the namespace wait | Apply `deploy/kube/manifests/envoy-gateway-openshell.yaml` manually only after Envoy Gateway is installed and `grpcRoute` is enabled |
300+
| HTTPS ingress (`grpcRoute.gateway.listener.protocol=HTTPS`) connection resets or TLS handshake hangs | Envoy terminates TLS but the gateway pod still expects TLS, so the plaintext backend hop fails | Set `server.disableTls=true` so Envoy forwards plaintext to the pod; verify the listener `certificateRefs` Secret exists in the release namespace and `openshell status` over `https://<host>` |
301+
| HTTPS ingress returns `Unauthenticated` after connecting | TLS terminates at Envoy, so the gateway never sees a client cert; no OIDC issuer is configured for identity | Configure `server.oidc.issuer` and register with `openshell gateway add https://<host> --oidc-issuer <url>`, or set `server.auth.allowUnauthenticatedUsers=true` for a trusted-proxy/dev cluster |
300302

301303
## Reporting
302304

‎deploy/helm/openshell/README.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -151,8 +151,9 @@ add `ci/values-spire.yaml` to the OpenShell release values files.
151151
| grpcRoute.gateway.className | string | `"eg"` | GatewayClass to reference. Envoy Gateway installs one named "eg". |
152152
| grpcRoute.gateway.create | bool | `false` | When true, a Gateway resource is created in the release namespace. Set to false and provide name/namespace to attach to a pre-existing Gateway. |
153153
| grpcRoute.gateway.listener.allowedRoutes | string | `"Same"` | "Same" restricts attached routes to the release namespace; "All" allows any namespace. |
154-
| grpcRoute.gateway.listener.port | int | `80` | Listener port for the generated Gateway resource. |
155-
| grpcRoute.gateway.listener.protocol | string | `"HTTP"` | Listener protocol for the generated Gateway resource. |
154+
| grpcRoute.gateway.listener.port | int | `80` | Listener port for the generated Gateway resource. Use 443 with protocol HTTPS. |
155+
| grpcRoute.gateway.listener.protocol | string | `"HTTP"` | Listener protocol for the generated Gateway resource: HTTP or HTTPS. HTTPS terminates TLS at the Envoy Gateway listener; pair it with server.disableTls=true so Envoy forwards plaintext to the gateway pod, and use OIDC for client identity (the gateway never sees the client cert). |
156+
| grpcRoute.gateway.listener.tls.certificateRefs | list | `[]` | certificateRefs for the HTTPS listener. Required when protocol is HTTPS. Each entry needs a `name` pointing at a kubernetes.io/tls Secret in the Gateway's namespace. May reference a cert-manager-issued Secret or the existing openshell-server-tls Secret (its SANs must include the external hostname). |
156157
| grpcRoute.gateway.name | string | `""` | Name of the Gateway resource. Defaults to the chart fullname. |
157158
| grpcRoute.gateway.namespace | string | `""` | Namespace of the Gateway referenced by the GRPCRoute parentRef. Defaults to the release namespace. |
158159
| grpcRoute.hostnames | list | `[]` | Hostnames the GRPCRoute matches on. Leave empty to match all hosts. |
Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
# Gateway API overlay with TLS termination at the Envoy Gateway listener.
5+
#
6+
# Exercises the HTTPS listener branch of templates/gateway.yaml for
7+
# `helm template`/lint coverage. Envoy Gateway terminates TLS using the
8+
# referenced kubernetes.io/tls Secret and forwards plaintext to the gateway pod,
9+
# so the gateway runs with TLS disabled and uses OIDC for client identity.
10+
#
11+
# The certificate Secret (openshell-ingress-tls) and the OIDC issuer below are
12+
# placeholders for render coverage; a real deployment must provide a valid TLS
13+
# Secret in the release namespace and a reachable OIDC issuer.
14+
15+
grpcRoute:
16+
enabled: true
17+
gateway:
18+
create: true
19+
className: "eg"
20+
listener:
21+
port: 443
22+
protocol: HTTPS
23+
tls:
24+
certificateRefs:
25+
- name: openshell-ingress-tls
26+
hostnames: []
27+
28+
server:
29+
# Envoy terminates TLS at the edge; the gateway listens plaintext behind it.
30+
disableTls: true
31+
oidc:
32+
issuer: "https://keycloak.example.com/realms/openshell"
33+
audience: "openshell-cli"

‎deploy/helm/openshell/templates/gateway.yaml‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,9 +12,21 @@ metadata:
1212
spec:
1313
gatewayClassName: {{ .Values.grpcRoute.gateway.className }}
1414
listeners:
15-
- name: http
15+
- name: {{ ternary "https" "http" (eq .Values.grpcRoute.gateway.listener.protocol "HTTPS") }}
1616
port: {{ .Values.grpcRoute.gateway.listener.port }}
1717
protocol: {{ .Values.grpcRoute.gateway.listener.protocol }}
18+
{{- if eq .Values.grpcRoute.gateway.listener.protocol "HTTPS" }}
19+
{{- if not .Values.grpcRoute.gateway.listener.tls.certificateRefs }}
20+
{{- fail "grpcRoute.gateway.listener.tls.certificateRefs is required when grpcRoute.gateway.listener.protocol is HTTPS" }}
21+
{{- end }}
22+
{{- if not .Values.server.disableTls }}
23+
{{- fail "grpcRoute.gateway.listener.protocol=HTTPS terminates TLS at Envoy Gateway, which forwards plaintext gRPC to the gateway pod; set server.disableTls=true so the pod listens plaintext (this chart does not render a BackendTLSPolicy for re-encryption to a TLS backend)" }}
24+
{{- end }}
25+
tls:
26+
mode: Terminate
27+
certificateRefs:
28+
{{- toYaml .Values.grpcRoute.gateway.listener.tls.certificateRefs | nindent 10 }}
29+
{{- end }}
1830
allowedRoutes:
1931
namespaces:
2032
from: {{ .Values.grpcRoute.gateway.listener.allowedRoutes }}

‎deploy/helm/openshell/values.yaml‎

Lines changed: 14 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -372,9 +372,21 @@ grpcRoute:
372372
namespace: ""
373373
# Listener settings (only used when gateway.create is true).
374374
listener:
375-
# -- Listener port for the generated Gateway resource.
375+
# -- Listener port for the generated Gateway resource. Use 443 with protocol HTTPS.
376376
port: 80
377-
# -- Listener protocol for the generated Gateway resource.
377+
# -- Listener protocol for the generated Gateway resource: HTTP or HTTPS.
378+
# HTTPS terminates TLS at the Envoy Gateway listener; pair it with
379+
# server.disableTls=true so Envoy forwards plaintext to the gateway pod,
380+
# and use OIDC for client identity (the gateway never sees the client cert).
378381
protocol: HTTP
379382
# -- "Same" restricts attached routes to the release namespace; "All" allows any namespace.
380383
allowedRoutes: Same
384+
# TLS settings for the listener. Used only when protocol is HTTPS
385+
# (mode is always Terminate).
386+
tls:
387+
# -- certificateRefs for the HTTPS listener. Required when protocol is
388+
# HTTPS. Each entry needs a `name` pointing at a kubernetes.io/tls Secret
389+
# in the Gateway's namespace. May reference a cert-manager-issued Secret
390+
# or the existing openshell-server-tls Secret (its SANs must include the
391+
# external hostname).
392+
certificateRefs: []

‎docs/kubernetes/ingress.mdx‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -77,6 +77,64 @@ openshell gateway add http://<external-ip> --name production
7777
openshell status
7878
```
7979

80+
This setup is plaintext end-to-end and is intended for development. For external access, terminate TLS at the gateway as shown below.
81+
82+
## HTTPS (TLS termination)
83+
84+
Envoy Gateway can terminate TLS at the listener and forward plaintext to the OpenShell gateway pod:
85+
86+
```text
87+
client → HTTPS → Envoy Gateway (terminate TLS) → plaintext → openshell gateway pod
88+
```
89+
90+
Envoy Gateway only terminates TLS here — it does not perform OIDC. Do not enable an Envoy Gateway OIDC `SecurityPolicy` in front of the gateway: that flow relies on browser redirects and cookies and cannot work with the OpenShell CLI or headless agents. Instead, the OpenShell gateway validates an OIDC bearer token that the client sends in the gRPC `authorization` metadata, which Envoy forwards untouched.
91+
92+
Because Envoy terminates TLS, the OpenShell gateway never sees a client certificate, so client mTLS cannot provide identity on this path. Use OIDC bearer tokens for client identity instead.
93+
94+
For headless agents and CI, the CLI obtains the token via the OAuth2 client-credentials grant (no browser): set `OPENSHELL_OIDC_CLIENT_SECRET` to the OAuth client secret before running `openshell gateway add`. The client id comes from `--oidc-client-id` (default `openshell-cli`); pass it explicitly if your IdP client uses a different id. Interactive human users get the browser-based Authorization Code + PKCE flow by default.
95+
96+
### Provide a TLS certificate
97+
98+
Create a `kubernetes.io/tls` Secret in the `openshell` namespace with the certificate for your external hostname:
99+
100+
```shell
101+
kubectl -n openshell create secret tls openshell-ingress-tls \
102+
--cert=tls.crt --key=tls.key
103+
```
104+
105+
The Secret may also be issued by cert-manager, or you can reference the chart's existing `openshell-server-tls` Secret if its SANs include the external hostname.
106+
107+
### Install with HTTPS termination
108+
109+
Enable an HTTPS listener, point it at the Secret, disable gateway-pod TLS so Envoy forwards plaintext, and configure an OIDC issuer for client identity:
110+
111+
```shell
112+
helm upgrade --install openshell \
113+
oci://ghcr.io/nvidia/openshell/helm-chart \
114+
--version <version> \
115+
--namespace openshell \
116+
--set grpcRoute.enabled=true \
117+
--set grpcRoute.gateway.create=true \
118+
--set grpcRoute.gateway.className=eg \
119+
--set grpcRoute.gateway.listener.protocol=HTTPS \
120+
--set grpcRoute.gateway.listener.port=443 \
121+
--set 'grpcRoute.gateway.listener.tls.certificateRefs[0].name=openshell-ingress-tls' \
122+
--set server.disableTls=true \
123+
--set server.oidc.issuer=https://<issuer> \
124+
--set 'grpcRoute.hostnames[0]=<external-hostname>'
125+
```
126+
127+
Keep the certificate Secret in the release namespace. Referencing a Secret in another namespace requires a `ReferenceGrant`.
128+
129+
### Register over HTTPS
130+
131+
```shell
132+
openshell gateway add https://<external-hostname> --name production --oidc-issuer https://<issuer>
133+
openshell status
134+
```
135+
136+
See [Authentication](/kubernetes/setup) for OIDC issuer, audience, and roles configuration.
137+
80138
## SSH Relay
81139

82140
Sandbox SSH uses the gateway endpoint registered with the CLI. No separate Helm SSH host or port values are required.

0 commit comments

Comments
 (0)