Skip to content

Commit 7d843b3

Browse files
committed
feat(helm)!: confine gateway workspace permissions with an admission policy
Install a ValidatingAdmissionPolicy, on by default in managed and operator workspace modes, that matches only the gateway ServiceAccount. It admits Secret, Pod, Sandbox, Service, ServiceAccount, and NetworkPolicy writes only in namespaces labeled as owned by this gateway and namespaces matching the operator selector. Secret writes are also admitted in the credential namespace when the Kubernetes Secrets credential driver is enabled. Namespace writes are admitted only for namespaces owned by this gateway, judged by their existing labels. - Skip rendering the policy when rbac.create or rbac.clusterScoped.create is false; a cluster-admin applies it with the other cluster-scoped objects. - Require Kubernetes 1.30. - Fail rendering with operatorNamespaceFile or set-based operator selectors, which the policy cannot evaluate. admissionPolicy.enabled set to false opts out. - Add e2e checks in managed and operator modes that send server-side dry-run requests as the gateway ServiceAccount outside its namespaces, including Pod creation in the sandbox and credential namespace, and assert the policy rejects them. - Document the policy and opt-out, raise the minimum Kubernetes version in the docs and support matrix, and add policy denials to the cluster debugging skill. Signed-off-by: Kris Hicks <[email protected]>
1 parent 71440b2 commit 7d843b3

13 files changed

Lines changed: 498 additions & 17 deletions

File tree

‎crates/openshell-driver-kubernetes/README.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,16 @@ workspace namespace modes via `workspace_mode`:
2727
gateway state; it never deletes or otherwise accesses the operator-managed
2828
Kubernetes namespace.
2929

30+
In managed and operator mode, a chart-installed `ValidatingAdmissionPolicy`
31+
matching only the gateway ServiceAccount limits the ClusterRole's workspace
32+
grants: Secret, Pod, Sandbox, Service, ServiceAccount, and NetworkPolicy writes
33+
are admitted only in namespaces labeled as owned by this gateway and namespaces
34+
matching the operator label selector, with Secret writes also admitted in the
35+
credential namespace, and Namespace writes only for namespaces this gateway
36+
owns. The
37+
gateway cannot patch namespaces, so it cannot add ownership labels to an
38+
existing namespace. The policy is on by default and can be disabled.
39+
3040
When the gateway configures `[openshell.gateway.otlp]`, Kubernetes
3141
compute-driver spans export to the same OTLP/gRPC collector with the service
3242
name `openshell-driver-kubernetes`. The driver preserves the gateway trace

‎deploy/helm/openshell/README.md‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,14 +20,17 @@ multiple pre-provisioned workspace namespaces.
2020
## Cluster-scoped vs namespaced objects
2121

2222
Most objects in this chart are namespaced and land in the release namespace.
23-
Only two are cluster-scoped:
23+
These are cluster-scoped:
2424

2525
| Object | Default name |
2626
| --- | --- |
2727
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
2828
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |
29+
| `ValidatingAdmissionPolicy` | `<fullname>-workspace-scope-<release namespace>` |
30+
| `ValidatingAdmissionPolicyBinding` | `<fullname>-workspace-scope-<release namespace>` |
2931

30-
By default the release creates both, so an install by a cluster-admin is
32+
The admission policy and its binding render only in `managed` and `operator`
33+
workspace modes. By default the release creates all of them, so an install by a cluster-admin is
3134
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
3235
split the install in two.
3336

@@ -44,6 +47,10 @@ helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <ver
4447
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
4548
```
4649

50+
In `managed` and `operator` workspace modes, also add
51+
`--show-only templates/workspace-admission-policy.yaml`, and pass
52+
`--set admissionPolicy.enabled=false` to the namespace-admin install below.
53+
4754
A namespace-admin then installs and upgrades the release with cluster-scoped
4855
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
4956
or the equivalent `--set`:
@@ -73,7 +80,7 @@ currently held`.
7380
| --- | --- | --- |
7481
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
7582
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
76-
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |
83+
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false`, `admissionPolicy.enabled=false` |
7784

7885
`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
7986
is needed there.
@@ -116,7 +123,18 @@ kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
116123
helm.sh/resource-policy=keep --overwrite
117124
```
118125

119-
The objects then survive the upgrade that sets the flag, and the cluster-admin
126+
In `managed` and `operator` workspace modes, the namespace-admin release also
127+
sets `admissionPolicy.enabled=false`, which deletes the admission policy and its
128+
binding while the gateway keeps its cluster-wide permissions. Annotate them too:
129+
130+
```shell
131+
kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-<namespace>" \
132+
helm.sh/resource-policy=keep --overwrite
133+
kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-<namespace>" \
134+
helm.sh/resource-policy=keep --overwrite
135+
```
136+
137+
The objects then survive the upgrade that sets the flags, and the cluster-admin
120138
owns them from that point on. Fresh installs need no such step.
121139

122140
`rbac.clusterScoped.create` is independent of
@@ -309,6 +327,7 @@ discovery endpoint or its TLS CA.
309327

310328
| Key | Type | Default | Description |
311329
|-----|------|---------|-------------|
330+
| admissionPolicy.enabled | bool | `true` | Install a ValidatingAdmissionPolicy that rejects gateway requests to create, update, or delete Secrets, Pods, Sandboxes, Services, ServiceAccounts, NetworkPolicies, and Namespaces outside gateway-owned managed namespaces and operator-selected namespaces. Secret writes are also admitted in the credential namespace. |
312331
| affinity | object | `{}` | Affinity rules for the gateway pod. |
313332
| agentSandbox.preflight.enabled | bool | `true` | Check the live cluster for a supported Agent Sandbox API before rendering gateway resources. Disable only for offline rendering and linting. |
314333
| certManager.caSecretName | string | `"openshell-ca-tls"` | Secret created for the intermediate CA (Certificate with isCA: true). |

‎deploy/helm/openshell/README.md.gotmpl‎

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -20,14 +20,17 @@ multiple pre-provisioned workspace namespaces.
2020
## Cluster-scoped vs namespaced objects
2121

2222
Most objects in this chart are namespaced and land in the release namespace.
23-
Only two are cluster-scoped:
23+
These are cluster-scoped:
2424

2525
| Object | Default name |
2626
| --- | --- |
2727
| `ClusterRole` | `<fullname>-node-reader-<release namespace>` |
2828
| `ClusterRoleBinding` | `<fullname>-node-reader-<release namespace>` |
29+
| `ValidatingAdmissionPolicy` | `<fullname>-workspace-scope-<release namespace>` |
30+
| `ValidatingAdmissionPolicyBinding` | `<fullname>-workspace-scope-<release namespace>` |
2931

30-
By default the release creates both, so an install by a cluster-admin is
32+
The admission policy and its binding render only in `managed` and `operator`
33+
workspace modes. By default the release creates all of them, so an install by a cluster-admin is
3134
unchanged. On clusters where cluster-scoped RBAC is owned by a different team,
3235
split the install in two.
3336

@@ -44,6 +47,10 @@ helm template openshell oci://ghcr.io/nvidia/openshell/helm-chart --version <ver
4447
--show-only templates/clusterrolebinding.yaml | kubectl apply -f -
4548
```
4649

50+
In `managed` and `operator` workspace modes, also add
51+
`--show-only templates/workspace-admission-policy.yaml`, and pass
52+
`--set admissionPolicy.enabled=false` to the namespace-admin install below.
53+
4754
A namespace-admin then installs and upgrades the release with cluster-scoped
4855
objects omitted, using [`ci/values-namespace-admin.yaml`](ci/values-namespace-admin.yaml)
4956
or the equivalent `--set`:
@@ -73,7 +80,7 @@ currently held`.
7380
| --- | --- | --- |
7481
| `shared` | built-in `admin` only | `rbac.create=false`, cluster-admin pre-creates all gateway RBAC |
7582
| `shared` | `admin` plus the sandbox permissions in the namespace | `rbac.clusterScoped.create=false` |
76-
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false` |
83+
| `managed`, `operator` | built-in `admin` only | `rbac.clusterScoped.create=false`, `admissionPolicy.enabled=false` |
7784

7885
`managed` and `operator` render no namespaced sandbox `Role`, so no extra grant
7986
is needed there.
@@ -116,7 +123,18 @@ kubectl annotate clusterrolebinding "openshell-node-reader-<namespace>" \
116123
helm.sh/resource-policy=keep --overwrite
117124
```
118125

119-
The objects then survive the upgrade that sets the flag, and the cluster-admin
126+
In `managed` and `operator` workspace modes, the namespace-admin release also
127+
sets `admissionPolicy.enabled=false`, which deletes the admission policy and its
128+
binding while the gateway keeps its cluster-wide permissions. Annotate them too:
129+
130+
```shell
131+
kubectl annotate validatingadmissionpolicy "openshell-workspace-scope-<namespace>" \
132+
helm.sh/resource-policy=keep --overwrite
133+
kubectl annotate validatingadmissionpolicybinding "openshell-workspace-scope-<namespace>" \
134+
helm.sh/resource-policy=keep --overwrite
135+
```
136+
137+
The objects then survive the upgrade that sets the flags, and the cluster-admin
120138
owns them from that point on. Fresh installs need no such step.
121139

122140
`rbac.clusterScoped.create` is independent of

‎deploy/helm/openshell/ci/values-namespace-admin.yaml‎

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,8 +3,9 @@
33

44
# Namespace-admin overlay — renders only namespaced objects.
55
#
6-
# Use this when a cluster-admin applies the gateway ClusterRole and
7-
# ClusterRoleBinding once, out of band, and the OpenShell release is installed
6+
# Use this when a cluster-admin applies the gateway ClusterRole,
7+
# ClusterRoleBinding, and admission policy once, out of band, and the OpenShell
8+
# release is installed
89
# and upgraded by an installer that holds no cluster-scoped permissions.
910
#
1011
# Generate the cluster-scoped objects for the cluster-admin step from the same
@@ -15,11 +16,18 @@
1516
# --set rbac.clusterScoped.create=true \
1617
# --set agentSandbox.preflight.enabled=false \
1718
# --show-only templates/clusterrole.yaml \
18-
# --show-only templates/clusterrolebinding.yaml | kubectl apply -f -
19+
# --show-only templates/clusterrolebinding.yaml \
20+
# --show-only templates/workspace-admission-policy.yaml | kubectl apply -f -
21+
#
22+
# Omit the admission policy template in shared workspace mode, which renders
23+
# no policy.
1924
#
2025
# The gateway ServiceAccount name and release namespace are unchanged, so the
2126
# pre-created ClusterRoleBinding still matches this release.
2227

2328
rbac:
2429
clusterScoped:
2530
create: false
31+
32+
admissionPolicy:
33+
enabled: false

‎deploy/helm/openshell/templates/_helpers.tpl‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -261,6 +261,17 @@ shared workspace mode.
261261
{{- uniq $names | toJson -}}
262262
{{- end }}
263263

264+
{{/*
265+
Whether to render the gateway workspace admission policy. Shared mode grants
266+
the gateway nothing cluster-wide that the policy covers.
267+
*/}}
268+
{{- define "openshell.admissionPolicyEnabled" -}}
269+
{{- $workspaceMode := .Values.server.drivers.kubernetes.workspaceMode | default "shared" -}}
270+
{{- if and .Values.admissionPolicy.enabled (ne $workspaceMode "shared") -}}
271+
true
272+
{{- end -}}
273+
{{- end }}
274+
264275
{{/*
265276
Namespace where Kubernetes Secret-backed provider credentials live.
266277
*/}}
Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
{{- if include "openshell.admissionPolicyEnabled" . }}
5+
{{- $kubernetes := .Values.server.drivers.kubernetes }}
6+
{{- $workspaceMode := $kubernetes.workspaceMode }}
7+
{{- $gatewayId := .Values.server.sandboxJwt.gatewayId | default (include "openshell.fullname" .) }}
8+
{{- $credentialSecret := "false" }}
9+
{{- if .Values.server.credentialDrivers.kubernetesSecrets.enabled }}
10+
{{- $credentialSecret = printf "request.resource.group == '' && request.resource.resource == 'secrets' && variables.targetNamespace.metadata.name == '%s'" (include "openshell.credentialKubernetesSecretsNamespace" .) }}
11+
{{- end }}
12+
{{- $managed := "false" }}
13+
{{- if eq $workspaceMode "managed" }}
14+
{{- if not (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $gatewayId) }}
15+
{{- fail "admissionPolicy requires the gateway ID (server.sandboxJwt.gatewayId or the release fullname) to be a DNS-1123 label" }}
16+
{{- end }}
17+
{{- $managed = printf "('openshell.ai/managed-by' in variables.labels && variables.labels['openshell.ai/managed-by'] == 'openshell' && 'openshell.ai/gateway-id' in variables.labels && variables.labels['openshell.ai/gateway-id'] == '%s')" $gatewayId }}
18+
{{- end }}
19+
{{- $operator := "false" }}
20+
{{- if eq $workspaceMode "operator" }}
21+
{{- if $kubernetes.operatorNamespaceFile }}
22+
{{- fail "admissionPolicy cannot enforce server.drivers.kubernetes.operatorNamespaceFile; use operatorNamespaceLabel or set admissionPolicy.enabled=false" }}
23+
{{- end }}
24+
{{- $terms := list }}
25+
{{- range splitList "," $kubernetes.operatorNamespaceLabel }}
26+
{{- $term := trim . }}
27+
{{- if not (regexMatch "^([a-z0-9]([-a-z0-9.]*[a-z0-9])?/)?[A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?=([A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?)?$" $term) }}
28+
{{- fail (printf "admissionPolicy supports only key=value terms in server.drivers.kubernetes.operatorNamespaceLabel, got %q; set admissionPolicy.enabled=false to use other selectors" $term) }}
29+
{{- end }}
30+
{{- $pair := splitn "=" 2 $term }}
31+
{{- $terms = append $terms (printf "('%s' in variables.labels && variables.labels['%s'] == '%s')" $pair._0 $pair._0 $pair._1) }}
32+
{{- end }}
33+
{{- $operator = printf "(%s)" (join " && " $terms) }}
34+
{{- end }}
35+
apiVersion: admissionregistration.k8s.io/v1
36+
kind: ValidatingAdmissionPolicy
37+
metadata:
38+
name: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }}
39+
labels:
40+
{{- include "openshell.labels" . | nindent 4 }}
41+
spec:
42+
failurePolicy: Fail
43+
matchConstraints:
44+
resourceRules:
45+
- apiGroups: [""]
46+
apiVersions: ["v1"]
47+
operations: ["CREATE", "UPDATE", "DELETE"]
48+
resources: ["secrets", "pods", "services", "serviceaccounts", "namespaces"]
49+
- apiGroups: ["networking.k8s.io"]
50+
apiVersions: ["v1"]
51+
operations: ["CREATE", "UPDATE", "DELETE"]
52+
resources: ["networkpolicies"]
53+
- apiGroups: ["agents.x-k8s.io"]
54+
apiVersions: ["*"]
55+
operations: ["CREATE", "UPDATE", "DELETE"]
56+
resources: ["sandboxes", "sandboxes/status"]
57+
matchConditions:
58+
- name: gateway-service-account
59+
expression: >-
60+
request.userInfo.username ==
61+
'system:serviceaccount:{{ .Release.Namespace }}:{{ include "openshell.serviceAccountName" . }}'
62+
variables:
63+
- name: isNamespace
64+
expression: "request.resource.group == '' && request.resource.resource == 'namespaces'"
65+
- name: targetNamespace
66+
expression: >-
67+
variables.isNamespace
68+
? (request.operation == 'CREATE' ? object : oldObject)
69+
: namespaceObject
70+
- name: labels
71+
expression: >-
72+
has(variables.targetNamespace.metadata.labels)
73+
? variables.targetNamespace.metadata.labels
74+
: {}
75+
- name: managed
76+
expression: {{ $managed | quote }}
77+
- name: operator
78+
expression: {{ $operator | quote }}
79+
- name: credentialSecret
80+
expression: {{ $credentialSecret | quote }}
81+
validations:
82+
- expression: >-
83+
variables.isNamespace
84+
? variables.managed
85+
: (variables.credentialSecret || variables.managed || variables.operator)
86+
messageExpression: >-
87+
'the OpenShell gateway may not ' + request.operation + ' ' +
88+
request.resource.resource + ' in namespace ' +
89+
variables.targetNamespace.metadata.name
90+
reason: Forbidden
91+
---
92+
apiVersion: admissionregistration.k8s.io/v1
93+
kind: ValidatingAdmissionPolicyBinding
94+
metadata:
95+
name: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }}
96+
labels:
97+
{{- include "openshell.labels" . | nindent 4 }}
98+
spec:
99+
policyName: {{ include "openshell.fullname" . }}-workspace-scope-{{ .Release.Namespace }}
100+
validationActions: ["Deny"]
101+
{{- end }}

0 commit comments

Comments
 (0)