Skip to content

Commit 71c7bf0

Browse files
committed
feat(saw): sandbox web UIs on their own route, signed in with Keycloak
A sandbox with `ui: {route: true}` in its SAW-BOM profile (data-science: the default workspace's notebook) gets a route <vm>-<workspace>-<sandbox>-ui, a VM and Service port, and its callback registered on the Keycloak client openshell-dashboard. In the VM, per UI: - oauth2-proxy on 0.0.0.0:<proxyPort> (Keycloak, PKCE) admitting only the workspace owner and sandboxUiProxy.allowedUsers; - a TCP relay that lets at most 16 connections through to `openshell forward service` (OpenShell allows 20 per sandbox, NVIDIA/OpenShell#3494), queues the rest, retries refused connections, lets idle keep-alive connections give way, and re-sends a GET that gets no answer; - the forward to the sandbox's port 18789. OpenClaw runs in trusted-proxy mode for such a sandbox: it takes the user from X-Forwarded-Email (oauth2-proxy's preferred_username) over loopback only, for the same users, so the Control UI needs no gateway token. The CLI and TUI use the gateway secret as gateway.auth.password, written back after `config set` removes it (openclaw/openclaw#162216). The secret is kept across runs and the gateway restarted so config changes apply. sandboxUiProxy.trustedProxy.enabled turns it off. Installer fixes found on a live cluster: - a sandbox reporting an error after a VM restart gets 90 s to recover before it is recreated, and a recreate waits for the deletion; - onboarding waits until the sandbox accepts exec (the readiness check matched "Ready" while it was still Provisioning, so OpenClaw was never started after a reboot). Co-Authored-By: Claude
1 parent d5c44af commit 71c7bf0

16 files changed

Lines changed: 1537 additions & 79 deletions

File tree

‎charts/openshell-saw/files/installer/apply_bom.py‎

Lines changed: 673 additions & 24 deletions
Large diffs are not rendered by default.

‎charts/openshell-saw/files/prepare.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ OIDC_ISSUER_URL="{{ include "openshell-sandbox.oidcIssuerUrl" . }}"
1717
OIDC_KEYCLOAK_NAME="{{ .Values.oidc.keycloakName }}"
1818
OIDC_REALM="{{ .Values.oidc.realm }}"
1919
KEYCLOAK_NS="{{ include "openshell-sandbox.keycloakNamespace" . }}"
20+
UI_ROUTE_HOSTS="{{ range $e := include "openshell-sandbox.sandboxUi" . | fromJsonArray }}{{ $e.host }} {{ end }}"
2021
SCRIPTS_DIR="/scripts"
2122

2223
# --- Phase 1: tools ---
@@ -30,7 +31,7 @@ echo "Disk source is a registry/HTTP import; no golden image bootstrap needed."
3031
{{- end }}
3132

3233
# --- Phase 3: dashboard redirect URI in Keycloak ---
33-
if [[ "${DASHBOARD_ENABLED}" == "true" && -n "${OIDC_ISSUER_URL}" ]]; then
34+
if [[ ( "${DASHBOARD_ENABLED}" == "true" || -n "${UI_ROUTE_HOSTS// /}" ) && -n "${OIDC_ISSUER_URL}" ]]; then
3435
source "${SCRIPTS_DIR}/register-keycloak-redirect.sh"
3536
else
3637
echo "Dashboard or OIDC issuer not configured; skipping Keycloak redirect registration."
Lines changed: 57 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,58 +1,68 @@
11
#!/usr/bin/env bash
2-
# Phase: register this VM's dashboard redirect URI on the shared Keycloak client.
3-
# Cluster-side only; the dashboard itself is started by the in-VM installer.
2+
# Phase: register this VM's redirect URIs on the shared Keycloak client: the
3+
# dashboard's (the *-webui route) and one per sandbox web UI route
4+
# (UI_ROUTE_HOSTS). Cluster-side only; the proxies run in the VM.
45
# Expects: VM_NAME, NS, OIDC_KEYCLOAK_NAME, OIDC_REALM, KEYCLOAK_NS,
5-
# DASHBOARD_CLIENT_ID, OIDC_ISSUER_URL
6+
# DASHBOARD_CLIENT_ID, OIDC_ISSUER_URL, DASHBOARD_ENABLED, UI_ROUTE_HOSTS
67

7-
WEBUI_ROUTE_HOST="$(kubectl get route "${VM_NAME}-webui" -n "${NS}" -o jsonpath='{.spec.host}' 2>/dev/null || true)"
8-
if [[ -z "${WEBUI_ROUTE_HOST}" ]]; then
9-
echo "WARNING: ${VM_NAME}-webui route not found — skipping dashboard setup"
10-
return 0 2>/dev/null || true
8+
HOSTS=()
9+
if [[ "${DASHBOARD_ENABLED:-false}" == "true" ]]; then
10+
WEBUI_ROUTE_HOST="$(kubectl get route "${VM_NAME}-webui" -n "${NS}" -o jsonpath='{.spec.host}' 2>/dev/null || true)"
11+
if [[ -n "${WEBUI_ROUTE_HOST}" ]]; then
12+
HOSTS+=("${WEBUI_ROUTE_HOST}")
13+
else
14+
echo "WARNING: ${VM_NAME}-webui route not found — skipping the dashboard redirect URI"
15+
fi
16+
fi
17+
for host in ${UI_ROUTE_HOSTS:-}; do
18+
HOSTS+=("${host}")
19+
done
20+
if [[ ${#HOSTS[@]} -eq 0 ]]; then
21+
echo "No redirect URIs to register."
22+
return 0 2>/dev/null || exit 0
1123
fi
1224

13-
DASHBOARD_REDIRECT_URL="https://${WEBUI_ROUTE_HOST}/oauth2/callback"
14-
15-
# Register redirect URI on Keycloak client
1625
KC_ADMIN_SECRET="${OIDC_KEYCLOAK_NAME}-initial-admin"
1726
KC_ADMIN_USER="$(kubectl get secret "${KC_ADMIN_SECRET}" -n "${KEYCLOAK_NS}" -o jsonpath='{.data.username}' 2>/dev/null | base64 -d || true)"
1827
KC_ADMIN_PASS="$(kubectl get secret "${KC_ADMIN_SECRET}" -n "${KEYCLOAK_NS}" -o jsonpath='{.data.password}' 2>/dev/null | base64 -d || true)"
1928
KC_BASE="$(echo "${OIDC_ISSUER_URL}" | sed 's#/realms/.*##')"
20-
if [[ -n "${KC_ADMIN_USER}" && -n "${KC_BASE}" ]]; then
21-
echo "Registering redirect URI on Keycloak client '${DASHBOARD_CLIENT_ID}'..."
22-
KC_TOKEN_RESPONSE="$(curl -sk -X POST "${KC_BASE}/realms/master/protocol/openid-connect/token" \
23-
-d "grant_type=password" -d "client_id=admin-cli" \
24-
-d "username=${KC_ADMIN_USER}" -d "password=${KC_ADMIN_PASS}")"
25-
KC_ADMIN_TOKEN="$(echo "${KC_TOKEN_RESPONSE}" | jq -r '.access_token // empty')"
26-
if [[ -z "${KC_ADMIN_TOKEN}" ]]; then
27-
echo "WARNING: Keycloak admin token fetch failed: $(echo "${KC_TOKEN_RESPONSE}" | jq -r '.error_description // .error // "no response / unparseable response"')"
28-
fi
29-
if [[ -n "${KC_ADMIN_TOKEN}" ]]; then
30-
CLIENT_UUID="$(curl -sk -H "Authorization: Bearer ${KC_ADMIN_TOKEN}" \
31-
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients?clientId=${DASHBOARD_CLIENT_ID}" \
32-
| jq -r '.[0].id // empty')"
33-
if [[ -n "${CLIENT_UUID}" ]]; then
34-
CLIENT_JSON="$(curl -sk -H "Authorization: Bearer ${KC_ADMIN_TOKEN}" \
35-
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients/${CLIENT_UUID}")"
36-
WEBUI_ORIGIN="https://${WEBUI_ROUTE_HOST}"
37-
UPDATED_JSON="$(echo "${CLIENT_JSON}" | jq \
38-
--arg redirect "${DASHBOARD_REDIRECT_URL}" --arg origin "${WEBUI_ORIGIN}" \
39-
'.redirectUris = ((.redirectUris // []) + [$redirect] | unique) |
40-
.webOrigins = ((.webOrigins // []) + [$origin] | unique)')"
41-
HTTP_CODE="$(curl -sk -o /dev/null -w '%{http_code}' -X PUT \
42-
-H "Authorization: Bearer ${KC_ADMIN_TOKEN}" -H "Content-Type: application/json" \
43-
-d "${UPDATED_JSON}" \
44-
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients/${CLIENT_UUID}")"
45-
if [[ "${HTTP_CODE}" == "204" ]]; then
46-
echo "Redirect URI registered: ${DASHBOARD_REDIRECT_URL}"
47-
else
48-
echo "WARNING: failed to register redirect URI (HTTP ${HTTP_CODE}) — dashboard OIDC login will fail"
49-
fi
50-
else
51-
echo "WARNING: Keycloak client '${DASHBOARD_CLIENT_ID}' not found — dashboard OIDC login will fail"
52-
fi
53-
else
54-
echo "WARNING: could not obtain Keycloak admin token — dashboard OIDC login will fail"
55-
fi
29+
if [[ -z "${KC_ADMIN_USER}" || -z "${KC_BASE}" ]]; then
30+
echo "WARNING: Keycloak admin credentials not found — OIDC login through ${HOSTS[*]} will fail"
31+
return 0 2>/dev/null || exit 0
32+
fi
33+
34+
echo "Registering redirect URIs on Keycloak client '${DASHBOARD_CLIENT_ID}': ${HOSTS[*]}"
35+
KC_TOKEN_RESPONSE="$(curl -sk -X POST "${KC_BASE}/realms/master/protocol/openid-connect/token" \
36+
-d "grant_type=password" -d "client_id=admin-cli" \
37+
-d "username=${KC_ADMIN_USER}" -d "password=${KC_ADMIN_PASS}")"
38+
KC_ADMIN_TOKEN="$(echo "${KC_TOKEN_RESPONSE}" | jq -r '.access_token // empty')"
39+
if [[ -z "${KC_ADMIN_TOKEN}" ]]; then
40+
echo "WARNING: Keycloak admin token fetch failed: $(echo "${KC_TOKEN_RESPONSE}" | jq -r '.error_description // .error // "no response / unparseable response"')"
41+
return 0 2>/dev/null || exit 0
42+
fi
43+
CLIENT_UUID="$(curl -sk -H "Authorization: Bearer ${KC_ADMIN_TOKEN}" \
44+
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients?clientId=${DASHBOARD_CLIENT_ID}" \
45+
| jq -r '.[0].id // empty')"
46+
if [[ -z "${CLIENT_UUID}" ]]; then
47+
echo "WARNING: Keycloak client '${DASHBOARD_CLIENT_ID}' not found — OIDC login will fail"
48+
return 0 2>/dev/null || exit 0
49+
fi
50+
51+
REDIRECTS="$(printf 'https://%s/oauth2/callback\n' "${HOSTS[@]}" | jq -R . | jq -s .)"
52+
ORIGINS="$(printf 'https://%s\n' "${HOSTS[@]}" | jq -R . | jq -s .)"
53+
CLIENT_JSON="$(curl -sk -H "Authorization: Bearer ${KC_ADMIN_TOKEN}" \
54+
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients/${CLIENT_UUID}")"
55+
# Added, never removed: other VMs register on the same client.
56+
UPDATED_JSON="$(echo "${CLIENT_JSON}" | jq \
57+
--argjson redirects "${REDIRECTS}" --argjson origins "${ORIGINS}" \
58+
'.redirectUris = ((.redirectUris // []) + $redirects | unique) |
59+
.webOrigins = ((.webOrigins // []) + $origins | unique)')"
60+
HTTP_CODE="$(curl -sk -o /dev/null -w '%{http_code}' -X PUT \
61+
-H "Authorization: Bearer ${KC_ADMIN_TOKEN}" -H "Content-Type: application/json" \
62+
-d "${UPDATED_JSON}" \
63+
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients/${CLIENT_UUID}")"
64+
if [[ "${HTTP_CODE}" == "204" ]]; then
65+
echo "Redirect URIs registered: $(echo "${REDIRECTS}" | jq -r 'join(" ")')"
5666
else
57-
echo "WARNING: Keycloak admin credentials not found — dashboard OIDC login will fail"
67+
echo "WARNING: failed to register redirect URIs (HTTP ${HTTP_CODE}) — OIDC login will fail"
5868
fi

‎charts/openshell-saw/templates/_helpers.tpl‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -266,3 +266,44 @@ rpc = "openshell.v1.OpenShell/SubmitPolicyAnalysis"
266266
phases = ["validate"]
267267
{{- end }}
268268
{{- end }}
269+
270+
271+
{{/*
272+
Sandbox UI routes: sandboxUi entries with the route host filled in,
273+
<vm>-<workspace>-<sandbox>-ui.apps.<clusterDomain> (at most 62 characters
274+
in its first label with 19-character names), or entry.host when set.
275+
*/}}
276+
{{- define "openshell-sandbox.sandboxUi" -}}
277+
{{- $root := . -}}
278+
{{- $out := list -}}
279+
{{- $seen := dict -}}
280+
{{- range $e := .Values.sandboxUi | default list -}}
281+
{{- $ws := $e.workspace | default "" | toString -}}
282+
{{- $sb := $e.sandbox | default "" | toString -}}
283+
{{- if not (and (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $ws) (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $sb)) -}}
284+
{{- fail (printf "sandboxUi: workspace %q and sandbox %q must be lowercase DNS labels" $ws $sb) -}}
285+
{{- end -}}
286+
{{- if not (and $e.proxyPort $e.forwardPort) -}}
287+
{{- fail (printf "sandboxUi %s/%s needs proxyPort and forwardPort" $ws $sb) -}}
288+
{{- end -}}
289+
{{- $label := printf "%s-%s-%s-ui" (include "openshell-sandbox.fullname" $root) $ws $sb -}}
290+
{{- if gt (len $label) 63 -}}
291+
{{- fail (printf "sandboxUi route label %q is %d characters; DNS labels allow 63" $label (len $label)) -}}
292+
{{- end -}}
293+
{{- $host := $e.host | default "" -}}
294+
{{- if and (not $host) $root.Values.global -}}
295+
{{- if $root.Values.global.clusterDomain -}}
296+
{{- $host = printf "%s.apps.%s" $label $root.Values.global.clusterDomain -}}
297+
{{- end -}}
298+
{{- end -}}
299+
{{- $port := int $e.proxyPort -}}
300+
{{- if hasKey $seen (toString $port) -}}
301+
{{- fail (printf "sandboxUi proxyPort %d is used twice" $port) -}}
302+
{{- end -}}
303+
{{- $_ := set $seen (toString $port) true -}}
304+
{{- $out = append $out (dict "workspace" $ws "sandbox" $sb "name" $label "host" $host
305+
"proxyPort" $port "forwardPort" (int $e.forwardPort)
306+
"portName" (printf "ui-%d" $port)) -}}
307+
{{- end -}}
308+
{{- toJson $out -}}
309+
{{- end }}

‎charts/openshell-saw/templates/configmap-installer.yaml‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ changes here reach an existing VM after a restart (unlike cloud-init).
3131
{{- if and (eq $signMode "enforce") (not $trustKeys) (or (not $signIdentity) (not $signIssuer)) }}
3232
{{- fail "signing.mode enforce requires signing.trustKeys or both signing.identity and signing.issuer" }}
3333
{{- end }}
34+
{{- $trusted := .Values.sandboxUiProxy.trustedProxy | default dict }}
3435
{{- $config := dict
3536
"vmName" (include "openshell-sandbox.fullname" .)
3637
"namespace" .Release.Namespace
@@ -52,7 +53,18 @@ changes here reach an existing VM after a restart (unlike cloud-init).
5253
"identity" (.Values.harness.cosign.identity | default "")
5354
"issuer" (.Values.harness.cosign.issuer | default "https://token.actions.githubusercontent.com")))
5455
"prune" (dict "mode" $pruneMode "sandboxes" (.Values.prune.sandboxes | default false))
55-
"secrets" (include "openshell-sandbox.providerSecrets" . | fromJsonArray) }}
56+
"secrets" (include "openshell-sandbox.providerSecrets" . | fromJsonArray)
57+
"sandboxUi" (include "openshell-sandbox.sandboxUi" . | fromJsonArray)
58+
"sandboxUiProxy" (dict
59+
"targetPort" (int .Values.sandboxUiProxy.targetPort)
60+
"allowedUsers" (prepend (.Values.sandboxUiProxy.allowedUsers | default list) .Values.accessControl.owner | compact | uniq)
61+
"image" .Values.dashboard.proxyImage
62+
"clientId" .Values.dashboard.clientId
63+
"insecureSkipTlsVerify" (.Values.dashboard.insecureSkipIssuerTlsVerify | default false)
64+
"trustedProxy" (dict
65+
"enabled" (eq (toString $trusted.enabled) "true")
66+
"cidrs" ($trusted.cidrs | default (list "127.0.0.1/32" "::1/128"))
67+
"deviceAutoApprove" (ne (toString $trusted.deviceAutoApprove) "false"))) }}
5668
apiVersion: v1
5769
kind: ConfigMap
5870
metadata:

‎charts/openshell-saw/templates/job-prepare.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@ metadata:
88
name: {{ include "openshell-sandbox.fullname" . }}-prepare
99
labels:
1010
{{- include "openshell-sandbox.labels" . | nindent 4 }}
11+
annotations:
12+
# A Sync hook, recreated on every sync (the steps are idempotent): a
13+
# Job cannot change once created, and new sandbox UI routes need their
14+
# Keycloak callbacks registered. BeforeHookCreation also replaces the
15+
# plain Job older charts created under this name.
16+
argocd.argoproj.io/hook: Sync
17+
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
1118
spec:
1219
backoffLimit: {{ .Values.job.backoffLimit }}
1320
activeDeadlineSeconds: {{ .Values.job.activeDeadlineSeconds }}
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{{- /* One route per sandbox web UI (sandboxUi), to the oauth2-proxy the
2+
installer runs for it in the VM. */ -}}
3+
{{- range $e := include "openshell-sandbox.sandboxUi" . | fromJsonArray }}
4+
---
5+
apiVersion: route.openshift.io/v1
6+
kind: Route
7+
metadata:
8+
name: {{ $e.name }}
9+
labels:
10+
{{- include "openshell-sandbox.labels" $ | nindent 4 }}
11+
openshell.pattern/sandbox-ui: "true"
12+
annotations:
13+
openshell.pattern/workspace: {{ $e.workspace | quote }}
14+
openshell.pattern/sandbox: {{ $e.sandbox | quote }}
15+
spec:
16+
{{- if $e.host }}
17+
host: {{ $e.host }}
18+
{{- end }}
19+
to:
20+
kind: Service
21+
name: {{ include "openshell-sandbox.fullname" $ }}-gateway
22+
weight: 100
23+
port:
24+
targetPort: {{ $e.portName }}
25+
tls:
26+
termination: edge
27+
insecureEdgeTerminationPolicy: Redirect
28+
{{- end }}

‎charts/openshell-saw/templates/service.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,9 @@ spec:
2525
port: 22
2626
targetPort: 22
2727
protocol: TCP
28+
{{- range $e := include "openshell-sandbox.sandboxUi" . | fromJsonArray }}
29+
- name: {{ $e.portName }}
30+
port: {{ $e.proxyPort }}
31+
targetPort: {{ $e.proxyPort }}
32+
protocol: TCP
33+
{{- end }}

‎charts/openshell-saw/templates/virtualmachine.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,11 @@ spec:
114114
- name: ssh
115115
port: 22
116116
protocol: TCP
117+
{{- range $e := include "openshell-sandbox.sandboxUi" . | fromJsonArray }}
118+
- name: {{ $e.portName }}
119+
port: {{ $e.proxyPort }}
120+
protocol: TCP
121+
{{- end }}
117122
firmware:
118123
bootloader:
119124
efi:

‎charts/openshell-saw/values.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -175,6 +175,30 @@ route:
175175
# Computed from global.clusterDomain when empty.
176176
dashboardHost: ""
177177

178+
# Sandbox web UIs (OpenClaw / NemoClaw control UI) on their own routes. The
179+
# saw-users chart fills the list from the user's SAW-BOM profiles (sandboxes
180+
# with ui.route: true): [{workspace, sandbox, proxyPort, forwardPort}]. For
181+
# each, the VM runs `openshell forward service` from forwardPort to the
182+
# sandbox's targetPort and an oauth2-proxy on proxyPort (Keycloak, client
183+
# dashboard.clientId), and the route <vm>-<workspace>-<sandbox>-ui sends to
184+
# that proxy. Only the owner (accessControl.owner, matched against the
185+
# Keycloak preferred_username) and allowedUsers get through. The ports are
186+
# VM interface ports, so changing the list takes a VM restart.
187+
sandboxUi: []
188+
sandboxUiProxy:
189+
targetPort: 18789
190+
allowedUsers: []
191+
# OpenClaw's trusted-proxy auth: the sandbox's gateway takes the user from
192+
# the oauth2-proxy's X-Forwarded-User (allowed: the same users) instead of
193+
# asking for its gateway token, and approves their UI devices. Only
194+
# requests from `cidrs` are trusted; the forward arrives over loopback.
195+
# The CLI and TUI in the sandbox use the gateway secret as a password.
196+
# Off: the UI asks for the token (make sandbox-ui prints it).
197+
trustedProxy:
198+
enabled: true
199+
cidrs: ["127.0.0.1/32", "::1/128"]
200+
deviceAutoApprove: true
201+
178202
# OpenShell Dashboard + oauth2-proxy, run inside the VM by the installer.
179203
# Needs oidc.issuerUrl (or global.clusterDomain) and a known webui host.
180204
dashboard:

0 commit comments

Comments
 (0)