Skip to content

Commit 19cd57c

Browse files
committed
feat(saw): sandbox web UIs on their own route, signed in with Keycloak
A sandbox with `ui: {route: true}` in its SAW-BOM profile (data-science: the default workspace's notebook) gets a route <vm>-<workspace>-<sandbox>-ui, a VM and Service port, and its callback registered on the Keycloak client openshell-dashboard. In the VM, per UI: - oauth2-proxy on 0.0.0.0:<proxyPort> (Keycloak, PKCE) admitting only the workspace owner and sandboxUiProxy.allowedUsers; - a TCP relay that lets at most 16 connections through to `openshell forward service` (OpenShell allows 20 per sandbox, NVIDIA/OpenShell#3494), queues the rest, retries refused connections, lets idle keep-alive connections give way, and re-sends a GET that gets no answer; - the forward to the sandbox's port 18789. OpenClaw runs in trusted-proxy mode for such a sandbox: it takes the user from X-Forwarded-Email (oauth2-proxy's preferred_username) over loopback only, for the same users, so the Control UI needs no gateway token. The CLI and TUI use the gateway secret as gateway.auth.password, written back after `config set` removes it (openclaw/openclaw#162216). The secret is kept across runs and the gateway restarted so config changes apply. sandboxUiProxy.trustedProxy.enabled turns it off. Installer fixes found on a live cluster: - a sandbox reporting an error after a VM restart gets 90 s to recover before it is recreated, and a recreate waits for the deletion; - onboarding waits until the sandbox accepts exec (the readiness check matched "Ready" while it was still Provisioning, so OpenClaw was never started after a reboot). Co-Authored-By: Claude
1 parent 4768f02 commit 19cd57c

16 files changed

Lines changed: 1534 additions & 77 deletions

File tree

‎charts/openshell-saw/files/installer/apply_bom.py‎

Lines changed: 670 additions & 22 deletions
Large diffs are not rendered by default.

‎charts/openshell-saw/files/prepare.sh‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@ OIDC_ISSUER_URL="{{ include "openshell-sandbox.oidcIssuerUrl" . }}"
1717
OIDC_KEYCLOAK_NAME="{{ .Values.oidc.keycloakName }}"
1818
OIDC_REALM="{{ .Values.oidc.realm }}"
1919
KEYCLOAK_NS="{{ include "openshell-sandbox.keycloakNamespace" . }}"
20+
UI_ROUTE_HOSTS="{{ range $e := include "openshell-sandbox.sandboxUi" . | fromJsonArray }}{{ $e.host }} {{ end }}"
2021
SCRIPTS_DIR="/scripts"
2122

2223
# --- Phase 1: tools ---
@@ -30,7 +31,7 @@ echo "Disk source is a registry/HTTP import; no golden image bootstrap needed."
3031
{{- end }}
3132

3233
# --- Phase 3: dashboard redirect URI in Keycloak ---
33-
if [[ "${DASHBOARD_ENABLED}" == "true" && -n "${OIDC_ISSUER_URL}" ]]; then
34+
if [[ ( "${DASHBOARD_ENABLED}" == "true" || -n "${UI_ROUTE_HOSTS// /}" ) && -n "${OIDC_ISSUER_URL}" ]]; then
3435
source "${SCRIPTS_DIR}/register-keycloak-redirect.sh"
3536
else
3637
echo "Dashboard or OIDC issuer not configured; skipping Keycloak redirect registration."
Lines changed: 57 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -1,58 +1,68 @@
11
#!/usr/bin/env bash
2-
# Phase: register this VM's dashboard redirect URI on the shared Keycloak client.
3-
# Cluster-side only; the dashboard itself is started by the in-VM installer.
2+
# Phase: register this VM's redirect URIs on the shared Keycloak client: the
3+
# dashboard's (the *-webui route) and one per sandbox web UI route
4+
# (UI_ROUTE_HOSTS). Cluster-side only; the proxies run in the VM.
45
# Expects: VM_NAME, NS, OIDC_KEYCLOAK_NAME, OIDC_REALM, KEYCLOAK_NS,
5-
# DASHBOARD_CLIENT_ID, OIDC_ISSUER_URL
6+
# DASHBOARD_CLIENT_ID, OIDC_ISSUER_URL, DASHBOARD_ENABLED, UI_ROUTE_HOSTS
67

7-
WEBUI_ROUTE_HOST="$(kubectl get route "${VM_NAME}-webui" -n "${NS}" -o jsonpath='{.spec.host}' 2>/dev/null || true)"
8-
if [[ -z "${WEBUI_ROUTE_HOST}" ]]; then
9-
echo "WARNING: ${VM_NAME}-webui route not found — skipping dashboard setup"
10-
return 0 2>/dev/null || true
8+
HOSTS=()
9+
if [[ "${DASHBOARD_ENABLED:-false}" == "true" ]]; then
10+
WEBUI_ROUTE_HOST="$(kubectl get route "${VM_NAME}-webui" -n "${NS}" -o jsonpath='{.spec.host}' 2>/dev/null || true)"
11+
if [[ -n "${WEBUI_ROUTE_HOST}" ]]; then
12+
HOSTS+=("${WEBUI_ROUTE_HOST}")
13+
else
14+
echo "WARNING: ${VM_NAME}-webui route not found — skipping the dashboard redirect URI"
15+
fi
16+
fi
17+
for host in ${UI_ROUTE_HOSTS:-}; do
18+
HOSTS+=("${host}")
19+
done
20+
if [[ ${#HOSTS[@]} -eq 0 ]]; then
21+
echo "No redirect URIs to register."
22+
return 0 2>/dev/null || exit 0
1123
fi
1224

13-
DASHBOARD_REDIRECT_URL="https://${WEBUI_ROUTE_HOST}/oauth2/callback"
14-
15-
# Register redirect URI on Keycloak client
1625
KC_ADMIN_SECRET="${OIDC_KEYCLOAK_NAME}-initial-admin"
1726
KC_ADMIN_USER="$(kubectl get secret "${KC_ADMIN_SECRET}" -n "${KEYCLOAK_NS}" -o jsonpath='{.data.username}' 2>/dev/null | base64 -d || true)"
1827
KC_ADMIN_PASS="$(kubectl get secret "${KC_ADMIN_SECRET}" -n "${KEYCLOAK_NS}" -o jsonpath='{.data.password}' 2>/dev/null | base64 -d || true)"
1928
KC_BASE="$(echo "${OIDC_ISSUER_URL}" | sed 's#/realms/.*##')"
20-
if [[ -n "${KC_ADMIN_USER}" && -n "${KC_BASE}" ]]; then
21-
echo "Registering redirect URI on Keycloak client '${DASHBOARD_CLIENT_ID}'..."
22-
KC_TOKEN_RESPONSE="$(curl -sk -X POST "${KC_BASE}/realms/master/protocol/openid-connect/token" \
23-
-d "grant_type=password" -d "client_id=admin-cli" \
24-
-d "username=${KC_ADMIN_USER}" -d "password=${KC_ADMIN_PASS}")"
25-
KC_ADMIN_TOKEN="$(echo "${KC_TOKEN_RESPONSE}" | jq -r '.access_token // empty')"
26-
if [[ -z "${KC_ADMIN_TOKEN}" ]]; then
27-
echo "WARNING: Keycloak admin token fetch failed: $(echo "${KC_TOKEN_RESPONSE}" | jq -r '.error_description // .error // "no response / unparseable response"')"
28-
fi
29-
if [[ -n "${KC_ADMIN_TOKEN}" ]]; then
30-
CLIENT_UUID="$(curl -sk -H "Authorization: Bearer ${KC_ADMIN_TOKEN}" \
31-
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients?clientId=${DASHBOARD_CLIENT_ID}" \
32-
| jq -r '.[0].id // empty')"
33-
if [[ -n "${CLIENT_UUID}" ]]; then
34-
CLIENT_JSON="$(curl -sk -H "Authorization: Bearer ${KC_ADMIN_TOKEN}" \
35-
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients/${CLIENT_UUID}")"
36-
WEBUI_ORIGIN="https://${WEBUI_ROUTE_HOST}"
37-
UPDATED_JSON="$(echo "${CLIENT_JSON}" | jq \
38-
--arg redirect "${DASHBOARD_REDIRECT_URL}" --arg origin "${WEBUI_ORIGIN}" \
39-
'.redirectUris = ((.redirectUris // []) + [$redirect] | unique) |
40-
.webOrigins = ((.webOrigins // []) + [$origin] | unique)')"
41-
HTTP_CODE="$(curl -sk -o /dev/null -w '%{http_code}' -X PUT \
42-
-H "Authorization: Bearer ${KC_ADMIN_TOKEN}" -H "Content-Type: application/json" \
43-
-d "${UPDATED_JSON}" \
44-
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients/${CLIENT_UUID}")"
45-
if [[ "${HTTP_CODE}" == "204" ]]; then
46-
echo "Redirect URI registered: ${DASHBOARD_REDIRECT_URL}"
47-
else
48-
echo "WARNING: failed to register redirect URI (HTTP ${HTTP_CODE}) — dashboard OIDC login will fail"
49-
fi
50-
else
51-
echo "WARNING: Keycloak client '${DASHBOARD_CLIENT_ID}' not found — dashboard OIDC login will fail"
52-
fi
53-
else
54-
echo "WARNING: could not obtain Keycloak admin token — dashboard OIDC login will fail"
55-
fi
29+
if [[ -z "${KC_ADMIN_USER}" || -z "${KC_BASE}" ]]; then
30+
echo "WARNING: Keycloak admin credentials not found — OIDC login through ${HOSTS[*]} will fail"
31+
return 0 2>/dev/null || exit 0
32+
fi
33+
34+
echo "Registering redirect URIs on Keycloak client '${DASHBOARD_CLIENT_ID}': ${HOSTS[*]}"
35+
KC_TOKEN_RESPONSE="$(curl -sk -X POST "${KC_BASE}/realms/master/protocol/openid-connect/token" \
36+
-d "grant_type=password" -d "client_id=admin-cli" \
37+
-d "username=${KC_ADMIN_USER}" -d "password=${KC_ADMIN_PASS}")"
38+
KC_ADMIN_TOKEN="$(echo "${KC_TOKEN_RESPONSE}" | jq -r '.access_token // empty')"
39+
if [[ -z "${KC_ADMIN_TOKEN}" ]]; then
40+
echo "WARNING: Keycloak admin token fetch failed: $(echo "${KC_TOKEN_RESPONSE}" | jq -r '.error_description // .error // "no response / unparseable response"')"
41+
return 0 2>/dev/null || exit 0
42+
fi
43+
CLIENT_UUID="$(curl -sk -H "Authorization: Bearer ${KC_ADMIN_TOKEN}" \
44+
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients?clientId=${DASHBOARD_CLIENT_ID}" \
45+
| jq -r '.[0].id // empty')"
46+
if [[ -z "${CLIENT_UUID}" ]]; then
47+
echo "WARNING: Keycloak client '${DASHBOARD_CLIENT_ID}' not found — OIDC login will fail"
48+
return 0 2>/dev/null || exit 0
49+
fi
50+
51+
REDIRECTS="$(printf 'https://%s/oauth2/callback\n' "${HOSTS[@]}" | jq -R . | jq -s .)"
52+
ORIGINS="$(printf 'https://%s\n' "${HOSTS[@]}" | jq -R . | jq -s .)"
53+
CLIENT_JSON="$(curl -sk -H "Authorization: Bearer ${KC_ADMIN_TOKEN}" \
54+
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients/${CLIENT_UUID}")"
55+
# Added, never removed: other VMs register on the same client.
56+
UPDATED_JSON="$(echo "${CLIENT_JSON}" | jq \
57+
--argjson redirects "${REDIRECTS}" --argjson origins "${ORIGINS}" \
58+
'.redirectUris = ((.redirectUris // []) + $redirects | unique) |
59+
.webOrigins = ((.webOrigins // []) + $origins | unique)')"
60+
HTTP_CODE="$(curl -sk -o /dev/null -w '%{http_code}' -X PUT \
61+
-H "Authorization: Bearer ${KC_ADMIN_TOKEN}" -H "Content-Type: application/json" \
62+
-d "${UPDATED_JSON}" \
63+
"${KC_BASE}/admin/realms/${OIDC_REALM}/clients/${CLIENT_UUID}")"
64+
if [[ "${HTTP_CODE}" == "204" ]]; then
65+
echo "Redirect URIs registered: $(echo "${REDIRECTS}" | jq -r 'join(" ")')"
5666
else
57-
echo "WARNING: Keycloak admin credentials not found — dashboard OIDC login will fail"
67+
echo "WARNING: failed to register redirect URIs (HTTP ${HTTP_CODE}) — OIDC login will fail"
5868
fi

‎charts/openshell-saw/templates/_helpers.tpl‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,3 +259,44 @@ rpc = "openshell.v1.OpenShell/SubmitPolicyAnalysis"
259259
phases = ["validate"]
260260
{{- end }}
261261
{{- end }}
262+
263+
264+
{{/*
265+
Sandbox UI routes: sandboxUi entries with the route host filled in,
266+
<vm>-<workspace>-<sandbox>-ui.apps.<clusterDomain> (at most 62 characters
267+
in its first label with 19-character names), or entry.host when set.
268+
*/}}
269+
{{- define "openshell-sandbox.sandboxUi" -}}
270+
{{- $root := . -}}
271+
{{- $out := list -}}
272+
{{- $seen := dict -}}
273+
{{- range $e := .Values.sandboxUi | default list -}}
274+
{{- $ws := $e.workspace | default "" | toString -}}
275+
{{- $sb := $e.sandbox | default "" | toString -}}
276+
{{- if not (and (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $ws) (regexMatch "^[a-z0-9]([-a-z0-9]*[a-z0-9])?$" $sb)) -}}
277+
{{- fail (printf "sandboxUi: workspace %q and sandbox %q must be lowercase DNS labels" $ws $sb) -}}
278+
{{- end -}}
279+
{{- if not (and $e.proxyPort $e.forwardPort) -}}
280+
{{- fail (printf "sandboxUi %s/%s needs proxyPort and forwardPort" $ws $sb) -}}
281+
{{- end -}}
282+
{{- $label := printf "%s-%s-%s-ui" (include "openshell-sandbox.fullname" $root) $ws $sb -}}
283+
{{- if gt (len $label) 63 -}}
284+
{{- fail (printf "sandboxUi route label %q is %d characters; DNS labels allow 63" $label (len $label)) -}}
285+
{{- end -}}
286+
{{- $host := $e.host | default "" -}}
287+
{{- if and (not $host) $root.Values.global -}}
288+
{{- if $root.Values.global.clusterDomain -}}
289+
{{- $host = printf "%s.apps.%s" $label $root.Values.global.clusterDomain -}}
290+
{{- end -}}
291+
{{- end -}}
292+
{{- $port := int $e.proxyPort -}}
293+
{{- if hasKey $seen (toString $port) -}}
294+
{{- fail (printf "sandboxUi proxyPort %d is used twice" $port) -}}
295+
{{- end -}}
296+
{{- $_ := set $seen (toString $port) true -}}
297+
{{- $out = append $out (dict "workspace" $ws "sandbox" $sb "name" $label "host" $host
298+
"proxyPort" $port "forwardPort" (int $e.forwardPort)
299+
"portName" (printf "ui-%d" $port)) -}}
300+
{{- end -}}
301+
{{- toJson $out -}}
302+
{{- end }}

‎charts/openshell-saw/templates/configmap-installer.yaml‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ changes here reach an existing VM after a restart (unlike cloud-init).
3131
{{- if and (eq $signMode "enforce") (not $trustKeys) (or (not $signIdentity) (not $signIssuer)) }}
3232
{{- fail "signing.mode enforce requires signing.trustKeys or both signing.identity and signing.issuer" }}
3333
{{- end }}
34+
{{- $trusted := .Values.sandboxUiProxy.trustedProxy | default dict }}
3435
{{- $config := dict
3536
"vmName" (include "openshell-sandbox.fullname" .)
3637
"namespace" .Release.Namespace
@@ -49,7 +50,18 @@ changes here reach an existing VM after a restart (unlike cloud-init).
4950
"insecureSkipTlsVerify" (.Values.dashboard.insecureSkipIssuerTlsVerify | default false))
5051
"signing" (dict "mode" $signMode "trustKeys" $trustKeys "identity" $signIdentity "issuer" $signIssuer)
5152
"prune" (dict "mode" $pruneMode "sandboxes" (.Values.prune.sandboxes | default false))
52-
"secrets" (include "openshell-sandbox.providerSecrets" . | fromJsonArray) }}
53+
"secrets" (include "openshell-sandbox.providerSecrets" . | fromJsonArray)
54+
"sandboxUi" (include "openshell-sandbox.sandboxUi" . | fromJsonArray)
55+
"sandboxUiProxy" (dict
56+
"targetPort" (int .Values.sandboxUiProxy.targetPort)
57+
"allowedUsers" (prepend (.Values.sandboxUiProxy.allowedUsers | default list) .Values.accessControl.owner | compact | uniq)
58+
"image" .Values.dashboard.proxyImage
59+
"clientId" .Values.dashboard.clientId
60+
"insecureSkipTlsVerify" (.Values.dashboard.insecureSkipIssuerTlsVerify | default false)
61+
"trustedProxy" (dict
62+
"enabled" (eq (toString $trusted.enabled) "true")
63+
"cidrs" ($trusted.cidrs | default (list "127.0.0.1/32" "::1/128"))
64+
"deviceAutoApprove" (ne (toString $trusted.deviceAutoApprove) "false"))) }}
5365
apiVersion: v1
5466
kind: ConfigMap
5567
metadata:

‎charts/openshell-saw/templates/job-prepare.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,13 @@ metadata:
88
name: {{ include "openshell-sandbox.fullname" . }}-prepare
99
labels:
1010
{{- include "openshell-sandbox.labels" . | nindent 4 }}
11+
annotations:
12+
# A Sync hook, recreated on every sync (the steps are idempotent): a
13+
# Job cannot change once created, and new sandbox UI routes need their
14+
# Keycloak callbacks registered. BeforeHookCreation also replaces the
15+
# plain Job older charts created under this name.
16+
argocd.argoproj.io/hook: Sync
17+
argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
1118
spec:
1219
backoffLimit: {{ .Values.job.backoffLimit }}
1320
activeDeadlineSeconds: {{ .Values.job.activeDeadlineSeconds }}
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{{- /* One route per sandbox web UI (sandboxUi), to the oauth2-proxy the
2+
installer runs for it in the VM. */ -}}
3+
{{- range $e := include "openshell-sandbox.sandboxUi" . | fromJsonArray }}
4+
---
5+
apiVersion: route.openshift.io/v1
6+
kind: Route
7+
metadata:
8+
name: {{ $e.name }}
9+
labels:
10+
{{- include "openshell-sandbox.labels" $ | nindent 4 }}
11+
openshell.pattern/sandbox-ui: "true"
12+
annotations:
13+
openshell.pattern/workspace: {{ $e.workspace | quote }}
14+
openshell.pattern/sandbox: {{ $e.sandbox | quote }}
15+
spec:
16+
{{- if $e.host }}
17+
host: {{ $e.host }}
18+
{{- end }}
19+
to:
20+
kind: Service
21+
name: {{ include "openshell-sandbox.fullname" $ }}-gateway
22+
weight: 100
23+
port:
24+
targetPort: {{ $e.portName }}
25+
tls:
26+
termination: edge
27+
insecureEdgeTerminationPolicy: Redirect
28+
{{- end }}

‎charts/openshell-saw/templates/service.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,9 @@ spec:
2525
port: 22
2626
targetPort: 22
2727
protocol: TCP
28+
{{- range $e := include "openshell-sandbox.sandboxUi" . | fromJsonArray }}
29+
- name: {{ $e.portName }}
30+
port: {{ $e.proxyPort }}
31+
targetPort: {{ $e.proxyPort }}
32+
protocol: TCP
33+
{{- end }}

‎charts/openshell-saw/templates/virtualmachine.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,11 @@ spec:
114114
- name: ssh
115115
port: 22
116116
protocol: TCP
117+
{{- range $e := include "openshell-sandbox.sandboxUi" . | fromJsonArray }}
118+
- name: {{ $e.portName }}
119+
port: {{ $e.proxyPort }}
120+
protocol: TCP
121+
{{- end }}
117122
firmware:
118123
bootloader:
119124
efi:

‎charts/openshell-saw/values.yaml‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -167,6 +167,30 @@ route:
167167
# Computed from global.clusterDomain when empty.
168168
dashboardHost: ""
169169

170+
# Sandbox web UIs (OpenClaw / NemoClaw control UI) on their own routes. The
171+
# saw-users chart fills the list from the user's SAW-BOM profiles (sandboxes
172+
# with ui.route: true): [{workspace, sandbox, proxyPort, forwardPort}]. For
173+
# each, the VM runs `openshell forward service` from forwardPort to the
174+
# sandbox's targetPort and an oauth2-proxy on proxyPort (Keycloak, client
175+
# dashboard.clientId), and the route <vm>-<workspace>-<sandbox>-ui sends to
176+
# that proxy. Only the owner (accessControl.owner, matched against the
177+
# Keycloak preferred_username) and allowedUsers get through. The ports are
178+
# VM interface ports, so changing the list takes a VM restart.
179+
sandboxUi: []
180+
sandboxUiProxy:
181+
targetPort: 18789
182+
allowedUsers: []
183+
# OpenClaw's trusted-proxy auth: the sandbox's gateway takes the user from
184+
# the oauth2-proxy's X-Forwarded-User (allowed: the same users) instead of
185+
# asking for its gateway token, and approves their UI devices. Only
186+
# requests from `cidrs` are trusted; the forward arrives over loopback.
187+
# The CLI and TUI in the sandbox use the gateway secret as a password.
188+
# Off: the UI asks for the token (make sandbox-ui prints it).
189+
trustedProxy:
190+
enabled: true
191+
cidrs: ["127.0.0.1/32", "::1/128"]
192+
deviceAutoApprove: true
193+
170194
# OpenShell Dashboard + oauth2-proxy, run inside the VM by the installer.
171195
# Needs oidc.issuerUrl (or global.clusterDomain) and a known webui host.
172196
dashboard:

0 commit comments

Comments
 (0)