Repository navigation
feat(opencode-router): MVP multi-user multi-session opencode web via Kubernetes pod isolation - #2
Merged
Merged
Conversation
Introduces the opencode-router package — a reverse proxy that dynamically provisions per-user OpenCode pods on Kubernetes. Includes README (what/why) and ADR-001 documenting architectural decisions (how). https://claude.ai/code/session_01Fr7xuy5tERDbTxZ8H5DtaK
Adds the opencode-router package — a Node.js HTTP/WebSocket reverse proxy that dynamically provisions per-user Kubernetes Pods and PVCs for isolated OpenCode sessions. Uses @kubernetes/client-node for pod lifecycle management and http-proxy for transparent request proxying. - config.ts: environment-based configuration with validation - pod-manager.ts: idempotent PVC/Pod creation, idle cleanup, activity tracking - index.ts: HTTP server with WebSocket upgrade support and graceful shutdown - Dockerfile: multi-stage Node 22 Alpine build https://claude.ai/code/session_01Fr7xuy5tERDbTxZ8H5DtaK
Adds a SolidJS + Vite SPA (packages/opencode-router-app/) that lets users choose which git repo to clone when first visiting their OpenCode session. Reuses @opencode-ai/ui components (Button, TextField, Logo, ThemeProvider) for consistent look and feel. Router changes: - New API endpoints: GET /api/status, POST /api/sessions - Static file serving for SPA assets with index.html SPA fallback - ensurePod() now accepts optional repoUrl parameter - New getPodState() for checking pod status without provisioning - Replaces hardcoded LOADING_HTML with SPA-driven loading screen - Multi-stage Dockerfile includes SPA build https://claude.ai/code/session_01Fr7xuy5tERDbTxZ8H5DtaK
Covers the full Kubernetes deployment: namespace, RBAC, Secret, ConfigMap, router Deployment/Service, Ingress with oauth2-proxy (Traefik and nginx examples), Docker image build, dynamically created resources, capacity planning, network policies, and operational runbook. Written so an agent translating this into Pulumi/Terraform/Helm manifests understands both the what and the why for each resource. https://claude.ai/code/session_01Fr7xuy5tERDbTxZ8H5DtaK
Add pod-level (runAsUser/Group 1000, fsGroup 1000, runAsNonRoot) and container-level (no privilege escalation, drop ALL caps, RuntimeDefault seccomp) securityContext to ensurePod() so user pods comply with restricted PSS and the PVC mounted at /root is writable by UID 1000 via fsGroup. Also add securityContext to the git-init init container so cloned files are owned by UID 1000 rather than root.
…r pods Document the decision to add pod/container-level securityContext (UID 1000, fsGroup 1000, drop ALL caps, RuntimeDefault seccomp) to user pods so they comply with Kubernetes PSS restricted level and PVCs are writable by UID 1000.
Add COPY patches/ patches/ before bun install so frozen-lockfile resolution works. Copy all workspace package.json files so bun can validate the lockfile. Use monorepo root as build context. Use --ignore-scripts to skip native compilation (tree-sitter-powershell etc) that is not needed for the router or router-app builds. Fix bun run --cwd syntax: flag must come after 'run' subcommand. Removes the broken npm ci + package-lock.json approach.
…nd pod-level securityContext PSS restricted:latest requires seccompProfile.type RuntimeDefault on every container including init containers. The git-init init container was missing this field, causing pod creation to fail with a 403 Forbidden from the K8s admission controller when a user provides a git repo URL.
…onfig, pull secrets, auth bypass - Fix loadFromCluster() silently producing undefined server URL outside cluster; detect SA token file presence to choose loadFromDefault() instead - Fix opencode serve command: --host → --hostname, add explicit --port flag - Add imagePullSecretName config + pod spec imagePullSecrets (IMAGE_PULL_SECRET_NAME env) - Add DEV_EMAIL fallback in getEmail() for local dev without oauth2-proxy - Add DEV_POD_PROXY_TARGET to bypass unreachable pod IPs outside the cluster - Fix redirect loop in router-app: use VITE_ROUTER_URL env to redirect to router instead of back to Vite root when pod is running - Add dev scripts: create-local-kubeconfig.sh, port-forward-pod.sh - Add .env.local.example for both router and router-app packages - Add npm run dev script that auto-sources .env.local - Update deployment.md with local dev section and corrected serve command
…on router
Instead of having the SPA redirect to a different URL (causing dev/prod divergence),
the router now proxies the setup UI to the Vite dev server when DEV_VITE_URL is set.
- router/src/index.ts: proxy to DEV_VITE_URL (HTTP + WS) when no pod is running
- router/src/config.ts: add devViteUrl from DEV_VITE_URL env var
- router-app/src/app.tsx: revert to plain window.location.replace('/') - no more VITE_ROUTER_URL
- router-app/vite.config.ts: remove /api proxy - Vite is now a pure HMR backend
- .env.local.example: swap VITE_ROUTER_URL → DEV_VITE_URL, single entry point is localhost:3002
Dev flow: open localhost:3002 (router), not localhost:5173 (Vite)
HMR still works via WS proxied through the router
Replaces the single per-user pod model with one pod+PVC per (email, repoUrl, branch).
Router changes:
- getSessionHash(email, repoUrl, branch) → 12-char hex session identity
- Pods named opencode-session-<hash>, PVCs opencode-pvc-<hash>
- Annotations: opencode.ai/repo-url, opencode.ai/branch (enables session listing)
- git-init: clone + checkout branch (create from HEAD if branch doesn't exist remotely)
- New API: GET /api/sessions (list), POST /api/sessions {repoUrl,branch} → {hash,url}
GET /api/sessions/:hash (state)
- Routing: /code/:hash/* → proxy to session pod (strips prefix before proxying)
/ → always serves SPA or Vite dev proxy (never hijacked by running pod)
- updateLastActivity/getPodIP now take hash directly (not email)
- listUserSessions(email): lists all pods with matching email annotation
SPA changes:
- api.ts: Session type, listSessions(), createSession(repoUrl, branch), getSessionState(hash)
- app.tsx: session list view with links to /code/:hash + New Session button
- setup-form.tsx: add branch field, onCreated receives (hash, url)
- loading-screen.tsx: polls /api/sessions/:hash, redirects to session URL on running
GET /api/sessions now returns {email, sessions:[]} instead of just an array.
Email comes from the authenticated request context, so it's always present
even when the user has no sessions yet.
SPA api.ts: SessionsResponse type, listSessions() returns {email, sessions}
SPA app.tsx: destructures {email, sessions} directly from listSessions()
git refuses to operate on directories owned by a different UID. This happens when the PVC was previously written by root (from an earlier broken pod run) and the init container runs as UID 1000. Add 'git config --global --add safe.directory /workspace' before any git operations so the init container always succeeds regardless of PVC ownership.
…iting gitconfig Two issues with the previous approach: 1. 'git config --global' needs a writable $HOME — init container has HOME=/ (UID 1000, no home dir) causing 'could not lock config file //gitconfig: Permission denied' 2. The global config write is fragile for ephemeral containers anyway Fix: - Set HOME=/tmp on the init container (writable, ephemeral) - Use 'git -c safe.directory=/workspace' inline flag on all git operations instead of writing to global config — no file write needed, applies per-command
… computing name The pod naming changed from opencode-user-<email-hash> to opencode-session-<session-hash> (where session hash includes email+repo+branch), so computing the name from email alone no longer works. New approach: list running pods by label selector, auto-select if only one is running, prompt the user to pick if multiple sessions are running.
…ession cookie Problem: opencode's SPA makes requests to absolute paths (/session, /provider, etc.). When the browser is at /code/<hash>, these requests hit the router without the /code/<hash> prefix and were falling through to the Vite proxy instead of the opencode pod. Fix: set an 'oc-session' cookie when the browser navigates to /code/<hash>. All subsequent requests that don't match /code/:hash or /api/ are routed to the session pod identified by the cookie. This covers all absolute-path API and WebSocket requests made by the opencode SPA. Flow: GET /code/<hash> → set cookie oc-session=<hash>, proxy to pod (strip prefix) GET /session → read cookie, proxy to same pod WS /pty/... → read cookie, proxy WS to same pod GET /api/sessions → handled by router (not proxied) GET / → no cookie or stale cookie → SPA
…uterDomain> Replaces /code/:hash path prefix + cookie hack with proper subdomain isolation. Architecture: opencode-router.domain → router SPA (session list, new session form) <hash>.opencode-router.domain → opencode pod for that session Router changes: - getSessionHash(host): extracts 12-char hex from subdomain, null for root domain - HTTP handler: session subdomain → proxy directly to pod (no prefix stripping needed) - WS handler: same subdomain check for WebSocket upgrades - ROUTER_DOMAIN env var required (was optional with fallback — removed fallback) - Session URLs in API responses: proto://<hash>.<routerDomain> (proto from X-Forwarded-Proto) - listUserSessions: takes IncomingMessage to derive scheme; builds full subdomain URLs - api.ts: sessionUrl() helper builds full URL for POST /api/sessions and GET /api/sessions/:hash Config: - ROUTER_DOMAIN is now required (required() throws on startup if missing) - .env.local: add ROUTER_DOMAIN=opencode-router.no-panic.org
… without rebuild - getSessionHash(): strip port from routerDomain before suffix-match so 'abc123.localhost:3002' correctly matches ROUTER_DOMAIN=localhost:3002 - .env.local.example: ROUTER_DOMAIN=localhost:3002 (*.localhost resolves to 127.0.0.1 natively — no /etc/hosts or DNS needed) - .env.local.example: NODE_TLS_REJECT_UNAUTHORIZED=0 for self-signed k3s cert - package.json dev script: 'bun --env-file=.env.local --watch src/index.ts' (runs TS natively, no build step, hot-reload on source changes) - .env.local.example: converted from 'export KEY=val' shell to plain dotenv - create-local-kubeconfig.sh: update instructions, document TOKEN_DURATION=168h
…rojects The init container clones the repo directly into /workspace (subPath 'projects' on the PVC), which maps to <PVC>/projects/. The main container mounts the full PVC at /root with no subPath, so the repo is at /root/projects/ inside the container. Without workingDir set, opencode serve used process.cwd() (likely / or /root) and failed to discover the git project. Setting workingDir: '/root/projects' makes opencode start in the correct directory.
…ounts Switch all /root paths to /home/opencode to match the proper non-root user home directory in the opencode image. The homelab Dockerfile now creates the opencode user with -h /home/opencode (not /root), so mounts must align: - PVC mounted at /home/opencode (was /root) - ConfigMap mounted at /home/opencode/.opencode (was /root/.opencode) - workingDir: /home/opencode/projects (was /root/projects)
…sion URLs Add ROUTE_SUFFIX env var (default: "") that is appended to the session hash to form the session subdomain: <hash><ROUTE_SUFFIX>.<ROUTER_DOMAIN> With ROUTE_SUFFIX="-oc" and ROUTER_DOMAIN="no-panic.org": abc123def456-oc.no-panic.org This keeps session hostnames at the first subdomain level, covered by the *.no-panic.org Universal SSL certificate, avoiding the need for a Cloudflare Advanced Certificate Manager subscription.
- Fix sessionUrl() in api.ts: was missing config.routeSuffix, causing POST /api/sessions to redirect to <hash>.<domain> instead of <hash><ROUTE_SUFFIX>.<domain> (e.g. <hash>-oc.no-panic.org) - README: add session URL pattern section explaining ROUTE_SUFFIX rationale (first-level subdomain for wildcard TLS coverage) - README: add ROUTER_DOMAIN and ROUTE_SUFFIX to configuration table - README: update per-user language to per-session (key is email+repo+branch) - README: fix RBAC block to include watch verb on pods
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
mrsimpson
added a commit
to digitaleraluhut/homelab
that referenced
this pull request
Apr 12, 2026
… with Cloudflare operator (#31) Implements the homelab infrastructure side of the multi-user multi-session opencode web feature (mrsimpson/opencode#2, addressing upstream issue anomalyco/opencode#20067). ## Intent `opencode serve` is a single-user server — everyone on the same instance shares sessions, filesystem, and API keys. This PR deploys an MVP solution on the homelab Kubernetes cluster: a router that gives each user a dedicated `opencode serve` pod per (repo, branch) session, with full isolation and GitHub OAuth protection. Each session gets its own **first-level subdomain** (`<hash>-oc.<domain>`), matching the Claude Code web model where separate coding sessions have separate URLs — and staying within Cloudflare Universal SSL wildcard coverage (`*.<domain>`) so no paid certificate product is needed. Since session hostnames are created at runtime, a new **Cloudflare operator sidecar** watches session pods and provisions/deprovisions DNS records and Traefik IngressRoutes on demand. ## Architecture ``` Browser │ HTTPS ▼ Cloudflare Tunnel → Traefik ├─ opencode-router.<domain> ──[oauth2-chain]──▶ opencode-router (:3000) │ │ setup UI: pick repo + branch │ │ POST /api/sessions │ ▼ │ K8s: create Pod + PVC │ (one per email+repo+branch) │ └─ <hash>-oc.<domain> ──[oauth2-chain]──▶ opencode-router (:3000) │ proxy by hash from Host header ▼ opencode-session-<hash> Pod (:4096) (opencode serve, isolated filesystem + credentials) ``` The router pod runs **two containers**: - `app` — the opencode-router: creates pods/PVCs via the K8s API, proxies traffic to the correct session pod by hash - `cloudflare-operator` — sidecar: watches session pods and manages per-session DNS records, Cloudflare Tunnel routes, and Traefik IngressRoutes On pod **ADDED**: operator creates a Cloudflare CNAME (`<hash>-oc.<domain>` → tunnel), a tunnel ingress rule, and two Traefik `IngressRoute` CRDs (reusing the router's existing oauth2-chain middleware). On pod **DELETED**: all three are removed. ## Major changes **New: `images/opencode-cloudflare-operator/`** — standalone Node.js operator image - `src/index.ts` — K8s pod watch loop with exponential backoff reconnect; health server on `:8080` - `src/cloudflare.ts` — Cloudflare API: DNS record CRUD + Tunnel ingress rule CRUD; idempotent against concurrent replica writes (catches "already exists" on POST) - `src/ingressroute.ts` — creates/deletes Traefik `IngressRoute` CRDs per session, mirroring the pattern ExposedWebApp uses for the main router domain - `src/config.ts` — all config via env vars (`CF_API_TOKEN`, `CF_ZONE_ID`, `CF_TUNNEL_ID`, `DOMAIN`, `ROUTE_SUFFIX`, `INGRESSROUTE_NAMESPACE`, `OAUTH2_CHAIN_MIDDLEWARE`, `ROUTER_SERVICE_NAME`, …) - `build.sh` — multi-arch Docker build + push to GHCR **New: `packages/apps/opencode-router/`** — Pulumi package - Namespace `opencode-router` with `restricted` PSS - ServiceAccount + Role + RoleBinding: pod/PVC management for router + `traefik.io/ingressroutes` create/delete for operator - Secrets: `opencode-api-keys` (Anthropic key for session pods), `opencode-router-cf-credentials` (Cloudflare API token for operator) - ConfigMap: `opencode-config-dir` with shared `opencode.json` - ExternalSecret: GHCR pull secret (explicit — namespace is pre-created, so ExposedWebApp's auto-create path is skipped) - `ExposedWebApp` for the router deployment with operator as `extraContainers` sidecar; env vars `ROUTER_DOMAIN` + `ROUTE_SUFFIX=-oc` set on the router container - Build script: `images/opencode-router/build.sh` **Extended: `packages/core/components/src/ExposedWebApp.ts`** - `image` type widened to `string | pulumi.Output<string>` - `extraContainers?: object[]` — appends sidecar containers alongside `app` - `serviceAccountName?: string` — sets pod ServiceAccount - `probes?: { readinessProbe?, livenessProbe? }` — container health probes - `namespace` exposed as public property - `allowOverwrite: true` on the Cloudflare DNS record **New: `images/opencode-router/build.sh`** — builds the opencode-router image from the fork monorepo **Updated: `src/index.ts`** — wires `createOpencodeRouter` into the root stack with `cfOperatorImage`, `tunnelId`, `apiToken` config values **Updated: `docs/howto/`** — `expose-web-app.md` and `add-oauth-protection.md` updated to reflect ExposedWebApp extensions ## Side effects - The `opencode-router` ServiceAccount has `traefik.io/ingressroutes` RBAC — scoped to the `opencode-router` namespace only - With 2 router replicas, both operator sidecars watch the same pods; all provisioning operations are idempotent (DNS: GET-before-POST + catch "already exists"; IngressRoute: HTTP 409 no-op; Tunnel route: check before PUT) - Session pods, PVCs, DNS records, Tunnel routes, and IngressRoutes are **not Pulumi-managed** — created/deleted at runtime, not visible in `pulumi up` output - Cloudflare Tunnel config is mutated at runtime by the operator; the static Pulumi-managed catch-all rule must remain last (the operator preserves this invariant)
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses anomalyco#20067 (as MVP / alternative approach)
Intent
opencode serveruns a single-user server — everyone hitting the same instance shares sessions, filesystem, and API keys. This is the core problem raised in anomalyco#20067.This PR takes a proxy-based approach rather than adding auth into
opencode serveitself: a lightweight router sits in front of oneopencode servepod per session, providing:opencode serveitself needs no auth changesThis is an MVP. It is opinionated about the deployment target (Kubernetes) and auth mechanism (oauth2-proxy), but the router itself is a standard Node.js HTTP/WebSocket proxy that could front other auth mechanisms.
Architecture
Each session gets a dedicated subdomain (
<hash>-oc.<domain>) — the router identifies which pod to proxy to by extracting the hash from theHostheader. This is the same pattern used by Claude Code web, where each coding session has its own URL.Major changes
New:
packages/opencode-router/— HTTP/WebSocket reverse proxysrc/pod-manager.ts— creates/deletes Kubernetes Pods and PVCs on demand; session key issha256(email + repoUrl + branch)[0:12]; fullrestrictedPSS-compliant securityContext; git-init init container clones the repo on first pod creationsrc/api.ts—GET /api/sessions,POST /api/sessions {repoUrl, branch},GET /api/sessions/:hash; session URLs includeROUTE_SUFFIXfor first-level subdomain TLS coveragesrc/index.ts— HTTP server + WebSocket upgrade handler;getSessionHash()extracts session fromHostheader; background idle pod cleanupsrc/config.ts—ROUTER_DOMAIN(required base domain),ROUTE_SUFFIX(default"", set to-ocin production), plus K8s/storage configDockerfile— multi-stage build from monorepo rootdocs/deployment.md— full Kubernetes deployment guide (RBAC, Secrets, Ingress, WebSocket config)docs/adr-001-per-user-pod-isolation.md— rationale for proxy pattern over in-process authdocs/adr-002-restricted-pss-securitycontext.md— Kubernetes security hardeningNew:
packages/opencode-router-app/— session management SPAGET /api/sessions/:hash)ROUTE_SUFFIXdesign — session hostnames are<hash><ROUTE_SUFFIX>.<ROUTER_DOMAIN>. The suffix (e.g.-oc) keeps sessions at the first subdomain level, covered by a standard*.<domain>wildcard TLS certificate. Without it, sessions at<hash>.opencode-router.<domain>(second level) would require a paid certificate product.Side effects
ROUTER_DOMAINis a required env varROUTE_SUFFIXdefaults to""(backward compatible); set to-ocin productionDEV_EMAILenables local dev auth bypass; must never be set in production