Repository navigation
feat(opencode-router): deploy per-session isolated OpenCode instances with Cloudflare operator - #31
Merged
Merged
Conversation
Build opencode-router from the opencode monorepo root with the Dockerfile at packages/opencode-router/Dockerfile. Version is read from the monorepo package.json; tag format is <version>-homelab.<revision>.
- packages/apps/opencode-router/package.json + tsconfig.json - packages/apps/opencode-router/src/index.ts: 15 K8s resources (Namespace/PSS-restricted, SA, Role, RoleBinding, Secret, ConfigMap, ExternalSecret, Deployment/2-replica/non-root, Service, 3 Traefik middlewares, 2 IngressRoutes, Cloudflare DNS CNAME) - src/index.ts: import createOpencodeRouter + call with opencodeConfig - package.json: add @mrsimpson/homelab-app-opencode-router dependency Cloudflare zoneId sourced from homelabConfig.cloudflare.zoneId; tunnelCname from baseInfra.cloudflare.tunnelCname.
All homelab-4.3.x tasks closed: - 4.3.1: pod-manager.ts securityContext fix (in opencode fork) - 4.3.2: ADR-002 written (in opencode fork) - 4.3.3: images/opencode-router/build.sh - 4.3.4: packages/apps/opencode-router package.json + tsconfig.json - 4.3.5: full Pulumi index.ts (15 K8s resources) - 4.3.6: wired into root src/index.ts + package.json - 4.3.7: Pulumi config set (gitignored Pulumi.dev.yaml) - 4.3.8: tsc --noEmit + biome check both clean Key decisions added: Cloudflare DNS wiring pattern, optional cfg.cloudflare field, opencodeImage/routerImage Pulumi config keys, ADR-002 reference.
pulumi preview: +15 to create (opencode-router resources), 108 unchanged. Zero errors, zero deletions, zero unexpected modifications. Clean preview.
…Store and key refs - ClusterSecretStore name: cluster-secret-store -> pulumi-esc - data keys: ghcr-credentials/auth -> github-username + github-token (matches the pattern used by all other working ExternalSecrets in the cluster) - Template updated to build dockerconfigjson from username+token (same as opencode ns)
- ExternalSecret fix: wrong store name + key refs corrected - GHCR PAT expiry: diagnosis and resolution documented - Deployment verified: 2/2 pods Running, all 15 resources live - Public URL https://opencode-router.no-panic.org/ returns 401 (correct)
…t env + plan updates - Add IMAGE_PULL_SECRET_NAME=ghcr-pull-secret to router Deployment env so user pods get imagePullSecrets set (needed to pull ghcr.io/mrsimpson/opencode from GHCR) - Update dev plan with local dev decisions: DEV_EMAIL, Vite proxy, kubeconfig, port-forward
…_DOMAIN env - Add wildcard IngressRoute for *.opencode-router.<domain> → router service (HostRegexp matching 12-char hex session hashes) - Add wildcard Cloudflare DNS CNAME for *.opencode-router.<domain> - Pass ROUTER_DOMAIN env var to router Deployment - Fix health probes: /api/status → /api/sessions (status endpoint removed)
Key decisions logged: - *.localhost:3002 subdomain routing for local dev (no /etc/hosts) - pnpm dev now runs TS directly via bun --watch (no rebuild needed) - NODE_TLS_REJECT_UNAUTHORIZED=0 for self-signed k3s cert in dev - SA token expiry: use TOKEN_DURATION=168h for 7-day tokens WIP documented: - Workspace directory not properly mounted in session pods - Branch detection should auto-detect from repo, not require manual entry
Replace -h /root with -h /home/opencode in adduser so the opencode user (UID 1000) has a conventional home directory. /root as a non-root user's home is confusing and unconventional. Aligns with pod-manager volume mounts which now mount the PVC at /home/opencode.
Refactor opencode-router to use ExposedWebApp instead of manually creating ~15 K8s resources. ExposedWebApp now handles Deployment, Service, OAuth2-Proxy auth (middlewares + IngressRoutes), main DNS CNAME, and GHCR pull secret. ExposedWebApp gains two general-purpose fields: `serviceAccountName` (for setting a ServiceAccount on the pod spec) and `probes` (readiness/liveness). The `namespace` property is now publicly exposed for dependency wiring. The opencode-router retains app-specific resources: Namespace, RBAC (ServiceAccount/Role/RoleBinding), Secret, ConfigMap, wildcard IngressRoute for session subdomains, and wildcard DNS record. https://claude.ai/code/session_017UpZCGUzQchsP77wsRPNmx
- ExposedWebApp.ts JSDoc: document serviceAccountName, probes, and the OAuth2-Proxy middleware naming contract (<name>-oauth2-chain) - docs/howto/expose-web-app.md: rewrite for current HomelabContext + ExposedWebApp API (was describing old ingress-nginx pattern) - docs/howto/add-oauth-protection.md: rewrite for centralized oauth2-proxy with AuthType.OAUTH2_PROXY (was describing defunct per-app OAuth sidecar pattern) - packages/apps/opencode-router/README.md: new, covers intent, architecture, custom image dependency, and config variables - .vibe/development-plan: research and key decisions for the ExposedWebApp integration refactor https://claude.ai/code/session_017UpZCGUzQchsP77wsRPNmx
…ix type casts Two regressions from the ExposedWebApp refactor: 1. ExternalSecret for GHCR pull secret was silently dropped. ExposedWebApp auto-creates it only when isCreatingNamespace=true, but we pre-create the namespace and pass it in — so auto-create never fires. Explicit ExternalSecret resource restored in createOpencodeRouter(). Deployment now dependsOn pullSecret. 2. 'as unknown as string' type casts in image/env fields removed. ExposedWebApp.image widened to 'string | pulumi.Output<string>', and env values already accept Output<string>, so no casts needed. Note: main DNS record (opencode-router.<domain>) is created automatically by ExposedWebApp via HomelabContext's injected cloudflare config — not a regression, just clarified in comments. Also: 'void roleBinding/apiKeysSecret/configMap' statements replaced with proper Pulumi dependsOn on the ExposedWebApp call for RBAC and pullSecret resources.
When a DNS record for the same name/type already exists in Cloudflare (e.g. created by a previous Pulumi run under a different URN, or migrated from manually-created records), the Cloudflare API returns an error without allowOverwrite=true. This happened when opencode-router's DNS record moved from a top-level resource (old manual code) to an ExposedWebApp child resource — the old Pulumi state entry was deleted but the Cloudflare record persisted.
…code-router domain Universal SSL only covers *.no-panic.org (one level deep). Session subdomains (*.opencode-router.no-panic.org) are two levels deep and get a TLS handshake failure without an explicit cert. Added CertificatePack (advanced, Let's Encrypt, DNS-01/txt validation) covering both opencode-router.no-panic.org and *.opencode-router.no-panic.org, scoped to the opencode-router app package — not base infra, since this is app-specific.
*.opencode-router.<domain> is a second-level wildcard, not covered by Cloudflare's Universal SSL (free plan only issues *.no-panic.org). Advanced Certificate Manager costs $10/month. Session URLs are now <hash>.ocsession.<domain> — a first-level subdomain covered by the existing *.no-panic.org wildcard cert. No cert changes needed. - ROUTER_DOMAIN env var: ocsession.<domain> - Wildcard IngressRoute: *.ocsession.<domain> - Wildcard DNS CNAME: *.ocsession.<domain> - Removes CertificatePack resource (not needed, not available on free plan)
…L activation Cloudflare Universal SSL requires a proxied DNS record for the parent hostname (ocsession.no-panic.org) before it issues certs covering *.ocsession.no-panic.org. Without it, the edge never activates for this subdomain tree and all TLS handshakes fail. Added ocsession.<domain> CNAME alongside the wildcard *.ocsession.<domain>.
…S + tunnel routes Replace failed wildcard/second-level subdomain approach with an operator sidecar that provisions DNS records + tunnel routes on demand. Problem: Cloudflare Universal SSL covers *.no-panic.org (first level only). Session URLs were *.opencode-router.no-panic.org (second level) → TLS failure. Advanced Certificate Manager requires a paid plan. Solution: session URLs become <hash>-oc.no-panic.org (first level, covered by *.no-panic.org). The Cloudflare operator sidecar watches for session pods and creates/deletes the DNS record + tunnel ingress route per session. Changes: - ExposedWebApp: add extraContainers[] field (sidecar containers) - ExposedWebApp: RBAC now includes 'watch' verb on pods (needed by operator) - opencode-router index.ts: remove wildcard IngressRoute + CertificatePack, add CF credentials Secret, add operator sidecar via extraContainers, set ROUTER_DOMAIN=<base domain> + ROUTE_SUFFIX=-oc on router container - src/index.ts: pass cfOperatorImage, tunnelId, apiToken to router config - images/opencode-cloudflare-operator/: operator runtime (Node.js, k8s watch, Cloudflare API), Dockerfile, build.sh - packages/apps/opencode-cloudflare-operator/: Pulumi package stub (operator runs as sidecar — this package is not wired in directly) - Pulumi config: routerImage=0.0.1-homelab.3, cfOperatorImage=0.1.0-homelab.1 opencode fork (router-webapp branch): ROUTE_SUFFIX env var support added, session URL construction + hash extraction updated accordingly.
…re operator sidecar - Add ingressroute.ts to operator: creates/deletes two IngressRoute CRDs per session (<hash>-oc-app + <hash>-oc-signin), mirroring ExposedWebApp's OAuth2 pattern, so Traefik can route <hash>-oc.<domain> traffic to the router service - Fix operator race condition: createDnsRecord catches 'already exists' Cloudflare error from concurrent replica writes instead of surfacing as unhandled error - Add ingressRouteNamespace, oauth2ChainMiddleware, routerServiceName to operator config (env vars INGRESSROUTE_NAMESPACE, OAUTH2_CHAIN_MIDDLEWARE, ROUTER_SERVICE_NAME) - Extend RBAC Role with traefik.io/ingressroutes create/delete/get/list - Pass three new env vars to operator sidecar in Pulumi - Update README: correct architecture diagram (first-level subdomains, operator), fix resource table, add operator section and correct config table - Update plan file with post-deployment architecture decisions - Operator image: ghcr.io/mrsimpson/opencode-cloudflare-operator:0.1.0-homelab.3
1 task done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the homelab infrastructure side of the multi-user multi-session opencode web feature (mrsimpson/opencode#2, addressing upstream issue anomalyco/opencode#20067).
Intent
opencode serveis a single-user server — everyone on the same instance shares sessions, filesystem, and API keys. This PR deploys an MVP solution on the homelab Kubernetes cluster: a router that gives each user a dedicatedopencode servepod per (repo, branch) session, with full isolation and GitHub OAuth protection.Each session gets its own first-level subdomain (
<hash>-oc.<domain>), matching the Claude Code web model where separate coding sessions have separate URLs — and staying within Cloudflare Universal SSL wildcard coverage (*.<domain>) so no paid certificate product is needed.Since session hostnames are created at runtime, a new Cloudflare operator sidecar watches session pods and provisions/deprovisions DNS records and Traefik IngressRoutes on demand.
Architecture
The router pod runs two containers:
app— the opencode-router: creates pods/PVCs via the K8s API, proxies traffic to the correct session pod by hashcloudflare-operator— sidecar: watches session pods and manages per-session DNS records, Cloudflare Tunnel routes, and Traefik IngressRoutesOn pod ADDED: operator creates a Cloudflare CNAME (
<hash>-oc.<domain>→ tunnel), a tunnel ingress rule, and two TraefikIngressRouteCRDs (reusing the router's existing oauth2-chain middleware).On pod DELETED: all three are removed.
Major changes
New:
images/opencode-cloudflare-operator/— standalone Node.js operator imagesrc/index.ts— K8s pod watch loop with exponential backoff reconnect; health server on:8080src/cloudflare.ts— Cloudflare API: DNS record CRUD + Tunnel ingress rule CRUD; idempotent against concurrent replica writes (catches "already exists" on POST)src/ingressroute.ts— creates/deletes TraefikIngressRouteCRDs per session, mirroring the pattern ExposedWebApp uses for the main router domainsrc/config.ts— all config via env vars (CF_API_TOKEN,CF_ZONE_ID,CF_TUNNEL_ID,DOMAIN,ROUTE_SUFFIX,INGRESSROUTE_NAMESPACE,OAUTH2_CHAIN_MIDDLEWARE,ROUTER_SERVICE_NAME, …)build.sh— multi-arch Docker build + push to GHCRNew:
packages/apps/opencode-router/— Pulumi packageopencode-routerwithrestrictedPSStraefik.io/ingressroutescreate/delete for operatoropencode-api-keys(Anthropic key for session pods),opencode-router-cf-credentials(Cloudflare API token for operator)opencode-config-dirwith sharedopencode.jsonExposedWebAppfor the router deployment with operator asextraContainerssidecar; env varsROUTER_DOMAIN+ROUTE_SUFFIX=-ocset on the router containerimages/opencode-router/build.shExtended:
packages/core/components/src/ExposedWebApp.tsimagetype widened tostring | pulumi.Output<string>extraContainers?: object[]— appends sidecar containers alongsideappserviceAccountName?: string— sets pod ServiceAccountprobes?: { readinessProbe?, livenessProbe? }— container health probesnamespaceexposed as public propertyallowOverwrite: trueon the Cloudflare DNS recordNew:
images/opencode-router/build.sh— builds the opencode-router image from the fork monorepoUpdated:
src/index.ts— wirescreateOpencodeRouterinto the root stack withcfOperatorImage,tunnelId,apiTokenconfig valuesUpdated:
docs/howto/—expose-web-app.mdandadd-oauth-protection.mdupdated to reflect ExposedWebApp extensionsSide effects
opencode-routerServiceAccount hastraefik.io/ingressroutesRBAC — scoped to theopencode-routernamespace onlypulumi upoutput