Skip to content

feat(opencode-router): deploy per-session isolated OpenCode instances with Cloudflare operator - #31

Merged
mrsimpson merged 22 commits into
mainfrom
opencode-router-2
Apr 12, 2026
Merged

mrsimpson merged 22 commits into
mainfrom
opencode-router-2

Conversation

@mrsimpson

@mrsimpson mrsimpson commented Apr 12, 2026 •

Copy link
Copy Markdown
Member

Implements the homelab infrastructure side of the multi-user multi-session opencode web feature (mrsimpson/opencode#2, addressing upstream issue anomalyco/opencode#20067).

Intent

opencode serve is a single-user server — everyone on the same instance shares sessions, filesystem, and API keys. This PR deploys an MVP solution on the homelab Kubernetes cluster: a router that gives each user a dedicated opencode serve pod per (repo, branch) session, with full isolation and GitHub OAuth protection.

Each session gets its own first-level subdomain (<hash>-oc.<domain>), matching the Claude Code web model where separate coding sessions have separate URLs — and staying within Cloudflare Universal SSL wildcard coverage (*.<domain>) so no paid certificate product is needed.

Since session hostnames are created at runtime, a new Cloudflare operator sidecar watches session pods and provisions/deprovisions DNS records and Traefik IngressRoutes on demand.

Architecture

Browser
  │  HTTPS
  ▼
Cloudflare Tunnel → Traefik
  ├─ opencode-router.<domain>  ──[oauth2-chain]──▶  opencode-router (:3000)
  │                                                       │ setup UI: pick repo + branch
  │                                                       │ POST /api/sessions
  │                                                       ▼
  │                                              K8s: create Pod + PVC
  │                                              (one per email+repo+branch)
  │
  └─ <hash>-oc.<domain>  ──[oauth2-chain]──▶  opencode-router (:3000)
                                                    │ proxy by hash from Host header
                                                    ▼
                                           opencode-session-<hash> Pod (:4096)
                                           (opencode serve, isolated filesystem + credentials)

The router pod runs two containers:

  • app — the opencode-router: creates pods/PVCs via the K8s API, proxies traffic to the correct session pod by hash
  • cloudflare-operator — sidecar: watches session pods and manages per-session DNS records, Cloudflare Tunnel routes, and Traefik IngressRoutes

On pod ADDED: operator creates a Cloudflare CNAME (<hash>-oc.<domain> → tunnel), a tunnel ingress rule, and two Traefik IngressRoute CRDs (reusing the router's existing oauth2-chain middleware).

On pod DELETED: all three are removed.

Major changes

New: images/opencode-cloudflare-operator/ — standalone Node.js operator image

  • src/index.ts — K8s pod watch loop with exponential backoff reconnect; health server on :8080
  • src/cloudflare.ts — Cloudflare API: DNS record CRUD + Tunnel ingress rule CRUD; idempotent against concurrent replica writes (catches "already exists" on POST)
  • src/ingressroute.ts — creates/deletes Traefik IngressRoute CRDs per session, mirroring the pattern ExposedWebApp uses for the main router domain
  • src/config.ts — all config via env vars (CF_API_TOKEN, CF_ZONE_ID, CF_TUNNEL_ID, DOMAIN, ROUTE_SUFFIX, INGRESSROUTE_NAMESPACE, OAUTH2_CHAIN_MIDDLEWARE, ROUTER_SERVICE_NAME, …)
  • build.sh — multi-arch Docker build + push to GHCR

New: packages/apps/opencode-router/ — Pulumi package

  • Namespace opencode-router with restricted PSS
  • ServiceAccount + Role + RoleBinding: pod/PVC management for router + traefik.io/ingressroutes create/delete for operator
  • Secrets: opencode-api-keys (Anthropic key for session pods), opencode-router-cf-credentials (Cloudflare API token for operator)
  • ConfigMap: opencode-config-dir with shared opencode.json
  • ExternalSecret: GHCR pull secret (explicit — namespace is pre-created, so ExposedWebApp's auto-create path is skipped)
  • ExposedWebApp for the router deployment with operator as extraContainers sidecar; env vars ROUTER_DOMAIN + ROUTE_SUFFIX=-oc set on the router container
  • Build script: images/opencode-router/build.sh

Extended: packages/core/components/src/ExposedWebApp.ts

  • image type widened to string | pulumi.Output<string>
  • extraContainers?: object[] — appends sidecar containers alongside app
  • serviceAccountName?: string — sets pod ServiceAccount
  • probes?: { readinessProbe?, livenessProbe? } — container health probes
  • namespace exposed as public property
  • allowOverwrite: true on the Cloudflare DNS record

New: images/opencode-router/build.sh — builds the opencode-router image from the fork monorepo

Updated: src/index.ts — wires createOpencodeRouter into the root stack with cfOperatorImage, tunnelId, apiToken config values

Updated: docs/howto/ — expose-web-app.md and add-oauth-protection.md updated to reflect ExposedWebApp extensions

Side effects

  • The opencode-router ServiceAccount has traefik.io/ingressroutes RBAC — scoped to the opencode-router namespace only
  • With 2 router replicas, both operator sidecars watch the same pods; all provisioning operations are idempotent (DNS: GET-before-POST + catch "already exists"; IngressRoute: HTTP 409 no-op; Tunnel route: check before PUT)
  • Session pods, PVCs, DNS records, Tunnel routes, and IngressRoutes are not Pulumi-managed — created/deleted at runtime, not visible in pulumi up output
  • Cloudflare Tunnel config is mutated at runtime by the operator; the static Pulumi-managed catch-all rule must remain last (the operator preserves this invariant)

mrsimpson and others added 22 commits April 3, 2026 01:16
Build opencode-router from the opencode monorepo root with the Dockerfile
at packages/opencode-router/Dockerfile. Version is read from the monorepo
package.json; tag format is <version>-homelab.<revision>.
- packages/apps/opencode-router/package.json + tsconfig.json
- packages/apps/opencode-router/src/index.ts: 15 K8s resources
  (Namespace/PSS-restricted, SA, Role, RoleBinding, Secret, ConfigMap,
  ExternalSecret, Deployment/2-replica/non-root, Service, 3 Traefik
  middlewares, 2 IngressRoutes, Cloudflare DNS CNAME)
- src/index.ts: import createOpencodeRouter + call with opencodeConfig
- package.json: add @mrsimpson/homelab-app-opencode-router dependency

Cloudflare zoneId sourced from homelabConfig.cloudflare.zoneId;
tunnelCname from baseInfra.cloudflare.tunnelCname.
All homelab-4.3.x tasks closed:
- 4.3.1: pod-manager.ts securityContext fix (in opencode fork)
- 4.3.2: ADR-002 written (in opencode fork)
- 4.3.3: images/opencode-router/build.sh
- 4.3.4: packages/apps/opencode-router package.json + tsconfig.json
- 4.3.5: full Pulumi index.ts (15 K8s resources)
- 4.3.6: wired into root src/index.ts + package.json
- 4.3.7: Pulumi config set (gitignored Pulumi.dev.yaml)
- 4.3.8: tsc --noEmit + biome check both clean

Key decisions added: Cloudflare DNS wiring pattern, optional cfg.cloudflare
field, opencodeImage/routerImage Pulumi config keys, ADR-002 reference.
pulumi preview: +15 to create (opencode-router resources), 108 unchanged.
Zero errors, zero deletions, zero unexpected modifications. Clean preview.
…Store and key refs

- ClusterSecretStore name: cluster-secret-store -> pulumi-esc
- data keys: ghcr-credentials/auth -> github-username + github-token
  (matches the pattern used by all other working ExternalSecrets in the cluster)
- Template updated to build dockerconfigjson from username+token (same as opencode ns)
- ExternalSecret fix: wrong store name + key refs corrected
- GHCR PAT expiry: diagnosis and resolution documented
- Deployment verified: 2/2 pods Running, all 15 resources live
- Public URL https://opencode-router.no-panic.org/ returns 401 (correct)
…t env + plan updates

- Add IMAGE_PULL_SECRET_NAME=ghcr-pull-secret to router Deployment env so user pods
  get imagePullSecrets set (needed to pull ghcr.io/mrsimpson/opencode from GHCR)
- Update dev plan with local dev decisions: DEV_EMAIL, Vite proxy, kubeconfig, port-forward
…_DOMAIN env

- Add wildcard IngressRoute for *.opencode-router.<domain> → router service
  (HostRegexp matching 12-char hex session hashes)
- Add wildcard Cloudflare DNS CNAME for *.opencode-router.<domain>
- Pass ROUTER_DOMAIN env var to router Deployment
- Fix health probes: /api/status → /api/sessions (status endpoint removed)
Key decisions logged:
- *.localhost:3002 subdomain routing for local dev (no /etc/hosts)
- pnpm dev now runs TS directly via bun --watch (no rebuild needed)
- NODE_TLS_REJECT_UNAUTHORIZED=0 for self-signed k3s cert in dev
- SA token expiry: use TOKEN_DURATION=168h for 7-day tokens

WIP documented:
- Workspace directory not properly mounted in session pods
- Branch detection should auto-detect from repo, not require manual entry
Replace -h /root with -h /home/opencode in adduser so the opencode user (UID
1000) has a conventional home directory. /root as a non-root user's home is
confusing and unconventional. Aligns with pod-manager volume mounts which now
mount the PVC at /home/opencode.
Refactor opencode-router to use ExposedWebApp instead of manually creating
~15 K8s resources. ExposedWebApp now handles Deployment, Service, OAuth2-Proxy
auth (middlewares + IngressRoutes), main DNS CNAME, and GHCR pull secret.

ExposedWebApp gains two general-purpose fields: `serviceAccountName` (for
setting a ServiceAccount on the pod spec) and `probes` (readiness/liveness).
The `namespace` property is now publicly exposed for dependency wiring.

The opencode-router retains app-specific resources: Namespace, RBAC
(ServiceAccount/Role/RoleBinding), Secret, ConfigMap, wildcard IngressRoute
for session subdomains, and wildcard DNS record.

https://claude.ai/code/session_017UpZCGUzQchsP77wsRPNmx
- ExposedWebApp.ts JSDoc: document serviceAccountName, probes, and
  the OAuth2-Proxy middleware naming contract (<name>-oauth2-chain)
- docs/howto/expose-web-app.md: rewrite for current HomelabContext +
  ExposedWebApp API (was describing old ingress-nginx pattern)
- docs/howto/add-oauth-protection.md: rewrite for centralized
  oauth2-proxy with AuthType.OAUTH2_PROXY (was describing defunct
  per-app OAuth sidecar pattern)
- packages/apps/opencode-router/README.md: new, covers intent,
  architecture, custom image dependency, and config variables
- .vibe/development-plan: research and key decisions for the
  ExposedWebApp integration refactor

https://claude.ai/code/session_017UpZCGUzQchsP77wsRPNmx
…ix type casts

Two regressions from the ExposedWebApp refactor:

1. ExternalSecret for GHCR pull secret was silently dropped.
   ExposedWebApp auto-creates it only when isCreatingNamespace=true,
   but we pre-create the namespace and pass it in — so auto-create
   never fires. Explicit ExternalSecret resource restored in
   createOpencodeRouter(). Deployment now dependsOn pullSecret.

2. 'as unknown as string' type casts in image/env fields removed.
   ExposedWebApp.image widened to 'string | pulumi.Output<string>',
   and env values already accept Output<string>, so no casts needed.

Note: main DNS record (opencode-router.<domain>) is created
automatically by ExposedWebApp via HomelabContext's injected
cloudflare config — not a regression, just clarified in comments.

Also: 'void roleBinding/apiKeysSecret/configMap' statements replaced
with proper Pulumi dependsOn on the ExposedWebApp call for RBAC
and pullSecret resources.
When a DNS record for the same name/type already exists in Cloudflare
(e.g. created by a previous Pulumi run under a different URN, or
migrated from manually-created records), the Cloudflare API returns
an error without allowOverwrite=true.

This happened when opencode-router's DNS record moved from a top-level
resource (old manual code) to an ExposedWebApp child resource — the
old Pulumi state entry was deleted but the Cloudflare record persisted.
…code-router domain

Universal SSL only covers *.no-panic.org (one level deep). Session
subdomains (*.opencode-router.no-panic.org) are two levels deep and
get a TLS handshake failure without an explicit cert.

Added CertificatePack (advanced, Let's Encrypt, DNS-01/txt validation)
covering both opencode-router.no-panic.org and
*.opencode-router.no-panic.org, scoped to the opencode-router app
package — not base infra, since this is app-specific.
*.opencode-router.<domain> is a second-level wildcard, not covered by
Cloudflare's Universal SSL (free plan only issues *.no-panic.org).
Advanced Certificate Manager costs $10/month.

Session URLs are now <hash>.ocsession.<domain> — a first-level subdomain
covered by the existing *.no-panic.org wildcard cert. No cert changes
needed.

- ROUTER_DOMAIN env var: ocsession.<domain>
- Wildcard IngressRoute: *.ocsession.<domain>
- Wildcard DNS CNAME: *.ocsession.<domain>
- Removes CertificatePack resource (not needed, not available on free plan)
…L activation

Cloudflare Universal SSL requires a proxied DNS record for the parent
hostname (ocsession.no-panic.org) before it issues certs covering
*.ocsession.no-panic.org. Without it, the edge never activates for
this subdomain tree and all TLS handshakes fail.

Added ocsession.<domain> CNAME alongside the wildcard *.ocsession.<domain>.
…S + tunnel routes

Replace failed wildcard/second-level subdomain approach with an operator
sidecar that provisions DNS records + tunnel routes on demand.

Problem: Cloudflare Universal SSL covers *.no-panic.org (first level only).
Session URLs were *.opencode-router.no-panic.org (second level) → TLS failure.
Advanced Certificate Manager requires a paid plan.

Solution: session URLs become <hash>-oc.no-panic.org (first level, covered
by *.no-panic.org). The Cloudflare operator sidecar watches for session pods
and creates/deletes the DNS record + tunnel ingress route per session.

Changes:
- ExposedWebApp: add extraContainers[] field (sidecar containers)
- ExposedWebApp: RBAC now includes 'watch' verb on pods (needed by operator)
- opencode-router index.ts: remove wildcard IngressRoute + CertificatePack,
  add CF credentials Secret, add operator sidecar via extraContainers,
  set ROUTER_DOMAIN=<base domain> + ROUTE_SUFFIX=-oc on router container
- src/index.ts: pass cfOperatorImage, tunnelId, apiToken to router config
- images/opencode-cloudflare-operator/: operator runtime (Node.js, k8s watch,
  Cloudflare API), Dockerfile, build.sh
- packages/apps/opencode-cloudflare-operator/: Pulumi package stub
  (operator runs as sidecar — this package is not wired in directly)
- Pulumi config: routerImage=0.0.1-homelab.3, cfOperatorImage=0.1.0-homelab.1

opencode fork (router-webapp branch): ROUTE_SUFFIX env var support added,
  session URL construction + hash extraction updated accordingly.
…re operator sidecar

- Add ingressroute.ts to operator: creates/deletes two IngressRoute CRDs per
  session (<hash>-oc-app + <hash>-oc-signin), mirroring ExposedWebApp's OAuth2
  pattern, so Traefik can route <hash>-oc.<domain> traffic to the router service
- Fix operator race condition: createDnsRecord catches 'already exists' Cloudflare
  error from concurrent replica writes instead of surfacing as unhandled error
- Add ingressRouteNamespace, oauth2ChainMiddleware, routerServiceName to operator
  config (env vars INGRESSROUTE_NAMESPACE, OAUTH2_CHAIN_MIDDLEWARE, ROUTER_SERVICE_NAME)
- Extend RBAC Role with traefik.io/ingressroutes create/delete/get/list
- Pass three new env vars to operator sidecar in Pulumi
- Update README: correct architecture diagram (first-level subdomains, operator),
  fix resource table, add operator section and correct config table
- Update plan file with post-deployment architecture decisions
- Operator image: ghcr.io/mrsimpson/opencode-cloudflare-operator:0.1.0-homelab.3
@mrsimpson mrsimpson changed the title feat(opencode-router): per-session isolated OpenCode instances via Cloudflare operator sidecar feat(opencode-router): deploy per-session isolated OpenCode instances with Cloudflare operator Apr 12, 2026
@mrsimpson
mrsimpson merged commit e7bb721 into main Apr 12, 2026
@mrsimpson
mrsimpson deleted the opencode-router-2 branch April 12, 2026 08:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants