|
| 1 | +# Self-hosted in-cluster registry for air-gapped Cozystack |
| 2 | + |
| 3 | +- **Title:** `Self-hosted in-cluster registry for air-gapped Cozystack (bundle bootstrap → in-cluster source of truth)` |
| 4 | +- **Author(s):** `@gecube` |
| 5 | +- **Date:** `2026-06-24` |
| 6 | +- **Status:** Draft |
| 7 | + |
| 8 | +> Migrated from discussion [cozystack/cozystack#3029](https://github.com/cozystack/cozystack/discussions/3029) to the design-proposal process for review. |
| 9 | +
|
| 10 | +## Overview |
| 11 | + |
| 12 | +Cozystack's current air-gap story is "bring your own registry": operators must stand up and populate a mirror themselves, and mirror creation is explicitly out of scope. This proposal evolves that into a bundled, self-hosted flow. We ship an **offline bundle** (~4–10 GB per flavor/version) containing OCI images and Talos OS assets, bootstrap from a **throwaway local registry** on the admin machine, and then stand up a **self-hosted in-cluster registry** (`distribution` or `zot`) that becomes the persistent source of truth. Image pulls are redirected through the existing Talos/containerd registry-mirror mechanism, so after a single cutover the admin machine becomes dispensable. |
| 13 | + |
| 14 | +The key insight: a Cozystack cluster already runs the storage it needs (LINSTOR/Piraeus, SeaweedFS), so the in-cluster registry needs no new external dependency. |
| 15 | + |
| 16 | +## Scope and related proposals |
| 17 | + |
| 18 | +- Migrated together with the **[Coroot eBPF observability](../coroot-ebpf-observability/)** proposal (from discussion [#3028](https://github.com/cozystack/cozystack/discussions/3028)). The two are independent in substance; they are submitted as a pair. |
| 19 | +- **Tenant-owned registries are out of scope.** Cozystack already ships Harbor (system component and tenant-deployable app) with RBAC, projects, and robot accounts for multi-tenant push/pull. The in-cluster `registry.cozy-system` proposed here is read-mostly and digest-pinned, not a tenant build registry. |
| 20 | + |
| 21 | +## Context |
| 22 | + |
| 23 | +Air-gap delivery spans two artifact classes across two cluster tiers: |
| 24 | + |
| 25 | +**Artifact classes** |
| 26 | +- OCI container images — pulled by containerd at runtime. |
| 27 | +- Talos OS assets — kernel, initramfs, ISO, metal images (bare-metal) and the nocloud disk image (VMs). |
| 28 | + |
| 29 | +**Cluster tiers** |
| 30 | +- Management (root) cluster on bare-metal Talos nodes. |
| 31 | +- Tenant (leaf) clusters as Kamaji control planes and KubeVirt VMs running Talos. |
| 32 | + |
| 33 | +Today's docs only cover OCI images via mirrors, leaving Talos asset delivery fragmented and the bootstrap chicken-and-egg unaddressed. |
| 34 | + |
| 35 | +### The problem |
| 36 | + |
| 37 | +An operator with no internet egress cannot install or upgrade Cozystack without manually assembling a mirror, and even then has no guidance for delivering Talos boot media or for day-2 upgrades. The result is bespoke, error-prone per-site tooling. |
| 38 | + |
| 39 | +## Goals |
| 40 | + |
| 41 | +- A single offline bundle per flavor/version carries every OCI image and Talos asset required for a full install. |
| 42 | +- Bootstrap works from a throwaway registry on the admin machine, with no internet egress. |
| 43 | +- A self-hosted in-cluster registry becomes the persistent source of truth after bootstrap; the admin machine is no longer required. |
| 44 | +- Image redirection uses the Talos/containerd mirror mechanism, affecting system components, node images, and tenants alike. |
| 45 | +- Day-2 upgrades have a defined, preflight-guarded sequence. |
| 46 | + |
| 47 | +### Non-goals |
| 48 | + |
| 49 | +- Tenant-facing build/push registry (Harbor already covers this). |
| 50 | +- Cryptographic image signing/verification (deferred to Phase 2). |
| 51 | +- `paas-hosted` air-gap support (deferred to Phase 3 — no node-level containerd control). |
| 52 | + |
| 53 | +## Design |
| 54 | + |
| 55 | +**Rule 1 — Standardized tagging.** Cozystack publishes images consistently to `ghcr.io/cozystack/…` plus known upstream registries, so the image set is enumerable for a deterministic bundle. |
| 56 | + |
| 57 | +**Rule 2 — Air-gap requires a private registry.** Accepted as a given; this proposal specifies whose and where. |
| 58 | + |
| 59 | +**Rule 3 — Bootstrap with a local registry.** The admin loads the bundle into a temporary Docker registry: |
| 60 | + |
| 61 | +```bash |
| 62 | +docker run -d -p 5000:5000 --name cozy-bootstrap-registry \ |
| 63 | + -v /srv/cozy-registry:/var/lib/registry registry:2 |
| 64 | +cozystack images push --bundle cozystack-airgap-paas-full-v1.5.0.tar \ |
| 65 | + --to http://ADMIN_IP:5000 |
| 66 | +``` |
| 67 | + |
| 68 | +**Rule 4 — In-cluster registry post-bootstrap.** Once core components stabilize, the platform deploys a lightweight registry (`distribution` or `zot`) backed by SeaweedFS S3 or a LINSTOR PVC. After it is populated, the mirror endpoints are rewritten from the admin IP to the internal registry domain (e.g. `registry.cozy-system`), and the admin machine becomes dispensable. |
| 69 | + |
| 70 | +**Rule 5 — Redirection via containerd mirrors.** We favor the existing Talos/containerd `machine.registries.mirrors` mechanism over admission-based approaches (Kyverno or native CEL). Mirrors operate below Kubernetes and affect all image pulls — system components, node images, and tenants — making them more reliable for air-gap than Pod-spec-level mutation. |
| 71 | + |
| 72 | +```yaml |
| 73 | +machine: |
| 74 | + registries: |
| 75 | + mirrors: |
| 76 | + ghcr.io: |
| 77 | + endpoints: |
| 78 | + - https://registry.cozy-system # internal, preferred |
| 79 | + - https://ghcr.io # fallback |
| 80 | +``` |
| 81 | +
|
| 82 | +### Bootstrap → cutover |
| 83 | +
|
| 84 | +```mermaid |
| 85 | +flowchart TD |
| 86 | + A[Admin machine: load offline bundle] --> B[Throwaway local registry :5000] |
| 87 | + B --> C[Talos nodes PXE boot via bootbox/matchbox] |
| 88 | + C --> D[Core components deploy, pulling from admin registry] |
| 89 | + D --> E[Deploy in-cluster registry distribution/zot] |
| 90 | + E --> F[Populate in-cluster registry from bundle] |
| 91 | + F --> G{Cutover: rewrite mirror endpoints<br/>admin IP → registry.cozy-system} |
| 92 | + G --> H[Admin machine dispensable] |
| 93 | +``` |
| 94 | + |
| 95 | +Management cluster nodes and tenant VM workers converge on `registry.cozy-system` for OCI pulls. Talos boot media (management only) remains an irreducible bootstrap responsibility served via bootbox/matchbox. Tenant VMs import the Talos nocloud disk image via KubeVirt CDI from the in-cluster source. |
| 96 | + |
| 97 | +### Compatibility across distributions |
| 98 | + |
| 99 | +| Bundle | Owns node OS? | Mirror mechanism | Air-gap story | |
| 100 | +|---------------|-----------------|-----------------------------------|---------------| |
| 101 | +| `paas-full` | ✅ Talos | ✅ `machine.registries.mirrors` | **Full** | |
| 102 | +| `distro-full` | ✅ Talos | ✅ | **Full** | |
| 103 | +| `paas-hosted` | ❌ External K8s | ❌ Cannot set host containerd | **Partial** | |
| 104 | + |
| 105 | +Phase 1 targets `paas-full` and `distro-full`. `paas-hosted` lacks node-level containerd control and is deferred. |
| 106 | + |
| 107 | +## User-facing changes |
| 108 | + |
| 109 | +- A `cozystack images` CLI surface (or equivalent) for building, pushing, and verifying bundles. |
| 110 | +- Per-release air-gap bundles published as release assets (proposed — see open questions). |
| 111 | +- Documented air-gap install and upgrade runbooks. |
| 112 | + |
| 113 | +## Upgrade and rollback compatibility |
| 114 | + |
| 115 | +Day-2 upgrades require pre-loading the new version's artifacts into the live in-cluster registry **before** bumping the platform version: |
| 116 | + |
| 117 | +1. Generate the new-version bundle on a connected machine. |
| 118 | +2. Transfer via removable media. |
| 119 | +3. Push OCI images and Talos assets to the in-cluster registry. |
| 120 | +4. Run a preflight verification confirming all required digests are present. |
| 121 | +5. Only then trigger the platform version bump and Flux reconciliation. |
| 122 | + |
| 123 | +Critical sequencing: bumping before pre-loading triggers immediate `ImagePullBackOff`. The preflight check guards against incomplete transfers. Old images are garbage-collected post-verification as a deliberate step, never automatically. Rollback re-seeds from the retained offline bundle. |
| 124 | + |
| 125 | +## Security |
| 126 | + |
| 127 | +Phase 1 is explicitly a PoC posture: |
| 128 | + |
| 129 | +**Provided** |
| 130 | +- Transport-level tamper protection via TLS with a baked-in root CA distributed to all nodes. |
| 131 | +- Containerd validation of registry certificates. |
| 132 | + |
| 133 | +**Not provided (deferred to Phase 2)** |
| 134 | +- Cryptographic image signing/verification (cosign/sigstore). |
| 135 | +- Provenance/SBOM attestation. |
| 136 | +- At-rest encryption (a storage-layer concern). |
| 137 | + |
| 138 | +The bootstrap registry may use `tls.insecureSkipVerify: true` temporarily; the persistent in-cluster registry must use proper TLS with CA trust. |
| 139 | + |
| 140 | +## Failure and edge cases |
| 141 | + |
| 142 | +- **Bump before pre-load** → immediate `ImagePullBackOff`; preflight check is the guard. |
| 143 | +- **Total registry outage** → blocks new pulls only; already-running pods keep their cached images. Mitigated by multi-replica HA backing storage. |
| 144 | +- **Incomplete bundle transfer** → caught by preflight digest verification before any version bump. |
| 145 | +- **DNS for `registry.cozy-system` unresolvable at containerd level** → node cannot pull; the registry domain must resolve on every node, not just in-cluster. |
| 146 | +- **Admission policies (Kyverno/CEL) cannot rewrite system/node images** → only see Pod specs, so they are unsuitable for air-gap redirection. |
| 147 | + |
| 148 | +## Testing |
| 149 | + |
| 150 | +- Unit: bundle manifest enumeration and digest verification. |
| 151 | +- Integration: throwaway-registry bootstrap on a `paas-full` cluster with egress firewalled off. |
| 152 | +- e2e: full offline install, then a day-2 upgrade via pre-load → preflight → bump. |
| 153 | +- Manual: cutover (mirror endpoint rewrite) with no half-migrated nodes. |
| 154 | + |
| 155 | +## Rollout |
| 156 | + |
| 157 | +- **Phase 0** — Community discussion and direction validation (this proposal). |
| 158 | +- **Phase 1 (PoC)** — Bundle tooling, offline bootbox seeding, throwaway admin registry, in-cluster `distribution`/`zot`, tenant Talos nocloud + images from in-cluster source, TLS with baked-in CA. Targets `paas-full` and `distro-full`. |
| 159 | +- **Phase 2** — Native CEL `ValidatingAdmissionPolicy` for cheap provenance constraints; cosign/sigstore integration; at-rest encryption. |
| 160 | +- **Phase 3** — `paas-hosted` support; tenant cluster hardening; upgrade/garbage-collection tooling. |
| 161 | + |
| 162 | +## Open questions |
| 163 | + |
| 164 | +1. In-cluster registry implementation: `distribution` vs. `zot` vs. reusing Harbor? |
| 165 | +2. Addressing strategy: Service/VIP vs. DNS name? |
| 166 | +3. Bundle granularity: per-flavor or single superset? |
| 167 | +4. Acceptable to defer `paas-hosted` support? |
| 168 | +5. Tooling location: `cozystack` CLI, separate repo, or installer? |
| 169 | +6. Should the project publish per-release air-gap bundles as release assets? |
| 170 | +7. Worth implementing delta bundles for smaller upgrade transfers? |
| 171 | + |
| 172 | +## Alternatives considered |
| 173 | + |
| 174 | +- **Admission-based redirection (Kyverno / native CEL).** Rejected as the primary mechanism: admission webhooks only see Pod specs and cannot rewrite system-component or node-image pulls, so they cannot cover the air-gap surface. CEL constraints may still serve a Phase 2 provenance role. |
| 175 | +- **Bring-your-own registry only (status quo).** Rejected: leaves Talos asset delivery and the bootstrap chicken-and-egg unsolved, and pushes bespoke tooling onto every operator. |
| 176 | +- **Reusing Harbor as the platform mirror.** Possible but heavier than needed for a read-mostly, digest-pinned source of truth; kept as an open question rather than the default. |
0 commit comments