Skip to content

Commit 836d2ec

Browse files
committed
Add design proposal: self-hosted in-cluster registry for air-gapped Cozystack
Migrated from discussion cozystack/cozystack#3029 to the design-proposal process for review. Signed-off-by: Gaál György <[email protected]>
1 parent fbfc6ba commit 836d2ec

1 file changed

Lines changed: 176 additions & 0 deletions

File tree

  • design-proposals/airgap-in-cluster-registry
Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# Self-hosted in-cluster registry for air-gapped Cozystack
2+
3+
- **Title:** `Self-hosted in-cluster registry for air-gapped Cozystack (bundle bootstrap → in-cluster source of truth)`
4+
- **Author(s):** `@gecube`
5+
- **Date:** `2026-06-24`
6+
- **Status:** Draft
7+
8+
> Migrated from discussion [cozystack/cozystack#3029](https://github.com/cozystack/cozystack/discussions/3029) to the design-proposal process for review.
9+
10+
## Overview
11+
12+
Cozystack's current air-gap story is "bring your own registry": operators must stand up and populate a mirror themselves, and mirror creation is explicitly out of scope. This proposal evolves that into a bundled, self-hosted flow. We ship an **offline bundle** (~4–10 GB per flavor/version) containing OCI images and Talos OS assets, bootstrap from a **throwaway local registry** on the admin machine, and then stand up a **self-hosted in-cluster registry** (`distribution` or `zot`) that becomes the persistent source of truth. Image pulls are redirected through the existing Talos/containerd registry-mirror mechanism, so after a single cutover the admin machine becomes dispensable.
13+
14+
The key insight: a Cozystack cluster already runs the storage it needs (LINSTOR/Piraeus, SeaweedFS), so the in-cluster registry needs no new external dependency.
15+
16+
## Scope and related proposals
17+
18+
- Migrated together with the **[Coroot eBPF observability](../coroot-ebpf-observability/)** proposal (from discussion [#3028](https://github.com/cozystack/cozystack/discussions/3028)). The two are independent in substance; they are submitted as a pair.
19+
- **Tenant-owned registries are out of scope.** Cozystack already ships Harbor (system component and tenant-deployable app) with RBAC, projects, and robot accounts for multi-tenant push/pull. The in-cluster `registry.cozy-system` proposed here is read-mostly and digest-pinned, not a tenant build registry.
20+
21+
## Context
22+
23+
Air-gap delivery spans two artifact classes across two cluster tiers:
24+
25+
**Artifact classes**
26+
- OCI container images — pulled by containerd at runtime.
27+
- Talos OS assets — kernel, initramfs, ISO, metal images (bare-metal) and the nocloud disk image (VMs).
28+
29+
**Cluster tiers**
30+
- Management (root) cluster on bare-metal Talos nodes.
31+
- Tenant (leaf) clusters as Kamaji control planes and KubeVirt VMs running Talos.
32+
33+
Today's docs only cover OCI images via mirrors, leaving Talos asset delivery fragmented and the bootstrap chicken-and-egg unaddressed.
34+
35+
### The problem
36+
37+
An operator with no internet egress cannot install or upgrade Cozystack without manually assembling a mirror, and even then has no guidance for delivering Talos boot media or for day-2 upgrades. The result is bespoke, error-prone per-site tooling.
38+
39+
## Goals
40+
41+
- A single offline bundle per flavor/version carries every OCI image and Talos asset required for a full install.
42+
- Bootstrap works from a throwaway registry on the admin machine, with no internet egress.
43+
- A self-hosted in-cluster registry becomes the persistent source of truth after bootstrap; the admin machine is no longer required.
44+
- Image redirection uses the Talos/containerd mirror mechanism, affecting system components, node images, and tenants alike.
45+
- Day-2 upgrades have a defined, preflight-guarded sequence.
46+
47+
### Non-goals
48+
49+
- Tenant-facing build/push registry (Harbor already covers this).
50+
- Cryptographic image signing/verification (deferred to Phase 2).
51+
- `paas-hosted` air-gap support (deferred to Phase 3 — no node-level containerd control).
52+
53+
## Design
54+
55+
**Rule 1 — Standardized tagging.** Cozystack publishes images consistently to `ghcr.io/cozystack/…` plus known upstream registries, so the image set is enumerable for a deterministic bundle.
56+
57+
**Rule 2 — Air-gap requires a private registry.** Accepted as a given; this proposal specifies whose and where.
58+
59+
**Rule 3 — Bootstrap with a local registry.** The admin loads the bundle into a temporary Docker registry:
60+
61+
```bash
62+
docker run -d -p 5000:5000 --name cozy-bootstrap-registry \
63+
-v /srv/cozy-registry:/var/lib/registry registry:2
64+
cozystack images push --bundle cozystack-airgap-paas-full-v1.5.0.tar \
65+
--to http://ADMIN_IP:5000
66+
```
67+
68+
**Rule 4 — In-cluster registry post-bootstrap.** Once core components stabilize, the platform deploys a lightweight registry (`distribution` or `zot`) backed by SeaweedFS S3 or a LINSTOR PVC. After it is populated, the mirror endpoints are rewritten from the admin IP to the internal registry domain (e.g. `registry.cozy-system`), and the admin machine becomes dispensable.
69+
70+
**Rule 5 — Redirection via containerd mirrors.** We favor the existing Talos/containerd `machine.registries.mirrors` mechanism over admission-based approaches (Kyverno or native CEL). Mirrors operate below Kubernetes and affect all image pulls — system components, node images, and tenants — making them more reliable for air-gap than Pod-spec-level mutation.
71+
72+
```yaml
73+
machine:
74+
registries:
75+
mirrors:
76+
ghcr.io:
77+
endpoints:
78+
- https://registry.cozy-system # internal, preferred
79+
- https://ghcr.io # fallback
80+
```
81+
82+
### Bootstrap → cutover
83+
84+
```mermaid
85+
flowchart TD
86+
A[Admin machine: load offline bundle] --> B[Throwaway local registry :5000]
87+
B --> C[Talos nodes PXE boot via bootbox/matchbox]
88+
C --> D[Core components deploy, pulling from admin registry]
89+
D --> E[Deploy in-cluster registry distribution/zot]
90+
E --> F[Populate in-cluster registry from bundle]
91+
F --> G{Cutover: rewrite mirror endpoints<br/>admin IP → registry.cozy-system}
92+
G --> H[Admin machine dispensable]
93+
```
94+
95+
Management cluster nodes and tenant VM workers converge on `registry.cozy-system` for OCI pulls. Talos boot media (management only) remains an irreducible bootstrap responsibility served via bootbox/matchbox. Tenant VMs import the Talos nocloud disk image via KubeVirt CDI from the in-cluster source.
96+
97+
### Compatibility across distributions
98+
99+
| Bundle | Owns node OS? | Mirror mechanism | Air-gap story |
100+
|---------------|-----------------|-----------------------------------|---------------|
101+
| `paas-full` | ✅ Talos | ✅ `machine.registries.mirrors` | **Full** |
102+
| `distro-full` | ✅ Talos | ✅ | **Full** |
103+
| `paas-hosted` | ❌ External K8s | ❌ Cannot set host containerd | **Partial** |
104+
105+
Phase 1 targets `paas-full` and `distro-full`. `paas-hosted` lacks node-level containerd control and is deferred.
106+
107+
## User-facing changes
108+
109+
- A `cozystack images` CLI surface (or equivalent) for building, pushing, and verifying bundles.
110+
- Per-release air-gap bundles published as release assets (proposed — see open questions).
111+
- Documented air-gap install and upgrade runbooks.
112+
113+
## Upgrade and rollback compatibility
114+
115+
Day-2 upgrades require pre-loading the new version's artifacts into the live in-cluster registry **before** bumping the platform version:
116+
117+
1. Generate the new-version bundle on a connected machine.
118+
2. Transfer via removable media.
119+
3. Push OCI images and Talos assets to the in-cluster registry.
120+
4. Run a preflight verification confirming all required digests are present.
121+
5. Only then trigger the platform version bump and Flux reconciliation.
122+
123+
Critical sequencing: bumping before pre-loading triggers immediate `ImagePullBackOff`. The preflight check guards against incomplete transfers. Old images are garbage-collected post-verification as a deliberate step, never automatically. Rollback re-seeds from the retained offline bundle.
124+
125+
## Security
126+
127+
Phase 1 is explicitly a PoC posture:
128+
129+
**Provided**
130+
- Transport-level tamper protection via TLS with a baked-in root CA distributed to all nodes.
131+
- Containerd validation of registry certificates.
132+
133+
**Not provided (deferred to Phase 2)**
134+
- Cryptographic image signing/verification (cosign/sigstore).
135+
- Provenance/SBOM attestation.
136+
- At-rest encryption (a storage-layer concern).
137+
138+
The bootstrap registry may use `tls.insecureSkipVerify: true` temporarily; the persistent in-cluster registry must use proper TLS with CA trust.
139+
140+
## Failure and edge cases
141+
142+
- **Bump before pre-load** → immediate `ImagePullBackOff`; preflight check is the guard.
143+
- **Total registry outage** → blocks new pulls only; already-running pods keep their cached images. Mitigated by multi-replica HA backing storage.
144+
- **Incomplete bundle transfer** → caught by preflight digest verification before any version bump.
145+
- **DNS for `registry.cozy-system` unresolvable at containerd level** → node cannot pull; the registry domain must resolve on every node, not just in-cluster.
146+
- **Admission policies (Kyverno/CEL) cannot rewrite system/node images** → only see Pod specs, so they are unsuitable for air-gap redirection.
147+
148+
## Testing
149+
150+
- Unit: bundle manifest enumeration and digest verification.
151+
- Integration: throwaway-registry bootstrap on a `paas-full` cluster with egress firewalled off.
152+
- e2e: full offline install, then a day-2 upgrade via pre-load → preflight → bump.
153+
- Manual: cutover (mirror endpoint rewrite) with no half-migrated nodes.
154+
155+
## Rollout
156+
157+
- **Phase 0** — Community discussion and direction validation (this proposal).
158+
- **Phase 1 (PoC)** — Bundle tooling, offline bootbox seeding, throwaway admin registry, in-cluster `distribution`/`zot`, tenant Talos nocloud + images from in-cluster source, TLS with baked-in CA. Targets `paas-full` and `distro-full`.
159+
- **Phase 2** — Native CEL `ValidatingAdmissionPolicy` for cheap provenance constraints; cosign/sigstore integration; at-rest encryption.
160+
- **Phase 3** — `paas-hosted` support; tenant cluster hardening; upgrade/garbage-collection tooling.
161+
162+
## Open questions
163+
164+
1. In-cluster registry implementation: `distribution` vs. `zot` vs. reusing Harbor?
165+
2. Addressing strategy: Service/VIP vs. DNS name?
166+
3. Bundle granularity: per-flavor or single superset?
167+
4. Acceptable to defer `paas-hosted` support?
168+
5. Tooling location: `cozystack` CLI, separate repo, or installer?
169+
6. Should the project publish per-release air-gap bundles as release assets?
170+
7. Worth implementing delta bundles for smaller upgrade transfers?
171+
172+
## Alternatives considered
173+
174+
- **Admission-based redirection (Kyverno / native CEL).** Rejected as the primary mechanism: admission webhooks only see Pod specs and cannot rewrite system-component or node-image pulls, so they cannot cover the air-gap surface. CEL constraints may still serve a Phase 2 provenance role.
175+
- **Bring-your-own registry only (status quo).** Rejected: leaves Talos asset delivery and the bootstrap chicken-and-egg unsolved, and pushes bespoke tooling onto every operator.
176+
- **Reusing Harbor as the platform mirror.** Possible but heavier than needed for a read-mostly, digest-pinned source of truth; kept as an open question rather than the default.

0 commit comments

Comments
 (0)