|
| 1 | +# Implementation Plan v3 — Lightweight container-cluster e2e lane (cozystack#3238) |
| 2 | + |
| 3 | +> **Framework decision: chainsaw-first.** This lane is built as a follow-up **on top of |
| 4 | +> PR #2826** (`feat/e2e-chainsaw-example`, OPEN, mergeable), which migrates the whole app |
| 5 | +> suite BATS→Chainsaw. Revised after plannotator feedback (6 notes). |
| 6 | +> |
| 7 | +> Foundation (from the k8s-distribution doc, feedback #1): platform **`default` variant + |
| 8 | +> `cozypkg add`** composition — not "enable the paas bundle". Verified: `templates/sources.yaml` |
| 9 | +> registers **every** PackageSource unconditionally, so `cozypkg add cozystack.<anything>` |
| 10 | +> (incl. `bucket-application`) works standalone with no bundle enabled. |
| 11 | +
|
| 12 | +--- |
| 13 | + |
| 14 | +## 0. Where chainsaw-first lands the work (feedback #6) |
| 15 | + |
| 16 | +#2826 is a **full** migration, not a pilot. Confirmed structure on the branch: |
| 17 | + |
| 18 | +- App tests: `hack/e2e-chainsaw/<suite>/chainsaw-test.yaml` (+ manifests) for **all** 22 apps. |
| 19 | + Shared shell in `hack/e2e-chainsaw/_lib/`. Global `hack/e2e-chainsaw/.chainsaw.yaml` |
| 20 | + (namespace `tenant-test`, `parallel: 1`, crust-gather `catch` on failure, JUnit report). |
| 21 | +- **Cluster bootstrap + install stay BATS** even post-migration: |
| 22 | + `hack/e2e-prepare-cluster.bats` (3 QEMU Talos VMs), `hack/e2e-install-cozystack.bats` |
| 23 | + (helm install + platform Package + `e2e-post-install-prep.sh`), `hack/e2e-test-openapi.bats`. |
| 24 | + **Only the app suite is chainsaw.** |
| 25 | +- Run target: `make -C packages/core/testing test-chainsaw CHAINSAW_SUITES="…"` → |
| 26 | + `cd hack/e2e-chainsaw && chainsaw test ${CHAINSAW_SUITES}`. |
| 27 | +- **Selection already chainsaw-aware:** `hack/select-e2e.sh` was rewritten to map changed files → |
| 28 | + suite dir names under `hack/e2e-chainsaw/`; the CI "Select E2E tests" step feeds them into |
| 29 | + `CHAINSAW_SUITES` (empty ⇒ whole suite). |
| 30 | + |
| 31 | +**Implication for #3238:** the lane's *differentiators* (container cluster, lite package set, |
| 32 | +local-path SC, cilium-only CNI) all live in the **provisioning + install phase, which is still |
| 33 | +BATS**. The app-test phase reuses #2826's chainsaw suites **unchanged**, scoped to the 17 |
| 34 | +survivors via `CHAINSAW_SUITES`. So chainsaw-first does **not** change §1/§b/§c below; it changes |
| 35 | +only §d (run/selection) and the sequencing (§e) — and it makes feedback #5 free. |
| 36 | + |
| 37 | +--- |
| 38 | + |
| 39 | +## 1. Package composition — `default` variant + a lite preset (feedback #1, #2) |
| 40 | + |
| 41 | +The k8s-distribution doc's path: operator `variant=generic`; platform **`default`** variant |
| 42 | +(*"registers PackageSources for all packages, installs nothing"* — verified); then `cozypkg add` |
| 43 | +per package. `cozypkg add` is a **different mechanism** from `bundles.enabledPackages` (which only |
| 44 | +toggles `.optional` packages): it creates a standalone `Package` CR against any registered |
| 45 | +PackageSource, so it reaches postgres-application, kafka-application, **and bucket-application** |
| 46 | +with no paas/iaas bundle. `dependsOn` is an ordering gate, not an auto-installer → the lane must |
| 47 | +add the **full closure**. |
| 48 | + |
| 49 | +Per feedback #2, codify the closure as a **lite preset** rather than dozens of ad-hoc `cozypkg add`: |
| 50 | +- **A1 (prototype):** a checked-in list (e.g. `hack/e2e-lite-packages.txt`) + an install-script |
| 51 | + loop of `cozypkg add`. No Go/bundle change. |
| 52 | +- **A2 (durable, feedback #2's "bundle/preset for local tests"):** a named `lite`/`e2e` platform |
| 53 | + variant (`values-lite.yaml` + entry in `cmd/cozystack-operator/main.go:678`) or a dedicated |
| 54 | + bundle block. Reproducible for local dev; touches the Go binary + `helm-unittest`. Do after A1. |
| 55 | + |
| 56 | +**Closure (17 survivors):** |
| 57 | +- *Engine/control:* `cozystack-engine`, `cozystack-basics`, `tenant-application` (creates the |
| 58 | + `tenant-test` tenant the suites install into — matches `.chainsaw.yaml`'s namespace), |
| 59 | + `cert-manager`, `gateway-api-crds`, `reloader`, `snapshot-controller`, |
| 60 | + `cozystack-scheduler` (**variant `default`**, not `linstor` — §b), `securitygroup-controller`. |
| 61 | +- *Networking:* `networking` **`cilium-generic`** (or `cilium` on Talos-in-Docker, or `noop` if the |
| 62 | + cluster ships Cilium already — §c). |
| 63 | +- *Storage (new pkg):* the local-path/host-path package (§b) → default SC. |
| 64 | +- *Object storage:* `seaweedfs-application`, `objectstorage-controller` (bucket/harbor COSI). |
| 65 | +- *LB:* `metallb` or Cilium LB-IPAM (§c, R-C1). |
| 66 | +- *Operators:* `etcd-operator`, `postgres-operator`, `mariadb-operator`, `kafka-operator`, |
| 67 | + `clickhouse-operator`, `foundationdb-operator`, `redis-operator`, `mongodb-operator`. |
| 68 | +- *App defs:* `etcd-application`, `gateway-application`, `external-dns`(+`-application`), |
| 69 | + `kuberture`, `bucket-application`, `harbor-application`, `postgres-application`, |
| 70 | + `clickhouse-application`, `foundationdb-application`, `kafka-application`, `mariadb-application`, |
| 71 | + `mongodb-application`, `openbao-application`, `qdrant-application`, `redis-application`. |
| 72 | +- `serviceexposure` needs nothing extra (controller ships in `cozystack-engine`). |
| 73 | + |
| 74 | +(For `etcd.bats`/`etcd` chainsaw suite's `workloadmonitor`, add `victoria-metrics-operator` + |
| 75 | +`monitoring-agents` — the operator+CRD, not the whole tenant monitoring stack; see R-D3.) |
| 76 | + |
| 77 | +--- |
| 78 | + |
| 79 | +## (b) Default StorageClass swap — new package, non-Rancher (feedback #2, #3) |
| 80 | + |
| 81 | +**Finding (verified):** the linstor chart renders **no `StorageClass`**. `local` (default) + |
| 82 | +`replicated` are created imperatively by `hack/e2e-post-install-prep.sh:69-102` after ZFS pools. |
| 83 | +The 15 SURVIVES suites use `storageClass: ""` → they need only **one default SC**, name irrelevant |
| 84 | +(`local`/`replicated`-by-name are used only by the 5 NEEDS-REAL suites). |
| 85 | + |
| 86 | +**Swap:** ship a small `packages/system/<local-storage>` package (declarative, Flux-managed — |
| 87 | +not a `kubectl apply` in a script, per feedback #2) wired only into the lite preset; drop |
| 88 | +`linstor`/`linstor-scheduler`/`linstor-gui`. |
| 89 | + |
| 90 | +**Provisioner — DECIDED: OpenEBS LocalPV-Hostpath** (non-Rancher, feedback #3). |
| 91 | + |
| 92 | +Implementation-time correction to the earlier ranking: `csi-driver-host-path` (the previous #1) ships no upstream Helm chart — only raw `deploy/kubernetes-*` manifests — so it does not fit cozystack's vendor-an-upstream-chart package pattern (`nfs-driver`'s Makefile does `helm pull csi-driver-nfs --untar`). Its only advantage over the alternatives, CSI snapshots, is not needed by the 17 survivors (they only need PVCs to bind; none snapshot in e2e). So it is the most work for no benefit here. |
| 93 | + |
| 94 | +OpenEBS LocalPV-Hostpath is CNCF, non-Rancher, and publishes an official Helm chart (`localpv-provisioner` at `https://openebs.github.io/dynamic-localpv-provisioner`) → vendors cleanly via the `nfs-driver` pattern. Dynamic provisioning + `WaitForFirstConsumer`. The chart's own `hostpathClass.isDefaultClass: true` mints the default StorageClass (named `local` for parity with the LINSTOR default SC name), so no separate SC template is needed. Images stay on their upstream registry (like `nfs-driver`), so no image-mirroring step. |
| 95 | + |
| 96 | +Rejected: `csi-driver-host-path` (no chart, snapshot advantage unused); `local-static-provisioner` (no dynamic provisioning); `nfs-driver`+NFS server (heavier); Rancher local-path (excluded by feedback #3). |
| 97 | + |
| 98 | +**`cozystack-scheduler` coupling:** `system.yaml:28` uses variant `linstor`; the lite preset must |
| 99 | +use **`default`** (as `isp-hosted` does, `system.yaml:33`). No linstor `VolumeSnapshotClass` will |
| 100 | +exist — use the host-path CSI's snapshot class if a suite needs one. |
| 101 | + |
| 102 | +--- |
| 103 | + |
| 104 | +## (c) Cilium-only networking — 👍 (feedback #4) |
| 105 | + |
| 106 | +Unchanged and confirmed: cilium-only variants **already exist** in `sources/networking.yaml` — |
| 107 | +`cilium` (Talos) and `cilium-generic`. Base `system/cilium/values.yaml` is a full primary CNI |
| 108 | +(`ipam.mode: kubernetes`, `kubeProxyReplacement: true`, `gatewayAPI.enabled: true`); the kube-ovn |
| 109 | +coupling lives only in the `values-kubeovn.yaml` overlay, omitted by these variants. **No new CNI |
| 110 | +code** — pick `cilium` (Talos-in-Docker), `cilium-generic` (kind), or `noop` (only if a |
| 111 | +pre-installed Cilium exists; kindnet won't satisfy `gateway`/`securitygroup`). |
| 112 | + |
| 113 | +Lost with kube-ovn (none affect the 17): VPC subnets/overlays (iaas, off), `multus`, `cozy-proxy`. |
| 114 | +**Tenant isolation survives** — `packages/apps/tenant/templates/networkpolicy.yaml` is 100% |
| 115 | +`cilium.io/v2` CNP/CCNP, zero kube-ovn, so `tenant.spec.isolated: true` and `securitygroup` work. |
| 116 | + |
| 117 | +**R-C1 — LB IP:** `gateway` needs a `LoadBalancer` VIP (→ `Programmed`). MetalLB L2 or Cilium |
| 118 | +LB-IPAM pinned to a docker-bridge range. `serviceexposure` uses `externalIPs`, needs none. |
| 119 | + |
| 120 | +--- |
| 121 | + |
| 122 | +## (d) Workflow / lane — chainsaw run + chainsaw-aware selection (feedback #5, #6) |
| 123 | + |
| 124 | +Add a **parallel `e2e-container` job** to `pull-requests.yaml`, alongside the QEMU lane (which |
| 125 | +keeps the 5 NEEDS-REAL suites). It reuses #2826's chainsaw machinery end to end: |
| 126 | + |
| 127 | +1. **Provision (BATS, no nested virt).** New `hack/e2e-prepare-cluster-container.bats` (or script): |
| 128 | + **Talos-in-Docker** (`talosctl cluster create --provisioner docker --cni none`) preferred for |
| 129 | + Talos fidelity (KubePrism `localhost:7445` → `cilium` variant applies unchanged); **kind** as |
| 130 | + the lighter alt (→ `cilium-generic` + `k8sServiceHost`=kind API). No `_out/assets/nocloud-*` |
| 131 | + dependency → skips image-build gating. |
| 132 | +2. **Install (BATS) via the lite preset (§1).** Fork `e2e-install-cozystack.bats`: operator |
| 133 | + `variant=generic`, platform **`default`** variant, `cozypkg add` the §1 closure (or apply the |
| 134 | + lite preset). Replace `e2e-post-install-prep.sh` with a container variant (install the storage |
| 135 | + package + LB pool; **no LINSTOR**). Keep the cilium-leak-healer step (CNI-level flake). |
| 136 | + Still creates `tenant-test` so `.chainsaw.yaml`'s namespace holds. |
| 137 | +3. **Run the app suite (Chainsaw, unchanged).** |
| 138 | + `make -C packages/core/testing test-chainsaw CHAINSAW_SUITES="<selected 17-subset>"`. The |
| 139 | + chainsaw suites are reused **verbatim** from #2826 — no per-suite edits. |
| 140 | +4. **Selection (feedback #5 — satisfied natively).** Reuse #2826's chainsaw-aware |
| 141 | + `hack/select-e2e.sh` (already maps diffs → suite dirs). Add a lane filter that **intersects** |
| 142 | + its output with the **17-survivor allowlist**, then sets `CHAINSAW_SUITES`. Diff-scoping is |
| 143 | + preserved; the 5 NEEDS-REAL suites (`vminstance`, `kubernetes-latest/previous`, |
| 144 | + `kubernetes-oidc-system/customconfig`) are never dispatched to this lane. Codify the allowlist |
| 145 | + next to the selector (or derive the excluded set from a `needs: [virt|drbd]` label on those |
| 146 | + suites). |
| 147 | +5. **Report:** reuse the `chainsaw-report.xml` collect/upload steps. |
| 148 | + |
| 149 | +**Conventions.** Chainsaw suites already comply with `docs/agents/e2e-testing.md` (declarative |
| 150 | +asserts poll existence+state; `catch`/`finally` diagnostics; JUnit). The **new BATS provisioning/ |
| 151 | +install** scripts must still follow the repo rules (event-driven `until` before `kubectl wait`, no |
| 152 | +`EXIT`/`RETURN` traps, fail-fast on HR readiness) and `hack/cozytest.sh` limits (only `@test`; |
| 153 | +dash `/bin/sh`, no `pipefail`; `set -u`). Chainsaw v0.2.15 gotchas (filter-as-list conditions; |
| 154 | +"actual" is a projection; bare-ref `error` = must-not-exist) are #2826's concern, inherited as-is. |
| 155 | + |
| 156 | +--- |
| 157 | + |
| 158 | +## (e) Sequencing — build on #2826 (feedback #6) |
| 159 | + |
| 160 | +- **Hard dependency: #2826 must merge (or this lane rebases onto it) first.** It is OPEN and |
| 161 | + mergeable on `main` as of 2026-07-09. Do not build a BATS app lane — it would be thrown away. |
| 162 | +- **Framework-agnostic work can start now** and lands identically either way: §b storage package, |
| 163 | + §c CNI variant selection + LB wiring, §1 lite preset/closure. Prototype these against a |
| 164 | + `default`-variant install on a dev stand independent of #2826. |
| 165 | +- **Framework-coupled work waits on #2826:** §d.3/§d.4 (the `test-chainsaw CHAINSAW_SUITES` run + |
| 166 | + the chainsaw-aware `select-e2e.sh` lane filter) must target #2826's files, so land after it. |
| 167 | +- Watch for churn on #2826 (suite dir names, `select-e2e.sh` mapping, `.chainsaw.yaml`) — the |
| 168 | + lane's allowlist and selector filter key off those. |
| 169 | + |
| 170 | +--- |
| 171 | + |
| 172 | +## Open risks |
| 173 | +- **R-B1 (dissolved) — bucket.** Under `default`+`cozypkg`, `bucket-application`'s PackageSource is |
| 174 | + registered like any other → addable with **no iaas bundle**. Blocker only exists on the |
| 175 | + (rejected) "enable paas bundle" route. |
| 176 | +- **R-C1 — LB IP on a container network** (§c). Range must be routable on the kind/docker bridge or |
| 177 | + `gateway` hangs. |
| 178 | +- **R-C2 — preset drift.** Hand-maintained closure can rot. Derive it from the PackageSource |
| 179 | + `dependsOn` graph (same yq logic `select-e2e.sh` uses) or add a CI completeness check. |
| 180 | +- **R-D1 — Talos-in-Docker vs kind fidelity.** `cilium` (Talos) values hard-code KubePrism |
| 181 | + `localhost:7445`; kind → `cilium-generic` + kind API endpoint. Prototype both. |
| 182 | +- **R-D2 — seaweedfs/CNPG weight.** bucket/harbor need seaweedfs → CNPG `seaweedfs-db` + raft |
| 183 | + quorum (~5-6 min) on the default SC; validate host-path CSI satisfies its `WaitForFirstConsumer` |
| 184 | + PVCs before assuming bucket/harbor pass. |
| 185 | +- **R-D3 — monitoring weight.** The lite closure can skip the full VictoriaMetrics/Grafana stack; |
| 186 | + only `etcd`'s `workloadmonitor` needs `victoria-metrics-operator` + `monitoring-agents`. |
| 187 | +- **R-E1 — #2826 is not merged.** The whole chainsaw-first plan is gated on it (§e). If #2826 |
| 188 | + stalls, the framework-agnostic work (§1/§b/§c) still has standalone value. |
| 189 | +- **R-E2 — chainsaw `parallel: 1`.** The shared `.chainsaw.yaml` runs suites sequentially (bucket's |
| 190 | + fixed port-forward port blocks parallelism). The lightweight lane inherits this → its wall-clock |
| 191 | + win comes from a faster *cluster* (no QEMU/DRBD/kube-ovn), not from parallel suites, unless |
| 192 | + #2826 later makes ports dynamic. |
| 193 | + |
| 194 | +--- |
| 195 | + |
| 196 | +## Suggested delivery order (chainsaw-first) |
| 197 | +1. **Track/land #2826** — hard prerequisite for §d (feedback #6). |
| 198 | +2. **Storage package** (§b): `csi-driver-host-path` + default SC. Standalone, unit-testable now. |
| 199 | +3. **CNI + LB** (§c): select `cilium`/`cilium-generic`, wire MetalLB/Cilium-LBIPAM pool. |
| 200 | +4. **Lite preset** (§1, A1 first; derive closure from the dependsOn graph per R-C2). Prove on a |
| 201 | + dev stand with the `default` platform variant + `cozypkg add`. |
| 202 | +5. **Container provisioning + install** (§d.1/§d.2, BATS): Talos-in-Docker first. |
| 203 | +6. **Wire onto #2826:** lane filter in `select-e2e.sh` (17-allowlist intersection) + the |
| 204 | + `e2e-container` job running `test-chainsaw CHAINSAW_SUITES` (§d.3/§d.4, feedback #5), parallel |
| 205 | + to the QEMU lane. |
| 206 | +7. Optionally promote A1 → A2 (named `lite` variant/bundle) for local dev testing (feedback #2). |
0 commit comments