Skip to content

Commit 9f0e3ca

Browse files
docs(e2e): add lightweight container-cluster e2e lane proposal
Design for a fast, diff-scoped e2e lane on a container cluster (Talos-in-Docker / kind) that runs the 17 of 22 app suites which do not need nested KVM or DRBD-replicated storage (#3238). Chainsaw-first: builds on the BATS->Chainsaw migration (#2826); the lane's differentiators live in the BATS provisioning/install phase (cilium-only CNI, a local-path default StorageClass, and a default-variant + cozypkg lite package set), while the app suites are reused verbatim and scoped via CHAINSAW_SUITES. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent e89617d commit 9f0e3ca

1 file changed

Lines changed: 206 additions & 0 deletions

File tree

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# Implementation Plan v3 — Lightweight container-cluster e2e lane (cozystack#3238)
2+
3+
> **Framework decision: chainsaw-first.** This lane is built as a follow-up **on top of
4+
> PR #2826** (`feat/e2e-chainsaw-example`, OPEN, mergeable), which migrates the whole app
5+
> suite BATS→Chainsaw. Revised after plannotator feedback (6 notes).
6+
>
7+
> Foundation (from the k8s-distribution doc, feedback #1): platform **`default` variant +
8+
> `cozypkg add`** composition — not "enable the paas bundle". Verified: `templates/sources.yaml`
9+
> registers **every** PackageSource unconditionally, so `cozypkg add cozystack.<anything>`
10+
> (incl. `bucket-application`) works standalone with no bundle enabled.
11+
12+
---
13+
14+
## 0. Where chainsaw-first lands the work (feedback #6)
15+
16+
#2826 is a **full** migration, not a pilot. Confirmed structure on the branch:
17+
18+
- App tests: `hack/e2e-chainsaw/<suite>/chainsaw-test.yaml` (+ manifests) for **all** 22 apps.
19+
Shared shell in `hack/e2e-chainsaw/_lib/`. Global `hack/e2e-chainsaw/.chainsaw.yaml`
20+
(namespace `tenant-test`, `parallel: 1`, crust-gather `catch` on failure, JUnit report).
21+
- **Cluster bootstrap + install stay BATS** even post-migration:
22+
`hack/e2e-prepare-cluster.bats` (3 QEMU Talos VMs), `hack/e2e-install-cozystack.bats`
23+
(helm install + platform Package + `e2e-post-install-prep.sh`), `hack/e2e-test-openapi.bats`.
24+
**Only the app suite is chainsaw.**
25+
- Run target: `make -C packages/core/testing test-chainsaw CHAINSAW_SUITES="…"` →
26+
`cd hack/e2e-chainsaw && chainsaw test ${CHAINSAW_SUITES}`.
27+
- **Selection already chainsaw-aware:** `hack/select-e2e.sh` was rewritten to map changed files →
28+
suite dir names under `hack/e2e-chainsaw/`; the CI "Select E2E tests" step feeds them into
29+
`CHAINSAW_SUITES` (empty ⇒ whole suite).
30+
31+
**Implication for #3238:** the lane's *differentiators* (container cluster, lite package set,
32+
local-path SC, cilium-only CNI) all live in the **provisioning + install phase, which is still
33+
BATS**. The app-test phase reuses #2826's chainsaw suites **unchanged**, scoped to the 17
34+
survivors via `CHAINSAW_SUITES`. So chainsaw-first does **not** change §1/§b/§c below; it changes
35+
only §d (run/selection) and the sequencing (§e) — and it makes feedback #5 free.
36+
37+
---
38+
39+
## 1. Package composition — `default` variant + a lite preset (feedback #1, #2)
40+
41+
The k8s-distribution doc's path: operator `variant=generic`; platform **`default`** variant
42+
(*"registers PackageSources for all packages, installs nothing"* — verified); then `cozypkg add`
43+
per package. `cozypkg add` is a **different mechanism** from `bundles.enabledPackages` (which only
44+
toggles `.optional` packages): it creates a standalone `Package` CR against any registered
45+
PackageSource, so it reaches postgres-application, kafka-application, **and bucket-application**
46+
with no paas/iaas bundle. `dependsOn` is an ordering gate, not an auto-installer → the lane must
47+
add the **full closure**.
48+
49+
Per feedback #2, codify the closure as a **lite preset** rather than dozens of ad-hoc `cozypkg add`:
50+
- **A1 (prototype):** a checked-in list (e.g. `hack/e2e-lite-packages.txt`) + an install-script
51+
loop of `cozypkg add`. No Go/bundle change.
52+
- **A2 (durable, feedback #2's "bundle/preset for local tests"):** a named `lite`/`e2e` platform
53+
variant (`values-lite.yaml` + entry in `cmd/cozystack-operator/main.go:678`) or a dedicated
54+
bundle block. Reproducible for local dev; touches the Go binary + `helm-unittest`. Do after A1.
55+
56+
**Closure (17 survivors):**
57+
- *Engine/control:* `cozystack-engine`, `cozystack-basics`, `tenant-application` (creates the
58+
`tenant-test` tenant the suites install into — matches `.chainsaw.yaml`'s namespace),
59+
`cert-manager`, `gateway-api-crds`, `reloader`, `snapshot-controller`,
60+
`cozystack-scheduler` (**variant `default`**, not `linstor` — §b), `securitygroup-controller`.
61+
- *Networking:* `networking` **`cilium-generic`** (or `cilium` on Talos-in-Docker, or `noop` if the
62+
cluster ships Cilium already — §c).
63+
- *Storage (new pkg):* the local-path/host-path package (§b) → default SC.
64+
- *Object storage:* `seaweedfs-application`, `objectstorage-controller` (bucket/harbor COSI).
65+
- *LB:* `metallb` or Cilium LB-IPAM (§c, R-C1).
66+
- *Operators:* `etcd-operator`, `postgres-operator`, `mariadb-operator`, `kafka-operator`,
67+
`clickhouse-operator`, `foundationdb-operator`, `redis-operator`, `mongodb-operator`.
68+
- *App defs:* `etcd-application`, `gateway-application`, `external-dns`(+`-application`),
69+
`kuberture`, `bucket-application`, `harbor-application`, `postgres-application`,
70+
`clickhouse-application`, `foundationdb-application`, `kafka-application`, `mariadb-application`,
71+
`mongodb-application`, `openbao-application`, `qdrant-application`, `redis-application`.
72+
- `serviceexposure` needs nothing extra (controller ships in `cozystack-engine`).
73+
74+
(For `etcd.bats`/`etcd` chainsaw suite's `workloadmonitor`, add `victoria-metrics-operator` +
75+
`monitoring-agents` — the operator+CRD, not the whole tenant monitoring stack; see R-D3.)
76+
77+
---
78+
79+
## (b) Default StorageClass swap — new package, non-Rancher (feedback #2, #3)
80+
81+
**Finding (verified):** the linstor chart renders **no `StorageClass`**. `local` (default) +
82+
`replicated` are created imperatively by `hack/e2e-post-install-prep.sh:69-102` after ZFS pools.
83+
The 15 SURVIVES suites use `storageClass: ""` → they need only **one default SC**, name irrelevant
84+
(`local`/`replicated`-by-name are used only by the 5 NEEDS-REAL suites).
85+
86+
**Swap:** ship a small `packages/system/<local-storage>` package (declarative, Flux-managed —
87+
not a `kubectl apply` in a script, per feedback #2) wired only into the lite preset; drop
88+
`linstor`/`linstor-scheduler`/`linstor-gui`.
89+
90+
**Provisioner — DECIDED: OpenEBS LocalPV-Hostpath** (non-Rancher, feedback #3).
91+
92+
Implementation-time correction to the earlier ranking: `csi-driver-host-path` (the previous #1) ships no upstream Helm chart — only raw `deploy/kubernetes-*` manifests — so it does not fit cozystack's vendor-an-upstream-chart package pattern (`nfs-driver`'s Makefile does `helm pull csi-driver-nfs --untar`). Its only advantage over the alternatives, CSI snapshots, is not needed by the 17 survivors (they only need PVCs to bind; none snapshot in e2e). So it is the most work for no benefit here.
93+
94+
OpenEBS LocalPV-Hostpath is CNCF, non-Rancher, and publishes an official Helm chart (`localpv-provisioner` at `https://openebs.github.io/dynamic-localpv-provisioner`) → vendors cleanly via the `nfs-driver` pattern. Dynamic provisioning + `WaitForFirstConsumer`. The chart's own `hostpathClass.isDefaultClass: true` mints the default StorageClass (named `local` for parity with the LINSTOR default SC name), so no separate SC template is needed. Images stay on their upstream registry (like `nfs-driver`), so no image-mirroring step.
95+
96+
Rejected: `csi-driver-host-path` (no chart, snapshot advantage unused); `local-static-provisioner` (no dynamic provisioning); `nfs-driver`+NFS server (heavier); Rancher local-path (excluded by feedback #3).
97+
98+
**`cozystack-scheduler` coupling:** `system.yaml:28` uses variant `linstor`; the lite preset must
99+
use **`default`** (as `isp-hosted` does, `system.yaml:33`). No linstor `VolumeSnapshotClass` will
100+
exist — use the host-path CSI's snapshot class if a suite needs one.
101+
102+
---
103+
104+
## (c) Cilium-only networking — 👍 (feedback #4)
105+
106+
Unchanged and confirmed: cilium-only variants **already exist** in `sources/networking.yaml` —
107+
`cilium` (Talos) and `cilium-generic`. Base `system/cilium/values.yaml` is a full primary CNI
108+
(`ipam.mode: kubernetes`, `kubeProxyReplacement: true`, `gatewayAPI.enabled: true`); the kube-ovn
109+
coupling lives only in the `values-kubeovn.yaml` overlay, omitted by these variants. **No new CNI
110+
code** — pick `cilium` (Talos-in-Docker), `cilium-generic` (kind), or `noop` (only if a
111+
pre-installed Cilium exists; kindnet won't satisfy `gateway`/`securitygroup`).
112+
113+
Lost with kube-ovn (none affect the 17): VPC subnets/overlays (iaas, off), `multus`, `cozy-proxy`.
114+
**Tenant isolation survives** — `packages/apps/tenant/templates/networkpolicy.yaml` is 100%
115+
`cilium.io/v2` CNP/CCNP, zero kube-ovn, so `tenant.spec.isolated: true` and `securitygroup` work.
116+
117+
**R-C1 — LB IP:** `gateway` needs a `LoadBalancer` VIP (→ `Programmed`). MetalLB L2 or Cilium
118+
LB-IPAM pinned to a docker-bridge range. `serviceexposure` uses `externalIPs`, needs none.
119+
120+
---
121+
122+
## (d) Workflow / lane — chainsaw run + chainsaw-aware selection (feedback #5, #6)
123+
124+
Add a **parallel `e2e-container` job** to `pull-requests.yaml`, alongside the QEMU lane (which
125+
keeps the 5 NEEDS-REAL suites). It reuses #2826's chainsaw machinery end to end:
126+
127+
1. **Provision (BATS, no nested virt).** New `hack/e2e-prepare-cluster-container.bats` (or script):
128+
**Talos-in-Docker** (`talosctl cluster create --provisioner docker --cni none`) preferred for
129+
Talos fidelity (KubePrism `localhost:7445` → `cilium` variant applies unchanged); **kind** as
130+
the lighter alt (→ `cilium-generic` + `k8sServiceHost`=kind API). No `_out/assets/nocloud-*`
131+
dependency → skips image-build gating.
132+
2. **Install (BATS) via the lite preset (§1).** Fork `e2e-install-cozystack.bats`: operator
133+
`variant=generic`, platform **`default`** variant, `cozypkg add` the §1 closure (or apply the
134+
lite preset). Replace `e2e-post-install-prep.sh` with a container variant (install the storage
135+
package + LB pool; **no LINSTOR**). Keep the cilium-leak-healer step (CNI-level flake).
136+
Still creates `tenant-test` so `.chainsaw.yaml`'s namespace holds.
137+
3. **Run the app suite (Chainsaw, unchanged).**
138+
`make -C packages/core/testing test-chainsaw CHAINSAW_SUITES="<selected 17-subset>"`. The
139+
chainsaw suites are reused **verbatim** from #2826 — no per-suite edits.
140+
4. **Selection (feedback #5 — satisfied natively).** Reuse #2826's chainsaw-aware
141+
`hack/select-e2e.sh` (already maps diffs → suite dirs). Add a lane filter that **intersects**
142+
its output with the **17-survivor allowlist**, then sets `CHAINSAW_SUITES`. Diff-scoping is
143+
preserved; the 5 NEEDS-REAL suites (`vminstance`, `kubernetes-latest/previous`,
144+
`kubernetes-oidc-system/customconfig`) are never dispatched to this lane. Codify the allowlist
145+
next to the selector (or derive the excluded set from a `needs: [virt|drbd]` label on those
146+
suites).
147+
5. **Report:** reuse the `chainsaw-report.xml` collect/upload steps.
148+
149+
**Conventions.** Chainsaw suites already comply with `docs/agents/e2e-testing.md` (declarative
150+
asserts poll existence+state; `catch`/`finally` diagnostics; JUnit). The **new BATS provisioning/
151+
install** scripts must still follow the repo rules (event-driven `until` before `kubectl wait`, no
152+
`EXIT`/`RETURN` traps, fail-fast on HR readiness) and `hack/cozytest.sh` limits (only `@test`;
153+
dash `/bin/sh`, no `pipefail`; `set -u`). Chainsaw v0.2.15 gotchas (filter-as-list conditions;
154+
"actual" is a projection; bare-ref `error` = must-not-exist) are #2826's concern, inherited as-is.
155+
156+
---
157+
158+
## (e) Sequencing — build on #2826 (feedback #6)
159+
160+
- **Hard dependency: #2826 must merge (or this lane rebases onto it) first.** It is OPEN and
161+
mergeable on `main` as of 2026-07-09. Do not build a BATS app lane — it would be thrown away.
162+
- **Framework-agnostic work can start now** and lands identically either way: §b storage package,
163+
§c CNI variant selection + LB wiring, §1 lite preset/closure. Prototype these against a
164+
`default`-variant install on a dev stand independent of #2826.
165+
- **Framework-coupled work waits on #2826:** §d.3/§d.4 (the `test-chainsaw CHAINSAW_SUITES` run +
166+
the chainsaw-aware `select-e2e.sh` lane filter) must target #2826's files, so land after it.
167+
- Watch for churn on #2826 (suite dir names, `select-e2e.sh` mapping, `.chainsaw.yaml`) — the
168+
lane's allowlist and selector filter key off those.
169+
170+
---
171+
172+
## Open risks
173+
- **R-B1 (dissolved) — bucket.** Under `default`+`cozypkg`, `bucket-application`'s PackageSource is
174+
registered like any other → addable with **no iaas bundle**. Blocker only exists on the
175+
(rejected) "enable paas bundle" route.
176+
- **R-C1 — LB IP on a container network** (§c). Range must be routable on the kind/docker bridge or
177+
`gateway` hangs.
178+
- **R-C2 — preset drift.** Hand-maintained closure can rot. Derive it from the PackageSource
179+
`dependsOn` graph (same yq logic `select-e2e.sh` uses) or add a CI completeness check.
180+
- **R-D1 — Talos-in-Docker vs kind fidelity.** `cilium` (Talos) values hard-code KubePrism
181+
`localhost:7445`; kind → `cilium-generic` + kind API endpoint. Prototype both.
182+
- **R-D2 — seaweedfs/CNPG weight.** bucket/harbor need seaweedfs → CNPG `seaweedfs-db` + raft
183+
quorum (~5-6 min) on the default SC; validate host-path CSI satisfies its `WaitForFirstConsumer`
184+
PVCs before assuming bucket/harbor pass.
185+
- **R-D3 — monitoring weight.** The lite closure can skip the full VictoriaMetrics/Grafana stack;
186+
only `etcd`'s `workloadmonitor` needs `victoria-metrics-operator` + `monitoring-agents`.
187+
- **R-E1 — #2826 is not merged.** The whole chainsaw-first plan is gated on it (§e). If #2826
188+
stalls, the framework-agnostic work (§1/§b/§c) still has standalone value.
189+
- **R-E2 — chainsaw `parallel: 1`.** The shared `.chainsaw.yaml` runs suites sequentially (bucket's
190+
fixed port-forward port blocks parallelism). The lightweight lane inherits this → its wall-clock
191+
win comes from a faster *cluster* (no QEMU/DRBD/kube-ovn), not from parallel suites, unless
192+
#2826 later makes ports dynamic.
193+
194+
---
195+
196+
## Suggested delivery order (chainsaw-first)
197+
1. **Track/land #2826** — hard prerequisite for §d (feedback #6).
198+
2. **Storage package** (§b): `csi-driver-host-path` + default SC. Standalone, unit-testable now.
199+
3. **CNI + LB** (§c): select `cilium`/`cilium-generic`, wire MetalLB/Cilium-LBIPAM pool.
200+
4. **Lite preset** (§1, A1 first; derive closure from the dependsOn graph per R-C2). Prove on a
201+
dev stand with the `default` platform variant + `cozypkg add`.
202+
5. **Container provisioning + install** (§d.1/§d.2, BATS): Talos-in-Docker first.
203+
6. **Wire onto #2826:** lane filter in `select-e2e.sh` (17-allowlist intersection) + the
204+
`e2e-container` job running `test-chainsaw CHAINSAW_SUITES` (§d.3/§d.4, feedback #5), parallel
205+
to the QEMU lane.
206+
7. Optionally promote A1 → A2 (named `lite` variant/bundle) for local dev testing (feedback #2).

0 commit comments

Comments
 (0)