Skip to content

Commit c9d3e4a

Browse files
test(e2e): skip etcd-adoption backup in the upgrade lane, keep seaweedfs baseline
The etcd v1alpha2 adoption snapshot cannot succeed in an e2e sandbox: it needs a trusted-cert (ACME) external S3 endpoint, which a sandbox (example.org, no DNS/ACME, self-signed in-cluster cert) cannot provide — matching the dev10 findings. Set migrations.etcdAdoptSkipBackup=true in the platform Package so migration 50 runs the real adoption without the infeasible snapshot. Keep seaweedfs on the root tenant (the historical default install-cozystack.bats also uses) for a realistic baseline, but drop the fragile cozy-backups-creds wait — the adoption no longer depends on the seaweedfs S3/bucket/creds chain. Co-Authored-By: Claude Opus 4.8 <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 52e30c6 commit c9d3e4a

2 files changed

Lines changed: 30 additions & 12 deletions

File tree

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -196,6 +196,7 @@ Separate from the per-PR app suite, the **upgrade lane** tests the most common u
196196
- **Cross-boundary state** other than CRs (the pre-upgrade CrashLoop/unbound-PV baseline and recorded PV bindings) is written to `/workspace/_out/upgrade-baseline/*.txt` by seed suites and diffed by the verify suites. The all-HRs-Ready gate + migration-stamp advance (the `cozystack-version` ConfigMap reaching `migrations.targetVersion`) are the authoritative health signals; the CrashLoop name-diff is a warning-only heuristic (pod names churn across an upgrade).
197197
- **Trigger — a `upgrade-e2e` job in `pull-requests.yaml`, parallel to `e2e`.** It is a sibling with the same `needs` (no dependency between them), so it adds no critical-path wall-clock. It runs on the release-promotion PR (`release` label) or any PR a maintainer opts in with the **`upgrade-e2e`** label. **Advisory / non-blocking** — its own check, gates nothing.
198198
- **The heaviest workload (a tenant Kubernetes cluster spanning the upgrade — the #2931 worker-rollover path) is env-gated OFF** behind `UPGRADE_E2E_TENANT_K8S`, via an early `exit 0` in a single `script` step (Chainsaw v0.2.15 has no test-level `skip`). It is the flakiest surface (nested KVM + DRBD + the #3231 tenant-worker image import) and needs dev-stand validation before being enabled in CI; the DB/Redis canaries + platform-health checks are the always-on core.
199+
- **The lane sets `migrations.etcdAdoptSkipBackup: true`** in the baseline platform Package. An etcd-adoption safety snapshot needs a trusted-cert external S3 endpoint the sandbox can't provide (`example.org`, no DNS/ACME, self-signed in-cluster cert), so it is skipped while the adoption itself still runs. `etcdAdoptSkipBackup` is a first-class, default-off platform value — real upgrades still snapshot.
199200

200201
## In-flight direction (not yet the merged standard)
201202

‎hack/e2e-upgrade-install-previous.bats‎

Lines changed: 29 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -75,12 +75,19 @@
7575
}
7676

7777
@test "Create platform Package and reconcile previous version" {
78-
# Minimal, version-stable platform config: networking + publishing only. These
79-
# keys have been valid since the 1.4 line, so the same manifest installs on the
80-
# baseline AND is what the current chart re-renders on upgrade. Deliberately
81-
# lean (no monitoring/seaweedfs bundle) — the seeded canaries don't need them,
82-
# and a leaner baseline is faster and less flaky while still exercising the
83-
# operators/CRDs/storage/tenant/app upgrade surface.
78+
# Version-stable platform config: networking + publishing (valid since the 1.4
79+
# line, so the same manifest installs on the baseline AND is re-rendered by the
80+
# current chart on upgrade), plus migrations.etcdAdoptSkipBackup.
81+
#
82+
# etcdAdoptSkipBackup makes the etcd v1alpha2 adoption migration (run during the
83+
# upgrade) adopt the live etcd WITHOUT its pre-adoption S3 safety snapshot. That
84+
# snapshot is structurally impossible in an e2e sandbox: the Etcd backup
85+
# strategy has no caCert field, so it needs a trusted-cert (ACME) external S3
86+
# endpoint, which a sandbox on example.org with no DNS/ACME cannot provide
87+
# (proven on dev10 — the snapshot only completes against a real ACME S3 host).
88+
# The adoption itself (pod/PVC re-ownership, operator swap, etcd 3.5->3.6 roll)
89+
# still runs — that is what this lane exercises. Set here (no strict values
90+
# schema on either version) so it is in place when the current chart upgrades.
8491
kubectl apply -f - <<EOF
8592
apiVersion: cozystack.io/v1alpha1
8693
kind: Package
@@ -99,6 +106,8 @@ spec:
99106
publishing:
100107
host: "example.org"
101108
apiServerEndpoint: "https://192.168.123.10:6443"
109+
migrations:
110+
etcdAdoptSkipBackup: true
102111
EOF
103112

104113
# Configure storage (LINSTOR pool + StorageClasses) and the MetalLB pool in the
@@ -128,14 +137,22 @@ EOF
128137
}
129138

130139
@test "Configure root tenant (baseline)" {
131-
# Enable just enough of the root tenant to host a child tenant + apps: etcd
132-
# (tenant control plane) and ingress. Monitoring/seaweedfs are intentionally
133-
# left off (see the lean-baseline rationale above).
140+
# Mirror a realistic root tenant: etcd (a legacy etcd.aenix.io cluster for the
141+
# v1alpha2 adoption migration to exercise on upgrade), ingress, and seaweedfs.
142+
# seaweedfs is the historical default (install-cozystack.bats enables it too),
143+
# so the baseline resembles a real cluster being upgraded. The etcd-adoption
144+
# migration does NOT depend on the seaweedfs backup chain here — that snapshot
145+
# is skipped via migrations.etcdAdoptSkipBackup (see the platform Package
146+
# above), so we only need the seaweedfs HR itself Ready, not the deeper
147+
# S3/bucket/creds projection.
134148
kubectl patch tenants/root -n tenant-root --type merge \
135-
-p '{"spec":{"host":"example.org","ingress":true,"etcd":true,"isolated":true}}'
149+
-p '{"spec":{"host":"example.org","ingress":true,"etcd":true,"isolated":true,"seaweedfs":true}}'
136150

137-
timeout 60 sh -ec 'until kubectl get hr -n tenant-root etcd ingress tenant-root >/dev/null 2>&1; do sleep 1; done'
138-
kubectl wait hr/etcd hr/ingress hr/tenant-root -n tenant-root --timeout=10m --for=condition=ready
151+
timeout 60 sh -ec 'until kubectl get hr -n tenant-root etcd ingress seaweedfs tenant-root >/dev/null 2>&1; do sleep 1; done'
152+
# seaweedfs installs as a serial chain (seaweedfs-db CNPG -> seaweedfs-system
153+
# raft -> seaweedfs wrapper), ~5-6min idle and longer under load; the
154+
# tenant-root parent flips Ready only after every child, so gate at 20m.
155+
kubectl wait hr/etcd hr/ingress hr/seaweedfs hr/tenant-root -n tenant-root --timeout=20m --for=condition=ready
139156
}
140157

141158
@test "Create isolated test tenant (baseline)" {

0 commit comments

Comments
 (0)