Kubernetes app: MachineDeployment.replicas hardcoded to 2 defeats nodeGroup minReplicas — render-side OIDC e2e churns worker DRBD, flaking kubernetes-previous #3231

TL;DR

The Kubernetes app chart hardcodes MachineDeployment.spec.replicas: 2, ignoring the nodeGroup's minReplicas. As a result the two render-side kubernetes-oidc-* e2e tests — which set minReplicas: 0 and delete their Kubernetes CR after ~6 s specifically to avoid a full cluster bring-up — still provision two 20 GiB replicated (DRBD) worker DataVolumes each. The create/delete churn of those volumes overlaps the very next test, kubernetes-previous, and transiently exhausts a LINSTOR satellite's ZFS pool. kubernetes-previous's worker root disks then fail placement with ResourceExhausted: Not enough available nodes for ~6 min, its worker VMs boot late, cilium can't finish initialising before the fixed 12-minute node-join wait, and the test fails with cni plugin not initialized.

This is a chart defect (scale-from-0 is broken by the hardcoded replicas) surfacing as an intermittent e2e failure. It can reproduce for any PR, because the PR E2E job runs the PR merged into main.

Affected

• Chart: any nodeGroup with minReplicas: 0 can never scale from zero — the MachineDeployment always starts at 2 workers.
• E2E: kubernetes-previous flakes on storage-pressure timing. Observed on the E2E run of feat(release): immutable tags & rc→stable promotion (#2677) #3017 (merge into a0db4447), but not caused by that PR — the churn originates in the OIDC tests on main.

Symptom

hack/e2e-apps/kubernetes-previous.bats → node-join wait times out:

=== node-join failed: fewer than 2 tenant nodes Ready within 12m — diagnostics follow ===
Ready   False   KubeletNotReady   container runtime network not ready:
  NetworkReady=false reason:NetworkPluginNotReady
  message:Network plugin returns error: cni plugin not initialized

cilium is not broken — its agents are mid-startup with 0 restarts (one Running/not-ready, one Pending). They start late only because the worker nodes register late.

Root cause chain

1. Chart hardcode. packages/apps/kubernetes/templates/cluster.yaml:

kind: MachineDeployment
metadata:
  annotations:
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-min-size: "{{ $group.minReplicas }}"
    cluster.x-k8s.io/cluster-api-autoscaler-node-group-max-size: "{{ $group.maxReplicas }}"
spec:
  replicas: 2   # hardcoded — ignores $group.minReplicas

minReplicas/maxReplicas are wired only to the autoscaler annotations; spec.replicas is a literal 2. On HR reconcile CAPI immediately creates 2 Machines → 2 KubevirtMachines → 2 worker DataVolumes, regardless of minReplicas: 0. The in-tenant cluster-autoscaler would only scale down later (once the control plane + autoscaler are up, after the scale-down delay).

2. Render-side OIDC tests inadvertently provision workers. hack/e2e-apps/kubernetes-oidc-system.bats and kubernetes-oidc-customconfig.bats set:

nodeGroups:
  md0: { minReplicas: 0, maxReplicas: 1, roles: [worker] }

and delete the CR ~6 s later without waiting for readiness (they only assert render-side objects). Because of (1), each still spins up 2 × 20 GiB replicated DRBD worker DataVolumes. The CR-delete races provisioning, so the MachineSet respawns replacements (3 distinct worker UUIDs observed for oidc-byo), producing a burst of zvol create/destroy churn.

3. Pool exhaustion hits the next test. The churn overlaps kubernetes-previous's own 2 worker placements, on top of the resident tenant-root/platform baseline sharing the same LINSTOR data pool. A satellite runs out of ZFS space:

ERROR REPORT 6A4BC7F1 — Node: srv3 — 2026-07-06 16:18:47
Command 'zfs create -V 20976048KB data/pvc-6ae90bea-…_00000' returned with exitcode 1.
cannot create 'data/pvc-6ae90bea-…_00000': out of space

(pvc-6ae90bea = the kubernetes-test-previous-version worker root disk.) autoPlace=3 needs a replica on each of srv1/srv2/srv3; srv3 can't fit → placement one node short:

CreateVolume failed … ResourceExhausted … 'Not enough available nodes' …
  the storage pools have to have at least '20971520' free space   # 20971520 KiB = 20 GiB (= required_bytes 21474836480)
  Replica count: 3   Node names: srv1 srv2 srv3   Storage pool name: data

This repeats for ~6 min (16:17:31 → 16:23:38). An OIDC worker PVC itself hit the same error mid-window and then failed to delete:

16:20:29  ProvisioningFailed  pvc kubernetes-oidc-byo-md0-…    ResourceExhausted: Not enough available nodes
16:24:46  VolumeFailedDelete  pvc-edcc55c7 … zfs destroy … : dataset is busy   (srv1)

4. Cascade to the failure. Late PVC bind (16:23:38) → late worker VM boot (16:25) → kubelets register 16:27–16:28 → cilium DaemonSet schedules on the freshly-joined nodes but hasn't finished CNI init by the 12-minute deadline (16:29:56) → nodes stay NotReady → exit 1.

Evidence

• Run: actions/runs/28798030083 (job "E2E Tests"), cozyreport artifact.
• CI tests the merge ref: HEAD is now at 8a06e21d7 Merge <pr-head> into a0db4447… — so main's OIDC tests run for every PR.
• OIDC worker DataVolumes (CDI importer pods, kubernetes/events.txt): importer-kubernetes-oidc-{byo,system}-md0-…-disk-system, scheduled across srv1/srv2/srv3.
• srv3 zfs create … out of space — linstor/error-reports ErrorReport 6A4BC7F1 (quoted above); companion 6A4BC7EB on srv1 = failed delete, dataset is busy.
• Unit check: 20971520 in the autoplacer output is KiB (× 1024 = 21474836480 B = the requested 20 GiB) — a full-replica requirement, not metadata slack.
• Files: packages/apps/kubernetes/templates/cluster.yaml (MD replicas: 2); hack/e2e-apps/kubernetes-oidc-{system,customconfig}.bats; hack/e2e-apps/run-kubernetes.sh (node-join timeout 12m, drains).

Why the existing mitigations don't catch it

hack/e2e-apps/run-kubernetes.sh has two storage barriers, both in the file-scope cozy_cleanup:

• cozy_wait_linstor_pool_free 90 300 runs at the bats-file boundary (after kubernetes-latest, before OIDC). It passed: LINSTOR ZFS pool free: smallest satellite srv2 has 100 GiB (>= 90 GiB threshold). The exhaustion is created ~5 min later, between the OIDC sub-tests and kubernetes-previous — a window with no pool-free gate.
• cozy_wait_tenant_drained 300 timed out (did not drain within 300s), but only because of an unrelated orphaned PVC (default-chk-clickhouse-test-keeper-cluster1-0-2-0, 1Gi, from an earlier test) that makes it a guaranteed 300 s no-op.

Proposed fixes

1. Chart (root cause): make MachineDeployment.spec.replicas respect the nodeGroup's minReplicas (e.g. replicas: {{ $group.minReplicas }}), or omit the field for autoscaled groups, so scale-from-0 works and a minReplicas: 0 group provisions no workers until needed.
2. E2E (immediate unblock): drop nodeGroups from the render-side OIDC test CRs — no nodeGroup → no MachineDeployment → no worker DataVolumes. The OIDC assertions (KCP extraArgs, authn-config Secret, KeycloakClient) don't need workers.
3. Harness (hardening): add a cozy_wait_linstor_pool_free between intra-file kube sub-tests (not only at file boundaries), and clean up the orphaned clickhouse keeper PVC so cozy_wait_tenant_drained isn't a permanent no-op.

Notes

• Not fixable by rebasing a PR — the churn lives in main and PR E2E already runs the merge into main.
• The OIDC tests are render-side by design (reasonable); the chart's hardcoded replicas is what defeats that intent. Fix (1) is the durable one; (2) unblocks E2E immediately.