Skip to content

Commit b6ac249

Browse files
authored
test(e2e): bump sandbox tenant storage quota 100Gi -> 200Gi (#3206)
## What this PR does Raises the E2E sandbox tenant's `resourceQuotas.storage` from 100Gi to 200Gi so back-to-back tenant Kubernetes tests (`kubernetes-latest.bats` -> `kubernetes-previous.bats`) do not collide with tenant quota during the second cluster's CDI import. The 100Gi default in `packages/apps/tenant` is unchanged — this touches only the E2E test tenant CR in `hack/e2e-install-cozystack.bats`. ### Root cause Both kubernetes-*.bats tests provision a 2-worker tenant control plane whose worker root disks are populated by CDI. CDI's second-phase importer allocates a scratch PVC of the same 20Gi as the target disk before the qcow2 unpack, so per-tenant peak footprint is: ``` 2 × 20Gi target + 2 × 21Gi scratch = ~82Gi during import ``` collapsing to 40Gi at rest. `kubernetes-latest`'s `cozy_cleanup` deletes the Kubernetes CR and blocks in `cozy_wait_tenant_drained` (300s) waiting for KubeVirt VMs / VMIs / namespace PVCs to disappear. The API-level PVC delete returns quickly, but the LINSTOR CSI detach + DRBD resource release on the underlying nodes lags past that budget. During the overlap window the deleted PVCs still count against `ResourceQuota.status.used.requests.storage` because the storage provisioner has not finished releasing the volume. `kubernetes-previous` then starts its own CDI import with 40Gi of stale `kubernetes-latest` quota still charged, and CDI's scratch PVC create for the second worker (used 80Gi + scratch 21Gi = 101Gi) is rejected by the tenant-quota admission check at 100Gi. The second worker's DataVolume stalls in `ImportInProgress`, the tenant node never joins, and the 12-minute node-join deadline in `run-kubernetes.sh` fails `kubernetes-previous.bats`. ### Evidence `cozyreport.tgz` from the failing E2E run of PR #3044 (run 28730757213): ``` cdi-deployment log: scratch PVC API create errored: persistentvolumeclaims "…-qxl8r-disk-system-scratch" is forbidden: exceeded quota: tenant-quota, requested: requests.storage=22763536384 (21Gi), used: requests.storage=85899345920 (80Gi), limited: requests.storage=100Gi ``` The tenant-quota `status.used.requests.storage` in the snapshot decays back to 40Gi by end-of-run — confirming the leak is a transient window during the second test's start-up, not a permanent leak. ### Relation to prior work Overlaps in scope with #2947 (`ci(e2e): bump tenant storage quota 100Gi -> 200Gi`), which bundles this same bump alongside three other e2e stabilization changes (crust-gather tag, harbor BucketClaim timeout, cilium IP-conflict healer). #2947 has been stale since 2026-06-17 with the maintainer suggesting it be closed. This PR isolates just the quota-bump so it can land on its own scope; the other three changes are orthogonal and can ship separately if #2947 stays dormant. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Updated end-to-end coverage for tenant creation with isolated mode to reflect higher storage quota expectations. * Improved quota validation in test scenarios to better match supported storage behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents a44d11d + 3ab9f95 commit b6ac249

1 file changed

Lines changed: 12 additions & 2 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -370,7 +370,17 @@ spec:
370370
resourceQuotas:
371371
cpu: "60"
372372
memory: "128Gi"
373-
storage: "100Gi"
373+
# 200Gi so back-to-back tenant Kubernetes tests
374+
# (kubernetes-latest, kubernetes-previous) don't run into
375+
# ResourceQuota during CDI's second-phase scratch PVC allocation.
376+
# Each tenant provisions 2 worker VMs × 20Gi disk + 21Gi CDI scratch
377+
# during import; when kubernetes-latest teardown's DRBD detach lags
378+
# past cozy_wait_tenant_drained, the leftover 40Gi of latest's worker
379+
# PVCs stays counted against tenant-quota while kubernetes-previous
380+
# is already asking for its own 40Gi + ~21Gi scratch: the scratch
381+
# PVC create call trips the 100Gi ceiling and the second worker's
382+
# DataVolume stalls in ImportInProgress indefinitely.
383+
storage: "200Gi"
374384
seaweedfs: false
375385
EOF
376386
timeout 60 sh -ec 'until kubectl get hr/tenant-test -n tenant-root >/dev/null 2>&1; do sleep 2; done'
@@ -381,7 +391,7 @@ EOF
381391
timeout 60 sh -ec 'until [ "$(kubectl get quota -n tenant-test --no-headers 2>/dev/null | wc -l)" -ge 1 ]; do sleep 1; done'
382392
kubectl get quota -n tenant-test \
383393
-o jsonpath='{range .items[*]}{.spec.hard.requests\.memory}{" "}{.spec.hard.requests\.storage}{"\n"}{end}' \
384-
| grep -qx '137438953472 100Gi'
394+
| grep -qx '137438953472 200Gi'
385395

386396
# Assert LimitRange defaults for containers
387397
kubectl get limitrange -n tenant-test \

0 commit comments

Comments
 (0)