Skip to content

Commit 99a28d8

Browse files
myasnikovdaniilmyasnikovdaniil
authored
test(e2e): migrate app suite from BATS to Chainsaw and wire into CI (#2826)
## What this PR does Migrates the E2E **app** test suite from BATS to [Kyverno Chainsaw](https://github.com/kyverno/chainsaw) — a declarative, Kubernetes-native E2E framework (CNCF, part of the Kyverno project; successor to KUTTL) — and wires it into CI as the replacement for the per-app BATS loop. This grows the original pilot (postgres + bucket) into the full suite. All 22 app suites are ported under `hack/e2e-chainsaw/` and the `hack/e2e-apps/*.bats` files are removed: `postgres`, `bucket`, `mariadb`, `mongodb`, `redis`, `qdrant`, `clickhouse`, `kafka`, `etcd`, `openbao`, `harbor`, `foundationdb`, `external-dns`, `kuberture`, `vminstance`, `gateway`, `kubernetes-latest`, `kubernetes-previous`, `kubernetes-oidc-system`, `kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure`. **Approach** - **Declarative** suites assert on `status.conditions` and concrete fields. The `timeout N sh -ec "until kubectl get ..."` + `kubectl wait` pair that appeared ~100 times collapses into a single `assert` that polls existence and state together, with a structured diff on failure and automatic `events`/`describe`/`podLogs` capture via `catch` (previously only `harbor.bats` did this, by hand). - **Imperative** suites keep their logic in `script` steps: `openbao` init/unseal, `kuberture` external-dns split-horizon probes, `vminstance`, the `gateway` admission/impersonation cases, and `kubernetes-latest`/`previous`, which wrap the relocated `hack/e2e-chainsaw/_lib/run-kubernetes.sh` verbatim (Kamaji bring-up, LB/NFS/ouroboros checks). - `gateway` tests derive the tenant apex from the namespace `namespace.cozystack.io/host` label at runtime, so they are host-independent. **CI** - The `e2e` job now runs `chainsaw test hack/e2e-chainsaw/` via a new `test-chainsaw` target and uploads the JUnit `chainsaw-report.xml`. - The `chainsaw` binary is added to the e2e-sandbox image. - `install-cozystack` and `test-openapi` stay BATS (cluster bootstrap + OpenAPI checks). - The per-app 3-retry loop is dropped — assertion polling replaces the fixed-timeout flakiness it papered over. **Validation** Ran against a development cluster: the DB/app, storage, and VM suites pass; the three suites that depend on platform features not present on that cluster (`gateway`, `kuberture`, `external-dns`) and the two heavyweight `kubernetes-*` suites are exercised by this PR's CI run on a freshly installed platform. Note for reviewers: service-port asserts use the `(ports[*].port)` projection form rather than a number-literal filter (`` ports[?port == `N`] ``), because Chainsaw v0.2.15 mis-evaluates JMESPath number-literal comparisons. **2026-07-09: reconciled with main (~520 commits of drift)** - etcd suite re-ported to the v1alpha2 operator contracts (#2859): `readyMembers=3` gate, pod-label/`/scale`/WorkloadMonitor/metrics/defrag contracts, plus the `ETCD_E2E_S3_ROUNDTRIP`-gated backup round-trip driving `examples/backups/etcd` - 12 new `ingress-hostname-policy` gateway cases ported (apex derived at runtime from the tenant-root namespace label) - post-branch bats drift folded in: `mariadb-single` webhook guard, kafka `c1.small` presets, bucket/harbor 2m BucketClaim fail-fast, bucket readonly-denial promoted to a hard fail (cosi-driver v0.3.1), qdrant PVC-reclaim guard (#3059), kuberture fail-loud negations, SC-fallback-default test (#2872 B1) - 4 suites that only exist on main since the branch was cut are ported: `kubernetes-oidc-system`, `kubernetes-oidc-customconfig`, `securitygroup`, `serviceexposure` - CI seams: `nightly.yaml` converted to the chainsaw invocation, `run-kubernetes.sh` reconciled (Talos/CABPT waits, tenant drain, LINSTOR pool wait, talos-image-cache under `_lib/`), `e2e-capture-dataplane.sh` replicated in the Chainsaw global catch ### Release note ```release-note fix(kubernetes): tenant Kubernetes teardown no longer hangs when the cluster has no working nodes — the pre-delete hook now bounds its wait for the in-tenant HelmReleases and force-clears their Flux finalizers on timeout (#3271) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * End-to-end testing is now driven by a unified Chainsaw suite (platform apps, Kubernetes, networking, security groups, service exposure). * Test Impact Analysis selects affected suites, with an option to run the full suite. * CI produces a JUnit report artifact and collects suite-scoped diagnostics on failure. * **Bug Fixes** * CI E2E now fails fast and emits clearer Kubernetes/HelmRelease and sandbox bucket state diagnostics. * Improved teardown reliability with bounded wait + finalizer handling to avoid hangs. * **Documentation** * Updated E2E testing docs and local guidance to reflect the new Chainsaw-based workflow and suite paths. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents b99df11 + 8af4ed5 commit 99a28d8

107 files changed

Lines changed: 5461 additions & 3687 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gemini/styleguide.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -107,4 +107,4 @@ When reviewing `values.schema.json`:
107107
- Missing comments on generated code.
108108
- `go.sum` changes accompanying `go.mod` changes — expected and correct.
109109
- Fork relationships for vendored tooling images — intentional (e.g., `cozystack/kilo` fork is expected).
110-
- Absence of unit tests for vendored chart overrides — covered by E2E tests in `hack/e2e-apps/`.
110+
- Absence of unit tests for vendored chart overrides — covered by E2E tests in `hack/e2e-chainsaw/`.

‎.github/workflows/nightly.yaml‎

Lines changed: 27 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -294,28 +294,38 @@ jobs:
294294
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-openapi
295295
296296
# Always the FULL app suite — a nightly has no diff to scope against, so
297-
# Test Impact Analysis does not apply.
297+
# Test Impact Analysis does not apply (empty CHAINSAW_SUITES = whole suite).
298298
- name: Run E2E tests (full suite)
299299
run: |
300300
cd "/tmp/$SANDBOX_NAME"
301-
apps_list=$(ls hack/e2e-apps/*.bats | xargs -n1 basename | cut -d. -f1)
302-
failed_tests=""
303-
for app in $apps_list; do
304-
echo "::group::Testing $app"
305-
if make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-apps-$app; then
306-
echo "::endgroup::"; echo "✅ $app passed"
307-
else
308-
echo "::endgroup::"; echo "❌ $app failed"
309-
failed_tests="$failed_tests $app"
310-
echo "::group::Diagnostics for $app"
311-
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -50' || true
312-
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -30' || true
313-
echo "::endgroup::"
314-
fi
315-
done
316-
if [ -n "$failed_tests" ]; then echo "❌ Failed:$failed_tests"; exit 1; fi
301+
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-chainsaw CHAINSAW_SUITES=""; then
302+
echo "❌ Chainsaw E2E failed (see the assertion diffs above and diagnostics below)"
303+
echo "::group::Diagnostics: HelmRelease status"
304+
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -50' || true
305+
echo "::endgroup::"
306+
echo "::group::Diagnostics: recent events"
307+
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -30' || true
308+
echo "::endgroup::"
309+
exit 1
310+
fi
317311
echo "✅ All E2E tests passed"
318312
313+
- name: Collect chainsaw report
314+
if: always()
315+
run: |
316+
cd "/tmp/$SANDBOX_NAME"
317+
mkdir -p _out
318+
docker cp "$SANDBOX_NAME":/workspace/hack/e2e-chainsaw/chainsaw-report.xml \
319+
_out/chainsaw-report.xml || true
320+
321+
- name: Upload chainsaw report
322+
if: always()
323+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
324+
with:
325+
name: chainsaw-report
326+
path: /tmp/${{ env.SANDBOX_NAME }}/_out/chainsaw-report.xml
327+
if-no-files-found: ignore
328+
319329
- name: Collect report
320330
if: always()
321331
run: |

‎.github/workflows/pull-requests.yaml‎

Lines changed: 52 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -611,7 +611,7 @@ jobs:
611611
fi
612612
613613
# Test Impact Analysis (TIA): walk the PackageSource graph for the PR
614-
# diff and select only the bats files affected. Runs by default; opt
614+
# diff and select only the Chainsaw suites affected. Runs by default; opt
615615
# out via the `full-e2e` label to run the full suite.
616616
# `skip=true` means nothing to test (docs-only PR).
617617
- name: Select E2E tests
@@ -636,10 +636,17 @@ jobs:
636636
cd /tmp/$SANDBOX_NAME
637637
make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-openapi
638638
639-
# Run E2E tests. With `full-e2e` label the selector is skipped, its
640-
# outputs are empty, the != 'true' guard lets this step run, and the
641-
# full bats list is used. Without the label, the selector decides;
642-
# an empty selection (docs-only) skips this step entirely.
639+
# ▸ Run E2E tests (Chainsaw, scoped by Test Impact Analysis)
640+
#
641+
# The "Select E2E tests" step above picks the suites impacted by the PR
642+
# diff. With the `full-e2e` label that step is skipped, SELECTED_APPS is
643+
# empty, and we fall back to the whole suite; an empty selection on a
644+
# docs-only PR skips this step entirely (the skip != 'true' guard).
645+
#
646+
# Chainsaw discovers each named suite under hack/e2e-chainsaw/ and polls
647+
# every assertion to its own timeout, so the old per-app 3-retry loop is
648+
# gone — a failure is reported as a structured assertion diff, with
649+
# cluster-state diagnostics dumped below on a non-zero exit.
643650
- name: Run E2E tests
644651
id: e2e_tests
645652
if: ${{ steps.select.outputs.skip != 'true' }}
@@ -649,40 +656,50 @@ jobs:
649656
run: |
650657
cd /tmp/$SANDBOX_NAME
651658
if [ "$FULL_E2E" = "true" ] || [ -z "$SELECTED_APPS" ]; then
652-
apps_list=$(ls hack/e2e-apps/*.bats | xargs -n1 basename | cut -d. -f1)
659+
suites="" # empty => test-chainsaw runs the whole suite
660+
echo "Running full chainsaw suite"
653661
else
654-
apps_list="$SELECTED_APPS"
662+
suites="$SELECTED_APPS"
663+
echo "Running selected chainsaw suites: $suites"
655664
fi
656-
failed_tests=""
657-
for app in $apps_list; do
658-
echo "::group::Testing $app"
659-
if make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-apps-$app; then
660-
echo "::endgroup::"
661-
echo "✅ Test $app completed successfully"
662-
else
663-
echo "::endgroup::"
664-
echo "❌ Test $app failed (no retry — see diagnostics below)"
665-
failed_tests="$failed_tests $app"
666-
echo "::group::Diagnostics for $app"
667-
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -50' || true
668-
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -30' || true
669-
# COSI state at the moment of failure — the CRDs ship no printer
670-
# columns, so pull the readiness fields explicitly (cozyreport
671-
# captures the same at end-of-job, but bucket states move between
672-
# a test failing and the report being taken).
673-
docker exec "$SANDBOX_NAME" sh -c '
674-
kubectl get bucketclaims.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.bucketReady,BUCKET:.status.bucketName" 2>&1
675-
kubectl get buckets.objectstorage.k8s.io -o custom-columns="NAME:.metadata.name,READY:.status.bucketReady,ID:.status.bucketID,CLAIMNS:.spec.bucketClaim.namespace" 2>&1
676-
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
677-
' || true
678-
echo "::endgroup::"
679-
fi
680-
done
681-
if [ -n "$failed_tests" ]; then
682-
echo "❌ Failed tests:$failed_tests"
665+
if ! make -C packages/core/testing SANDBOX_NAME=$SANDBOX_NAME test-chainsaw CHAINSAW_SUITES="$suites"; then
666+
echo "❌ Chainsaw E2E failed (see the assertion diffs above and diagnostics below)"
667+
echo "::group::Diagnostics: HelmRelease status"
668+
docker exec "$SANDBOX_NAME" sh -c 'kubectl get hr -A -o wide 2>&1 | tail -50' || true
669+
echo "::endgroup::"
670+
echo "::group::Diagnostics: recent events"
671+
docker exec "$SANDBOX_NAME" sh -c 'kubectl get events -A --sort-by=.lastTimestamp 2>&1 | tail -30' || true
672+
echo "::endgroup::"
673+
# COSI state at the moment of failure — the CRDs ship no printer
674+
# columns, so pull the readiness fields explicitly (cozyreport
675+
# captures the same at end-of-job, but bucket states move between
676+
# a test failing and the report being taken).
677+
echo "::group::Diagnostics: COSI bucket state"
678+
docker exec "$SANDBOX_NAME" sh -c '
679+
kubectl get bucketclaims.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,READY:.status.bucketReady,BUCKET:.status.bucketName" 2>&1
680+
kubectl get buckets.objectstorage.k8s.io -o custom-columns="NAME:.metadata.name,READY:.status.bucketReady,ID:.status.bucketID,CLAIMNS:.spec.bucketClaim.namespace" 2>&1
681+
kubectl get bucketaccesses.objectstorage.k8s.io -A -o custom-columns="NAMESPACE:.metadata.namespace,NAME:.metadata.name,GRANTED:.status.accessGranted,CLAIM:.spec.bucketClaimName" 2>&1
682+
' || true
683+
echo "::endgroup::"
683684
exit 1
684685
fi
685-
echo "✅ All E2E tests passed"
686+
echo "✅ All selected E2E tests passed"
687+
688+
- name: Collect chainsaw report
689+
if: always()
690+
run: |
691+
cd /tmp/$SANDBOX_NAME
692+
mkdir -p _out
693+
docker cp "$SANDBOX_NAME":/workspace/hack/e2e-chainsaw/chainsaw-report.xml \
694+
_out/chainsaw-report.xml || true
695+
696+
- name: Upload chainsaw report
697+
if: always()
698+
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
699+
with:
700+
name: chainsaw-report
701+
path: /tmp/${{ env.SANDBOX_NAME }}/_out/chainsaw-report.xml
702+
if-no-files-found: ignore
686703

687704
# ▸ Collect debug information (always runs)
688705
- name: Collect report

‎Makefile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ assets-cozypkg-%:
105105

106106
test:
107107
make -C packages/core/testing apply
108-
make -C packages/core/testing test
108+
make -C packages/core/testing e2e
109109

110110
unit-tests: helm-unit-tests bats-unit-tests go-unit-tests rd-presets-check test-check-readiness
111111

0 commit comments

Comments
 (0)