Skip to content

Commit 33bd161

Browse files
committed
test(helm): sweep packages/tests so the library suite stands on its own
The helm unit sweep iterates packages/apps, core, extra, system and library, and never visits packages/tests. The cozy-lib-tests suite — the whole test coverage for a helper library 34 charts consume by symlink — runs only because packages/library/cozy-lib/Makefile carries a line delegating its own 'test' target to it. Deleting that one line takes the visited-package count for the suite from one to zero and the executions from one to zero, with the sweep still exiting 0 and still printing "All Helm unit tests passed": the no-suites backstop is satisfied by the other directories. Nothing else in the sweep could have caught it. Every check it makes — the Makefile test, the `make -n test` probe, the collection of failing directories — runs inside a directory the loop already reached, so a package that never enters the loop takes all of them with it. What is left is tests_found, and that counts suites across the whole run rather than per package, so the other directories keep it satisfied. Add the directory to the loop. It holds one package today, so the sweep grows from 76 executed suites to 77, and that suite costs between 0.1s and 0.3s over six runs against a sweep of roughly 31 seconds. The delegation stays for the ergonomics of `make -C packages/library/cozy-lib test`, which means cozy-lib-tests now runs twice; the point is that neither run is load-bearing alone. Covered by a new bats file: one test that the sweep reaches the directory and executes what it finds there, one that a red suite there still fails the sweep rather than being swallowed. Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 1abc653 commit 33bd161

2 files changed

Lines changed: 92 additions & 5 deletions

File tree

‎hack/helm-unit-tests.bats‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
#!/usr/bin/env bats
2+
# -----------------------------------------------------------------------------
3+
# Unit tests for hack/helm-unit-tests.sh
4+
#
5+
# cozytest.sh's awk parser recognizes only @test blocks and a bare `}` on its
6+
# own line; there is no bats `run` or `$status`. Each test runs as a shell
7+
# function under `set -eu -x`, so assertions are direct shell tests that exit
8+
# non-zero on failure. setup()/teardown() are not honored — each test creates
9+
# and cleans its own scratch dir, and cleanup is the last statement of the body
10+
# rather than a `trap ... EXIT`: that trap replaces the one the bats binary
11+
# installs for its own bookkeeping, and a test failing under it can print no TAP
12+
# line at all. Last, not before the assertions, so a failure leaves the scratch
13+
# tree behind for inspection.
14+
#
15+
# The sweep resolves its package directories relative to the working directory,
16+
# so each test runs it from a synthetic tree holding nothing but the packages it
17+
# is about. That keeps the assertions exact and keeps the real repo's suites,
18+
# which take minutes, out of a unit test.
19+
#
20+
# Run with: hack/cozytest.sh hack/helm-unit-tests.bats
21+
# -----------------------------------------------------------------------------
22+
23+
SWEEP="$PWD/hack/helm-unit-tests.sh"
24+
25+
# A package whose `test` target announces itself, so "the sweep reached this
26+
# directory" and "the suite actually ran" are two separate observations.
27+
make_package() {
28+
mkdir -p "$1"
29+
printf 'test:\n\t@echo %s\n' "$2" > "$1/Makefile"
30+
}
31+
32+
@test "the sweep runs the suites under packages/tests" {
33+
# packages/tests holds cozy-lib-tests, the whole test suite for the helper
34+
# library that 34 charts consume by symlink. The sweep did not visit the
35+
# directory: the suite ran only because packages/library/cozy-lib/Makefile
36+
# carries a line delegating its own `test` target to it. Deleting that one
37+
# line took the library's coverage to zero with the sweep still green and
38+
# still printing "All Helm unit tests passed", because the no-suites
39+
# backstop was satisfied by the other package directories.
40+
#
41+
# Nothing else in the sweep could have caught it. Every check it makes —
42+
# the Makefile test, the `make -n test` probe, the failure collection —
43+
# runs inside a directory it already visited, so a package that drops out
44+
# of the loop takes all of them with it and leaves only the backstop,
45+
# which counts suites across the whole run rather than per package.
46+
tmp=$(mktemp -d)
47+
make_package "$tmp/packages/tests/cozy-lib-tests" SWEEP_REACHED_TESTS
48+
output=$(cd "$tmp" && "$SWEEP" 2>/dev/null)
49+
if ! printf '%s\n' "$output" | grep -q "Running tests in packages/tests/cozy-lib-tests"; then
50+
echo "the sweep never visited packages/tests; it printed:" >&2
51+
printf '%s\n' "$output" >&2
52+
exit 1
53+
fi
54+
printf '%s\n' "$output" | grep -q SWEEP_REACHED_TESTS
55+
rm -rf "$tmp"
56+
}
57+
58+
@test "a suite failing under packages/tests fails the sweep" {
59+
# Visiting the directory is worth nothing if a red suite there is swallowed:
60+
# the delegation line the sweep is replacing propagated the failure, and the
61+
# replacement has to as well.
62+
tmp=$(mktemp -d)
63+
mkdir -p "$tmp/packages/tests/cozy-lib-tests"
64+
printf 'test:\n\t@exit 1\n' > "$tmp/packages/tests/cozy-lib-tests/Makefile"
65+
rc=0
66+
(cd "$tmp" && "$SWEEP" >/dev/null 2>&1) || rc=$?
67+
if [ "$rc" -eq 0 ]; then
68+
echo "a failing suite under packages/tests left the sweep green" >&2
69+
exit 1
70+
fi
71+
rm -rf "$tmp"
72+
}

‎hack/helm-unit-tests.sh‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,10 +3,25 @@ set -eu
33

44
# Script to run unit tests for all Helm charts.
55
# It iterates through directories in packages/apps, packages/core,
6-
# packages/extra, packages/system, and packages/library and runs the 'test'
7-
# Makefile target if it exists. Keep this list in step with the loop below:
8-
# packages/core carries suites of its own, so dropping it to match a stale
9-
# comment would silently stop running them.
6+
# packages/extra, packages/system, packages/library and packages/tests and runs
7+
# the 'test' Makefile target if it exists. Keep this list in step with the loop
8+
# below: packages/core carries suites of its own, so dropping it to match a
9+
# stale comment would silently stop running them.
10+
#
11+
# packages/tests is on the list because every check below runs INSIDE a
12+
# directory the loop already reached — the Makefile test, the `make -n test`
13+
# probe, the failure collection. A package that never enters the loop takes all
14+
# of them with it and leaves only tests_found, which counts suites across the
15+
# whole run rather than per package, so it stays satisfied by the other
16+
# directories. That is what happened here, silently:
17+
# packages/tests/cozy-lib-tests ran only via a delegating 'test' target in
18+
# packages/library/cozy-lib/Makefile, and deleting that line dropped the
19+
# library's entire coverage while this sweep stayed green and still reported
20+
# success. The delegation can stay for `make -C packages/library/cozy-lib test`
21+
# ergonomics; it is no longer the only thing running the suite. The cost of
22+
# keeping it is that a full sweep executes cozy-lib-tests twice, once through
23+
# each route — 0.1s to 0.3s over six runs — so the duplicate pair of
24+
# `Running tests in ...` lines in the log is expected rather than a bug.
1025

1126
FAILED_DIRS_FILE="$(mktemp)"
1227
trap 'rm -f "$FAILED_DIRS_FILE"' EXIT
@@ -33,7 +48,7 @@ check_and_run_test() {
3348
return 0
3449
}
3550

36-
for package_dir in packages/apps packages/core packages/extra packages/system packages/library; do
51+
for package_dir in packages/apps packages/core packages/extra packages/system packages/library packages/tests; do
3752
if [ ! -d "$package_dir" ]; then
3853
echo "Warning: Directory $package_dir does not exist, skipping..." >&2
3954
continue

0 commit comments

Comments
 (0)