Skip to content

Commit 546e33b

Browse files
committed
fix(e2e): bound every remaining read in the report collector
Four sections of hack/cozyreport.sh read with a wall-clock ceiling and the rest -- nodes, namespaces, events, HelmReleases, packages, packagesources, PVCs, cert-manager, COSI, kamaji, the Talos node reads and every module gate -- read with none. The tarball is written on the collector's last line, so a single read that never returns does not truncate the report, it loses it, on the run where it is the only evidence left. Each read now goes through the shape that fits it: the existing bounded reader for a read whose output is a file, the existing selector for one that feeds a loop, and a new probe for the `if kubectl get deploy ...` gates. A gate is a read like any other, and an unbounded one is worse than the reads it guards, since the module below it never starts. Its cut-off is recorded, because skipping the module leaves the same empty tree as a cluster that never had the component; a gate that answers "not installed" is recorded nowhere, since that is the cluster answering correctly and a marker firing on every install without KubeVirt is one a reader learns to skip. The reads that stay on the bare wrapper say so where they sit; what they share is that the output is not the reader's to write, going into a variable or through a pipeline that transforms it, so there is no moment at which a marker could attach. The header claimed a per-section list of what was bounded, and the tests froze that list plus a per-section count of the unbounded reads. Both are gone. The list had already gone stale, naming four sections while the file had ten, and the counts were an undercount by construction -- they matched a redirect to `$VAR/`, which no gate, selector or command substitution has. One test replaces both: no read below the library guard runs without a ceiling, checked by looking for a counterexample. A ceiling per read is not a ceiling on the report, and the header now says so where it used to imply the opposite. The pod, VM, VMI and services walks carry a count cap and a time budget; the walks over nodes, namespaces, HelmReleases, packages, PVCs and certificates carry neither, so their cost is the cluster's to set. What this removes is the read that never returns. The bounded reader now names the tool it ran and describes what that tool returned. It served kubectl alone until this change gave it talosctl and the four host commands, so a `df` that hit a stale mount left a marker reading "kubectl exited 1 ... not because the object ended" -- two false statements in one line, in an artifact whose premise is that its notes can be believed. The exec reader keeps the remote command's whole message in the listing it belongs to. `kubectl exec` appends its own `command terminated with exit code N` after whatever the command said, so quoting the last line there drops the cause and keeps the trailer, and sending the message away from the file leaves a zero-byte listing -- the "returned nothing, or never ran" question this collector exists to answer. Its cut-off note also stops asserting that the archive is beside it: a stream that got members across is kept and truncated, one killed before its first byte leaves zero bytes and the call site's emptiness check removes it, and the reader runs before that check, so it names both. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 9c5c621 commit 546e33b

7 files changed

Lines changed: 1084 additions & 269 deletions

File tree

‎.github/workflows/e2e-tag.yaml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -281,10 +281,16 @@ jobs:
281281
# green and the only trace is in the raw log. This branch's own standard is
282282
# that a bound which fires is recorded, and that has to hold for the outer
283283
# bound too.
284+
#
285+
# The wording says what `outcome` establishes and no more. A step can end
286+
# non-zero without any ceiling firing -- in e2e-tag.yaml the body opens with
287+
# a `cd` into a sandbox tree an earlier step may never have created, and the
288+
# default shell is `bash -e` -- so naming the timeout as the cause would be
289+
# the same false-mechanism claim the collector refuses to make for a 137.
284290
- name: Report collection overran its ceiling
285291
if: always() && steps.collect_report.outcome == 'failure'
286292
run: |
287-
echo "::warning title=cozyreport::Collect report did not finish within its timeout-minutes; cozyreport.tgz is missing or partial for this run"
293+
echo "::warning title=cozyreport::Collect report ended non-zero -- its timeout-minutes fired, or the step failed before the collector ran; cozyreport.tgz is missing or partial for this run"
288294
289295
- name: Upload cozyreport.tgz
290296
if: always()

‎.github/workflows/nightly.yaml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -356,10 +356,14 @@ jobs:
356356
# green and the only trace is in the raw log. This branch's own standard is
357357
# that a bound which fires is recorded, and that has to hold for the outer
358358
# bound too.
359+
#
360+
# The wording says what `outcome` establishes and no more: a step can end
361+
# non-zero without any ceiling firing, so naming the timeout as the cause
362+
# would be the false-mechanism claim the collector refuses to make for a 137.
359363
- name: Report collection overran its ceiling
360364
if: always() && steps.collect_report.outcome == 'failure'
361365
run: |
362-
echo "::warning title=cozyreport::Collect report did not finish within its timeout-minutes; cozyreport.tgz is missing or partial for this run"
366+
echo "::warning title=cozyreport::Collect report ended non-zero -- its timeout-minutes fired, or the step failed before the collector ran; cozyreport.tgz is missing or partial for this run"
363367
364368
- name: Upload cozyreport.tgz
365369
if: always()

‎.github/workflows/pull-requests.yaml‎

Lines changed: 11 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -770,10 +770,12 @@ jobs:
770770
# where it is the only evidence left. `|| true` guards against a
771771
# non-zero exit, not against a read that does not return.
772772
#
773-
# 30 minutes is measured against the step rather than divided out of the
774-
# job: it takes 3-42s across sampled runs, on failed e2e jobs as well as
775-
# green ones. That leaves five sixths of the job budget for the steps
776-
# that upload what this one produced.
773+
# 30 minutes is measured against the step: it takes 3-42s across sampled
774+
# runs, on failed e2e jobs as well as green ones. It RESERVES nothing --
775+
# `timeout-minutes` caps a step, it does not carve budget out of the job,
776+
# so a suite that has already burned 175 of the 180 leaves this step five
777+
# minutes and the upload is cancelled with everything else. What the cap
778+
# buys is that a hung read cannot be the reason.
777779
# `continue-on-error` because the ceiling must not change what the job
778780
# REPORTS. `|| true` on the make call is this step's standing statement
779781
# that collecting diagnostics is never fatal, and a step killed at its
@@ -794,10 +796,14 @@ jobs:
794796
# green and the only trace is in the raw log. This branch's own standard is
795797
# that a bound which fires is recorded, and that has to hold for the outer
796798
# bound too.
799+
#
800+
# The wording says what `outcome` establishes and no more: a step can end
801+
# non-zero without any ceiling firing, so naming the timeout as the cause
802+
# would be the false-mechanism claim the collector refuses to make for a 137.
797803
- name: Report collection overran its ceiling
798804
if: always() && steps.collect_report.outcome == 'failure'
799805
run: |
800-
echo "::warning title=cozyreport::Collect report did not finish within its timeout-minutes; cozyreport.tgz is missing or partial for this run"
806+
echo "::warning title=cozyreport::Collect report ended non-zero -- its timeout-minutes fired, or the step failed before the collector ran; cozyreport.tgz is missing or partial for this run"
801807
802808
- name: Upload cozyreport.tgz
803809
if: always()

‎docs/agents/e2e-testing.md‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎hack/cozyreport-talos.bats‎

Lines changed: 30 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,30 @@ COZYREPORT_LIB=1
1010
# shellcheck source=/dev/null
1111
. "$SCRIPT"
1212

13-
# cozytest.sh's parser ends an @test block at the first bare `}`, so keep the
14-
# talosctl mock at top level rather than nesting a function inside the test.
15-
talosctl() {
16-
printf '%s\n' "$*" >> "$calls"
17-
[ -z "${TALOSCTL_FAIL:-}" ] && return 0
18-
echo 'error: rpc error: code = Unavailable desc = connection refused' >&2
19-
return 1
13+
# A PATH stub, not a shell function. These reads now go through the bounded
14+
# reader, which runs them under `timeout` -- an external binary that execs its
15+
# argument and therefore cannot see a function defined in this shell. A function
16+
# mock is not merely bypassed, it is bypassed SILENTLY: the real talosctl runs, and
17+
# what the test then reports is whatever the machine happens to have installed.
18+
#
19+
# cozytest.sh's parser ends an @test block at the first bare `}`, so the helper
20+
# stays at top level rather than nested inside a test.
21+
talos_stub_dir() {
22+
_td=$1
23+
mkdir -p "$_td/bin"
24+
cat > "$_td/bin/talosctl" <<'STUB'
25+
#!/bin/sh
26+
printf '%s\n' "$*" >> "$TALOSCTL_CALLS"
27+
[ -z "${TALOSCTL_FAIL:-}" ] && exit 0
28+
echo 'error: rpc error: code = Unavailable desc = connection refused' >&2
29+
exit 1
30+
STUB
31+
chmod +x "$_td/bin/talosctl"
32+
# And a `timeout` that runs its command rather than timing it, so the assertions
33+
# do not depend on the host having GNU coreutils.
34+
printf '#!/bin/sh\n[ "$1" = "-k" ] && shift 2\nshift\nexec "$@"\n' > "$_td/bin/timeout"
35+
chmod +x "$_td/bin/timeout"
36+
COZYREPORT_BOUND="timeout -k 5 $COZYREPORT_READ_TIMEOUT"
2037
}
2138

2239
@test "Talos host diagnostics pass an endpoint and use valid dmesg flags" {
@@ -28,9 +45,11 @@ talosctl() {
2845
# cleanup line is never reached and the directory survives for inspection, which
2946
# is what you want from a test that just failed.
3047
tmp=$(mktemp -d)
48+
talos_stub_dir "$tmp"
3149
calls="$tmp/talosctl.calls"
3250

33-
cozyreport_collect_talos_node /workspace/talosconfig 192.0.2.11 "$tmp"
51+
TALOSCTL_CALLS="$calls" PATH="$tmp/bin:$PATH" \
52+
cozyreport_collect_talos_node /workspace/talosconfig 192.0.2.11 "$tmp"
3453

3554
[ "$(sed -n '1p' "$calls")" = "--talosconfig /workspace/talosconfig -e 192.0.2.11 -n 192.0.2.11 dmesg" ]
3655
[ "$(sed -n '2p' "$calls")" = "--talosconfig /workspace/talosconfig -e 192.0.2.11 -n 192.0.2.11 logs kubelet --tail=500" ]
@@ -45,9 +64,11 @@ talosctl() {
4564

4665
@test "a Talos node that refuses the connection leaves the reason in its files" {
4766
tmp=$(mktemp -d)
67+
talos_stub_dir "$tmp"
4868
calls="$tmp/talosctl.calls"
4969

50-
TALOSCTL_FAIL=1 cozyreport_collect_talos_node /workspace/talosconfig 192.0.2.11 "$tmp"
70+
TALOSCTL_FAIL=1 TALOSCTL_CALLS="$calls" PATH="$tmp/bin:$PATH" \
71+
cozyreport_collect_talos_node /workspace/talosconfig 192.0.2.11 "$tmp"
5172

5273
# The reads are best-effort and must not abort the report, but "did not abort"
5374
# and "produced nothing" are different outcomes and only one of them is about

0 commit comments

Comments
 (0)