Skip to content

Commit 3fc8e52

Browse files
committed
fix(e2e): select catch pod logs by the nested release instance label
The harbor, openbao and qdrant app charts deploy their workload through a nested HelmRelease named `<release>-system`, and pass `fullnameOverride: <release>` down to it. The inner chart therefore stamps app.kubernetes.io/instance from its own release name, with the suffix, while object names come from the override, without it. The catch blocks selected on the outer release name, so podLogs matched no pods and every failed run collected an empty result. A pod name says nothing about its instance label; the two come from different sources on purpose. In qdrant the unsuffixed value is therefore not retired but kept as a collector of its own: the parent chart's PVC-cleanup Job labels its pod template with the parent release name, and that suite deletes the parent inside a `try:`, so the Job is in scope while the test still runs. The comment there is explicit about how narrow that window is -- the hook always exits 0, so Helm calls it succeeded and `hook-succeeded` takes the pod away; what is left is an uninstall that hangs with Helm still waiting on it. Harbor renders the same hook and does not get the same collector. It has no `delete` op, so its parent release is uninstalled only in Chainsaw's cleanup phase, which no catch can observe: catch is a defer inside the step, while the cleaner runs from t.Cleanup, after the test function returns. That collector would be the same permanently-empty capture this commit removes. A comment records the asymmetry so the next reader does not restore it for symmetry. openbao renders no such hook at all. Each workload value is now also pinned on a resource in an assert the suite already runs, so the chart renaming a label fails a step instead of quietly emptying a capture. The comments state what that does and does not cover: it does not catch a wrong value typed into a selector, and it catches nothing when the run fails before the assert. Harbor's database was reached by no collector at all. It is selected on `cnpg.io/cluster`, CNPG's own primary label, rather than on the instance label CNPG merely derives from the Cluster name -- the primary label has nothing to drift from, and it is what the postgres suite and the WorkloadMonitors already use. Each lookup keeps stderr apart from the value it captures. kubectl exits 0 while logging a klog line about an API group it could not reach, and an aggregated APIService in that state is what a failed run usually looks like; folded into the value, that line makes the "nothing found" branch unreachable and prints as though it were a result. Telling those outcomes apart is one helper in hack/e2e-chainsaw/_lib/ rather than a copy per call site, separated from the kubectl call so it can be pinned without a cluster, the way etcd-probe.sh already is. Both the census and the qdrant cleanup listing report through it, differing only in wording, so the discrimination is implemented and tested once. The tests cover each outcome for both wordings, including the zero exit with a klog line on stderr -- the case a folded stream turned into a row. A collector that matches nothing cannot say so: podLogs is silent, and `kubectl get -l` exits zero on an empty match. Each test-wide catch prints the instance label of every pod in the namespace, and the qdrant cleanup catch reports its three outcomes apart from a single call -- printing what that call returned rather than running it again, which could put a healthy listing under a failure heading. That census runs on a degraded cluster, where kubectl otherwise retries discovery against an unreachable apiserver until something kills the script, so every lookup carries `--request-timeout`. Measured against an unreachable apiserver, a bounded call costs about five times the flag, since kubectl retries discovery five times before it gives up. Each script then carries an explicit op timeout sized above its own calls -- 2m for the single-call censuses, 5m for the qdrant cleanup listing, which makes three. An inner budget is what should fire: a SIGKILL at the op boundary yields nothing, where a kubectl that gives up leaves an error to print. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent df157da commit 3fc8e52

5 files changed

Lines changed: 477 additions & 8 deletions

File tree

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,109 @@
1+
# shellcheck shell=sh
2+
# Sourced by the chainsaw app Tests after `cd` to the repo root. Provides:
3+
# lookup_report — tell a diagnostic lookup's three outcomes apart,
4+
# with the wording left to the caller; no cluster
5+
# pod_label_census_report — lookup_report with the census wording
6+
# pod_label_census — print app.kubernetes.io/instance for every pod in
7+
# a namespace, keeping the outcomes apart
8+
# Sourcing has no side effects: it defines three functions and nothing else.
9+
# None uses `local`, which is not POSIX and so unavailable to the `sh` that
10+
# Chainsaw runs these steps under; the `_plc_` prefix is what keeps the
11+
# variables from colliding with a caller's names.
12+
#
13+
# Why this exists. A `podLogs` collector selects on a label, and it cannot
14+
# report that it matched nothing — an empty capture and a selector that has
15+
# gone stale look identical in the artifact. This prints the values that
16+
# actually exist, so a reader can see whether the selector above it can match.
17+
# Deliberately no selector is repeated here: a copy would only prove itself
18+
# right, and would keep matching after the collector's own selector drifted.
19+
#
20+
# The three outcomes are kept apart because each means something different and
21+
# two of them are easy to misread as the third:
22+
#
23+
# - the lookup did not run — NOT the same as "no pods"; saying so
24+
# would report an unreachable apiserver
25+
# as a healthy empty namespace
26+
# - it ran and matched nothing — a real, informative answer
27+
# - it ran and returned rows — the census proper
28+
#
29+
# stderr is kept apart from the captured value rather than folded in with
30+
# `2>&1`, and that is the load-bearing part. kubectl exits 0 while logging a
31+
# klog line about an API group it could not reach, and an aggregated APIService
32+
# in that state is what a failed run usually looks like (see
33+
# docs/agents/e2e-testing.md). Folded into the value, that line makes the
34+
# "no pods" branch unreachable and prints under the census heading as though it
35+
# were a pod. Separated, it becomes what it is: a caveat that the census may be
36+
# partial, printed alongside the answer rather than instead of it.
37+
#
38+
# `--request-timeout` bounds each attempt because on a degraded cluster kubectl
39+
# otherwise retries discovery until something kills the script. Measured
40+
# against an unreachable apiserver, a bounded call costs about five times the
41+
# flag, since kubectl retries discovery five times before it gives up; the
42+
# caller's op timeout is sized above that.
43+
#
44+
# The report half is separated from the kubectl call precisely so it can be
45+
# tested without a cluster — see hack/pod-label-census_test.bats, matching how
46+
# hack/e2e-chainsaw/_lib/etcd-probe.sh is covered.
47+
48+
# lookup_report RC STDOUT STDERR FAILED_MSG EMPTY_MSG ROWS_HEADING
49+
#
50+
# The outcome logic, with the wording left to the caller. Every diagnostic
51+
# lookup in a `catch` has the same three answers to keep apart and the same two
52+
# ways of confusing them, so they share one implementation and one set of tests
53+
# rather than a copy per call site.
54+
#
55+
# Always returns 0: this runs inside a `catch`, where a non-zero exit would mask
56+
# the test's own error.
57+
lookup_report() {
58+
_plc_rc=$1
59+
_plc_out=$2
60+
_plc_err=$3
61+
_plc_failed=$4
62+
_plc_empty=$5
63+
_plc_heading=$6
64+
65+
if [ "$_plc_rc" -ne 0 ]; then
66+
echo "$_plc_failed"
67+
[ -n "$_plc_err" ] && echo "$_plc_err"
68+
return 0
69+
fi
70+
71+
if [ -z "$_plc_out" ]; then
72+
echo "$_plc_empty"
73+
else
74+
echo "$_plc_heading"
75+
echo "$_plc_out"
76+
fi
77+
78+
if [ -n "$_plc_err" ]; then
79+
echo "the lookup also logged this, so the result above may be partial:"
80+
echo "$_plc_err"
81+
fi
82+
return 0
83+
}
84+
85+
# pod_label_census_report RC STDOUT STDERR NAMESPACE
86+
pod_label_census_report() {
87+
lookup_report "$1" "$2" "$3" \
88+
"pod label census FAILED (the lookup did not run; this is not 'no pods'):" \
89+
"pod label census: no pods in $4" \
90+
"pod label census, app.kubernetes.io/instance per pod:"
91+
}
92+
93+
# pod_label_census NAMESPACE
94+
pod_label_census() {
95+
_plc_ns=$1
96+
_plc_errfile=$(mktemp)
97+
98+
if _plc_stdout=$(kubectl -n "$_plc_ns" --request-timeout=15s get pods \
99+
-o 'custom-columns=POD:.metadata.name,INSTANCE:.metadata.labels.app\.kubernetes\.io/instance' \
100+
--no-headers 2>"$_plc_errfile"); then
101+
_plc_status=0
102+
else
103+
_plc_status=1
104+
fi
105+
106+
pod_label_census_report "$_plc_status" "$_plc_stdout" "$(cat "$_plc_errfile")" "$_plc_ns"
107+
rm -f "$_plc_errfile"
108+
return 0
109+
}

‎hack/e2e-chainsaw/harbor/chainsaw-test.yaml‎

Lines changed: 78 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -34,16 +34,79 @@ spec:
3434
format: yaml
3535
- script:
3636
content: kubectl -n cozy-objectstorage-controller get pods
37+
# The app chart wraps the workload in a nested HelmRelease named
38+
# `<release>-system`, and the chart behind it stamps
39+
# app.kubernetes.io/instance from its own release name. So the Harbor pods
40+
# carry `harbor-test-system`, and the unsuffixed value this collector used to
41+
# hold reached none of them, on every failed run.
42+
#
43+
# Coverage boundary, so the absence of a component here is not read as its
44+
# health: this selector reaches the workload of the nested release — core,
45+
# registry, portal, jobservice, nginx, exporter, and trivy when it is
46+
# enabled (the manifest for this suite disables it). The Redis pods carry
47+
# `harbor-test-redis`, so this selector misses them, but the script below
48+
# collects their logs by `rf[rs]-` name prefix. The CNPG database is covered
49+
# by the podLogs op below, on a different label.
50+
#
51+
# Nothing here collects the parent chart's post-delete cleanup Job, which
52+
# labels its pod template with the unsuffixed parent release name. That is
53+
# not an oversight: this suite has no `delete` op, so the parent release is
54+
# uninstalled only in Chainsaw's cleanup phase, and catch cannot observe it
55+
# -- catch is a defer inside the step, while the cleaner runs from t.Cleanup,
56+
# after the test function has returned. A collector on the unsuffixed value
57+
# would be a permanently-empty capture, the defect these selectors exist to
58+
# avoid. The qdrant suite deletes inside a `try:`, so there the Job is at
59+
# least in scope, and that suite carries the collector with its own note on
60+
# how narrow the window is.
3761
- podLogs:
38-
selector: app.kubernetes.io/instance=harbor-test
62+
selector: app.kubernetes.io/instance=harbor-test-system
3963
tail: 50
64+
# harbor-test readiness gates on this database, so its logs are the evidence
65+
# for a whole class of failed runs. Selected on `cnpg.io/cluster`, CNPG's own
66+
# primary label, rather than on app.kubernetes.io/instance, which CNPG only
67+
# derives from the Cluster name: the primary label has nothing to drift from,
68+
# and it is what the postgres suite and the WorkloadMonitors already use.
69+
- podLogs:
70+
selector: cnpg.io/cluster=harbor-test-db
71+
tail: 50
72+
# podLogs cannot report that it matched nothing, so an empty capture above and
73+
# a selector that has gone stale look identical. Print the instance label of
74+
# every pod in the namespace instead. Deliberately no selector is repeated
75+
# here: a copy would only prove itself right, and would keep matching after
76+
# the collector's own selector drifted. The census shows the values that
77+
# actually exist, so a reader can see whether the selector above can match.
78+
#
79+
# What is and is not guarded, so the next reader does not assume more:
80+
# the `-system` value above is also pinned on a resource in a `steps:` assert
81+
# below, so the chart renaming that label turns a step red -- that is the
82+
# drift this pin exists to catch. It does NOT catch a wrong value typed into the
83+
# selector here: the assert pins the label on the resource, this pins it in
84+
# the collector, and they are two literals. It also catches nothing when the
85+
# run fails before reaching that assert. The census below covers neither, and
86+
# is not meant to -- it answers what the labels actually are.
87+
# The `cnpg.io/cluster` selector has no pin at all; it rides on CNPG's
88+
# primary label instead, which is the reason for choosing it.
89+
- script:
90+
# The helper bounds each kubectl attempt with --request-timeout, which on
91+
# a degraded cluster costs about five times the flag; one call is ~75s at
92+
# worst, and this op timeout sits above that so an inner budget is what
93+
# fires. A SIGKILL at the op boundary would take the census out mid-write
94+
# and yield nothing, where a kubectl that gives up leaves an error to
95+
# print. The helper carries the rest of the rationale and is unit-tested
96+
# in hack/pod-label-census_test.bats.
97+
timeout: 2m
98+
content: |
99+
set -eu
100+
cd ../../..
101+
. hack/e2e-chainsaw/_lib/pod-label-census.sh
102+
pod_label_census "$NAMESPACE"
40103
# harbor-test readiness is gated by the inline RedisFailover
41104
# harbor-test-redis: its three sentinels stay NotReady until the spotahome
42105
# redis-operator elects a master and rewrites their monitored-master config.
43-
# None of the collectors above see that path: the harbor-test instance
44-
# selector matches neither the operator (namespace cozy-redis-operator) nor
45-
# the rfr-/rfs- pods, and the RedisFailover CRD ships no status subresource,
46-
# so there is nothing to assert on declaratively.
106+
# None of the collectors above see that path: the instance selector above
107+
# matches neither the operator (namespace cozy-redis-operator) nor the
108+
# rfr-/rfs- pods, which carry `harbor-test-redis`, and the RedisFailover CRD
109+
# ships no status subresource, so there is nothing to assert on declaratively.
47110
- describe:
48111
apiVersion: databases.spotahome.com/v1
49112
kind: RedisFailover
@@ -141,6 +204,16 @@ spec:
141204
kind: Deployment
142205
metadata:
143206
name: harbor-test-core
207+
# Pins the label the catch collector selects on. Harbor's matchLabels
208+
# carry only release+app, so pin the pod template, which is what
209+
# podLogs actually matches. Asserts are subset matches, so this costs
210+
# nothing and turns a silent drift of the instance label into a red
211+
# step instead of an empty capture.
212+
spec:
213+
template:
214+
metadata:
215+
labels:
216+
app.kubernetes.io/instance: harbor-test-system
144217
status:
145218
(conditions[?type == 'Available']):
146219
- status: "True"

‎hack/e2e-chainsaw/openbao/chainsaw-test.yaml‎

Lines changed: 38 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,41 @@ spec:
1515
apiVersion: helm.toolkit.fluxcd.io/v2
1616
kind: HelmRelease
1717
name: openbao-test
18+
# The app chart wraps the workload in a nested HelmRelease named
19+
# `<release>-system`, and the chart behind it stamps
20+
# app.kubernetes.io/instance from its own release name. So the pods carry
21+
# `openbao-test-system`, not `openbao-test`; the latter matches nothing.
1822
- podLogs:
19-
selector: app.kubernetes.io/instance=openbao-test
23+
selector: app.kubernetes.io/instance=openbao-test-system
2024
tail: 100
25+
# podLogs cannot report that it matched nothing, so an empty capture above and
26+
# a selector that has gone stale look identical. Print the instance label of
27+
# every pod in the namespace instead. Deliberately no selector is repeated
28+
# here: a copy would only prove itself right, and would keep matching after
29+
# the collector's own selector drifted.
30+
#
31+
# What is and is not guarded, so the next reader does not assume more:
32+
# the `-system` value above is also pinned on a resource in a `steps:` assert
33+
# below, so the chart renaming that label turns a step red -- that is the
34+
# drift this pin exists to catch. It does NOT catch a wrong value typed into the
35+
# selector here: the assert pins the label on the resource, this pins it in
36+
# the collector, and they are two literals. It also catches nothing when the
37+
# run fails before reaching that assert. The census below covers neither, and
38+
# is not meant to -- it answers what the labels actually are.
39+
- script:
40+
# The helper bounds each kubectl attempt with --request-timeout, which on
41+
# a degraded cluster costs about five times the flag; one call is ~75s at
42+
# worst, and this op timeout sits above that so an inner budget is what
43+
# fires. A SIGKILL at the op boundary would take the census out mid-write
44+
# and yield nothing, where a kubectl that gives up leaves an error to
45+
# print. The helper carries the rest of the rationale and is unit-tested
46+
# in hack/pod-label-census_test.bats.
47+
timeout: 2m
48+
content: |
49+
set -eu
50+
cd ../../..
51+
. hack/e2e-chainsaw/_lib/pod-label-census.sh
52+
pod_label_census "$NAMESPACE"
2153
steps:
2254
- name: create-openbao
2355
try:
@@ -58,6 +90,11 @@ spec:
5890
kind: Pod
5991
metadata:
6092
name: openbao-test-0
93+
# Pins the label the catch collector selects on. Asserts are subset
94+
# matches, so this costs nothing and turns a silent drift of the
95+
# instance label into a red step instead of an empty capture.
96+
labels:
97+
app.kubernetes.io/instance: openbao-test-system
6198
(status.containerStatuses[0].started): true
6299

63100
- name: init-and-unseal

0 commit comments

Comments
 (0)