Skip to content

Commit 974c280

Browse files
authored
test(e2e): price the layers above the sandbox on the node-join window (#3890)
## What this PR does Adds two readings to the tenant node-join path, taken on either side of the same wait the KVM counters already bracket: the runner VM kernel's own `/proc/stat`, and the per-thread CPU time of the QEMU processes the sandbox VMs run as. Every CPU reading on this failure path has a subject inside the sandbox: a cgroup, a tenant worker's threads, a sandbox node's own `/proc/stat`. Each of them can say a guest was given its ticks while getting no work done, and none can say what the layers above spent. That is where this suite's own tax is paid. The sandbox nodes' guest time is charged to the runner kernel as user time, and whatever the runner VM loses to the machine hosting it is charged to nobody the sandbox can see, so a node reporting `steal 0` and a node whose hypervisor was itself waiting look identical from inside. `/proc/stat` is not namespaced, so reading it from the sandbox container reads the runner VM's kernel. That is the only place steal on that layer appears, and it is read in one direction only: a steal that climbs proves this runner VM was preempted, while a zero proves nothing, because the column is filled only where the hypervisor exposes a paravirt steal clock and nothing in the capture observes whether this one does. The sandbox nodes a layer down can make the stronger claim, since this repository starts them with `accel=kvm`; nobody here starts the runner VM, so for this lane the answer is unknown until somebody reads a non-zero. The sandbox VMs are QEMU processes in the same PID namespace as the shell driving the suite, so their threads are a local read too. What that adds is not "which node is busy" on its own, since `cozy_capture_sandbox_node_cpu_time` already reads each node's own `/proc/stat` by name; it is that split *on the join window*, since the existing collector samples twice across a twelve-second interval inside the diagnostics block, after the join has already failed. Beyond the window it adds the host-side accounting of each vCPU thread, what the runner actually granted it, which no in-guest counter can report, and QEMU's non-vCPU IO and worker threads, which no in-guest counter sees either. It also names which node each process is, read from the `srv<N>/qemu.pid` files the bringup already writes: a container pid identifies nothing, and whether the starving guest is the control-plane node or a worker-carrying one is the discriminating question. With no usable pid file the capture stays anonymous and says so, as its KVM sibling does; the pid reads themselves are bounded like every other read here, since a pid file that never yields a byte must not hold the capture. Both are taken at the two points the KVM counters already use, for the same reason: the values are cumulative, so a rate needs a pair, and the interval worth measuring is the window the guest is failing to boot in rather than a few seconds after the deadline has expired. The thread reading runs the probe the tenant worker capture already uses rather than a copy of it, because that probe already answers the two outcomes that matter here and already names the three ways it can come up short, and a second copy would drift from it exactly at those failure paths. Each capture carries the layer it describes in its own legend. Three kernels in this report publish a `/proc/stat` and two captures now read QEMU threads, one layer apart in each direction, so a reader who takes one for the other is comparing a hypervisor against its own guest. The runner-kernel legend also carries the trap specific to that layer: nearly all of its user time IS guest time, because running the sandbox VMs is what that kernel does, so adding the two columns double-counts the whole run. Cost and blast radius: two bounded local reads per sample, no cluster access, nothing added to any diagnostics-phase budget window. The failure-path calls sit behind the same reasoning as the KVM reading, which the phase-budget guard in `hack/run-kubernetes-node-join_test.bats` now prices with a cost arm for each; the guards that enumerate collectors (the missing-`timeout` warning and the documented spend order) carry both new names. Tests are in `hack/run-kubernetes-runner-cpu_test.bats`, 44 cases, green under both runner forms. They cover the readings themselves, every legend claim a reader depends on, the bounded read and its knobs, the sample-number split, each way a capture can come up empty (a file with no `cpu` row, a container with no QEMU, a proc mount that yields nothing, a QEMU named and then gone), the routing of a message from a read that succeeded against one that failed, the per-node attribution and each way it can be incomplete, and placement guards of their own: both pairs must bracket the wait on both paths, nothing may be inserted between the three second readings, and each collector must report a shortfall to the job log in its own words rather than only to the report, since one call site is the passing path where the artifact is the one nobody downloads. Two guards exist because a legend can fail without any number being wrong: one asserts that the parse-from-the-last-parenthesis rule the legend states actually works on the lines the capture holds, and one that every bracketed legend line lands as a single line, since an apostrophe inside one of these single-quoted strings shreds it into one word per line while still passing any short grep. No documentation change: the KVM pair this mirrors is not described in `docs/agents/e2e-testing.md` either, and its documentation lives in the legends the captures carry. Context: #3513 ### Screenshots No UI changes. ### Downstream repositories One line of the bringup changes too: `hack/e2e-prepare-cluster.bats` now starts each sandbox QEMU with `-name guest=srv<N>,debug-threads=on`. QEMU names vCPU threads (`CPU N/KVM`) only when asked to, so without the flag every thread reports the bare process name and the vCPU/IO split this capture documents would not exist to read on this lane; a test pins the flag to the legend that relies on it. Walked the trigger map in `docs/agents/contributing.md` against the diff. Of the `hack/` triggers it names, `hack/e2e-prepare-cluster.bats` (ansible-cozystack, talm) is touched: the touched lines are the CI sandbox QEMU flags, which neither downstream repository restates -- both provision real machines, not the e2e sandbox. `hack/package.mk`, `hack/update-crd.sh` and `hack/upload-assets.sh` are untouched. The rest of the change is confined to the Chainsaw kubernetes suites' shared library and its unit tests, which no downstream repository restates. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note test(e2e): a tenant node-join failure now records the runner kernel's own CPU time and the sandbox VMs' per-thread QEMU CPU time on either side of the join wait, so steal on the layer above the sandbox and an uneven split across the three nodes are both visible in cozyreport.tgz ```
2 parents c5e175a + b1b8cbe commit 974c280

6 files changed

Lines changed: 1969 additions & 36 deletions

File tree

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 444 additions & 35 deletions
Large diffs are not rendered by default.

‎hack/e2e-chainsaw/kubernetes-latest/chainsaw-test.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,11 @@ spec:
4343
# guest Talos capture
4444
# ghcr-mirror state, access log and warm-up Job
4545
# talos-image-cache diagnosis
46+
# The three bracket pairs -- the sandbox kernel KVM counters, the runner
47+
# kernel CPU time and the sandbox QEMU thread time -- sit outside this
48+
# order on purpose: each is ungated, one bounded read a side, taken
49+
# before the node-join wait and right after it on both paths, and their
50+
# cost arms live in the budget guard rather than in this list.
4651
# and then, after the phase rather than inside it, the in-process tenant
4752
# crust-gather snapshot.
4853
#

‎hack/e2e-chainsaw/kubernetes-previous/chainsaw-test.yaml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,11 @@ spec:
3737
# guest Talos capture
3838
# ghcr-mirror state, access log and warm-up Job
3939
# talos-image-cache diagnosis
40+
# The three bracket pairs -- the sandbox kernel KVM counters, the runner
41+
# kernel CPU time and the sandbox QEMU thread time -- sit outside this
42+
# order on purpose: each is ungated, one bounded read a side, taken
43+
# before the node-join wait and right after it on both paths, and their
44+
# cost arms live in the budget guard rather than in this list.
4045
# and then, after the phase rather than inside it, the in-process tenant
4146
# crust-gather snapshot.
4247
#

‎hack/e2e-prepare-cluster.bats‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,7 +97,13 @@ EOF
9797

9898
@test "Boot QEMU VMs" {
9999
for i in 1 2 3; do
100+
# `debug-threads=on` names the vCPU threads (`CPU N/KVM`) so the QEMU
101+
# thread capture on the node-join failure path can tell a vCPU from an IO
102+
# thread; without it every thread reports the bare process name. QEMU only
103+
# renames threads when the flag is passed, so the capture's legend and its
104+
# fixtures are pinned to this line carrying it.
100105
qemu-system-x86_64 -machine type=pc,accel=kvm -cpu host -smp 8 -m 24576 \
106+
-name guest=srv${i},debug-threads=on \
101107
-device virtio-net,netdev=net0,mac=52:54:00:12:34:5${i} \
102108
-netdev tap,id=net0,ifname=cozy-srv${i},script=no,downscript=no \
103109
-drive file=srv${i}/system.img,if=virtio,format=raw,cache=unsafe \

‎hack/run-kubernetes-node-join_test.bats‎

Lines changed: 82 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,6 +225,15 @@ stub_collectors() {
225225
# pass-through, so a `mount` that hung would hang the suite with no ceiling to
226226
# stop it.
227227
cozy_capture_sandbox_kvm_exits() { printf 'kvm-exits-stub\n'; }
228+
# The two readings beside it, stubbed on the same ground and for a sharper
229+
# version of it: neither issues a kubectl read, so neither contributes to the
230+
# audit below, and un-stubbed they do not probe the host -- they READ it. One
231+
# cats the runner's real /proc/stat and the other walks the whole of /proc,
232+
# opening a comm file per process on the machine running `make unit-tests`.
233+
# The result of a case here would then be set by whatever else that machine
234+
# happens to be running, which is the property this file removes.
235+
cozy_capture_runner_kernel_cpu_time() { printf 'runner-kernel-cpu-stub\n'; }
236+
cozy_capture_sandbox_qemu_thread_cpu() { printf 'sandbox-qemu-thread-stub\n'; }
228237
}
229238

230239
# The collectors below are deliberately NOT in stub_collectors, and that is
@@ -885,6 +894,8 @@ assert_file_lacks_pattern() {
885894
cozy_capture_tenant_worker_cpu_throttle() { :; }
886895
cozy_capture_tenant_worker_network_counters() { :; }
887896
cozy_capture_sandbox_kvm_exits() { :; }
897+
cozy_capture_runner_kernel_cpu_time() { :; }
898+
cozy_capture_sandbox_qemu_thread_cpu() { :; }
888899
ghcr_mirror_diagnose() { :; }
889900
kubectl() { :; }
890901
cozy_report_node_join_failure test-latest-version
@@ -952,6 +963,8 @@ assert_file_lacks_pattern() {
952963
cozy_capture_tenant_worker_cpu_throttle() { :; }
953964
cozy_capture_tenant_worker_network_counters() { :; }
954965
cozy_capture_sandbox_kvm_exits() { :; }
966+
cozy_capture_runner_kernel_cpu_time() { :; }
967+
cozy_capture_sandbox_qemu_thread_cpu() { :; }
955968
cozy_capture_tenant_worker_block_io() { :; }
956969
ghcr_mirror_diagnose() { :; }
957970
cozy_capture_sandbox_node_cpu_time() { :; }
@@ -1028,6 +1041,8 @@ assert_file_lacks_pattern() {
10281041
cozy_capture_tenant_worker_cpu_throttle() { :; }
10291042
cozy_capture_tenant_worker_network_counters() { :; }
10301043
cozy_capture_sandbox_kvm_exits() { :; }
1044+
cozy_capture_runner_kernel_cpu_time() { :; }
1045+
cozy_capture_sandbox_qemu_thread_cpu() { :; }
10311046
cozy_capture_tenant_worker_block_io() { :; }
10321047
ghcr_mirror_diagnose() { :; }
10331048
cozy_report_node_join_failure test-latest-version
@@ -1062,6 +1077,8 @@ assert_file_lacks_pattern() {
10621077
cozy_capture_tenant_worker_cpu_throttle() { :; }
10631078
cozy_capture_tenant_worker_network_counters() { :; }
10641079
cozy_capture_sandbox_kvm_exits() { :; }
1080+
cozy_capture_runner_kernel_cpu_time() { :; }
1081+
cozy_capture_sandbox_qemu_thread_cpu() { :; }
10651082
cozy_capture_tenant_worker_block_io() { :; }
10661083
ghcr_mirror_diagnose() { :; }
10671084
cozy_report_node_join_failure test-latest-version
@@ -1362,6 +1379,16 @@ assert_file_lacks_pattern() {
13621379
# ahead of the console in wall clock, and one bounded read is a cost this
13631380
# arm can state exactly, unlike a walk whose size follows the sandbox.
13641381
cozy_capture_sandbox_kvm_exits) ahead=$((ahead + bound + grace)) ;;
1382+
# The other two halves of the same shape, and priced identically. The
1383+
# runner-kernel reading is one bounded read of /proc/stat; the QEMU
1384+
# thread reading is one bounded probe walk of this container's own /proc
1385+
# plus three bounded pid-file reads, all under the same ceiling the arm
1386+
# prices. Each pair's first half is taken before the wait, so declining
1387+
# either here would orphan a reading taken eighteen minutes earlier
1388+
# rather than save one. Counted all the same, because they do sit ahead
1389+
# of the console in wall clock.
1390+
cozy_capture_runner_kernel_cpu_time) ahead=$((ahead + bound + grace)) ;;
1391+
cozy_capture_sandbox_qemu_thread_cpu) ahead=$((ahead + bound + grace)) ;;
13651392
cozy_capture_tenant_worker_network_counters) ahead=$((ahead + walk)) ;;
13661393
# One walk apiece and read once, so each costs what the expression above
13671394
# prices a capped walk at. Both sit ahead of the console on the same
@@ -1422,7 +1449,10 @@ assert_file_lacks_pattern() {
14221449
# missing from it. So the list is derived here rather than restated: every
14231450
# function that guards its bounded call with `command -v timeout` has to
14241451
# appear in that sentence.
1425-
warn=$(grep -n 'timeout is not on PATH' "$lib" | head -n 1 | cut -d: -f1)
1452+
# Anchored on the phrase unique to the phase warning itself: the captures now
1453+
# write their own [bounds] notes carrying "timeout is not on PATH", so the
1454+
# first match of that phrase is no longer this sentence.
1455+
warn=$(grep -n 'run UNBOUNDED' "$lib" | head -n 1 | cut -d: -f1)
14261456
if [ -z "$warn" ]; then
14271457
echo "expected the phase to still warn when timeout is missing" >&2
14281458
exit 1
@@ -1483,6 +1513,8 @@ assert_file_lacks_pattern() {
14831513
'cozy_capture_tenant_worker_block_io:block IO counter:_cozy_cadvisor_node_stream _cozy_virt_launcher_listing' \
14841514
'cozy_capture_sandbox_node_cpu_time:sandbox node CPU time:cozy_capture_sandbox_node_cpu_time' \
14851515
'cozy_capture_sandbox_kvm_exits:sandbox kernel KVM counters:cozy_capture_sandbox_kvm_exits' \
1516+
'cozy_capture_runner_kernel_cpu_time:runner kernel CPU time:cozy_capture_runner_kernel_cpu_time' \
1517+
'cozy_capture_sandbox_qemu_thread_cpu:sandbox QEMU per-thread CPU time:cozy_capture_sandbox_qemu_thread_cpu' \
14861518
'cozy_capture_tenant_worker_thread_cpu:worker per-thread CPU time:cozy_capture_tenant_worker_thread_cpu _cozy_virt_launcher_listing' \
14871519
'ghcr_mirror_diagnose:ghcr-mirror:ghcr_mirror_diagnose _ghcr_mirror_bounded_read' \
14881520
'talos_image_cache_diagnose:talos-image-cache:talos_image_cache_diagnose _talos_image_cache_bounded_read'; do
@@ -1520,6 +1552,8 @@ ${carrier}
15201552
cozy_capture_tenant_worker_network_counters) phrase='network counter' ;;
15211553
cozy_capture_sandbox_node_cpu_time) phrase='sandbox node CPU time' ;;
15221554
cozy_capture_sandbox_kvm_exits) phrase='sandbox kernel KVM counters' ;;
1555+
cozy_capture_runner_kernel_cpu_time) phrase='runner kernel CPU time' ;;
1556+
cozy_capture_sandbox_qemu_thread_cpu) phrase='sandbox QEMU per-thread CPU time' ;;
15231557
cozy_capture_tenant_worker_thread_cpu) phrase='worker per-thread CPU time' ;;
15241558
ghcr_mirror_diagnose) phrase='ghcr-mirror' ;;
15251559
talos_image_cache_diagnose) phrase='talos-image-cache' ;;
@@ -1601,6 +1635,12 @@ ${carrier}
16011635
# out or withholds: declining it would leave that earlier reading with
16021636
# nothing to pair against rather than save the phase any time.
16031637
cozy_capture_sandbox_kvm_exits) continue ;;
1638+
# The other two halves of that same pair, exempt on the same ground and not
1639+
# on their own: each was read once before the node-join wait, so what runs
1640+
# here is the second half of a measurement rather than a collector the
1641+
# budget may hand out or withhold.
1642+
cozy_capture_runner_kernel_cpu_time) continue ;;
1643+
cozy_capture_sandbox_qemu_thread_cpu) continue ;;
16041644
*)
16051645
echo "$fn runs in the diagnostics block but this test has no phrase for it; add one here and to both chainsaw comments" >&2
16061646
exit 1
@@ -1635,6 +1675,45 @@ EOF
16351675
done
16361676
}
16371677

1678+
@test "nothing new runs between the KVM reading and the two beside it" {
1679+
lib=hack/e2e-chainsaw/_lib/run-kubernetes.sh
1680+
# The interval each of the three pairs divides by is whatever runs between its
1681+
# two readings, and on the failure path that is everything the block does before
1682+
# the second one. The KVM guard below pins what may precede ITS reading; this
1683+
# pins the gap between that reading and the two that follow it, which is where
1684+
# the other two intervals actually widen. Nothing belongs in there: the three
1685+
# readings are the second halves of three pairs and are meant to be adjacent.
1686+
block=$(grep -n '^cozy_report_node_join_failure() {' "$lib" | head -n 1 | cut -d: -f1)
1687+
kvm=$(awk -v b="$block" 'NR > b && /cozy_capture_sandbox_kvm_exits 2/ { print NR; exit }' "$lib")
1688+
last=$(awk -v b="$block" 'NR > b && /cozy_capture_sandbox_qemu_thread_cpu 2/ { print NR; exit }' "$lib")
1689+
for v in block kvm last; do
1690+
eval "n=\$$v"
1691+
if [ -z "$n" ]; then
1692+
echo "expected to read $v from $lib; without it this guard reports success for having lost its input" >&2
1693+
return 1
1694+
fi
1695+
done
1696+
if [ "$last" -le "$kvm" ]; then
1697+
echo "expected the QEMU thread reading (line $last) to follow the KVM one (line $kvm) inside the failure block" >&2
1698+
return 1
1699+
fi
1700+
# Matched on the call rather than on the `|| true` suffix, for the reason the
1701+
# KVM guard gives: the suffix says a call is phase-gated and says nothing about
1702+
# what costs time, and the dominant idiom in this block is `cozy_diag_read`
1703+
# with no suffix at all.
1704+
between=$(grep -nE '^ *(cozy_capture_[a-z_]+|cozy_report_[a-z_]+|cozy_diag_read|[a-z_]+_diagnose)( |$)' "$lib" \
1705+
| sed -E 's/:[[:space:]]*/:/; s/(:[a-z_]+) .*/\1/' \
1706+
| awk -F: -v a="$kvm" -v b="$last" '$1 > a && $1 < b { print $2 }')
1707+
case "$(printf '%s\n' $between)" in
1708+
'cozy_capture_runner_kernel_cpu_time') ;;
1709+
*)
1710+
echo "more than the runner kernel reading now runs between the KVM reading and the QEMU thread one, so those two intervals are wider than the wait by whatever it costs:" >&2
1711+
printf '%s\n' $between >&2
1712+
return 1
1713+
;;
1714+
esac
1715+
}
1716+
16381717
@test "the KVM counter readings sit on either side of the node-join wait on both paths" {
16391718
lib=hack/e2e-chainsaw/_lib/run-kubernetes.sh
16401719
# What this pair measures is decided by where its two calls sit, not by
@@ -1797,6 +1876,8 @@ EOF
17971876
cozy_capture_tenant_worker_cpu_throttle() { :; }
17981877
cozy_capture_tenant_worker_network_counters() { :; }
17991878
cozy_capture_sandbox_kvm_exits() { :; }
1879+
cozy_capture_runner_kernel_cpu_time() { :; }
1880+
cozy_capture_sandbox_qemu_thread_cpu() { :; }
18001881
cozy_capture_tenant_worker_block_io() { :; }
18011882
ghcr_mirror_diagnose() { :; }
18021883
kubectl() { :; }

0 commit comments

Comments
 (0)