Skip to content

Commit e95dee2

Browse files
committed
test(e2e): capture the tenant worker guest serial console
The tenant Kubernetes suites fail the node-Ready budget with zero Nodes registered, and the time is lost inside the guest between the worker VMI reaching Running and Talos requesting its apid certificate. Nothing collected today covers that window: the existing in-guest capture drives talosctl, so it needs the very apid the worker never reached, and on the management side there is neither a Node nor a certificate request to read. Turn on the guest console log for the suite's node group and collect each virt-launcher's guest-console-log container from the management cluster. Reads start at the beginning of the stream rather than tailing it, so a guest that repaints its console cannot push the boot output out of the window; every read is bounded, and the walk stops at a cap that records both counts, so the collector cannot become the unbounded term in a failure path that already runs several others under one deadline. A read that returns no console is not filed as a quiet guest. Three outcomes get three labels, because only one of them says anything about the guest: the read failed and returned nothing, the read succeeded and returned nothing, or it broke off part way. Each read's stderr is kept beside the capture rather than inside it, since kubectl's own "container is not valid for pod" would otherwise fill the file and a console that never started would read as one that had spoken. That case is the headline one here, so the Pod state and events that explain it are collected alongside, once for the whole selector. Enabling the console overrides a cluster-wide setting the platform holds for an open upstream defect, so the run is also an experiment, and its own failure mode is the one that hides best: the container wedges virt-launcher in Init, no worker boots, and the suite reports "fewer than 2 tenant nodes Ready" -- the same sentence the failure under study produces. Read as the known signature, the experiment's answer would be filed as the flake it was meant to explain. So the failure path names that case first, before any other diagnostic, and only when it is true. The passing path checks separately that the container attached at all, and reports the ratio it saw rather than a bare success, since the check is deliberately satisfied by any one Pod. It keeps three outcomes apart too: a read that did not answer, a selector that matched no Pod, and Pods that matched and carry nothing. It reads the Pod name alongside the container names so the last two cannot collapse, and both container lists so a future move of the container cannot fail a run over a diagnostic. It runs last, after everything the suite exists to prove, because a debugging aid must not preempt the assertions that cover the cluster. The capture runs before the talosctl one because it is the only of the two that can describe a worker which never reached apid, the shape this failure keeps taking. kubectl reports a warning and an empty result alike, on stderr with a zero status, so the exit status decides which artifact its output lands in. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent c20d47a commit e95dee2

4 files changed

Lines changed: 1204 additions & 6 deletions

File tree

‎hack/e2e-chainsaw/_lib/run-kubernetes.sh‎

Lines changed: 322 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -553,6 +553,289 @@ cozy_capture_tenant_talos_node() {
553553
"${output_dir}/kubelet.log" logs kubelet --tail=500 || true
554554
}
555555

556+
# Name the console experiment's own failure before the noise starts. Enabling
557+
# logSerialConsole punches through the platform's cluster-wide disable, and if
558+
# kubevirt/kubevirt#15989 still bites, guest-console-log wedges virt-launcher
559+
# in Init: the workers never boot, no Node registers, and the suite reports
560+
# "fewer than 2 tenant nodes Ready within 18m" -- byte-identical to the failure
561+
# this instrumentation was added to study. A triager reading that line files it
562+
# as the known flake and the experiment's answer is lost at the bottom of
563+
# POD-STATE.txt. A diagnostic whose own worst case is disguised as the bug it
564+
# investigates is worth less than none, so it says so itself, first, and only
565+
# when it is actually true.
566+
cozy_report_guest_console_wedge() {
567+
local rows
568+
569+
rows=$(timeout -k 5 30 kubectl -n tenant-test get pods \
570+
-l kubevirt.io=virt-launcher \
571+
-o jsonpath='{range .items[*]}{range .status.initContainerStatuses[?(@.name=="guest-console-log")]}{.ready}{" "}{end}{.metadata.name}{"\n"}{end}' \
572+
--request-timeout=30s 2>/dev/null) || return 0
573+
574+
case "${rows}" in
575+
*"false "*) ;;
576+
*) return 0 ;;
577+
esac
578+
579+
echo "=== guest-console-log did not start on the workers below: this is the console experiment failing, NOT the node-join failure it instruments (kubevirt/kubevirt#15989) ==="
580+
echo "=== the cluster-wide disableSerialConsoleLog in packages/system/kubevirt exists for this; drop logSerialConsole from the e2e node group and the suite returns to its prior behaviour ==="
581+
printf '%s\n' "${rows}" | grep '^false ' || true
582+
}
583+
584+
# Check that KubeVirt actually attached the guest console container, on the
585+
# path where the workers did boot. The capture below runs only when node-join
586+
# fails, so without this the suite would first learn that the node group's
587+
# logSerialConsole never took effect in the run that needed the console, when
588+
# it can no longer be recovered. The platform disables console logging
589+
# cluster-wide (see packages/system/kubevirt), so what this asks is whether
590+
# the per-VM override beat that on the KubeVirt version actually shipping.
591+
#
592+
# The read's own status is kept separate from the answer it returns, and the
593+
# two get different exit codes, because they are different claims. Piping
594+
# kubectl into grep would collapse them: a read that failed produces no output,
595+
# grep finds nothing, and an API blip would be reported as "the override did
596+
# not take effect" -- a cause that was never established, sending the next
597+
# reader at KubeVirt config instead of at the apiserver. A selector that
598+
# matched no Pod at all is the same trap one step further on, and is kept
599+
# apart from a Pod that matched and carries nothing -- the second is the
600+
# finding, the first is not a statement about any Pod.
601+
#
602+
# What it establishes is that SOME virt-launcher Pod in the namespace carries
603+
# the container, not that every one does. That is the direction worth erring
604+
# in for a check whose only job is to catch an inert setting: it can miss, but
605+
# it cannot fail a run over a Pod that is not this test's. The match runs over
606+
# the whole name=containers line, so a Pod named after the container would
607+
# satisfy it -- these names come from the Machine name, so that cannot happen
608+
# here, but a reader changing the format should know the glob spans both.
609+
#
610+
# 0 a Pod carries the container
611+
# 1 the read did not answer, or matched no Pod; not evidence either way
612+
# 2 Pods matched and none carries the container
613+
cozy_assert_guest_console_attached() {
614+
local init_names read_rc=0
615+
local attached total
616+
617+
# The Pod name is in the output for a reason that is not cosmetic: without
618+
# it, a Pod whose initContainers list is absent emits a bare newline, command
619+
# substitution strips it, and "Pods matched, none carries the container" is
620+
# byte-identical to "no Pod matched". Those two must not collapse -- the
621+
# first is the finding this whole check exists for, and on these workers it
622+
# is also the only shape it can take, since guest-console-log is the only
623+
# init container a DataVolume-booted virt-launcher Pod ever has.
624+
init_names=$(timeout -k 5 30 kubectl -n tenant-test get pods \
625+
-l kubevirt.io=virt-launcher \
626+
-o jsonpath='{range .items[*]}{.metadata.name}{"="}{.spec.initContainers[*].name}{" "}{.spec.containers[*].name}{"\n"}{end}' \
627+
--request-timeout=30s) || read_rc=$?
628+
if [ "${read_rc}" -ne 0 ]; then
629+
echo "could not read tenant virt-launcher Pods to verify the guest console container (exit ${read_rc})" >&2
630+
echo "this says nothing about logSerialConsole either way; continuing" >&2
631+
return 1
632+
fi
633+
634+
if [ -z "${init_names}" ]; then
635+
echo "no tenant virt-launcher Pod matched; nothing to verify about the guest console container" >&2
636+
echo "this says nothing about logSerialConsole either way; continuing" >&2
637+
return 1
638+
fi
639+
640+
case "${init_names}" in
641+
*guest-console-log*)
642+
# The positive outcome goes on the record too. This run doubles as the
643+
# revalidation of a platform-wide workaround, and a check that returns
644+
# 0 in silence leaves "the override worked" to be inferred from the
645+
# absence of a complaint -- the inference this collector refuses to
646+
# make everywhere else. Counted rather than asserted for every Pod: the
647+
# check is deliberately "some Pod carries it", so the ratio is the
648+
# honest way to say what was seen.
649+
attached=$(printf '%s\n' "${init_names}" | grep -c 'guest-console-log' || true)
650+
total=$(printf '%s\n' "${init_names}" | grep -c '=' || true)
651+
echo "» guest-console-log attached on ${attached} of ${total} virt-launcher Pods"
652+
return 0
653+
;;
654+
esac
655+
656+
echo "tenant worker virt-launcher Pods carry no guest-console-log container" >&2
657+
echo "the node group asked for logSerialConsole, so either the cluster-wide" >&2
658+
echo "disableSerialConsoleLog won the override or the field never reached the VM" >&2
659+
# Bounded like the read above, and for the same reason: these run on a path
660+
# that ends in exit 1, and a hang here would take the step's whole deadline
661+
# with it -- losing the tenant snapshot that the exit is supposed to reach.
662+
timeout -k 5 30 kubectl -n tenant-test get pods -l kubevirt.io=virt-launcher \
663+
-o jsonpath='{range .items[*]}{.metadata.name}{": "}{.spec.initContainers[*].name}{"\n"}{end}' \
664+
--request-timeout=30s || true
665+
timeout -k 5 30 kubectl -n tenant-test get virtualmachineinstances.kubevirt.io \
666+
-o jsonpath='{range .items[*]}{.metadata.name}{": logSerialConsole="}{.spec.domain.devices.logSerialConsole}{"\n"}{end}' \
667+
--request-timeout=30s || true
668+
return 2
669+
}
670+
671+
# Capture each tenant worker's guest serial console from the management
672+
# cluster. When a node group sets logSerialConsole, KubeVirt streams the
673+
# console into a guest-console-log container beside virt-launcher, and reading
674+
# it needs nothing from inside the guest: no talosctl, no client certificate,
675+
# no helper Pod, no reachable apid. That is the whole reason it exists. The
676+
# talosctl capture below can only describe a worker whose apid already answers,
677+
# so a worker that stalls earlier -- the case where no Node registers and no
678+
# certificate request is ever made -- is invisible to it by construction, and
679+
# the console is the only remaining surface.
680+
#
681+
# Reads start at the beginning of the stream rather than tailing it, because a
682+
# guest that repaints its console would push the boot output out of any tail
683+
# window while --limit-bytes truncates from the far end instead. That covers
684+
# only what kubectl returns: whatever the kubelet has already rotated out of
685+
# the container log (containerLogMaxSize, 10Mi x 5 by default) is gone before
686+
# the capture runs, and no read option recovers it.
687+
#
688+
# An absent guest-console-log container is the finding, not a gap: it says
689+
# console logging did not take effect on this run, which is a different fact
690+
# from a guest that printed nothing. kubectl's refusal is kept beside the
691+
# capture rather than inside it so the two stay distinguishable, and the
692+
# capture file says which of them happened.
693+
#
694+
# The Pod list is namespace-wide rather than scoped to one release. The two
695+
# virt-launcher captures immediately above the call site already select that
696+
# way, and suites run one at a time. Below the cap a Pod that does not belong
697+
# to this test costs one extra file named after itself and nothing else; at
698+
# the cap it can displace one of this suite's own, since kubectl returns the
699+
# list name-sorted. COLLECTION-TRUNCATED.txt is what keeps that visible.
700+
cozy_capture_tenant_serial_console() {
701+
local report_dir="${COZY_REPORT_DIR:-/workspace/_out/cozyreport}/snapshots/${COZY_SNAPSHOT_NAME:-kubernetes}/tenant-serial-console"
702+
local pods pod rc
703+
local list_rc=0
704+
local seen=0
705+
local silent=0
706+
local pod_err
707+
local max_pods=6
708+
local err_log="${report_dir}/setup-error.log"
709+
local warn_log="${report_dir}/READ-WARNINGS.txt"
710+
711+
mkdir -p "${report_dir}"
712+
# stderr goes to its own file rather than being folded into the captured
713+
# stdout: a warning kubectl prints on an otherwise successful call would
714+
# otherwise be read back as a Pod name. Which file it goes to is decided by
715+
# the exit status, not by the fact that something was written: an apiserver
716+
# Warning header or a deprecation notice arrives on stderr with a zero exit,
717+
# so a warning beside a healthy read belongs in READ-WARNINGS.txt as
718+
# elsewhere in this tree, while the stderr of a call that actually failed IS
719+
# the diagnosis and belongs in setup-error.log. The list is wrapped in
720+
# timeout for the same reason every read below it is: --request-timeout
721+
# bounds the HTTP request, not a client retrying against a wedged
722+
# apiserver, and a hang here would cost the whole
723+
# section rather than one Pod.
724+
pods=$(timeout -k 5 30 kubectl -n tenant-test get pods -l kubevirt.io=virt-launcher \
725+
-o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}' \
726+
--request-timeout=30s 2>"${warn_log}") || list_rc=$?
727+
if [ "${list_rc}" -ne 0 ]; then
728+
echo "failed to list tenant virt-launcher Pods for serial console capture" >&2
729+
mv "${warn_log}" "${err_log}" 2>/dev/null || true
730+
printf '%s\n' \
731+
"failed to list tenant virt-launcher Pods for serial console capture (exit ${list_rc})" \
732+
>>"${err_log}"
733+
return 1
734+
fi
735+
[ -s "${warn_log}" ] || rm -f "${warn_log}"
736+
if [ -z "${pods}" ]; then
737+
echo "no virt-launcher Pod found for serial console capture" >&2
738+
printf '%s\n' 'no virt-launcher Pod found in namespace tenant-test' \
739+
>"${err_log}"
740+
return 1
741+
fi
742+
743+
# Cap the walk. Each read costs up to 35s (30s plus the kill grace) and the
744+
# pool can reach maxReplicas, so an uncapped loop is an unbounded term inside
745+
# a failure path that already spends ~12m on other collectors under one
746+
# chainsaw op timeout — and the bundle this exists to produce is what gets
747+
# truncated when that op is killed. A cap that fired is recorded with both
748+
# counts rather than leaving the shortfall implicit, since a short listing
749+
# otherwise reads as a small pool rather than a truncated walk.
750+
for pod in ${pods}; do
751+
seen=$((seen + 1))
752+
if [ "${seen}" -gt "${max_pods}" ]; then
753+
echo "--- guest serial console capture stopped at ${max_pods} Pods ---"
754+
printf 'capture stopped after %s Pods; %s matched in total\n' \
755+
"${max_pods}" "$(printf '%s\n' ${pods} | wc -l | tr -d ' ')" \
756+
>"${report_dir}/COLLECTION-TRUNCATED.txt"
757+
break
758+
fi
759+
echo "--- capturing guest serial console: ${pod} ---"
760+
rc=0
761+
# stderr goes beside the capture, not into it. Folded in, kubectl's own
762+
# "container guest-console-log is not valid for pod ..." would make the
763+
# file non-empty and the console would look like it had said something.
764+
# That error is the headline case here -- it is what a container wedged in
765+
# Init looks like -- so it has to stay separable from console bytes. Which
766+
# name it lands under is decided by the exit status, the same way the list
767+
# read above decides it: kubectl writes warnings on stderr with a zero
768+
# status too, and a warning beside a healthy capture is not an error.
769+
pod_err="${report_dir}/${pod}.read-error.log"
770+
timeout -k 5 30 kubectl -n tenant-test logs "${pod}" \
771+
-c guest-console-log --limit-bytes=1048576 \
772+
>"${report_dir}/${pod}.log" 2>"${pod_err}" || rc=$?
773+
if [ ! -s "${pod_err}" ]; then
774+
rm -f "${pod_err}"
775+
pod_err=
776+
elif [ "${rc}" -eq 0 ]; then
777+
mv "${pod_err}" "${report_dir}/${pod}.READ-WARNINGS.txt" 2>/dev/null || true
778+
pod_err=
779+
fi
780+
# Tested before the exit-code line is appended, or nothing is ever empty.
781+
# Three outcomes, three labels, because only one of them is a statement
782+
# about what the guest did.
783+
if [ ! -s "${report_dir}/${pod}.log" ] && [ "${rc}" -ne 0 ]; then
784+
silent=$((silent + 1))
785+
# timeout kills the child without a word, so a read can fail having
786+
# written nothing to either stream. Naming a file that was never created
787+
# sends the reader looking for evidence that does not exist.
788+
if [ -n "${pod_err}" ]; then
789+
printf '%s\n' \
790+
'the read returned no console at all; see read-error.log and POD-STATE.txt' \
791+
>>"${report_dir}/${pod}.log"
792+
else
793+
printf '%s\n' \
794+
'the read failed silently and returned no console at all; see POD-STATE.txt' \
795+
>>"${report_dir}/${pod}.log"
796+
fi
797+
elif [ ! -s "${report_dir}/${pod}.log" ]; then
798+
silent=$((silent + 1))
799+
printf '%s\n' \
800+
'the read succeeded and returned nothing; see POD-STATE.txt for whether the container started' \
801+
>>"${report_dir}/${pod}.log"
802+
elif [ "${rc}" -ne 0 ]; then
803+
silent=$((silent + 1))
804+
if [ -n "${pod_err}" ]; then
805+
printf '%s\n' \
806+
'console output is truncated; see read-error.log and POD-STATE.txt' \
807+
>>"${report_dir}/${pod}.log"
808+
else
809+
printf '%s\n' \
810+
'console output is truncated; see POD-STATE.txt for the Pod state' \
811+
>>"${report_dir}/${pod}.log"
812+
fi
813+
fi
814+
printf '\n[capture exit code: %s]\n' "${rc}" >>"${report_dir}/${pod}.log"
815+
done
816+
817+
# An empty console log has two causes that are indistinguishable in the log
818+
# itself: the guest printed nothing, or the container never started and there
819+
# was no stream to read. The second is what kubevirt/kubevirt#15989 produces,
820+
# and it is the reading that would otherwise be filed as a quiet guest --
821+
# silence reported as a result is the failure this collector exists to
822+
# prevent, so it is not allowed to produce one. The Pod's own state and its
823+
# events are what separate the two, and they are read once for the whole
824+
# selector rather than per Pod: several silent workers are one finding, and
825+
# paying per Pod would put back the unbounded term the cap removed.
826+
if [ "${silent}" -gt 0 ]; then
827+
{
828+
printf '=== describe pods -l kubevirt.io=virt-launcher ===\n'
829+
timeout -k 5 30 kubectl -n tenant-test describe pods \
830+
-l kubevirt.io=virt-launcher --request-timeout=30s 2>&1 || true
831+
printf '\n=== Pod events in tenant-test (not filtered to virt-launcher) ===\n'
832+
timeout -k 5 30 kubectl -n tenant-test get events \
833+
--field-selector involvedObject.kind=Pod \
834+
--sort-by=.lastTimestamp --request-timeout=30s 2>&1 || true
835+
} >"${report_dir}/POD-STATE.txt"
836+
fi
837+
}
838+
556839
# Collect the guest-side evidence requested by issue #3513. This runs only after
557840
# the 18-minute Ready deadline has already failed. VMIs without a reported IP
558841
# are recorded before any Certificate or Pod is created; when at least one IP is
@@ -728,6 +1011,12 @@ ${ouroboros_addon}
7281011
diskSize: 20Gi
7291012
gpus: []
7301013
instanceType: u1.medium
1014+
# The failure this suite keeps hitting is a worker that stalls in the
1015+
# guest before Talos apid answers, which leaves nothing to read on the
1016+
# management side and nothing for talosctl to connect to. Turning this on
1017+
# attaches KubeVirt's guest-console-log container, the only artifact that
1018+
# covers that window.
1019+
logSerialConsole: true
7311020
maxReplicas: 10
7321021
minReplicas: 2
7331022
resources: {}
@@ -872,6 +1161,7 @@ EOF
8721161
# designed without this artifact. Every capture is guarded with `|| true`
8731162
# so a capture failure never masks the real `exit 1`.
8741163
echo "=== node-join failed: fewer than 2 tenant nodes Ready within 18m — diagnostics follow ==="
1164+
cozy_report_guest_console_wedge || true
8751165
kubectl --kubeconfig "tenantkubeconfig-${test_name}" describe nodes || true
8761166
kubectl -n tenant-test get hr || true
8771167

@@ -886,6 +1176,22 @@ EOF
8861176
kubectl -n tenant-test get pods -l kubevirt.io=virt-launcher -o wide || true
8871177
kubectl -n tenant-test describe pods -l kubevirt.io=virt-launcher || true
8881178

1179+
# (b1) Guest serial console, read from the management cluster. First of the
1180+
# in-guest captures because it is the only one that survives a worker which
1181+
# never reached apid — the dominant shape of this failure, where no Node
1182+
# registers and no certificate request is ever made. (b) below needs that
1183+
# same apid to answer, so it cannot describe this class at all.
1184+
# Two shapes of empty result are worth telling apart from a silent guest.
1185+
# A virt-launcher Pod still Pending has no containers at all yet, which is
1186+
# the ordinary shape of a worker that never booted. A Pod wedged in Init on
1187+
# guest-console-log is the opposite reading: the console container is what
1188+
# stopped it booting, and then neither a console nor an apid exists to
1189+
# explain anything else. The container being absent from a Running Pod is
1190+
# the rarest of the three, since the per-VM field outranks the cluster-wide
1191+
# disable by design and the green path asserts it attached.
1192+
echo "=== (b1) tenant worker guest serial console (management cluster, ns tenant-test) ==="
1193+
cozy_capture_tenant_serial_console || true
1194+
8891195
# (b) In-guest Talos kernel and kubelet logs. The tenant chart intentionally
8901196
# has no admin talosconfig, so mint a one-hour os:reader client from its
8911197
# existing cert-manager Issuer and run talosctl from a hardened Pod that can
@@ -1484,6 +1790,22 @@ EOF
14841790
exit 1
14851791
fi
14861792

1793+
# Last, after everything the suite exists to prove. This checks a debugging
1794+
# aid, not the product: a worker boots and serves either way, so letting it
1795+
# run first would let a KubeVirt-side change to a diagnostic preempt the
1796+
# assertions that actually cover the tenant cluster. It still runs on the
1797+
# passing path rather than beside the collector, because the collector only
1798+
# runs when node-join fails and would learn the setting was inert in the one
1799+
# run where the console can no longer be recovered. Exit 2 alone fails the
1800+
# suite; see cozy_assert_guest_console_attached for why its three outcomes
1801+
# are not interchangeable.
1802+
echo "» verifying the tenant worker guest console container attached"
1803+
attach_rc=0
1804+
cozy_assert_guest_console_attached || attach_rc=$?
1805+
if [ "${attach_rc}" -eq 2 ]; then
1806+
exit 1
1807+
fi
1808+
14871809
# Success: disarm the tenant-snapshot trap so it doesn't fire on the clean exit.
14881810
trap - EXIT
14891811
# Clean up: delete the test-scoped tenant API LoadBalancer (frees its MetalLB

0 commit comments

Comments
 (0)