You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e95dee2
Browse filesBrowse the repository at this point in the historyBrowse files
test(e2e): capture the tenant worker guest serial console
The tenant Kubernetes suites fail the node-Ready budget with zero
Nodes registered, and the time is lost inside the guest between the
worker VMI reaching Running and Talos requesting its apid
certificate. Nothing collected today covers that window: the existing
in-guest capture drives talosctl, so it needs the very apid the
worker never reached, and on the management side there is neither a
Node nor a certificate request to read.
Turn on the guest console log for the suite's node group and collect
each virt-launcher's guest-console-log container from the management
cluster. Reads start at the beginning of the stream rather than
tailing it, so a guest that repaints its console cannot push the boot
output out of the window; every read is bounded, and the walk stops
at a cap that records both counts, so the collector cannot become the
unbounded term in a failure path that already runs several others
under one deadline.
A read that returns no console is not filed as a quiet guest. Three
outcomes get three labels, because only one of them says anything
about the guest: the read failed and returned nothing, the read
succeeded and returned nothing, or it broke off part way. Each read's
stderr is kept beside the capture rather than inside it, since
kubectl's own "container is not valid for pod" would otherwise fill
the file and a console that never started would read as one that had
spoken. That case is the headline one here, so the Pod state and
events that explain it are collected alongside, once for the whole
selector.
Enabling the console overrides a cluster-wide setting the platform
holds for an open upstream defect, so the run is also an experiment,
and its own failure mode is the one that hides best: the container
wedges virt-launcher in Init, no worker boots, and the suite reports
"fewer than 2 tenant nodes Ready" -- the same sentence the failure
under study produces. Read as the known signature, the experiment's
answer would be filed as the flake it was meant to explain. So the
failure path names that case first, before any other diagnostic, and
only when it is true.
The passing path checks separately that the container attached at
all, and reports the ratio it saw rather than a bare success, since
the check is deliberately satisfied by any one Pod. It keeps three
outcomes apart too: a read that did not answer, a selector that
matched no Pod, and Pods that matched and carry nothing. It reads the
Pod name alongside the container names so the last two cannot
collapse, and both container lists so a future move of the container
cannot fail a run over a diagnostic. It runs last, after everything
the suite exists to prove, because a debugging aid must not preempt
the assertions that cover the cluster.
The capture runs before the talosctl one because it is the only of
the two that can describe a worker which never reached apid, the
shape this failure keeps taking. kubectl reports a warning and an
empty result alike, on stderr with a zero status, so the exit status
decides which artifact its output lands in.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
echo"=== guest-console-log did not start on the workers below: this is the console experiment failing, NOT the node-join failure it instruments (kubevirt/kubevirt#15989) ==="
580
+
echo"=== the cluster-wide disableSerialConsoleLog in packages/system/kubevirt exists for this; drop logSerialConsole from the e2e node group and the suite returns to its prior behaviour ==="
581
+
printf'%s\n'"${rows}"| grep '^false '||true
582
+
}
583
+
584
+
# Check that KubeVirt actually attached the guest console container, on the
585
+
# path where the workers did boot. The capture below runs only when node-join
586
+
# fails, so without this the suite would first learn that the node group's
587
+
# logSerialConsole never took effect in the run that needed the console, when
588
+
# it can no longer be recovered. The platform disables console logging
589
+
# cluster-wide (see packages/system/kubevirt), so what this asks is whether
590
+
# the per-VM override beat that on the KubeVirt version actually shipping.
591
+
#
592
+
# The read's own status is kept separate from the answer it returns, and the
593
+
# two get different exit codes, because they are different claims. Piping
594
+
# kubectl into grep would collapse them: a read that failed produces no output,
595
+
# grep finds nothing, and an API blip would be reported as "the override did
596
+
# not take effect" -- a cause that was never established, sending the next
597
+
# reader at KubeVirt config instead of at the apiserver. A selector that
598
+
# matched no Pod at all is the same trap one step further on, and is kept
599
+
# apart from a Pod that matched and carries nothing -- the second is the
600
+
# finding, the first is not a statement about any Pod.
601
+
#
602
+
# What it establishes is that SOME virt-launcher Pod in the namespace carries
603
+
# the container, not that every one does. That is the direction worth erring
604
+
# in for a check whose only job is to catch an inert setting: it can miss, but
605
+
# it cannot fail a run over a Pod that is not this test's. The match runs over
606
+
# the whole name=containers line, so a Pod named after the container would
607
+
# satisfy it -- these names come from the Machine name, so that cannot happen
608
+
# here, but a reader changing the format should know the glob spans both.
609
+
#
610
+
# 0 a Pod carries the container
611
+
# 1 the read did not answer, or matched no Pod; not evidence either way
612
+
# 2 Pods matched and none carries the container
613
+
cozy_assert_guest_console_attached() {
614
+
local init_names read_rc=0
615
+
local attached total
616
+
617
+
# The Pod name is in the output for a reason that is not cosmetic: without
618
+
# it, a Pod whose initContainers list is absent emits a bare newline, command
619
+
# substitution strips it, and "Pods matched, none carries the container" is
620
+
# byte-identical to "no Pod matched". Those two must not collapse -- the
621
+
# first is the finding this whole check exists for, and on these workers it
622
+
# is also the only shape it can take, since guest-console-log is the only
623
+
# init container a DataVolume-booted virt-launcher Pod ever has.
624
+
init_names=$(timeout -k 5 30 kubectl -n tenant-test get pods \
0 commit comments