Skip to content

Commit e09b46f

Browse files
test(site-router): run the suite in CI instead of parking it
The suite was parked as chainsaw-test.yaml.disabled because its appliance image was not published. That is no longer true, and two paths supply the ref between them. On a PR touching a vyos path the build-vyos job builds the golden image and stamps the real ref@digest into packages/apps/site-router/images/vyos-router-disk.tag through a patch fragment Finalize merges before E2E runs. On every other PR that job is skipped, and the ref arrives from the base branch instead: the root `build:` recipe builds packages/system/vyos-router-image before packages/core/installer, so the cozystack-packages artifact pushed from main already carries the stamped .tag and finalize's overlay copies it in. The other two reasons the header gave are stale as well -- _probe-lib.sh is the live T13 implementation rather than a set of seams, and the firewall syntax it drives is the one the chart ships. Renaming the file is the whole of the mechanism: the selector enumerates a suite by its chainsaw-test.yaml, so the suite joins TIA selection and every full-suite escalation with nothing else to change. This needed #3817, which is now merged and in this branch through the merge below it. Un-parking makes site-router the first suite to sit downstream of cozystack-basics, and before that PR's propagation-hub exclusion the platform's namespace-and-policy package silently dropped from the full run to this one suite. Three guards pinned that and went red: the two escalation guards on cozystack-basics and the exact-set guard on system/postgres-operator, which reached this suite through the same hub. With the exclusion in the tree, cozystack-basics escalates again and postgres-operator is back to `harbor postgres`. Two CDI guards move from an exact set to membership, for one reason in both directions. kubevirt-cdi provisions the gateway's boot DataVolume, so it reaches this suite as well as vminstance, and its selection is now `site-router vminstance`. #3817's guard pinned `vminstance` alone and this branch's own guard pinned `site-router` alone; each would read the other's correct widening as a regression. What both rules actually owe is that their suite is IN the selection, so that is what they assert. Neither loses its teeth: dropping vminstance from the mapping still fails the first, and stripping site-router from the selector output -- the #3392 shape -- still fails the second. The stale claims inside the test go with the header: the three "TODO(T13): live -- implement ..." notes described _probe-lib.sh helpers that are written, and the port_security step called the VyOS 1.5 firewall syntax provisional after security-model.md recorded it as validated live. What the header now records instead is the one thing the two supply paths do not cover: when neither fires, the committed value is a bare v0.0.0 placeholder that no build path ever publishes, and the bring-up guard only rejects an EMPTY ref, so the run fails as a VM-never-ready timeout rather than naming the cause. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 7d7dccb commit e09b46f

3 files changed

Lines changed: 105 additions & 61 deletions

File tree

‎.github/workflows/pull-requests.yaml‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -548,8 +548,14 @@ jobs:
548548
# digest-pinned KubeVirt containerDisk pushed to the registry, and stamps the
549549
# ref@digest into packages/apps/site-router/images/vyos-router-disk.tag.
550550
# That stamp is captured as a patch fragment so finalize bundles the real digest
551-
# into the installer artifact (the deferred site-router e2e resolves it from
552-
# there). vm-default-images is an opt-in bundle package, so a non-opted-in e2e is
551+
# into the installer artifact, which is where the site-router e2e resolves it
552+
# from. That suite runs in CI, and when this job is skipped the ref comes from
553+
# the base branch instead: `make build` builds this package before
554+
# packages/core/installer, so the cozystack-packages artifact carries the
555+
# stamped .tag and finalize's overlay applies it. Only a base branch that has
556+
# not been built since the package landed leaves the committed placeholder in
557+
# place — see the note in hack/e2e-chainsaw/site-router/chainsaw-test.yaml.
558+
# vm-default-images is an opt-in bundle package, so a non-opted-in e2e is
553559
# unaffected either way.
554560
#
555561
# Path-scoped (plan.outputs.vyos): this heavy, privileged build only runs when

hack/e2e-chainsaw/site-router/chainsaw-test.yaml.disabled renamed to hack/e2e-chainsaw/site-router/chainsaw-test.yaml

Lines changed: 40 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -4,15 +4,33 @@
44
# GATE — the negative-security matrix (every forbidden flow dropped AND attributed
55
# to the guard that dropped it).
66
#
7-
# ── DEFERRED FROM CI ────────────────────────────────────────────────────────
8-
# This suite CANNOT run green in CI yet: the VyOS golden image is committed
9-
# DISABLED (T01) and the VyOS 1.5 firewall syntax is provisional (TODO(T13)). It
10-
# is therefore excluded from TIA selection AND from the full suite by an explicit
11-
# deferral in hack/select-e2e.sh, so the PR that adds it keeps CI green. It runs
12-
# only in the empirical validation, by hand:
7+
# ── RUNS IN CI ──────────────────────────────────────────────────────────────
8+
# The suite is enumerated like every other one (hack/select-e2e.sh finds a suite
9+
# by its chainsaw-test.yaml), so it runs on a site-router change and in every
10+
# full-suite escalation.
11+
#
12+
# The appliance it boots is stamped, not committed. Two paths supply the stamp
13+
# and between them they cover every run. On a PR that touches a vyos path the
14+
# build-vyos job publishes the golden image and writes the real ref@digest into
15+
# packages/apps/site-router/images/vyos-router-disk.tag through a patch fragment
16+
# the Finalize step merges before the E2E job reads it. On every other PR that
17+
# job is skipped (it is path-scoped on plan.outputs.vyos), and the ref instead
18+
# arrives from the base branch: the root `build:` recipe builds
19+
# packages/system/vyos-router-image before packages/core/installer, so the
20+
# cozystack-packages artifact pushed from main already carries the stamped .tag,
21+
# and finalize's overlay copies it onto this tree.
22+
#
23+
# What is NOT covered is legibility when neither path fires — a main build that
24+
# has not run since this package landed, or an overlay that could not pull the
25+
# artifact and logged a no-op. The committed value is then the bare v0.0.0
26+
# placeholder, which no build path in this repo ever publishes, and the guard in
27+
# the bring-up step below only rejects an EMPTY ref, so the run fails as an
28+
# eight-minute VM-never-ready timeout rather than naming the cause. Rejecting a
29+
# ref that carries no @sha256: digest would fix that; it is not done here.
30+
#
31+
# To drive it by hand against a dev cluster:
1332
# make test-chainsaw CHAINSAW_SUITES="site-router"
14-
# with the golden image published (or supplied via an uncommitted image override)
15-
# and the live-only script seams (_probe-lib.sh) implemented.
33+
# which needs the golden image published, or supplied via an image override.
1634
#
1735
# ── ONE Test, not many ──────────────────────────────────────────────────────
1836
# The topology is expensive (two nested KubeVirt VMs + IPsec), so it is brought
@@ -412,9 +430,11 @@ spec:
412430
is live (D8). The pod name is generated, so this selects by the KubeVirt
413431
label and polls (a non-empty result AND value=false — never a vacuous
414432
empty-list pass).
415-
TODO(T13): this depends on the guest source-filter confirm, which depends
416-
on the provisional VyOS 1.5 firewall syntax pushing successfully. Live
417-
validation must also cover the port_security live-toggle question the
433+
This rides on the guest source-filter confirm, and through it on the
434+
VyOS 1.5 firewall syntax pushing successfully — validated live against
435+
the shipped image (packages/apps/site-router/docs/security-model.md),
436+
so no longer provisional.
437+
What is still open is the port_security live-toggle question the
418438
reconciler flags (D8): whether kube-ovn v1.15.10 reconciles the LSP
419439
port_security field from an annotation flip on an already-wired port.
420440
try:
@@ -506,8 +526,8 @@ spec:
506526
From B, reach the backend behind A through the tunnel; the backend's
507527
/clientip must report B's REAL remote source (192.0.2.x), proving routed
508528
mode preserves the source (no NAT). THE routed-mode contract.
509-
TODO(T13): live — implement probe_from_b in _probe-lib.sh (drive B's
510-
guest). Validate source preservation end to end against the live image.
529+
The probe is driven from B's guest over SSH by probe_from_b_stdout
530+
(_probe-lib.sh), so the source the backend reports is B's real one.
511531
try:
512532
- script:
513533
timeout: 5m
@@ -532,8 +552,8 @@ spec:
532552
description: >-
533553
A multi-MB transfer B → backend across the tunnel must complete without
534554
stalling — the TCP MSS clamp (overlay MTU − 40) preventing PMTU black-holes.
535-
TODO(T13): live — implement the bulk transfer from B's guest; a stall
536-
(hang until timeout) is the failure signal.
555+
A stall — the transfer hanging until the step timeout rather than
556+
erroring — is the failure signal.
537557
try:
538558
- script:
539559
timeout: 5m
@@ -558,10 +578,11 @@ spec:
558578
establishing the responsible guard, not merely absence of connectivity.
559579
The one allowed flow (declared remote source → intended same-tenant
560580
destination) must pass with source preserved.
561-
TODO(T13): live — implement probe_from_b / hubble_dropped /
562-
guest_counter_incremented in _probe-lib.sh against the live gateway,
563-
Hubble, and the VyOS management API. The stubs fail loudly, so this GATE
564-
can never pass vacuously.
581+
The guards read the live gateway: guest counters over the VyOS
582+
management API, Cilium drops from the datapath drop monitor
583+
(_probe-lib.sh). Every helper fails loudly when its evidence is absent —
584+
an unreadable counter is not a flat one — so this GATE cannot pass
585+
vacuously.
565586
try:
566587
- script:
567588
timeout: 10m

‎hack/select-e2e_test.bats‎

Lines changed: 57 additions & 40 deletions
Original file line numberDiff line numberDiff line change
@@ -119,15 +119,25 @@ assert_full_suite() {
119119
}
120120

121121
@test "a CDI change selects the suite that exercises a VMDisk" {
122-
# kubevirt-cdi reaches vm-disk-application and nothing else runnable, so
123-
# until vm-disk-application mapped to a suite every CDI change ran all of
124-
# them. The vminstance suite creates a VMDisk and asserts the DataVolume
125-
# behind it, so it is the suite that covers CDI.
122+
# kubevirt-cdi reached no runnable suite until vm-disk-application mapped to
123+
# one, so every CDI change ran all of them. The vminstance suite creates a
124+
# VMDisk and asserts the DataVolume behind it, so it is the suite that
125+
# covers CDI, and this pins that it is selected.
126+
#
127+
# Membership, not equality: what the rule owes is that the VMDisk suite is
128+
# IN the selection, and CDI legitimately reaches more than one suite —
129+
# site-router's gateway boots from a CDI-provisioned DataVolume, so enabling
130+
# that suite widens this to `site-router vminstance`. An exact set would
131+
# read a correct widening as a regression. Dropping vminstance from the
132+
# mapping still fails this, which is what it guards.
126133
tmp=$(mktemp -d)
127134
cp -r packages/core/platform/sources "$tmp/sources"
128135
echo "packages/system/kubevirt-cdi/values.yaml" > "$tmp/diff"
129136
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
130-
[ "$output" = "vminstance" ]
137+
if ! echo "$output" | grep -wq vminstance; then
138+
echo "a CDI change must select the vminstance suite; got: $output" >&2
139+
exit 1
140+
fi
131141
rm -rf "$tmp"
132142
}
133143

@@ -999,18 +1009,17 @@ assert_full_suite() {
9991009
[ "$count" -eq 2 ]
10001010
}
10011011

1002-
@test "site-router is parked, so a change to its app escalates to the full suite" {
1003-
# The site-router suite is authored but cannot run until its appliance image
1004-
# is published, so it is parked as chainsaw-test.yaml.disabled and is not a
1005-
# suite as far as the selector is concerned. Its app therefore reaches no
1006-
# runnable suite and escalates, which is the #3330 rule rather than a
1007-
# special case — and is why parking it must NOT be spelled as a
1008-
# strip-the-name-from-the-output filter. Doing that leaves the walk still
1009-
# reaching the name, so any package whose only coverage ran through
1010-
# site-router (cozystack-basics and kubevirt-cdi both do, via kubevirt-cdi
1011-
# -> site-router-application) resolves to one suite, has it stripped, and
1012-
# selects NOTHING — a full-suite escalation silently turned into a skipped
1013-
# Chainsaw run and a green "E2E Tests" (#3392).
1012+
@test "a change to the site-router app selects its own suite" {
1013+
# The suite is live: it carries a chainsaw-test.yaml, so the selector sees it
1014+
# like any other and a change to the app it covers resolves to it and to
1015+
# nothing else. It must never be parked back by stripping the name from the
1016+
# selector output. Doing that leaves the walk still reaching the name, so any
1017+
# package whose only coverage runs through site-router (cozystack-basics and
1018+
# kubevirt-cdi both do, via kubevirt-cdi -> site-router-application) resolves
1019+
# to one suite, has it stripped, and selects NOTHING — a full-suite
1020+
# escalation silently turned into a skipped Chainsaw run and a green
1021+
# "E2E Tests" (#3392). Park it by renaming the file, the way
1022+
# hack/e2e-chainsaw/backup/ is parked, or not at all.
10141023
#
10151024
# These three clean up at the end of the body rather than from an EXIT trap:
10161025
# a test-level trap displaces the handler the bats binary installs, and a
@@ -1021,47 +1030,55 @@ assert_full_suite() {
10211030
cp -r packages/core/platform/sources "$tmp/sources"
10221031
echo "packages/apps/site-router/values.yaml" > "$tmp/diff"
10231032
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
1024-
assert_full_suite "$output"
1033+
[ "$output" = "site-router" ]
10251034
rm -rf "$tmp"
10261035
}
10271036

1028-
@test "the packages that reach a suite only through site-router still escalate" {
1029-
# The regression guard for the paragraph above, pinned on the two packages
1030-
# that actually have this shape. Both reach site-router-application through
1031-
# kubevirt-cdi and nothing else runnable, so both must run everything rather
1032-
# than selecting the parked suite or, worse, nothing at all.
1037+
@test "a change to kubevirt-cdi reaches the site-router suite" {
1038+
# CDI provisions the gateway's boot DataVolume, so it reaches
1039+
# site-router-application and a CDI change genuinely wants this suite.
1040+
#
1041+
# Membership, not equality. What must hold is that site-router is IN the
1042+
# selection: #3817 maps vm-disk-application onto vminstance, which gives CDI
1043+
# a second suite, and an exact-set assertion here would go red on that
1044+
# correct widening. Stripping the name from the output still fails this,
1045+
# which is the #3392 regression the guard exists for.
1046+
#
1047+
# cozystack-basics has the same shape today and is deliberately NOT pinned
1048+
# here. It is a propagation hub, its escalation is what the two escalation
1049+
# guards above own, and #3817 stops it propagating at all — so asserting a
1050+
# suite for it here would contradict those guards on the far side of that PR.
10331051
tmp=$(mktemp -d)
10341052
cp -r packages/core/platform/sources "$tmp/sources"
1035-
for pkg in packages/system/cozystack-basics/values.yaml \
1036-
packages/system/kubevirt-cdi/values.yaml; do
1037-
echo "$pkg" > "$tmp/diff"
1038-
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
1039-
assert_full_suite "$output"
1040-
done
1053+
echo "packages/system/kubevirt-cdi/values.yaml" > "$tmp/diff"
1054+
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
1055+
if ! echo "$output" | grep -wq site-router; then
1056+
echo "kubevirt-cdi must reach the site-router suite; got: $output" >&2
1057+
exit 1
1058+
fi
10411059
rm -rf "$tmp"
10421060
}
10431061

1044-
@test "editing the parked site-router suite selects nothing" {
1045-
# A *.disabled suite is registered nowhere and executed by nothing, so an
1046-
# edit to it cannot regress a test. This is the existing rule for a parked
1047-
# suite (hack/e2e-chainsaw/backup/ is the other one), not a new one.
1062+
@test "editing the site-router suite selects it" {
1063+
# A live suite is selected by an edit to its own directory, which is the
1064+
# ordinary per-suite rule and the thing parking it took away.
10481065
tmp=$(mktemp -d)
10491066
cp -r packages/core/platform/sources "$tmp/sources"
1050-
echo "hack/e2e-chainsaw/site-router/chainsaw-test.yaml.disabled" > "$tmp/diff"
1067+
echo "hack/e2e-chainsaw/site-router/chainsaw-test.yaml" > "$tmp/diff"
10511068
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
1052-
[ -z "$output" ]
1069+
[ "$output" = "site-router" ]
10531070
rm -rf "$tmp"
10541071
}
10551072

1056-
@test "the parked site-router suite is excluded from the full suite" {
1057-
# The full-suite escalation enumerates chainsaw-test.yaml, so parking the
1058-
# file keeps the suite out of every escalation without a second mechanism.
1073+
@test "the site-router suite is part of the full suite" {
1074+
# The full-suite escalation enumerates chainsaw-test.yaml, so a live suite
1075+
# joins every escalation without a second mechanism.
10591076
tmp=$(mktemp -d)
10601077
cp -r packages/core/platform/sources "$tmp/sources"
10611078
echo "packages/system/cilium/values.yaml" > "$tmp/diff"
10621079
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
1063-
if echo "$output" | grep -wq site-router; then
1064-
echo "site-router is parked and must not appear in the full suite; got: $output" >&2
1080+
if ! echo "$output" | grep -wq site-router; then
1081+
echo "site-router is live and must appear in the full suite; got: $output" >&2
10651082
exit 1
10661083
fi
10671084
rm -rf "$tmp"

0 commit comments

Comments
 (0)