Skip to content

Commit 813c618

Browse files
committed
fix(e2e): keep chainsaw asserts polling through transient API errors
Chainsaw v0.2.15 ends an assert or error operation on the first API error other than NotFound. kube-apiserver answers a GET with 500 when etcd changes its leader, so a leader change that lands inside a poll fails the step long before its own timeout, on whichever suite happens to be polling. The sandbox image now builds v0.2.15 from source with the fix proposed upstream in kyverno/chainsaw#2761, applied unchanged, until a chainsaw release carries it. The build runs the tests of the patched packages and a test that drives chainsaw built from the patched tree against a fake apiserver: a run of 500s must not fail an assert, and a persistent one must fail it at its timeout rather than at the first error. Closes: #4720 Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 1d813c9 commit 813c618

6 files changed

Lines changed: 500 additions & 5 deletions

File tree

‎.github/renovate.json‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,21 @@
8585
"matchFileNames": ["packages/system/cilium/images/cilium-operator/**"],
8686
"enabled": false
8787
},
88+
{
89+
"description": "Split golang patch from minor updates for the e2e sandbox builder, so the rule below can drop the minor bump without taking the patch bump with it. By default both land in one non-major bucket and only the newest release is proposed.",
90+
"matchManagers": ["dockerfile"],
91+
"matchFileNames": ["packages/core/testing/images/e2e-sandbox/**"],
92+
"matchPackageNames": ["golang"],
93+
"separateMinorPatch": true
94+
},
95+
{
96+
"description": "The e2e sandbox builds a patched chainsaw release with the Go minor upstream shipped that release with. A golang minor or major bump builds it with a toolchain upstream never shipped it with. Patch and digest updates of the same minor still arrive; move the minor by hand together with CHAINSAW_VERSION.",
97+
"matchManagers": ["dockerfile"],
98+
"matchFileNames": ["packages/core/testing/images/e2e-sandbox/**"],
99+
"matchPackageNames": ["golang"],
100+
"matchUpdateTypes": ["major", "minor"],
101+
"enabled": false
102+
},
88103
{
89104
"description": "kubevirt.io/csi-driver is pinned to an untagged pseudo-version commit and publishes no release tags, so Renovate's go datasource returns no-result and the lookup failure lingers on the Dependency Dashboard. Disable it; this module is advanced deliberately alongside the kubevirt-csi-driver image build.",
90105
"matchManagers": ["gomod"],

‎hack/e2e-chainsaw/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,4 +73,4 @@ chainsaw test postgres/
7373

7474
## CI integration
7575

76-
The `e2e` job in `.github/workflows/pull-requests.yaml` runs `make -C packages/core/testing test-chainsaw` (which execs `hack/e2e-chainsaw-run.sh` inside the e2e sandbox) and uploads `chainsaw-report.xml`. The runner calls `chainsaw test` once per suite directory, fails a suite whose pods outlive its cleanup by more than the default `timeouts.cleanup` (`LEAKED-PODS suite=<dir>` in the log; see `docs/agents/e2e-testing.md` §8 and `_lib/leak-allowlist.txt` for known, filed leaks), and merges the per-suite reports into that one file. A local `chainsaw test` as shown above runs the same tests without the leak check. The sandbox image (`packages/core/testing/images/e2e-sandbox/Dockerfile`) ships the `chainsaw` binary alongside `kubectl`/`helm`/`mc`/`jq`/`nc`.
76+
The `e2e` job in `.github/workflows/pull-requests.yaml` runs `make -C packages/core/testing test-chainsaw` (which execs `hack/e2e-chainsaw-run.sh` inside the e2e sandbox) and uploads `chainsaw-report.xml`. The runner calls `chainsaw test` once per suite directory, fails a suite whose pods outlive its cleanup by more than the default `timeouts.cleanup` (`LEAKED-PODS suite=<dir>` in the log; see `docs/agents/e2e-testing.md` §8 and `_lib/leak-allowlist.txt` for known, filed leaks), and merges the per-suite reports into that one file. A local `chainsaw test` as shown above runs the same tests without the leak check. The sandbox image (`packages/core/testing/images/e2e-sandbox/Dockerfile`) ships the `chainsaw` binary alongside `kubectl`/`helm`/`mc`/`jq`/`nc`, built from source with a patch that keeps an assert polling through transient API errors such as an etcd leader change; a released `chainsaw` run locally fails the assert on the first such error instead.

‎packages/core/testing/images/e2e-sandbox/Dockerfile‎

Lines changed: 52 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,61 @@
1+
# Chainsaw is built from source because v0.2.15 ends an assert on the first API
2+
# error other than NotFound, so an etcd leader change during a poll fails the
3+
# step long before its timeout. The patch keeps polling on transient server
4+
# errors. Go back to the release tarball, dropping this stage, patches/ and
5+
# chainsaw-transient/, once a chainsaw release carries kyverno/chainsaw#2761.
6+
# Go 1.26 and -trimpath follow the v0.2.15 release binary (go1.26.0); a newer
7+
# Go minor is not what that release was tested with.
8+
FROM --platform=$BUILDPLATFORM golang:1.26.8@sha256:3c3e25a4da13fd0478eed2df1eb35a0e667094a7124d3993a6a1d30f71c17e79 AS chainsaw
9+
ARG GOPROXY
10+
ARG CHAINSAW_VERSION=0.2.15
11+
ARG CHAINSAW_SOURCE_SHA256=04f3fc7dd8e958e4fc7d73383bc22b05827240dd2fe7a8e32ee99d8c0332da2a
12+
ARG TARGETOS
13+
ARG TARGETARCH
14+
ARG BUILDOS
15+
ARG BUILDARCH
16+
17+
WORKDIR /workspace
18+
19+
RUN curl -fsSL "https://github.com/kyverno/chainsaw/archive/refs/tags/v${CHAINSAW_VERSION}.tar.gz" -o /tmp/chainsaw.tar.gz \
20+
&& echo "${CHAINSAW_SOURCE_SHA256} /tmp/chainsaw.tar.gz" | sha256sum -c - \
21+
&& tar -xzf /tmp/chainsaw.tar.gz --strip=1 \
22+
&& rm -f /tmp/chainsaw.tar.gz
23+
24+
COPY patches /patches
25+
RUN git apply /patches/*.patch
26+
27+
# go build never compiles a _test.go, so nothing else runs the tests of the
28+
# packages the patches touch. The list is read into a variable first: a for
29+
# loop drops the exit status of a command substitution.
30+
RUN files=$(git apply --numstat /patches/*.patch) || exit 1; \
31+
for dir in $(printf '%s\n' "$files" | cut -f3 | grep '\.go$' | xargs -r -n1 dirname | sort -u); do \
32+
(cd "$dir" && go test .) || exit 1; \
33+
done
34+
35+
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -trimpath \
36+
-ldflags "-s -w -X github.com/kyverno/chainsaw/pkg/version.BuildVersion=${CHAINSAW_VERSION}" \
37+
-o /chainsaw .
38+
39+
# The test executes chainsaw, so a cross build tests a host build of the same
40+
# patched tree instead of the shipped binary. The tests skip without CHAINSAW,
41+
# and a skip reports ok, so a skip fails the step here.
42+
COPY chainsaw-transient /chainsaw-transient
43+
RUN if [ "$TARGETOS/$TARGETARCH" = "$BUILDOS/$BUILDARCH" ]; then \
44+
bin=/chainsaw; \
45+
else \
46+
bin=/tmp/chainsaw-host && CGO_ENABLED=0 go build -o "$bin" . || exit 1; \
47+
fi; \
48+
cd /chainsaw-transient && out=$(CHAINSAW="$bin" go test -count=1 -v .) || { printf '%s\n' "$out"; exit 1; }; \
49+
printf '%s\n' "$out"; \
50+
if printf '%s\n' "$out" | grep -q -- '--- SKIP'; then echo "chainsaw-transient skipped its tests" >&2; exit 1; fi
51+
152
FROM ubuntu:24.04@sha256:008173c23f95b170204355c12626cb5a965d779a7e1283b09e9cffbb1bf33ca3
253

354
ARG KUBECTL_VERSION=1.33.13
455
ARG TALOSCTL_VERSION=1.13.5
556
ARG HELM_VERSION=3.21.2
657
ARG COZYHR_VERSION=1.6.1
758
ARG CRUST_GATHER_VERSION=v0.15.2
8-
ARG CHAINSAW_VERSION=0.2.15
959
ARG MC_VERSION=RELEASE.2025-08-13T08-35-41Z
1060
ARG FLUX_VERSION=2.8.6
1161

@@ -41,9 +91,7 @@ RUN case "${TARGETARCH}" in \
4191
# `crust-gather serve`-able from the cozyreport artifact. Installs a static musl
4292
# binary as crust-gather.
4393
RUN curl -sSfL "https://github.com/crust-gather/crust-gather/raw/main/install.sh" | sh -s -- --version "${CRUST_GATHER_VERSION}" --bin-dir /usr/local/bin --yes
44-
RUN curl -sSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_${TARGETOS}_${TARGETARCH}.tar.gz" \
45-
| tar -xz -C /usr/local/bin chainsaw \
46-
&& chmod +x /usr/local/bin/chainsaw
94+
COPY --from=chainsaw /chainsaw /usr/local/bin/chainsaw
4795
RUN <<'EOF'
4896
cat <<'EOT' >> /etc/bash.bashrc
4997
. /etc/bash_completion
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
module github.com/cozystack/cozystack/e2e-sandbox/chainsaw-transient
2+
3+
go 1.26
Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
// Package transient runs a chainsaw built from the e2e sandbox's patched tree
2+
// against a fake apiserver that answers the asserted object's GET with
3+
// HTTP 500 and an empty reason, which is what kube-apiserver sends for an etcd
4+
// error such as `etcdserver: leader changed`.
5+
// Release v0.2.15 ends an assert on the first such answer; the image carries
6+
// a patch that keeps it polling until its own timeout.
7+
package transient
8+
9+
import (
10+
"encoding/json"
11+
"fmt"
12+
"net/http"
13+
"net/http/httptest"
14+
"os"
15+
"os/exec"
16+
"path/filepath"
17+
"sync/atomic"
18+
"testing"
19+
"time"
20+
)
21+
22+
const assertTimeout = 10 * time.Second
23+
24+
func writeJSON(w http.ResponseWriter, code int, v any) {
25+
w.Header().Set("Content-Type", "application/json")
26+
w.WriteHeader(code)
27+
_ = json.NewEncoder(w).Encode(v)
28+
}
29+
30+
func status(code int, reason, message string) map[string]any {
31+
return map[string]any{
32+
"kind": "Status", "apiVersion": "v1", "status": "Failure",
33+
"code": code, "reason": reason, "message": message,
34+
}
35+
}
36+
37+
// fakeAPIServer serves just enough discovery for chainsaw to resolve
38+
// v1/ConfigMap, and answers GET default/probe with 500 for the first
39+
// `failures` requests (all of them when failures < 0), then with the object.
40+
func fakeAPIServer(t *testing.T, failures int64) (*httptest.Server, *atomic.Int64) {
41+
t.Helper()
42+
var gets atomic.Int64
43+
mux := http.NewServeMux()
44+
mux.HandleFunc("/version", func(w http.ResponseWriter, _ *http.Request) {
45+
writeJSON(w, http.StatusOK, map[string]string{"major": "1", "minor": "33", "gitVersion": "v1.33.0"})
46+
})
47+
mux.HandleFunc("/api", func(w http.ResponseWriter, _ *http.Request) {
48+
writeJSON(w, http.StatusOK, map[string]any{
49+
"kind": "APIVersions", "versions": []string{"v1"},
50+
"serverAddressByClientCIDRs": []map[string]string{{"clientCIDR": "0.0.0.0/0", "serverAddress": "127.0.0.1"}},
51+
})
52+
})
53+
mux.HandleFunc("/apis", func(w http.ResponseWriter, _ *http.Request) {
54+
writeJSON(w, http.StatusOK, map[string]any{"kind": "APIGroupList", "apiVersion": "v1", "groups": []any{}})
55+
})
56+
mux.HandleFunc("/api/v1", func(w http.ResponseWriter, _ *http.Request) {
57+
verbs := []string{"create", "delete", "get", "list", "patch", "update", "watch"}
58+
writeJSON(w, http.StatusOK, map[string]any{
59+
"kind": "APIResourceList", "apiVersion": "v1", "groupVersion": "v1",
60+
"resources": []map[string]any{
61+
{"name": "namespaces", "singularName": "namespace", "namespaced": false, "kind": "Namespace", "verbs": verbs},
62+
{"name": "configmaps", "singularName": "configmap", "namespaced": true, "kind": "ConfigMap", "verbs": verbs},
63+
},
64+
})
65+
})
66+
mux.HandleFunc("/api/v1/namespaces/default", func(w http.ResponseWriter, _ *http.Request) {
67+
writeJSON(w, http.StatusOK, map[string]any{
68+
"kind": "Namespace", "apiVersion": "v1",
69+
"metadata": map[string]any{"name": "default"},
70+
"status": map[string]any{"phase": "Active"},
71+
})
72+
})
73+
mux.HandleFunc("/api/v1/namespaces/default/configmaps/probe", func(w http.ResponseWriter, r *http.Request) {
74+
if r.Method != http.MethodGet {
75+
writeJSON(w, http.StatusMethodNotAllowed, status(http.StatusMethodNotAllowed, "MethodNotAllowed", r.Method))
76+
return
77+
}
78+
n := gets.Add(1)
79+
if failures < 0 || n <= failures {
80+
writeJSON(w, http.StatusInternalServerError, status(http.StatusInternalServerError, "", "etcdserver: leader changed"))
81+
return
82+
}
83+
writeJSON(w, http.StatusOK, map[string]any{
84+
"kind": "ConfigMap", "apiVersion": "v1",
85+
"metadata": map[string]any{"name": "probe", "namespace": "default"},
86+
"data": map[string]string{"ready": "true"},
87+
})
88+
})
89+
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
90+
writeJSON(w, http.StatusNotFound, status(http.StatusNotFound, "NotFound", r.URL.Path))
91+
})
92+
srv := httptest.NewServer(mux)
93+
t.Cleanup(srv.Close)
94+
return srv, &gets
95+
}
96+
97+
func runChainsaw(t *testing.T, server string) (time.Duration, string, error) {
98+
t.Helper()
99+
bin := os.Getenv("CHAINSAW")
100+
// The image build sets CHAINSAW; `make go-module-tests` reaches this module
101+
// too, with no patched binary to test.
102+
if bin == "" {
103+
t.Skip("CHAINSAW is not set to the chainsaw binary under test")
104+
}
105+
dir := t.TempDir()
106+
kubeconfig := filepath.Join(dir, "kubeconfig")
107+
if err := os.WriteFile(kubeconfig, []byte(fmt.Sprintf(`apiVersion: v1
108+
kind: Config
109+
clusters:
110+
- name: fake
111+
cluster:
112+
server: %s
113+
contexts:
114+
- name: fake
115+
context:
116+
cluster: fake
117+
user: fake
118+
current-context: fake
119+
users:
120+
- name: fake
121+
user: {}
122+
`, server)), 0o600); err != nil {
123+
t.Fatal(err)
124+
}
125+
suite := filepath.Join(dir, "suite")
126+
if err := os.Mkdir(suite, 0o755); err != nil {
127+
t.Fatal(err)
128+
}
129+
if err := os.WriteFile(filepath.Join(suite, "chainsaw-test.yaml"), []byte(fmt.Sprintf(`apiVersion: chainsaw.kyverno.io/v1alpha1
130+
kind: Test
131+
metadata:
132+
name: transient
133+
spec:
134+
namespace: default
135+
skipDelete: true
136+
steps:
137+
- try:
138+
- assert:
139+
timeout: %s
140+
resource:
141+
apiVersion: v1
142+
kind: ConfigMap
143+
metadata:
144+
name: probe
145+
namespace: default
146+
data:
147+
ready: "true"
148+
`, assertTimeout)), 0o600); err != nil {
149+
t.Fatal(err)
150+
}
151+
cmd := exec.Command(bin, "test", "--test-dir", suite, "--no-color")
152+
cmd.Env = append(os.Environ(), "KUBECONFIG="+kubeconfig)
153+
start := time.Now()
154+
out, err := cmd.CombinedOutput()
155+
return time.Since(start), string(out), err
156+
}
157+
158+
// A run of 500s shorter than the timeout must not fail the assert: the poll
159+
// carries on and passes once the object comes back.
160+
func TestAssertSurvivesTransientServerErrors(t *testing.T) {
161+
srv, gets := fakeAPIServer(t, 5)
162+
elapsed, out, err := runChainsaw(t, srv.URL)
163+
if err != nil {
164+
t.Fatalf("chainsaw failed after %s and %d GETs of the object, want a pass once the 500s stop:\n%s", elapsed, gets.Load(), out)
165+
}
166+
if gets.Load() <= 5 {
167+
t.Fatalf("chainsaw passed after only %d GETs of the object, so it never saw the 500s:\n%s", gets.Load(), out)
168+
}
169+
}
170+
171+
// When the 500s never stop, the assert still fails, and it fails by its own
172+
// timeout rather than on the first error.
173+
func TestAssertFailsAtItsTimeoutOnPersistentServerErrors(t *testing.T) {
174+
srv, gets := fakeAPIServer(t, -1)
175+
elapsed, out, err := runChainsaw(t, srv.URL)
176+
if err == nil {
177+
t.Fatalf("chainsaw passed against an apiserver that only answers 500:\n%s", out)
178+
}
179+
if elapsed < assertTimeout || gets.Load() < 2 {
180+
t.Fatalf("chainsaw failed after %s and %d GETs of the object, want the %s assert timeout and more than one GET:\n%s", elapsed, gets.Load(), assertTimeout, out)
181+
}
182+
}

0 commit comments

Comments
 (0)