Skip to content

Commit 832e85b

Browse files
fix(e2e): keep chainsaw asserts polling through transient API errors (#4867)
## What this PR does Chainsaw asserts in the e2e suites can fail long before their timeout. Chainsaw v0.2.15 keeps polling only on NotFound and stops on any other API error. When etcd changes its leader, kube-apiserver answers the GET with a 500, and whichever suite happens to be polling goes red. Between 2026-09-27 and 10-11 this hit 6 of the 397 e2e attempts that reached chainsaw, which is 6 of the 23 assert failures in that window. Each one has `etcdserver: leader changed` in the apiserver logs within 0.2 s of the failure. Five of the six came after 10-09, and four of them were the computeplane ingress-nginx assert. The fix is kyverno/chainsaw#2761, which is still open and not in any release. Until it is released, the e2e sandbox image builds chainsaw v0.2.15 from source with that PR's commit applied unchanged, using the same Go minor and build flags as the v0.2.15 release binary. The build runs the tests of the patched packages. It also runs chainsaw against a fake apiserver that answers 500: a short run of 500s must not fail the assert, and an endless one must fail it at its timeout. Plain v0.2.15 fails both cases after the first GET. The Dockerfile header lists what to remove once a chainsaw release carries the fix. The README sentence added here has to go too, and so does the Renovate rule that keeps the builder on Go 1.26. The upstream PR also breaks one chainsaw test outside the packages it touches, `Test_runE` in `pkg/commands/assert`. Its expected string lacks the new `; operation terminated with error: context deadline exceeded` suffix. I found it with a full `go test ./...` on v0.2.15 with and without the patch. The image build tests only the patched packages, so that test does not run here. One trade-off: when the API stays broken, an assert or `error` step now runs to its timeout instead of failing within seconds, so a red run caused by a dead API gets longer. The proxmox e2e lane runs the chainsaw installed on its runner, not the sandbox image, so it does not get this fix. Known gaps, left for later: the new test covers only an assert on a named object, not one that lists objects or an `error` operation, and its skip guard does not notice if no test runs at all. The patch file has no `-whitespace` entry in `.gitattributes`, so `git diff --check` reports whitespace inside the upstream patch. A multi-platform build of the image would run the builder tests once per platform; the image is built for amd64 only. Closes #4720 ### Screenshots Not a UI change. ### Downstream repositories None of them builds or uses the e2e sandbox image; the change stays inside this repo's CI. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * End-to-end sandbox assertions now retry after transient Kubernetes API errors instead of failing immediately. Persistent errors still cause assertions to fail when their timeout expires. * **Testing** * Added checks that verify assertions recover after temporary API errors and fail within the expected timeout when errors persist. * **Maintenance** * Updated sandbox image build and dependency update settings for the Chainsaw testing tool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: cfa9c5a
2 parents b3a6c17 + df75adc commit 832e85b

6 files changed

Lines changed: 500 additions & 5 deletions

File tree

‎.github/renovate.json‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -85,6 +85,21 @@
8585
"matchFileNames": ["packages/system/cilium/images/cilium-operator/**"],
8686
"enabled": false
8787
},
88+
{
89+
"description": "Split golang patch from minor updates for the e2e sandbox builder, so the rule below can drop the minor bump without taking the patch bump with it. By default both land in one non-major bucket and only the newest release is proposed.",
90+
"matchManagers": ["dockerfile"],
91+
"matchFileNames": ["packages/core/testing/images/e2e-sandbox/**"],
92+
"matchPackageNames": ["golang"],
93+
"separateMinorPatch": true
94+
},
95+
{
96+
"description": "The e2e sandbox builds a patched chainsaw release with the Go minor upstream shipped that release with. A golang minor or major bump builds it with a toolchain upstream never shipped it with. Patch and digest updates of the same minor still arrive; move the minor by hand together with CHAINSAW_VERSION.",
97+
"matchManagers": ["dockerfile"],
98+
"matchFileNames": ["packages/core/testing/images/e2e-sandbox/**"],
99+
"matchPackageNames": ["golang"],
100+
"matchUpdateTypes": ["major", "minor"],
101+
"enabled": false
102+
},
88103
{
89104
"description": "The seaweedfs rebuild compiles one upstream release with the Go toolchain upstream's docker/Dockerfile.go_build uses at that tag, lays it over the upstream image of that tag, and `make update` in packages/system/seaweedfs re-pins the builder, the base image digest and the source checksum from there. A Renovate bump of any of them builds a combination upstream never shipped, and the next `make update` silently reverts it. Updates are disabled here; all of them move with the SeaweedFS version.",
90105
"matchManagers": ["dockerfile"],

‎hack/e2e-chainsaw/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,4 +81,4 @@ chainsaw test postgres/
8181

8282
## CI integration
8383

84-
The `e2e` job in `.github/workflows/pull-requests.yaml` runs `make -C packages/core/testing test-chainsaw` (which execs `hack/e2e-chainsaw-run.sh` inside the e2e sandbox) and uploads `chainsaw-report.xml`. The runner calls `chainsaw test` once per suite directory, fails a suite whose pods outlive its cleanup by more than the default `timeouts.cleanup` (`LEAKED-PODS suite=<dir>` in the log; see `docs/agents/e2e-testing.md` §8 and `_lib/leak-allowlist.txt` for known, filed leaks), and merges the per-suite reports into that one file. A local `chainsaw test` as shown above runs the same tests without the leak check. The sandbox image (`packages/core/testing/images/e2e-sandbox/Dockerfile`) ships the `chainsaw` binary alongside `kubectl`/`helm`/`mc`/`jq`/`nc`.
84+
The `e2e` job in `.github/workflows/pull-requests.yaml` runs `make -C packages/core/testing test-chainsaw` (which execs `hack/e2e-chainsaw-run.sh` inside the e2e sandbox) and uploads `chainsaw-report.xml`. The runner calls `chainsaw test` once per suite directory, fails a suite whose pods outlive its cleanup by more than the default `timeouts.cleanup` (`LEAKED-PODS suite=<dir>` in the log; see `docs/agents/e2e-testing.md` §8 and `_lib/leak-allowlist.txt` for known, filed leaks), and merges the per-suite reports into that one file. A local `chainsaw test` as shown above runs the same tests without the leak check. The sandbox image (`packages/core/testing/images/e2e-sandbox/Dockerfile`) ships the `chainsaw` binary alongside `kubectl`/`helm`/`mc`/`jq`/`nc`, built from source with a patch that keeps an assert polling through transient API errors such as an etcd leader change; a released `chainsaw` run locally fails the assert on the first such error instead.

‎packages/core/testing/images/e2e-sandbox/Dockerfile‎

Lines changed: 52 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,61 @@
1+
# Chainsaw is built from source because v0.2.15 ends an assert on the first API
2+
# error other than NotFound, so an etcd leader change during a poll fails the
3+
# step long before its timeout. The patch keeps polling on transient server
4+
# errors. Go back to the release tarball, dropping this stage, patches/ and
5+
# chainsaw-transient/, once a chainsaw release carries kyverno/chainsaw#2761.
6+
# Go 1.26 and -trimpath follow the v0.2.15 release binary (go1.26.0); a newer
7+
# Go minor is not what that release was tested with.
8+
FROM --platform=$BUILDPLATFORM golang:1.26.8@sha256:3c3e25a4da13fd0478eed2df1eb35a0e667094a7124d3993a6a1d30f71c17e79 AS chainsaw
9+
ARG GOPROXY
10+
ARG CHAINSAW_VERSION=0.2.15
11+
ARG CHAINSAW_SOURCE_SHA256=04f3fc7dd8e958e4fc7d73383bc22b05827240dd2fe7a8e32ee99d8c0332da2a
12+
ARG TARGETOS
13+
ARG TARGETARCH
14+
ARG BUILDOS
15+
ARG BUILDARCH
16+
17+
WORKDIR /workspace
18+
19+
RUN curl -fsSL "https://github.com/kyverno/chainsaw/archive/refs/tags/v${CHAINSAW_VERSION}.tar.gz" -o /tmp/chainsaw.tar.gz \
20+
&& echo "${CHAINSAW_SOURCE_SHA256} /tmp/chainsaw.tar.gz" | sha256sum -c - \
21+
&& tar -xzf /tmp/chainsaw.tar.gz --strip=1 \
22+
&& rm -f /tmp/chainsaw.tar.gz
23+
24+
COPY patches /patches
25+
RUN git apply /patches/*.patch
26+
27+
# go build never compiles a _test.go, so nothing else runs the tests of the
28+
# packages the patches touch. The list is read into a variable first: a for
29+
# loop drops the exit status of a command substitution.
30+
RUN files=$(git apply --numstat /patches/*.patch) || exit 1; \
31+
for dir in $(printf '%s\n' "$files" | cut -f3 | grep '\.go$' | xargs -r -n1 dirname | sort -u); do \
32+
(cd "$dir" && go test .) || exit 1; \
33+
done
34+
35+
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -trimpath \
36+
-ldflags "-s -w -X github.com/kyverno/chainsaw/pkg/version.BuildVersion=${CHAINSAW_VERSION}" \
37+
-o /chainsaw .
38+
39+
# The test executes chainsaw, so a cross build tests a host build of the same
40+
# patched tree instead of the shipped binary. The tests skip without CHAINSAW,
41+
# and a skip reports ok, so a skip fails the step here.
42+
COPY chainsaw-transient /chainsaw-transient
43+
RUN if [ "$TARGETOS/$TARGETARCH" = "$BUILDOS/$BUILDARCH" ]; then \
44+
bin=/chainsaw; \
45+
else \
46+
bin=/tmp/chainsaw-host && CGO_ENABLED=0 go build -o "$bin" . || exit 1; \
47+
fi; \
48+
cd /chainsaw-transient && out=$(CHAINSAW="$bin" go test -count=1 -v .) || { printf '%s\n' "$out"; exit 1; }; \
49+
printf '%s\n' "$out"; \
50+
if printf '%s\n' "$out" | grep -q -- '--- SKIP'; then echo "chainsaw-transient skipped its tests" >&2; exit 1; fi
51+
152
FROM ubuntu:24.04@sha256:008173c23f95b170204355c12626cb5a965d779a7e1283b09e9cffbb1bf33ca3
253

354
ARG KUBECTL_VERSION=1.33.13
455
ARG TALOSCTL_VERSION=1.13.5
556
ARG HELM_VERSION=3.21.2
657
ARG COZYHR_VERSION=1.6.1
758
ARG CRUST_GATHER_VERSION=v0.15.2
8-
ARG CHAINSAW_VERSION=0.2.15
959
ARG MC_VERSION=RELEASE.2025-08-13T08-35-41Z
1060
ARG FLUX_VERSION=2.8.6
1161

@@ -41,9 +91,7 @@ RUN case "${TARGETARCH}" in \
4191
# `crust-gather serve`-able from the cozyreport artifact. Installs a static musl
4292
# binary as crust-gather.
4393
RUN curl -sSfL "https://github.com/crust-gather/crust-gather/raw/main/install.sh" | sh -s -- --version "${CRUST_GATHER_VERSION}" --bin-dir /usr/local/bin --yes
44-
RUN curl -sSL "https://github.com/kyverno/chainsaw/releases/download/v${CHAINSAW_VERSION}/chainsaw_${TARGETOS}_${TARGETARCH}.tar.gz" \
45-
| tar -xz -C /usr/local/bin chainsaw \
46-
&& chmod +x /usr/local/bin/chainsaw
94+
COPY --from=chainsaw /chainsaw /usr/local/bin/chainsaw
4795
RUN <<'EOF'
4896
cat <<'EOT' >> /etc/bash.bashrc
4997
. /etc/bash_completion
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
module github.com/cozystack/cozystack/e2e-sandbox/chainsaw-transient
2+
3+
go 1.26
Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
// Package transient runs a chainsaw built from the e2e sandbox's patched tree
2+
// against a fake apiserver that answers the asserted object's GET with
3+
// HTTP 500 and an empty reason, which is what kube-apiserver sends for an etcd
4+
// error such as `etcdserver: leader changed`.
5+
// Release v0.2.15 ends an assert on the first such answer; the image carries
6+
// a patch that keeps it polling until its own timeout.
7+
package transient
8+
9+
import (
10+
"encoding/json"
11+
"fmt"
12+
"net/http"
13+
"net/http/httptest"
14+
"os"
15+
"os/exec"
16+
"path/filepath"
17+
"sync/atomic"
18+
"testing"
19+
"time"
20+
)
21+
22+
const assertTimeout = 10 * time.Second
23+
24+
func writeJSON(w http.ResponseWriter, code int, v any) {
25+
w.Header().Set("Content-Type", "application/json")
26+
w.WriteHeader(code)
27+
_ = json.NewEncoder(w).Encode(v)
28+
}
29+
30+
func status(code int, reason, message string) map[string]any {
31+
return map[string]any{
32+
"kind": "Status", "apiVersion": "v1", "status": "Failure",
33+
"code": code, "reason": reason, "message": message,
34+
}
35+
}
36+
37+
// fakeAPIServer serves just enough discovery for chainsaw to resolve
38+
// v1/ConfigMap, and answers GET default/probe with 500 for the first
39+
// `failures` requests (all of them when failures < 0), then with the object.
40+
func fakeAPIServer(t *testing.T, failures int64) (*httptest.Server, *atomic.Int64) {
41+
t.Helper()
42+
var gets atomic.Int64
43+
mux := http.NewServeMux()
44+
mux.HandleFunc("/version", func(w http.ResponseWriter, _ *http.Request) {
45+
writeJSON(w, http.StatusOK, map[string]string{"major": "1", "minor": "33", "gitVersion": "v1.33.0"})
46+
})
47+
mux.HandleFunc("/api", func(w http.ResponseWriter, _ *http.Request) {
48+
writeJSON(w, http.StatusOK, map[string]any{
49+
"kind": "APIVersions", "versions": []string{"v1"},
50+
"serverAddressByClientCIDRs": []map[string]string{{"clientCIDR": "0.0.0.0/0", "serverAddress": "127.0.0.1"}},
51+
})
52+
})
53+
mux.HandleFunc("/apis", func(w http.ResponseWriter, _ *http.Request) {
54+
writeJSON(w, http.StatusOK, map[string]any{"kind": "APIGroupList", "apiVersion": "v1", "groups": []any{}})
55+
})
56+
mux.HandleFunc("/api/v1", func(w http.ResponseWriter, _ *http.Request) {
57+
verbs := []string{"create", "delete", "get", "list", "patch", "update", "watch"}
58+
writeJSON(w, http.StatusOK, map[string]any{
59+
"kind": "APIResourceList", "apiVersion": "v1", "groupVersion": "v1",
60+
"resources": []map[string]any{
61+
{"name": "namespaces", "singularName": "namespace", "namespaced": false, "kind": "Namespace", "verbs": verbs},
62+
{"name": "configmaps", "singularName": "configmap", "namespaced": true, "kind": "ConfigMap", "verbs": verbs},
63+
},
64+
})
65+
})
66+
mux.HandleFunc("/api/v1/namespaces/default", func(w http.ResponseWriter, _ *http.Request) {
67+
writeJSON(w, http.StatusOK, map[string]any{
68+
"kind": "Namespace", "apiVersion": "v1",
69+
"metadata": map[string]any{"name": "default"},
70+
"status": map[string]any{"phase": "Active"},
71+
})
72+
})
73+
mux.HandleFunc("/api/v1/namespaces/default/configmaps/probe", func(w http.ResponseWriter, r *http.Request) {
74+
if r.Method != http.MethodGet {
75+
writeJSON(w, http.StatusMethodNotAllowed, status(http.StatusMethodNotAllowed, "MethodNotAllowed", r.Method))
76+
return
77+
}
78+
n := gets.Add(1)
79+
if failures < 0 || n <= failures {
80+
writeJSON(w, http.StatusInternalServerError, status(http.StatusInternalServerError, "", "etcdserver: leader changed"))
81+
return
82+
}
83+
writeJSON(w, http.StatusOK, map[string]any{
84+
"kind": "ConfigMap", "apiVersion": "v1",
85+
"metadata": map[string]any{"name": "probe", "namespace": "default"},
86+
"data": map[string]string{"ready": "true"},
87+
})
88+
})
89+
mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
90+
writeJSON(w, http.StatusNotFound, status(http.StatusNotFound, "NotFound", r.URL.Path))
91+
})
92+
srv := httptest.NewServer(mux)
93+
t.Cleanup(srv.Close)
94+
return srv, &gets
95+
}
96+
97+
func runChainsaw(t *testing.T, server string) (time.Duration, string, error) {
98+
t.Helper()
99+
bin := os.Getenv("CHAINSAW")
100+
// The image build sets CHAINSAW; `make go-module-tests` reaches this module
101+
// too, with no patched binary to test.
102+
if bin == "" {
103+
t.Skip("CHAINSAW is not set to the chainsaw binary under test")
104+
}
105+
dir := t.TempDir()
106+
kubeconfig := filepath.Join(dir, "kubeconfig")
107+
if err := os.WriteFile(kubeconfig, []byte(fmt.Sprintf(`apiVersion: v1
108+
kind: Config
109+
clusters:
110+
- name: fake
111+
cluster:
112+
server: %s
113+
contexts:
114+
- name: fake
115+
context:
116+
cluster: fake
117+
user: fake
118+
current-context: fake
119+
users:
120+
- name: fake
121+
user: {}
122+
`, server)), 0o600); err != nil {
123+
t.Fatal(err)
124+
}
125+
suite := filepath.Join(dir, "suite")
126+
if err := os.Mkdir(suite, 0o755); err != nil {
127+
t.Fatal(err)
128+
}
129+
if err := os.WriteFile(filepath.Join(suite, "chainsaw-test.yaml"), []byte(fmt.Sprintf(`apiVersion: chainsaw.kyverno.io/v1alpha1
130+
kind: Test
131+
metadata:
132+
name: transient
133+
spec:
134+
namespace: default
135+
skipDelete: true
136+
steps:
137+
- try:
138+
- assert:
139+
timeout: %s
140+
resource:
141+
apiVersion: v1
142+
kind: ConfigMap
143+
metadata:
144+
name: probe
145+
namespace: default
146+
data:
147+
ready: "true"
148+
`, assertTimeout)), 0o600); err != nil {
149+
t.Fatal(err)
150+
}
151+
cmd := exec.Command(bin, "test", "--test-dir", suite, "--no-color")
152+
cmd.Env = append(os.Environ(), "KUBECONFIG="+kubeconfig)
153+
start := time.Now()
154+
out, err := cmd.CombinedOutput()
155+
return time.Since(start), string(out), err
156+
}
157+
158+
// A run of 500s shorter than the timeout must not fail the assert: the poll
159+
// carries on and passes once the object comes back.
160+
func TestAssertSurvivesTransientServerErrors(t *testing.T) {
161+
srv, gets := fakeAPIServer(t, 5)
162+
elapsed, out, err := runChainsaw(t, srv.URL)
163+
if err != nil {
164+
t.Fatalf("chainsaw failed after %s and %d GETs of the object, want a pass once the 500s stop:\n%s", elapsed, gets.Load(), out)
165+
}
166+
if gets.Load() <= 5 {
167+
t.Fatalf("chainsaw passed after only %d GETs of the object, so it never saw the 500s:\n%s", gets.Load(), out)
168+
}
169+
}
170+
171+
// When the 500s never stop, the assert still fails, and it fails by its own
172+
// timeout rather than on the first error.
173+
func TestAssertFailsAtItsTimeoutOnPersistentServerErrors(t *testing.T) {
174+
srv, gets := fakeAPIServer(t, -1)
175+
elapsed, out, err := runChainsaw(t, srv.URL)
176+
if err == nil {
177+
t.Fatalf("chainsaw passed against an apiserver that only answers 500:\n%s", out)
178+
}
179+
if elapsed < assertTimeout || gets.Load() < 2 {
180+
t.Fatalf("chainsaw failed after %s and %d GETs of the object, want the %s assert timeout and more than one GET:\n%s", elapsed, gets.Load(), assertTimeout, out)
181+
}
182+
}

0 commit comments

Comments
 (0)