fix: assert-polling-error - #2761

…4867)

## What this PR does

Chainsaw asserts in the e2e suites can fail long before their timeout.
Chainsaw v0.2.15 keeps polling only on NotFound and stops on any other
API error. When etcd changes its leader, kube-apiserver answers the GET
with a 500, and whichever suite happens to be polling goes red. Between
2026-09-27 and 10-11 this hit 6 of the 397 e2e attempts that reached
chainsaw, which is 6 of the 23 assert failures in that window. Each one
has `etcdserver: leader changed` in the apiserver logs within 0.2 s of
the failure. Five of the six came after 10-09, and four of them were the
computeplane ingress-nginx assert.

The fix is kyverno/chainsaw#2761, which is still open and not in any
release. Until it is released, the e2e sandbox image builds chainsaw
v0.2.15 from source with that PR's commit applied unchanged, using the
same Go minor and build flags as the v0.2.15 release binary. The build
runs the tests of the patched packages. It also runs chainsaw against a
fake apiserver that answers 500: a short run of 500s must not fail the
assert, and an endless one must fail it at its timeout. Plain v0.2.15
fails both cases after the first GET.

The Dockerfile header lists what to remove once a chainsaw release
carries the fix. The README sentence added here has to go too, and so
does the Renovate rule that keeps the builder on Go 1.26.

The upstream PR also breaks one chainsaw test outside the packages it
touches, `Test_runE` in `pkg/commands/assert`. Its expected string lacks
the new `; operation terminated with error: context deadline exceeded`
suffix. I found it with a full `go test ./...` on v0.2.15 with and
without the patch. The image build tests only the patched packages, so
that test does not run here.

One trade-off: when the API stays broken, an assert or `error` step now
runs to its timeout instead of failing within seconds, so a red run
caused by a dead API gets longer. The proxmox e2e lane runs the chainsaw
installed on its runner, not the sandbox image, so it does not get this
fix.

Known gaps, left for later: the new test covers only an assert on a
named object, not one that lists objects or an `error` operation, and
its skip guard does not notice if no test runs at all. The patch file
has no `-whitespace` entry in `.gitattributes`, so `git diff --check`
reports whitespace inside the upstream patch. A multi-platform build of
the image would run the builder tests once per platform; the image is
built for amd64 only.

Closes #4720

### Screenshots

Not a UI change.

### Downstream repositories

None of them builds or uses the e2e sandbox image; the change stays
inside this repo's CI.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* End-to-end sandbox assertions now retry after transient Kubernetes API
errors instead of failing immediately. Persistent errors still cause
assertions to fail when their timeout expires.
* **Testing**
* Added checks that verify assertions recover after temporary API errors
and fail within the expected timeout when errors persist.
* **Maintenance**
* Updated sandbox image build and dependency update settings for the
Chainsaw testing tool.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Chainsaw v0.2.15 ends an assert or error operation on the first API
error other than NotFound. kube-apiserver answers a GET with 500 when
etcd changes its leader, so a leader change that lands inside a poll
fails the step long before its own timeout, on whichever suite happens
to be polling.

The sandbox image now builds v0.2.15 from source with the fix proposed
upstream in kyverno/chainsaw#2761, applied unchanged, until a chainsaw
release carries it. The build runs the tests of the patched packages
and a test that drives chainsaw built from the patched tree against a
fake apiserver: a run of 500s must not fail an assert, and a
persistent one must fail it at its timeout rather than at the first
error.

Closes: cozystack#4720

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>

Chainsaw v0.2.15 ends an assert or error operation on the first API
error other than NotFound. kube-apiserver answers a GET with 500 when
etcd changes its leader, so a leader change that lands inside a poll
fails the step long before its own timeout, on whichever suite happens
to be polling.

The sandbox image now builds v0.2.15 from source with the fix proposed
upstream in kyverno/chainsaw#2761, applied unchanged, until a chainsaw
release carries it. The build runs the tests of the patched packages
and a test that drives chainsaw built from the patched tree against a
fake apiserver: a run of 500s must not fail an assert, and a
persistent one must fail it at its timeout rather than at the first
error.

Closes: #4720

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <[email protected]>
(cherry picked from commit 813c618)
Signed-off-by: Myasnikov Daniil <[email protected]>

…4867)

## What this PR does

Chainsaw asserts in the e2e suites can fail long before their timeout.
Chainsaw v0.2.15 keeps polling only on NotFound and stops on any other
API error. When etcd changes its leader, kube-apiserver answers the GET
with a 500, and whichever suite happens to be polling goes red. Between
2026-09-27 and 10-11 this hit 6 of the 397 e2e attempts that reached
chainsaw, which is 6 of the 23 assert failures in that window. Each one
has `etcdserver: leader changed` in the apiserver logs within 0.2 s of
the failure. Five of the six came after 10-09, and four of them were the
computeplane ingress-nginx assert.

The fix is kyverno/chainsaw#2761, which is still open and not in any
release. Until it is released, the e2e sandbox image builds chainsaw
v0.2.15 from source with that PR's commit applied unchanged, using the
same Go minor and build flags as the v0.2.15 release binary. The build
runs the tests of the patched packages. It also runs chainsaw against a
fake apiserver that answers 500: a short run of 500s must not fail the
assert, and an endless one must fail it at its timeout. Plain v0.2.15
fails both cases after the first GET.

The Dockerfile header lists what to remove once a chainsaw release
carries the fix. The README sentence added here has to go too, and so
does the Renovate rule that keeps the builder on Go 1.26.

The upstream PR also breaks one chainsaw test outside the packages it
touches, `Test_runE` in `pkg/commands/assert`. Its expected string lacks
the new `; operation terminated with error: context deadline exceeded`
suffix. I found it with a full `go test ./...` on v0.2.15 with and
without the patch. The image build tests only the patched packages, so
that test does not run here.

One trade-off: when the API stays broken, an assert or `error` step now
runs to its timeout instead of failing within seconds, so a red run
caused by a dead API gets longer. The proxmox e2e lane runs the chainsaw
installed on its runner, not the sandbox image, so it does not get this
fix.

Known gaps, left for later: the new test covers only an assert on a
named object, not one that lists objects or an `error` operation, and
its skip guard does not notice if no test runs at all. The patch file
has no `-whitespace` entry in `.gitattributes`, so `git diff --check`
reports whitespace inside the upstream patch. A multi-platform build of
the image would run the builder tests once per platform; the image is
built for amd64 only.

Closes #4720

### Screenshots

Not a UI change.

### Downstream repositories

None of them builds or uses the e2e sandbox image; the change stays
inside this repo's CI.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* End-to-end sandbox assertions now retry after transient Kubernetes API
errors instead of failing immediately. Persistent errors still cause
assertions to fail when their timeout expires.
* **Testing**
* Added checks that verify assertions recover after temporary API errors
and fail within the expected timeout when errors persist.
* **Maintenance**
* Updated sandbox image build and dependency update settings for the
Chainsaw testing tool.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: cfa9c5a