test(e2e): fail the suite whose cert-manager Secrets outlive its cleanup - #4876

What this PR does

Follow-up to #4875 for #4704, and to the "Only pods are checked" limitation #4492 left open. cert-manager runs without --enable-certificate-owner-ref, so the Secret a Certificate writes outlives the Certificate and the release, and a later release of the same name finds and reuses the private key in it. The E2E leak guard only looked at pods, so a suite that deleted its app and left those Secrets behind still passed. This makes it fail.

• Attribution: a Secret is charged to the suite when all three hold after the suite's own cleanup. It carries the cert-manager.io/certificate-name annotation. It was created at or after the suite started, which does for a Secret what the pod UID baseline does, so a Secret an earlier suite left behind is not charged again. The root of its chain was created at or after the suite started too, where a Secret with no owners of its own is walked as if the Certificate the annotation names owned it, and from there up the Certificate's ownerReferences like any pod. A platform Secret re-issued mid-suite therefore traces back to a Certificate the install created and is not charged, while one whose Certificate is gone ends the walk at itself and is. A Secret that has owners, like the ones the Percona operator patches its CR onto, is walked through them.
• Reporting: charged Secrets get the same 300s budget and the same kubectl wait --for=delete as pods, and are reported under their own LEAKED-SECRETS suite=<dir> budget=300s count=N marker and leaked-secrets annotation, with one line per Secret naming its Certificate and root. The pod annotations and markers are unchanged, so a search for LEAKED-PODS still finds only pods; the one pod-facing text that changes is the inconclusive re-list error, which now says "a pod or Secret list". The allowlist stays per suite and covers both kinds, as it already exempted "any new leak" in a listed suite.
• The read is kubectl get secrets --all-namespaces --field-selector type=kubernetes.io/tls, so it does not carry every stored Helm release revision along, and the list is cut to metadata inside the pipe from kubectl, so the keys it holds never reach a shell variable or a here-string. A failed Secret list or Certificate read follows the rules that already apply to a failed pod list or owner read.

Merge after #4875. Reading the charts and suites on main, this guard fails three suites until #4875 is merged, and what it names there is the defect #4704 reports:

• qdrant: qdrant-test-ca, qdrant-test-tls
• redis: redis-secure.ca-tls, redis-secure.tls from the redis-tls test, and redis-flip.ca-tls, redis-flip.tls if cert-manager issues them in time, because in redis-tls-flip-rejected the TLS upgrade that the RedisFailover schema refuses still creates that release's Certificates first
• opensearch: opensearch-test.http-ca, opensearch-test-http-server

Each of them also spends the 300s budget waiting on those Secrets, 15 minutes of the job in all. The other suites that get cert-manager Secrets should stay clear: mariadb, etcd, seaweedfs and kubernetes delete theirs in post-delete hooks, mongodb's are owned by the PerconaServerMongoDB CR, the rabbitmq suite runs with TLS off, nats has no suite, and the ACME issuers never issue in the E2E sandbox. The CA copies made by the backup examples and by TenantProjection are Opaque Secrets without the annotation, so they are not read.

E2E criterion, registered before the run: on this branch as it is, the Run E2E tests log prints LEAKED-SECRETS suite=qdrant budget=300s count=2, LEAKED-SECRETS suite=opensearch budget=300s count=2 and LEAKED-SECRETS suite=redis budget=300s with a count of 2 or 4, each followed by the names above (checked-once= in place of budget=300s for a suite that failed on its own), and no other suite prints a LEAKED-SECRETS line except as allowlisted=, checked-once= or inconclusive=. With #4875 merged into it, no line matches LEAKED-SECRETS suite=.* budget=. Anything else is either a leak nobody listed here or a false positive to understand before merge.

Tests: hack/chainsaw-leak-guard_test.bats gains 15 tests, all against the stub kubectl (47 of 47 pass under Bats 1.14.0, and 47 of 47 through hack/cozytest.sh under dash; the new ones also pass with the jq 1.7 and bash 5.2 of Ubuntu 24.04, which the E2E sandbox image is built on). Every new rule was mutation-checked in a copy of the guard: dropping Secrets from the listing, dropping the creation-time filter, > instead of >= on it, dropping the Certificate as owner, letting the Certificate override a Secret's own owners, dropping the metadata cut, dropping pipefail, dropping the annotation filter, dropping the field selector, waiting on pod/ for a Secret, reporting Secrets under the pod marker, re-listing only pods, and skipping the root check for Secrets. Each one turns at least one named test red. shellcheck reports nothing on the guard and the runner. The Makefile's Bats unit list, run on main and on this branch in the same container, fails the same tests on both, all from that container's environment, and differs only by the 15 new tests.

Not covered here:

• Only Secrets cert-manager wrote are read. A leaked Secret of any other origin, or any other kind of object without node requests, is still not seen.
• harbor and bucket delete their ACME Certificates in their post-delete hooks but not the Secrets those write (<release>-ingress-tls, <release>-ui-tls). The E2E sandbox never completes an ACME order, so no such Secret exists there and this guard cannot see that leak.
• The Secret's own creationTimestamp is compared with the suite start across apiserver clocks, under the same no-skew-margin assumption the pod rule makes for root timestamps.
• kubernetes-latest and kubernetes-previous stay allowlisted for kamaji: deleted TenantControlPlane stays on its finalizer after a stalled reconcile #4489, and the exemption now hides a Secret leak there too.

Screenshots

Not a UI change.

Downstream repositories

• No downstream repository is affected by this change
• cozystack/website - follow-up:
• cozystack/terraform-provider-cozystack - follow-up:
• cozystack/ansible-cozystack - follow-up:
• cozystack/ccp - follow-up:
• cozystack/talm - follow-up:
• cozystack/cozyhr - follow-up:
• cozystack/cozy-proxy - follow-up:
• cozystack/cozystack-telemetry-server - follow-up:
• cozystack/external-apps-example - follow-up:
• cozystack/examples - follow-up:
• cozystack/community - follow-up:

The diff changes files under hack/ and docs/agents/ without moving or renaming any, and touches no make target, values file, CRD or package. The trigger map's hack/ entry is for cozystack/ccp, whose skills anchor on hack/package.mk, hack/common-envs.mk and make generate, none of which change here.

Release note

test(e2e): the E2E leak guard now also fails a suite that leaves behind the Secrets cert-manager wrote for its apps, reported as LEAKED-SECRETS with the Certificate and root of each.

Summary by CodeRabbit

• New Features
  • Leak checks now track cert-manager TLS Secrets alongside pods, attributing Secrets to the suite that created them.
  • Leaked pods and Secrets are reported separately. Unallowlisted leaks after a successful suite fail the check if they remain after the cleanup timeout.
• Documentation
  • Updated testing guidance to explain Secret attribution, cleanup timeouts, and leak reporting.