etcd 3.5.12 (etcd-operator v0.4.5) breaks consistent watches in k8s 1.35 tenants (RequestWatchProgress unsupported) #3080

Problem

Cozystack provisions modern Kubernetes tenants (e.g. k8s 1.35 via the Kamaji-backed Kubernetes app) but ships etcd-operator v0.4.5 (etcd.aenix.io/v1alpha1), whose default etcd image is quay.io/coreos/etcd:v3.5.12.

k8s ≥1.31 enables ConsistentListFromCache (GA, and locked-to-true in 1.35), which relies on the storage feature RequestWatchProgress. The apiserver only enables RequestWatchProgress when etcd is ≥3.5.13 (or ≥3.4.31). On etcd 3.5.12 it logs:

feature_support_checker.go: RequestWatchProgress feature is not supported by "https://etcd.<tenant>.svc:2379" endpoint

As a result, any consistent watch (e.g. ?watch=true&allowWatchBookmarks=true) fails with a 500 inside the watch stream:

{"type":"ERROR","object":{"kind":"Status","status":"Failure",
 "message":"a watch stream was requested by the client but the required storage feature RequestWatchProgress is disabled",
 "reason":"InternalError","code":500}}

Clients that watch aggregated/extension APIs (for example the Cozystack dashboard / cozyportal SPA, which lists resources via *.cozystack.io aggregated APIs with bookmarks) then get a continuous 502 storm on resource views.

Why it can't be worked around in the tenant

• ConsistentListFromCache is GA-locked in k8s 1.35: setting --feature-gates=ConsistentListFromCache=false makes kube-apiserver refuse to start (cannot set feature gate ConsistentListFromCache to false, feature is locked to true).
• The etcd.aenix.io/v1alpha1 EtcdCluster CRD has no spec.image or spec.version field, and etcd-operator v0.4.5 has no env/flag to override the default etcd image — so the etcd version cannot be raised per-cluster. It is fixed by the operator binary.

Fix

Ship the new etcd-operator (etcd-operator.cozystack.io/v1alpha2, which exposes spec.version and defaults a newer etcd) — tracked in #2859 — or, as a stopgap, bump the v0.4.5 default etcd image to ≥3.5.13.

Environment

• Cozystack 1.5.x (verified 1.5.0 / 1.5.1 / main all pin etcd-operator chart appVersion v0.4.5)
• Tenant: Kamaji-backed Kubernetes app, k8s 1.35
• etcd: quay.io/coreos/etcd:v3.5.12

Related: #2859.