Skip to content

fix(backport): batch for release-1.7, 2026-11-09 - #5246

Merged
myasnikovdaniil merged 96 commits into
release-1.7from
backport-batch/release-1.7-20261109
Nov 11, 2026
Merged

myasnikovdaniil merged 96 commits into
release-1.7from
backport-batch/release-1.7-20261109

Conversation

@cozystack-ci

@cozystack-ci cozystack-ci Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

What this PR does

Backports to release-1.7 from main, picked by the rules of the backport policy in merge order. A merged pull request comes as its own commits, picked with git cherry-pick -x so each keeps its author, message and sign-offs, and a merge commit that carries the original's merge message with a Backported-from: trailer naming it; a squash-merged one comes as one pick. backport-batch verify checks that shape. A pick listed as changed differs from the commit it came from; git range-diff <merge>^1..<merge>^2 <batch merge>^1..<batch merge>^2 shows how. A conflict or a failing check here is a backport task.

  • #4756 fix(backupstrategy-controller): run backup clients on a package-owned image, 2 commits, changed in the pick: 48d0cb2958a5
  • #4867 fix(e2e): keep chainsaw asserts polling through transient API errors, 1 commit
  • #4888 fix(ci): dispatch frozen-line builds with GITHUB_TOKEN, 1 commit
  • #4985 fix(etcd): stop counting canceled watch streams as failed gRPC requests, 1 commit
  • #5005 fix(tcp-balancer): cap HAProxy fds and threads by the pod's own limits, 1 commit
  • #4917 fix(kubernetes): run Proxmox CCM v0.16.1 and pin the Proxmox images by digest, 2 commits
  • #4974 fix(kubernetes-nodes): plan autoscaler capacity on the cores the VM actually gets, 1 commit
  • #4971 fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it, 2 commits
  • #4970 fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard, 3 commits
  • #4972 fix(e2e): validate the leak allowlist against every suite layout, 3 commits
  • #5082 fix(marketplace): pin a tapped repository to a digest instead of a mutable tag, 15 commits, changed in the pick: a31ae9090928, b326e0badd62, a3d883b4bf5b, f8509450f7ca
  • #4004 fix(dashboard): link a tenant row to its detail page, 2 commits
  • #5212 fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook, 1 commit
  • #5213 fix(kubernetes): move the OIDC users out of the bootstrap Job's script, 1 commit
  • #5208 fix(clickhouse): restore the archive a Backup was taken as, 3 commits
  • #5230 fix(backups): re-render the default backup objects when the S3 endpoint changes, 8 commits
  • #5133 fix(securitygroups): project a membership-only group to a policy Cilium accepts, 1 commit
  • #5091 fix(postgres): render the declared databases in a stable order, 1 commit
  • #5123 fix(mariadb-operator): derive the cluster name from the platform domain, 2 commits
  • #5151 fix(redis): run one Sentinel when Redis runs a single pod, 5 commits
  • #5124 fix(lineage, kubernetes): resolve CAPK node VMs to their cluster, 2 commits
  • #5184 fix(api): read pool usage under the quota key the controller counts, 1 commit
  • #5190 fix(platform): remove the dead registries value and patch-containerd Secret, 2 commits
  • #5211 fix(postgres): fail a backup job as soon as its cnpg Backup fails, 1 commit
  • #5155 fix(postgres): refuse system databases and never drop them, 1 commit
  • #5207 fix(mariadb): fail backup and restore jobs whose operator Job failed, 5 commits
  • #5024 fix(kubernetes, monitoring): send tenant cluster metrics to the tenant's vminsert, 1 commit

Maintainer overrides

  • #5083 fix(workloadmonitor): stop doubling the bucket- prefix in bucket Workload names: backport/skip, left out
  • #4973 fix(kubernetes-nodes): refuse a partial resources spec on every pool: backport/skip, left out
  • #5180 fix(tests): keep two unit tests green after the release image restamp: backport/skip, left out

Skipped

  • Rule 1, already on release-1.7: 19
  • Rule 2, the title type is not fix or security, or the scope is deps: 26
  • Rule 3, a breaking change: 5
  • Rule 4, touches platform migrations, targetVersion or a CRD, or makes a sizeable API change: 2
Rule 1, already on `release-1.7`
  • #4815 fix(vm-disk): name the importer error in the disk's WorkloadsReady message, its 2 commits picked, the last in 432336387019
  • #4293 fix(fluxcd): keep Helm uninstall working during flux-plunger recovery, its 1 commit picked, the last in 021040795b22
  • #4791 fix(kubernetes): keep NFS publish from hanging on a dead server, its 1 commit picked, the last in fa88bd1c4902
  • #4830 fix(multus): drop the CPU limit and make resources configurable, its 2 commits picked, the last in 363c8426bab7
  • #4865 fix(kafka): pass the container limits to the Kafka JVMs, its 3 commits picked, the last in fc38a540c02f
  • #4866 fix(linstor): mark volumes restored from a snapshot DRBD-initialized, its 1 commit picked, the last in f9550b71cc24
  • #4819 fix(seaweedfs): build SeaweedFS in-tree with a bounded S3 read-ahead, its 5 commits picked, the last in 7f4c3463d2a9
  • #3427 fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips, its 3 commits picked, the last in 4ac0cd581481
  • #4889 fix(monitoring): send alerts to every Alertmanager replica, its 1 commit picked, the last in 5033d53410ba
  • #4929 fix(backups): gate every default Strategy CR on its CRD, its 2 commits picked, the last in dee1eef5590b
  • #4914 fix(fluxcd): close the flux-plunger review follow-ups for the hook, drain and Warnings, its 7 commits picked, the last in a8e491646a06
  • #4939 fix(cluster-api): resync capi-operator every minute, its 1 commit picked, the last in cc8d6a4f05a0
  • #5027 fix(hack): keep the backup stack in the vminstance selective install, its 1 commit picked, the last in 8ba8386a2e5c
  • #4955 fix(vm-disk): let CDI adopt the PVC a Velero restore recreates, its 2 commits picked, the last in 691d7bce038e
  • #5071 fix(marketplace): do not install a connected repository's components on connect, its 2 commits picked, the last in 25f8f11187b9
  • #5086 fix(tenant): stop the gateway release waiting on TenantGateway listeners and certificates, its 1 commit picked, the last in 18b22495038b
  • #5087 fix(controller): carry healthCheckExprs and waitStrategy to existing releases, its 1 commit picked, the last in d2a452e89067
  • #5088 fix(ingress-nginx): drop the admission-webhook policy a rebase brought back, its 1 commit picked, the last in cf00b41e0e9a
  • #5142 fix(release): keep release bodies within GitHub's size limit, its 1 commit picked, the last in c7405158634e
Rule 2, the title type is not `fix` or `security`, or the scope is `deps`
  • #4845 docs(governance): welcome new contributors and link LF sanctions guidance, type docs
  • #4881 feat(bucket): add a ui flag to skip the s3manager web UI, type feat
  • #4899 chore: link the FOSSA license scan for CLOMonitor, type chore
  • #4898 feat(backups): back up Valkey with the Redis strategy, type feat
  • #4940 docs(release): add changelog for v1.6.5, type docs
  • #4861 feat(release): sign the release assets and attach a source SBOM, type feat
  • #3780 docs: move the operator guides out of docs/ to where readers find them, type docs
  • #4936 docs(release): check changelog claims against the code, type docs
  • #4959 docs(release): point changelog check B at the Talos installer profile, type docs
  • #3777 docs(agents): define where a new document belongs, type docs
  • #5025 test(harbor): pin the resources of the nginx proxy and the portal, type test
  • #5039 feat(keycloak): load additional CA files from Secret or ConfigMap keys, type feat
  • #4860 feat(kubernetes-nodes): trust a factory CA in the Proxmox template script, type feat
  • #5090 ci(e2e): end the Proxmox lane's SSH key with a newline, type ci
  • #4969 test(kubernetes): pin RetryOnFailure on the Proxmox csi release, type test
  • #5000 test(kubernetes): name the Proxmox fixture's node pool <cluster>-<pool>, type test
  • #5168 docs(agents): steer agents away from Helm lookup, type docs
  • #4997 feat(release): attach an SBOM to every release image, type feat
  • #3440 ci(api-review-gate): publish the gate as a commit status, type ci
  • #5034 docs: move the repo-only guides onto the website, type docs
  • #5215 feat(monitoring-agents): export PVC labels from kube-state-metrics, type feat
  • #5217 feat(monitoring-agents): keep application identity in container log streams, type feat
  • #5171 feat(api): expose HelmRelease failure state on Application status, type feat
  • #5237 feat(operator): let the platform source trust a private registry CA, type feat
  • #5106 feat(release): make backports automatic with backport-batch, type feat
  • #4983 feat(api): add the reservation evaluator for tenant quotas, type feat
Rule 3, a breaking change
  • #4790 fix(platform)!: refuse a networking variant change on a live cluster, ! in the title
  • #4796 fix(vm-disk)!: refuse a golden-image clone CDI would leave unbound, ! in the title
  • #5085 fix(kubernetes, opensearch, foundationdb)!: restrict managed application images, ! in the title
  • #4818 fix(keycloak)!: make the KMS proxy keep its keys across restarts, ! in the title
  • #5167 fix(monitoring-agents)!: isolate merged log payload under log_processed, ! in the title
Rule 4, touches platform migrations, `targetVersion` or a CRD, or makes a sizeable API change
  • #4941 fix(backups): keep the backupstrategy-controller Helm release under the Secret size limit, a sizeable API change (breaking-change: [v1alpha1] spec.spec.template: field "metadata" was removed, and 5 more), a CRD
  • #5223 fix(backups): report sizeBytes for MariaDB and CNPG backup artifacts, a CRD
{"line":"release-1.7","deferred":[]}

Screenshots

See the original pull requests.

Downstream repositories

Release note

fix(backupstrategy-controller): the ClickHouse (Altinity) and RabbitMQ backup strategy Pods now run a dedicated `backup-http-client` image (alpine + curl + jq + ca-certificates, non-root, ~7 MB) instead of the 349 MB `platform-migrations` image. Air-gapped mirrors need the new `ghcr.io/cozystack/cozystack/backup-http-client` repository.

fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout.

fix(ci): use GITHUB_TOKEN to start the initial packages build when freezing a release line

fix(etcd): the tenant etcd alert etcdHighNumberOfFailedGRPCRequests now counts only the gRPC codes that the etcd mixin and the platform copy of the alert count, so the Canceled watch streams of a Kubernetes API server that stops no longer fire it on every etcd member.

fix(tcp-balancer): HAProxy now takes its fd and thread caps from the replica's memory and CPU limits instead of the node, so a t1.nano balancer is no longer OOMKilled on nodes with many CPUs. maxconn follows the memory limit at about 16 connections per MiB, 2035 on t1.nano, and the pods roll once on upgrade.

fix(kubernetes): the Proxmox cloud-controller-manager now runs v0.16.1, and the Proxmox CCM and CSI driver images are pinned by digest. The v0.15.0 tag referenced before was re-pushed upstream to a build that panics on start, which left every new Proxmox worker with the uninitialized taint.

fix(marketplace): a tapped repository is pinned to a digest instead of following a mutable tag, so moving the tag upstream no longer changes what runs in the cluster; connect accepts oci://...@sha256:... and an optional keyless cosign verify policy (anchored to an exact identity and preserved across re-connects), and with verify flags the CLI verifies the signature and pins the verified digest at tap time.

fix(dashboard): tenant name in the tenant list links to its detail page now (where Delete is), and top level tenants get their Edit button back.

fix(cilium): the IPAM migration hook no longer fails with "Argument list too long" on clusters with more than a few nodes, which blocked the cilium upgrade and every package that depends on networking.

fix(kubernetes): the OIDC bootstrap Job no longer fails to start for a tenant cluster with about 240 or more oidc.users, the bindings now come from a ConfigMap the Job mounts.

fix(clickhouse): a RestoreJob now restores the archive its Backup was taken as, not the newest one. New Backups record the archive name, Backups taken before this release are restored best effort from the newest archive stamped no later than their takenAt, with a warning in the restore pod log. A ClickHouse BackupJob created again under a name whose Backup still exists now fails instead of reporting that Backup as its result.

fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand.

fix(securitygroups): a SecurityGroup with attachments and no rules no longer leaves an invalid CiliumNetworkPolicy behind; its backing policy is now accepted by Cilium and still allows or denies nothing.

fix(postgres): the init script renders the declared databases in a stable order, so an upgrade that leaves the databases unchanged no longer rewrites the init-script Secret.

fix(mariadb-operator): the MariaDB operator now uses the platform's networking.clusterDomain instead of a hard-coded cozy.local, so replication and User/Database/Grant reconciliation work on clusters with a non-default cluster domain.

fix(redis): an instance with `replicas: 1` now runs one Sentinel instead of three, without a Sentinel PodDisruptionBudget; on upgrade, the PDB the operator created earlier is deleted. Instances with two replicas or more are unchanged.

fix(lineage, kubernetes): the virt-launcher pods of Kubernetes worker nodes are now resolved to their Kubernetes application and marked as managed by Cozystack, so a SecurityGroup attached to a Kubernetes cluster now applies to its worker nodes, not only to its control plane.

fix(api): tenant quota admission now counts the parent pool's usage of extended resources and hugepages, so a child tenant can no longer get a share the parent has already used. After upgrading, updates to existing tenants may be rejected if their quota for these resources exceeds the parent's remaining budget. This includes updates that leave the quota unchanged.

fix(postgres): a Postgres BackupJob now fails as soon as its CloudNativePG Backup fails, with the recorded error, instead of after 30 minutes, and a BackupJob waiting on a CloudNativePG Backup reports a WaitingForCNPGBackup condition with the Backup's phase.

fix(postgres): databases now refuses the PostgreSQL system databases postgres, template0 and template1, and the init job never runs DROP DATABASE on them. A system database a previous version took over is handed back to the superuser on the next upgrade (event trigger removed, ownership and public schema restored, its <db>_admin and <db>_readonly roles dropped). A release that declares one of them must remove the entry before upgrading.

fix(mariadb): a MariaDB BackupJob or RestoreJob whose operator Job failed now ends Failed instead of Succeeded, and a failed backup no longer creates a Ready Backup without an archive. Backups created by failed runs before this fix are not removed; restoring from one can fail or select another dump from the shared prefix.

fix(kubernetes, monitoring): the vmagent of tenant Kubernetes clusters now writes to the tenant's vminsert and labels series with the owning tenant instead of `cozy-monitoring`. Previously no tenant cluster metric reached the tenant's Monitoring.

@github-actions github-actions Bot added area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/XXL This PR changes 1000+ lines, ignoring generated files labels Nov 9, 2026
@cozystack-ci

cozystack-ci Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

myasnikovdaniil

These conflicts wait for a maintainer, as resolving a conflict describes; unresolved, each goes to the next batch:

These change .github/workflows/, which the batch's token may not push, and wait for a maintainer to bring them over, as changes to workflows describes:

Updated by the backport batch on 2026-11-09.

@coderabbitai

coderabbitai Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

europrinter (yankawai) and others added 25 commits November 10, 2026 12:23
The tenant etcd copy of etcdHighNumberOfFailedGRPCRequests counted
every code other than OK. A Watch stream is counted once, when it ends,
and the streams of a kube-apiserver that stops end with Canceled. So
whenever an API server on that etcd goes away, the Watch method of
every member reads as 100 percent failed and the critical rule fires
on all of them.

The rules now count the codes the etcd mixin counts since
etcd-io/etcd#13127, the same list the platform copy in
monitoring-agents already uses. The grouping by method stays, so the
alert labels do not change.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <[email protected]>
(cherry picked from commit 1ca7c0d)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
… by digest

The Proxmox CCM was referenced by its bare v0.15.0 tag, and that tag no
longer names the image it named when the template was written. It was
re-pushed on 2026-09-27 (index sha256:d81702998ae1, replacing the
2026-08-06 build sha256:fbc553921145) to a binary whose gitVersion is
"0.15.0" without the leading v. component-base v0.37 parses that string
in the init() of its legacy metrics registry and panics:

  panic: version string "0.15.0" doesn't match expected regular
  expression: "^v(\d+\.\d+\.\d+)"

so the Deployment crash-loops, no Node ever loses the
node.cloudprovider.kubernetes.io/uninitialized taint, and workloads on a
new worker stay Pending. Any node that pulls the tag fresh gets it, which
is every fresh install.

v0.16 builds with goreleaser and stamps gitVersion as v{{.Version}}. The
configuration this template writes is read unchanged: the cluster entry
moved to go-proxmox-pool's ClusterConfig with the same url, insecure,
token_id, token_secret and region keys, and features.provider: capmox
still writes proxmox://<SMBIOS UUID>, the providerID capmox puts on the
Machine.

The reference now lives in images/proxmox-cloud-controller-manager.tag
as tag plus digest, the shape the chart already uses for the KubeVirt
cloud provider, so a moving tag can no longer change what runs.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <[email protected]>
(cherry picked from commit 05a347e)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
Without a global section HAProxy 3.4 sizes itself from the node.
maxconn comes from the nofile limit, cut to the built-in fd-hard-limit
of 1048576, so about 524k, and the thread count is the number of CPUs
the process may run on, whatever the CPU limit says. With one thread
that alone idles at about 85 MiB, and every thread adds about 0.6 MiB,
so a t1.nano replica with its 128Mi limit runs on a small node and is
OOMKilled at startup on a node with many CPUs.

fd-hard-limit is now 32 per MiB of the memory limit, from which HAProxy
derives about 16 connections per MiB, 2035 for the t1.nano defaults.
That leaves 64 KiB per connection against the roughly 35 KiB HAProxy
budgets per connection itself when it is given a memory limit with -m,
and the rest covers the process and the kernel socket buffers charged
to the same cgroup. It caps the derived value instead of setting
maxconn, because an explicit maxconn above half the hard nofile limit
stops HAProxy at startup (strict-limits is on by default), while
fd-hard-limit still follows a lower system limit.

thread-hard-limit is the CPU limit rounded up, at least 1. It only caps
the detected thread count, so a node with fewer CPUs is not forced to
run more threads than it has.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <[email protected]>
(cherry picked from commit 20545d0)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…ts (#4985)

## What this PR does

The tenant etcd chart has its own copy of
`etcdHighNumberOfFailedGRPCRequests`, and both of its rules count every
`grpc_code` other than `OK` as a failure. `grpc_server_handled_total`
counts a Watch stream once, when it ends, and the streams of a
kube-apiserver that goes away end with `Canceled`. So whenever an API
server on a tenant etcd stops, the Watch method of every member reads as
100 percent failed and the critical rule fires on all of them.

Both rules now count only
`Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded`,
the list the etcd mixin has used since etcd-io/etcd#13127 and the one
the platform copy in
`packages/system/monitoring-agents/alerts/etcd.yaml` already uses. The
grouping `BY (job, instance, grpc_service, grpc_method)`, the
thresholds, `for` and the annotations stay as they are, so the alerts
keep their labels.

Seen on a v1.6.5 cluster while a Kubernetes cluster on a tenant etcd was
created and deleted a few minutes later. vmalert had the critical alert
firing for `grpc_method="Watch"` on all three members with the value
100. At that point
`rate(grpc_server_handled_total{grpc_code!="OK"}[10m])` was 0.63 for
Watch with `Canceled`, and the only other non-OK codes were
`FailedPrecondition` on `RoleGet` and `UserGet` at 0.0017 each, which is
one call each. The same query an hour earlier returned nothing. The
platform copy runs in the same vmalert over the same series, and none of
the firing alerts came from it. Each alert also showed up seven times
there, once per tenant copy of these rules, because the root tenant's
vmalert loads the rules of every tenant namespace. This PR does not
change that.

`hack/etcd-grpc-alerts.bats` renders the chart's PrometheusRule with
`helm template` and replays it through `vmalert-tool` v1.136.0 pinned by
digest, the same image #4963 uses.
`hack/testdata/etcd-grpc-alerts/unittest.yaml` has three cases with
seven checks. Watch streams that end only with `Canceled` must not fire
at 6, 12 and 30 minutes. `Range` calls failing with `Unavailable` at 25
percent must fire the critical rule at 7 minutes and both rules at 12,
with the labels and the summary pinned. Both together must fire only for
`Range`. Against the rules on main the suite fails 4 of the 7 checks,
all of them on Watch alerts with the value 100. Dropping `Unavailable`
from the list, or grouping by `job` and `instance` only, fails 3 checks
each.

```console
$ bats hack/etcd-grpc-alerts.bats
1..1
ok 1 tenant etcd failed gRPC request alerts pass their vmalert-tool cases
```

The same run on the rules from main prints `not ok 1` and the four
failing checks. With Bats 1.14.0 at the commit CI pins, the suite passes
together with `bats-strict-setup`, `bats-no-negated-assert`,
`bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`,
`alert-severity-contract` and `image-pin-consistency`, 81 of 81. Like
the migration suites it needs docker, and only the image pull touches
the network. Helm 3.17.3 and 4.2.4 render the same rules. I have not run
the new rules in vmalert on a live cluster.

The rules file is the same on release-1.6 and release-1.7. The commit
cherry-picks onto release-1.6 without conflicts, and that backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.

### Screenshots

Not a UI change.

### Downstream repositories

A rule template in `packages/extra/etcd`, a Bats suite under `hack/` and
its test data. No values, schema, README, image or
`ApplicationDefinition` change, and the website does not mention these
rules, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(etcd): the tenant etcd alert etcdHighNumberOfFailedGRPCRequests now counts only the gRPC codes that the etcd mixin and the platform copy of the alert count, so the Canceled watch streams of a Kubernetes API server that stops no longer fire it on every etcd member.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Refined etcd gRPC failure alerts to target specific failure status
codes, reducing alerts for non-failure responses while preserving
existing thresholds and timing.
- Added coverage for canceled watch streams, unavailable range requests,
and combined failure scenarios, including warning and critical alert
evaluations.

- **Tests**
- Added integration validation for rendering etcd alert rules and
running them through the alert-testing tool, including failure handling
when prerequisites are unavailable.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: 432d02e
…gest

The CSI controller in the management cluster was referenced by the bare
tag ghcr.io/sergelogvinov/proxmox-csi-controller:v0.20.0, and the node
plugin in the tenant by the vendored chart's default, its appVersion as
a bare tag. That registry is the one whose proxmox-cloud-controller-
manager:v0.15.0 was re-pushed to a build that panics, so a tag there is
not a statement of what runs.

Both now carry the v0.20.0 digest: the controller through
images/proxmox-csi-controller.tag, like the cloud-controller-manager, and
the node plugin through repository and tag in the proxmox-csi values,
which also makes it visible to hack/lib/image-refs.sh and therefore to
the release multi-arch gate. Both digests are indexes for linux/amd64
and linux/arm64.

Pinning takes the version out of the vendored chart's hands: a chart
bump would bring a new appVersion while both pins kept naming the old
release, and the controller and the node plugin would drift apart from
the templates and from each other without anything noticing.
hack/proxmox-csi-driver-pin_test.bats fails unless the chart's
appVersion and both pins name the same release.

The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI
driver: registry.k8s.io publishes them through image promotion, which
does not overwrite a tag.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <[email protected]>
(cherry picked from commit 780b068)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
#5005)

## What this PR does

On a v1.6.5 cluster a TCPBalancer created with the chart defaults
(`t1.nano`, limits 250m CPU and 128Mi memory, `haproxy:3.4.4`)
crash-looped. Both pods were OOMKilled with a memory limit of 134217728,
7 restarts each in about 15 minutes, and HAProxy logged `[NOTICE] (8) :
Automatically setting global.maxconn to 524126.` On another cluster the
same chart picked maxconn 524251 and the pods kept running.

`haproxy.cfg` has no `global` section, so HAProxy sizes itself from the
node. maxconn comes from the nofile limit capped at the built-in
`fd-hard-limit` of 1048576 (`DEFAULT_MAXFD`), and the thread count is
the number of CPUs the process may run on, whatever its CPU limit is.
Both are paid for at startup, before any traffic, and the thread part
grows with the node.

This PR adds a `global` section sized from the replica's own limits,
read through the same `cozy-lib.resources.defaultingSanitize` helper the
Deployment uses, so `resources` and `resourcesPreset` resolve the same
way. For `t1.nano` it renders

```
global
    fd-hard-limit 4096
    thread-hard-limit 1
```

`fd-hard-limit` is 32 per MiB of the memory limit and HAProxy derives
maxconn from it, 2035 for `t1.nano`, which is about 64 KiB per
connection. For comparison `haproxy -m 128` picks maxconn 3600 on 3.4.4,
about 35 KiB per connection, and the difference is left for the process
itself and for the kernel socket buffers the cgroup also charges. An
explicit `maxconn` would read simpler, but `strict-limits` is on by
default and HAProxy refuses to start when it cannot raise nofile to
about twice that value. With 1024 soft and 4096 hard nofile `maxconn
4096` exits with `Cannot raise FD limit to 8251`, while `fd-hard-limit
8192` starts with maxconn 2018. `thread-hard-limit` is the CPU limit
rounded up, at least 1. It caps the detected thread count and never
forces more threads than the node has.

Reproduced with Docker Desktop on arm64 (12 vCPU, kernel 6.12) and the
same image, running the rendered config the way the Deployment mounts
it. A larger node is stood in for by adding `nbthread 96` or `nbthread
128` to the `global` section. Memory is `memory.peak` of the container
cgroup.

```sh
cd packages/apps/tcp-balancer && mkdir -p /tmp/cfg
helm template tb . --show-only templates/configmap.yaml | yq '.data["haproxy.cfg"]' > /tmp/cfg/haproxy.cfg
docker run --rm --memory 128m --memory-swap 128m --ulimit nofile=1048576:1048576 -v /tmp/cfg:/usr/local/etc/haproxy:ro haproxy:3.4.4
```

| run | main | this PR |
| --- | --- | --- |
| idle, 12 threads | maxconn 524258, peak 89.6 MiB | maxconn 2035, 1
thread, peak 18.7 MiB |
| idle, `nbthread 96` | maxconn 524127, worker exited with code 137,
OOMKilled | 1 thread, peak 20.8 MiB |
| idle, `nbthread 128` | worker exited with code 137, OOMKilled | 1
thread, peak 22.9 MiB |
| 2035 clients echoing 16 KiB through a backend for 25 s, 12 threads |
worker exited with code 137 about a second after the clients started |
running, peak 83.7 MiB |

One case this PR does not cover is clients and servers that both stop
reading. They fill kernel socket buffers that the cgroup charges, and
with this PR the 128Mi container was still OOMKilled after 1225 such
connections, which maxconn does not bound.

`helm unittest .` passes 13 of 13 with helm-unittest 1.1.2 as CI pins
it. Against the template on main the four new cases fail and the nine
existing ones pass, and each of six template mutations fails at least
one case. `haproxy -c` accepts the render for `t1.nano`, `t1.4xlarge`,
`m1.4xlarge`, a zero CPU limit, the whitelist, `kubernetesAndTalos` and
`tcp-with-proxy` inputs. A full `helm template` differs from main only
in the `global` section and the `checksum/config` annotation, so the
pods roll once on upgrade. I have not run the change on a live cluster.

The template is the same on release-1.6, and that backport is a separate
PR.

### Screenshots

Not a UI change.

### Downstream repositories

One chart template and its unittest suite in
`packages/apps/tcp-balancer`. No values key, schema, default, README or
package change, so no trigger map entry applies.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(tcp-balancer): HAProxy now takes its fd and thread caps from the replica's memory and CPU limits instead of the node, so a t1.nano balancer is no longer OOMKilled on nodes with many CPUs. maxconn follows the memory limit at about 16 connections per MiB, 2035 on t1.nano, and the pods roll once on upgrade.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **New Features**
- TCP Balancer now automatically configures HAProxy file-descriptor and
thread limits based on the selected or explicitly defined resource
settings.
- CPU-based thread limits are rounded up and always maintain a minimum
of one thread.
- Memory-based file-descriptor limits scale automatically with the
configured memory capacity.

- **Tests**
- Added coverage for default, preset-based, explicit resource, rounding,
and minimum-limit scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: 55aad6f
…ctually gets

On the kubevirt substrate resources.cpu is written into domain.cpu.cores
as spelled and decoded as YAML, so "010" gives the VM eight cores, while
the cluster-autoscaler capacity annotation parsed the same string as
decimal ten. The autoscaler then planned 9000m on a node that offers
7000m.

Keep the YAML-decoded value the range check already computes and start
the capacity annotation and the reserved-CPU check from it on kubevirt.
The kubelet reservations stay as they are: the worker machineconfig
sizes them from the decimal string, and the TalosConfigTemplate name is
a hash of that config, so changing them would roll every leading-zero
pool once. The annotation therefore subtracts the reservations the
machineconfig really carries (2 x 500m for "010") from the cores the VM
really has, which is the node's real allocatable. Neither template name
changes for any spelling, so no pool rolls. Proxmox parses the quantity
as decimal on both sides and is unchanged. The field description says
the value is decoded as YAML and that the reservations still read it as
a decimal.

This closes #4554 without the migration and the render-time refusal it
proposes: a refusal breaks stored pools on their next upgrade, a schema
pattern breaks updates of existing CRs, and a platform-wide migration of
every KubernetesNodes and nodeGroups entry is a lot of blast radius for a
spelling nobody has reported in a live cluster.

Closes #4554

Signed-off-by: Marian Koreniuk <[email protected]>
Assisted-by: LLM
(cherry picked from commit f791332)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…y digest (#4917)

## What this PR does

`v1.7.0-rc.1` and `release-1.7` ship the Proxmox
cloud-controller-manager as
`ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.15.0`, a bare
tag. That tag was re-pushed upstream on 2026-09-27: it used to name the
2026-08-06 build (`sha256:fbc553921145…`) and now names
`sha256:d81702998ae1…`, a binary whose gitVersion is `0.15.0` without
the leading `v`. component-base v0.37 parses that string in an `init()`
and panics:

```
panic: version string "0.15.0" doesn't match expected regular expression: "^v(\d+\.\d+\.\d+)"
```

The CCM Deployment crash-loops, no new Node loses
`node.cloudprovider.kubernetes.io/uninitialized`, and workloads on new
workers stay Pending. On a live Proxmox cluster this went unnoticed for
five days: the existing Nodes were already initialized, and the failure
only showed when the next new worker came up. Every fresh install pulls
the broken build, so as it stands v1.7.0 would be the first stable
release with the Proxmox substrate, shipping a CCM that cannot start.

Two commits, both cherry-pick onto `release-1.7` unchanged:

1. **`fix(kubernetes)`**: run v0.16.1, pinned as tag plus digest in
`images/proxmox-cloud-controller-manager.tag`. That is the same shape
the chart already uses for the KubeVirt cloud provider. v0.16 builds
with goreleaser and stamps the version with its `v`. The config the init
container writes still parses: the cluster entry moved to
go-proxmox-pool's `ClusterConfig` with the same `url`, `insecure`,
`token_id`, `token_secret` and `region` keys, and `features.provider:
capmox` still writes `proxmox://<SMBIOS UUID>`, the providerID capmox
puts on the Machine.
2. **`fix(kubernetes, proxmox-csi)`**: pin the CSI driver from the same
registry at its current v0.20.0, with no version change. The
management-side controller goes through
`images/proxmox-csi-controller.tag`. The tenant-side node plugin goes
through `repository` + `tag` in
`packages/system/proxmox-csi/values.yaml`, which also puts it in
`hack/lib/image-refs.sh` and so in front of the release multi-arch gate.
Pinning takes the version out of the vendored chart's hands, so
`hack/proxmox-csi-driver-pin_test.bats` now fails unless the chart's
`appVersion` and both pins name the same release.

All three digests are OCI indexes for `linux/amd64` and `linux/arm64`.
The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI
driver: registry.k8s.io publishes through image promotion, which does
not move a tag.

### Testing

- `helm unittest`: `packages/apps/kubernetes` 31 suites / 346 tests,
`packages/apps/kubernetes-nodes` 37 / 223, all passing.
- bats: the new test, plus `image-pin-consistency`,
`promote-rewrite-tags_test`, `overlay-main-images_test`,
`nightly-mirror_test`, `promote-retag_test`, `stitch-multiarch_test`,
`bats-strict-setup`, `bats-runner-coverage`: 107/107. The new test was
checked to fail on a node tag at another version.
- `git cherry-pick -x` of both commits onto `release-1.7` (315b335)
applies cleanly, and the Proxmox unit suites and the new bats test pass
there.
- v0.16.1 compatibility was checked against its source, as described
above. It has also run on a live Proxmox cluster, including the
initialisation of a new worker; the results are in the comment below.

### Screenshots

Not a UI change.

### Downstream repositories

The diff touches no app `values.schema.json` or defaults, adds or
renames no package, and moves nothing under `hack/`.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(kubernetes): the Proxmox cloud-controller-manager now runs v0.16.1, and the Proxmox CCM and CSI driver images are pinned by digest. The v0.15.0 tag referenced before was re-pushed upstream to a build that panics on start, which left every new Proxmox worker with the uninitialized taint.
```

Backported-from: 44848b9
cozy_guard_all_helmreleases reads the parent and the kubernetes-<test>-
addons. The worker pool is kubernetes-nodes-<test>-md0, an Application
HelmRelease the apiserver stamps with RetryOnFailure like the parent, on
the node-join path through the talos-reconcile Job it renders. Its name
does not start with the addon prefix, so a failed action on it was never
read, on the passing path or in the failure-path report.

Read it on both paths. The pool name is derived from the parent, the way
the addon prefix is, so the two cannot drift. The failure-path report
checks that the release exists first: a run that failed between the
parent and the pool apply has none, and that is a fact about where the
run stopped rather than a failed read.

Closes #4737

Signed-off-by: Marian Koreniuk <[email protected]>
Assisted-by: LLM
(cherry picked from commit cac1520)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
lg_allowlist_validate accepted only <suite>/chainsaw-test.y*ml, while the
selectors already see step-file suites and a Test nested below the suite
directory. A suite in one of those layouts could not be allowlisted at
all: the run stopped before the first suite. Validate the allowlist with
the rule hack/select-e2e.sh enumerates suites with, spelled the same way
on both sides and cross-referenced so they move together.

Describe the three layouts once in the e2e-chainsaw README, point the
agent doc at it, and list kubernetes-proxmox as the parked suite.

Closes #4659

Signed-off-by: Marian Koreniuk <[email protected]>
Assisted-by: LLM
(cherry picked from commit 782e0ab)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
The failure-path report checked the pool release with `get hr -o name`
and discarded the exit status and stderr, so a timeout, a kill, a
Forbidden or a refused connection all printed that the run ended before
the worker pool was applied. That is the conflation the guard wrapper
exists to refuse, on the path where a slow apiserver is most likely.

Read it with --ignore-not-found instead: exit 0 with empty output is an
absent release, exit 0 with a name is one to classify, and any non-zero
exit is a failed read named through _cozy_guard_read_failure.

The pool reads also skipped the report's own deadline and checked only
the phase clock, which says yes on every path that opens no phase.
Check the deadline before the existence read, like the listing and the
addons do, and name what was not read.

The stub's ABSENT marker now answers the flag the way kubectl does. New
cases pin the failed read and the deadline; the two budget cases are
re-timed for the extra clock reading.

Signed-off-by: Marian Koreniuk <[email protected]>
Assisted-by: LLM
(cherry picked from commit 4d7d336)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
… its regex

lg_suite_has_test dropped every directory with `! -type d`, but the
selector keeps a chainsaw-test.y*ml whatever its type, so Chainsaw can
fail loudly on a directory by that name. An allowlisted suite in that
shape failed validation and stopped the run before Chainsaw reported the
real error. Use the selector's own `\( ! -type d -o -name
chainsaw-test.yaml -o -name chainsaw-test.yml \)`, which still drops a
directory with a step-file name.

The step-file regex behind the find glob was never reached by a test:
install-01.yaml is already filtered by the glob. 1a-install.yaml passes
the glob and not the regex, so it pins the regex; a directory named
chainsaw-test.yaml and one named 01-install.yaml pin the type test.

The README said the runner recognises the same three layouts. It
recognises none: it hands every top-level directory to Chainsaw.

Signed-off-by: Marian Koreniuk <[email protected]>
Assisted-by: LLM
(cherry picked from commit cd983d4)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
The exit handler was armed after the KubernetesNodes apply, so a failing
pool apply exited under errexit with no handler installed: the parent
went unclassified, and the report's "pool release does not exist" line
described a run that could never reach it. Arm it right after the parent
Kubernetes apply instead. The snapshot half still no-ops until the
tenant kubeconfig is set, so the only change is that a failed pool apply
now gets the parent classified and the absent pool reported.

The ordering pin now also requires the handler to be armed before the
pool apply.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <[email protected]>
(cherry picked from commit 75bb70e)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…ctually gets (#4974)

On the kubevirt substrate `resources.cpu` is written into
`domain.cpu.cores` as spelled and decoded as YAML, so `"010"` gives the
VM eight cores, while the cluster-autoscaler capacity annotation parsed
the same string as decimal ten
(`capacity.cluster-autoscaler.kubernetes.io/cpu: "9000m"` on a node that
offers 7000m). This is the inconsistency reported in #4554 and in the
comment on #4546.

- The YAML-decoded value the range check already computes is kept, and
on kubevirt the capacity annotation and the reserved-CPU check start
from it.
- The kubelet reservations are not touched. The worker machineconfig
(`_talosconfigtemplate.tpl`) sizes them from the raw string read as a
decimal, and the TalosConfigTemplate name is a hash of that config, so
reading the decoded value there would roll every leading-zero pool once.
Instead the annotation subtracts the reservations the machineconfig
really carries from the cores the VM really has: for `"010"` that is
8000m − 2 × 500m = `"7000m"`, the node's real allocatable, while `"8"`
keeps `"7200m"` (2 × 400m). Tests pin both sides: the annotation and the
bootstrap `configRef` name in `resources_cpu_test.yaml`, and
`systemReserved.cpu` / `kubeReserved.cpu` in the machineconfig in
`talos_worker_config_test.yaml`.
- No pool rolls. Checked by rendering `nodegroup.yaml` on main and on
this branch for nine spellings (`"2"`, `2`, `2.0`, `"2.0"`, `"+2"`,
`"1e1"`, `"010"`, `999999`, `4294967295`): the only difference is the
capacity annotation of the `"010"` case, so the
`KubevirtMachineTemplate` name and the TalosConfigTemplate `configRef`
stay as they are. The machineconfig template itself is not changed.
- Proxmox parses the quantity as decimal for the machine template and
for the reservations alike, so it never had the inconsistency and is
unchanged (pinned by a test).
- The `resources` description says the value is decoded as YAML, what
that means for a leading zero, and that the reservations still read it
as a decimal. README, schema, the RD and the Go type are regenerated
from it.

This closes #4554 without the migration and the render-time refusal it
proposes. A refusal at render breaks a stored pool on its next upgrade,
a pattern in the OpenAPI schema breaks every update of an existing CR,
and a platform-wide migration over every `KubernetesNodes` and parent
`nodeGroups` entry is a lot of blast radius for a spelling nobody has
reported in a live cluster. With this change the autoscaler plans for
the node the VM really is. What is left of a leading zero is the octal
reading itself and reservations sized from the decimal value, and both
are documented. The unquoted zero of the issue is already harmless: it
is falsy, counts as unset, and renders the same bytes as an absent
field. If a migration is still wanted, please reopen.

Verified with `helm unittest .` in `packages/apps/kubernetes-nodes` (38
suites, 233 tests, 12 snapshots; pointing the annotation back at the
decimal string turns the `"010"` case red with `9000m`), `make generate`
(no diff), and a trial merge with #4973 (clean, 243 tests green).

Closes #4554


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* KubeVirt VM sizing and autoscaler capacity now reflect YAML-decoded
CPU values. For example, `"010"` represents eight VM cores, while
kubelet reservations interpret it as decimal ten. `"8"` and `"10"` yield
`7200m` and `9000m` of autoscaler capacity, respectively. Proxmox
continues to interpret `"010"` as ten cores, yielding `9000m` of
capacity.
* **Documentation**
* Clarified explicit worker CPU and memory sizing behavior, including
how leading-zero CPU values affect VM sizing, reservations, and capacity
calculations.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: 60d1f53
A tenant's CR lives in its parent's namespace, and the console recovered
the parent from the namespace name: `tenant-a-b` sits under `tenant-a`.
The root is the one level that naming skips. A tenant `acme` ordered in
`tenant-root` owns `tenant-acme`, not `tenant-root-acme`, so the prefix
test never matched a top-level tenant and `realParentNamespace` returned
undefined for it. That is most tenants on a normal cluster, and the Edit
button, offered only where the CR's namespace resolves, was missing for
all of them.

The ancestor labels carry the parent the name does not, so the lookup
falls back to `tenant-root` when the chain names it. Deriving the CR name
keys on the parent being the root rather than on the child's name
happening to start with it, which keeps a root-level tenant called
`root-x` from resolving to the CR `x`, and the row label agrees with it
rather than showing `x`.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
(cherry picked from commit e44e41d)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
The first commit says the two rules are cross-referenced so they move
together, but only lg_suite_has_test pointed at the selector. The
selector's suite enumeration comment now names lg_suite_has_test as well,
so a change to either side prompts the other.

The layout case also pins a chainsaw-test.yml that is a directory, the
other half of the type test: dropping that name from the find now turns
the case red.

Signed-off-by: Marian Koreniuk <[email protected]>
Assisted-by: LLM
(cherry picked from commit b61ad29)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…ard (#4970)

`cozy_guard_all_helmreleases` in
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` reads the parent
`kubernetes-<test>` release and the addons selected by the
`kubernetes-<test>-` prefix. The worker pool is
`kubernetes-nodes-<test>-md0`, an Application HelmRelease the aggregated
apiserver stamps with `RetryOnFailure` like the parent, and it sits on
the node-join path through the talos-reconcile Job it renders. Its name
does not start with the addon prefix, so a failed action on it was never
read, on the passing path or in the failure-path report.

This reads it on both paths, with the `required` strategy check the
parent gets, since the apiserver stamps both. The pool name is derived
from the parent (`cozy_pool_helmrelease_name`), the way the addon prefix
is, so the two cannot drift. The failure-path report checks that the
release exists before reading it, with `get hr --ignore-not-found -o
name`: a run whose pool apply failed has no pool release (exit 0, empty
output), and the report says so instead of reporting a failed read. For
that run to reach the report at all, `run_kubernetes_test` now arms the
exit handler (`_tenant_failure_on_exit`, with
`CURRENT_TENANT_PARENT_HR`) right after the parent `Kubernetes` apply
instead of after the `KubernetesNodes` apply: before, a failing pool
apply exited under errexit with no handler installed, and the parent
went unclassified. The snapshot half still no-ops until the tenant
kubeconfig is set. A non-zero exit (a timeout, a kill, a Forbidden) is
reported as the failed read it is, through `_cozy_guard_read_failure`,
never as absence. The pool reads are gated on the report's own deadline
as well as the phase clock, like the addon listing.

Unit tests in `hack/run-kubernetes-remediation_test.bats` cover a
failing pool release on the passing path, a clean one (the composite
returns zero), an empty strategy on the pool, the report with and
without the pool release, a cut-off existence read, and a pool read past
the report budget; the ordering pin now also requires the handler to be
armed before the pool apply. The kubectl stub gained an existence read:
an `ABSENT` marker prints nothing and exits 0 under `--ignore-not-found`
(NotFound without it, as kubectl does), and the failure markers apply to
it as `<name>.exists`.

Verified with `hack/cozytest.sh` and Bats on
`hack/run-kubernetes-remediation_test.bats` (51 tests),
`hack/run-kubernetes-node-join_test.bats` and
`hack/remediation-guard.bats`.

Closes #4737


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Kubernetes release checks now include worker-pool releases alongside
the parent release and add-ons, and flag missing or invalid install
strategies and remediation failures.
* Failure reports distinguish worker-pool releases that are absent from
those whose status cannot be read. Existing releases are assessed
against the phase gate and report time budget; when the budget is
exhausted, skipped checks are identified.
* Failure handling now begins immediately after the parent resource is
applied, covering errors that occur while applying the worker pool.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: 013b166
The tenant list rendered the name as plain text and put the row's only
link on an Edit button, so nothing in the list reached the detail page
that every other kind gets, tabs and a Delete action included. The only
way there was to open the edit form and cancel out of it.

The name cell becomes a link wherever Edit is offered, to the same CR.
The detail page reads the namespace from the tenant context, so a plain
click aligns the context to the row's parent first, the way Edit
already does. React runs that handler for a ctrl-, cmd- or shift-click
too, which the browser turns into a new tab or window, so it runs only
for the click that navigates its own tab; otherwise opening a tenant
beside the current one would move the tab left behind as well.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
(cherry picked from commit 7f55ff5)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
`lg_allowlist_validate` in `hack/e2e-chainsaw/_lib/leak-guard.sh`
accepted only `<suite>/chainsaw-test.y*ml`, while the selectors already
see step-file suites and a Test nested below the suite directory
(#4630). A suite in one of those layouts could not be put on the leak
allowlist at all: validation failed and the run stopped before the first
suite.

- `lg_suite_has_test` validates an allowlist entry with the rule
`hack/select-e2e.sh` enumerates suites with: `chainsaw-test.yaml`,
`chainsaw-test.yml`, or a numbered step file, anywhere in or below the
suite directory, with the selector's type test (a step-file name counts
unless it is a directory, a `chainsaw-test.y*ml` counts whatever its
type, so Chainsaw reports a directory by that name); a parked
`chainsaw-test.yaml.disabled` matches none. Both sides carry a comment
pointing at the other so they move together.
- `hack/e2e-chainsaw/README.md` describes the three layouts once, under
Layout, and lists `kubernetes-proxmox` as the parked suite.
`docs/agents/e2e-testing.md` points at it.
- Unit tests: every layout is accepted, including a `chainsaw-test.yaml`
that is a directory; a parked, empty, or not-a-step-file directory is
refused, including `1a-install.yaml` (passes the `find` glob, not the
step-file regex) and a directory named `01-install.yaml`.

Left as they are, with the reasons:
- `docs/agents/e2e-testing.md` calling `kubernetes-proxmox` the one
parked suite is correct on main now: `hack/e2e-chainsaw/backup/` was
removed in 9cc3a9e.
- The top-level globs in `hack/e2e-chainsaw-run.sh` are wider than
Chainsaw's step-file pattern, but they only refuse a test at the top
level and fail closed.
- A file under `examples/backups/<other>/` that matches no suite is
documented in `select-e2e.sh` as a docs-only demo that stays ignored.

Verified with Bats and `hack/cozytest.sh` on
`hack/chainsaw-leak-guard_test.bats` (34 tests) and
`hack/select-e2e_test.bats` (65 tests; one case, "a path owned by
several sources counts as covered if any one is", fails on my machine on
main as well, from the local yq build, and does not touch this change).

Closes #4659


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Documentation**
* Clarified that Chainsaw suites may use `chainsaw-test.yaml` or `.yml`,
numbered step files, or either layout in a subdirectory.
* Updated guidance on suite selection and noted that disabled test files
are not selected or allowlisted.
* Identified the parked Kubernetes Proxmox suite and its workflow
availability.
* **Bug Fixes**
  * Suite validation now recognizes tests in the documented layouts.
* Validation rejects empty directories, parked test files, invalid
step-file names, and directories named like step files.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: ba0b1ac
The Altinity strategy named each archive inside its Pod with date -u
and recorded nothing on the Backup, so a restore could only guess: it
listed the remote archives under the application's prefix and took the
newest. A RestoreJob of an older Backup therefore restored whichever
backup had run last.

The controller now names the archive before it creates the backup Job,
from the BackupJob's persisted StartedAt, so every reconcile and every
retried Pod of the Job writes the same one. The name is recorded in the
Backup's driverMetadata next to the strategy parameters, and a restore
renders it into the strategy Pod, which restores exactly that archive
and fails, naming it, when remote storage no longer holds it. The name
is recorded only when the Job's Pod was given it, so a Job created by
an earlier controller, or a strategy that names its own archives,
leaves the Backup without one instead of pointing it at an archive
that does not exist.

Because a retried Pod reuses the name, it waits on an attempt the
sidecar is still running instead of being refused with 423, and asks
for resume, so a local or partial remote backup an earlier attempt left
is completed rather than refused as existing.

Backups taken before names were recorded stay restorable: the Pod picks
the newest archive whose name is stamped no later than the Backup's
takenAt, which is set only after the backup Job finished, and logs a
warning that the choice is a best effort.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
(cherry picked from commit 47b4971)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
The hook fed the whole `kubectl get ciliumnodes -o json` to jq as one
argument. Cilium copies every Node label onto its CiliumNode, 22 to 30
KiB per node on Talos or with NFD and GPU labels, so the list passes the
128 KiB Linux allows a single argument within six nodes. jq then fails
to start with "Argument list too long", the hook Job fails before it
moves any node, and the cilium HelmRelease loops on UpgradeFailed with
the HelmReleases behind it stuck in DependencyNotReady.

The hook renders on every upgrade of cilium in cluster-pool mode, so a
cluster that already finished the migration hits the same failure on its
next values change.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
(cherry picked from commit a2d0e4c)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
## What this PR does

Fixes #3822

Tenant list showed name as plain text and the only link in a row was
Edit button, so tenant detail page with its tabs and Delete was
reachable only by opening edit form and cancelling out of it. Name cell
is a link to that page now, it appears exactly where Edit appears and
points to the same CR. Plain click switches selected tenant to the row's
parent first, same as Edit does, because detail page takes namespace
from selected tenant. Ctrl, cmd and shift click don't run that switch,
so the tab you open a tenant from stays where it was.

First commit fixes what the link depends on. Console recovers a tenant's
parent from namespace name (`tenant-a-b` sits under `tenant-a`), but
root is the one level that naming skips: tenant `acme` ordered in
`tenant-root` owns `tenant-acme`, not `tenant-root-acme`. So
`realParentNamespace` returned undefined for every top level tenant,
which is most tenants on a normal cluster, and Edit button has been
missing there for as long as it exists. Ancestor labels carry the parent
that the name does not. Root level tenant named `root-x` resolves to CR
`root-x` now, not `x`.

### Notes

I first tried to make links carry their tenant in the url (`?tenant=`),
so a detail page opened in new tab resolves on its own. That turned into
questions about tenant context itself: what to do with a tenant in a
link the user can't see, what localStorage may hold, and TenantNamespace
LIST ignoring `labelSelector`. Detail route addresses a Tenant CR by
relative name and takes namespace from selected tenant, and making that
robust is a refactor of tenant context and routing, which should start
with study and planning, not with patches on patches. So this PR only
closes the UX gap (no way from tenant list into a tenant's page), going
deeper is out of scope.

Known limit that comes with it: link has no tenant in it, so opening it
in a new tab (ctrl/cmd/middle click, open in new tab, copied url)
resolves against the tenant last selected in any tab, because selection
lives in shared localStorage. For a row directly under that tenant it
opens the right page. For a row deeper in the tree the page says
`Application <name> not found.` and there is no Delete. If that tenant
also has a direct child tenant with the same relative name, new tab
opens that other tenant's page with working Edit and Delete, and delete
confirm says only `Delete Tenant "<name>"?`. Plain click in the same tab
is not affected. Detail page's own tabs and Edit link already behave
this way, it's how detail route resolves its namespace today and fixing
it properly is the refactor above.

A row links, and offers Edit, exactly when namespace holding its Tenant
CR is one the apiserver returns to this user. It returns a tenant
namespace only if the user is subject of a RoleBinding in it, so
sub-tenant admin that cant act in `tenant-root` sees no link there.

No row level Delete button. Issue allows either that or a route to
detail page, I took the route so confirm and mutation stay in one place.

Covered by tests, every source change here fails a test when reverted:

- top level tenant, nested one, root level tenant named `root-x`, forest
root without ancestor labels
- bridged row whose real parent is invisible gets neither link nor Edit
- plain click switches to the row's parent and opens detail page
- ctrl click leaves selection and location alone

### Screenshots

Tenant list under root on a dev cluster. Every row name is a link to
tenant page now (`pr4078` is hovered), top level tenants had neither
link nor Edit before. Host column redacted.

<!-- SCREENSHOT: drag in 01-tenant-list-links.png here -->
<img width="1440" height="900" alt="01-tenant-list-links"
src="https://github.com/user-attachments/assets/39cd723a-1ac5-4d3b-8c36-b70ddc626961"
/>


### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

All changed files are under `packages/system/dashboard/images/console`,
console has no downstream copy (standalone UI repo is archived).

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(dashboard): tenant name in the tenant list links to its detail page now (where Delete is), and top level tenants get their Edit button back.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Root-level tenant names are now displayed without incorrectly removing
prefixes, and tenants ordered directly under the root are correctly
recognized in the hierarchy.
* Tenant rows link to their detail pages when the parent tenant is
accessible. Same-tab navigation updates the active tenant, while opening
a link in another tab leaves the current selection unchanged.
  * Edit links and navigation now use consistent tenant routes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: a2cc9a3
Every test in hack/altinity-backup-strategy.bats made a directory with
mktemp -d and never removed it, so each passing run of the file left
nine directories in TMPDIR, and they pile up on any machine that keeps
it between runs.

Each test now removes its directory as its last statement. A failed
assertion aborts the body before that line, so a failing test keeps
its stub log and rendered script for inspection.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
(cherry picked from commit d5559bb)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
The bootstrap Job wrote a ClusterRoleBinding per oidc.users entry into
its script, and the script reaches the container as one `sh -c`
argument. Linux caps a single argument at 128 KiB, which about 240
users cross at roughly 530 bytes each. The container then cannot start,
the post-install and post-upgrade hook fails, and the tenant cluster's
HelmRelease stays unready. The values schema puts no bound on
oidc.users, so a shared cluster reaches that size with ordinary
per-person entries.

The bindings and the list of their names now reach the Job through a
ConfigMap it mounts, so the script is the same size for any list. Names,
labels and roles of the bindings are unchanged, so the bindings already
in a tenant cluster are updated in place and none is orphaned.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>
(cherry picked from commit 7b7444f)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…#5212)

## What this PR does

Fixes #4968. The cilium
IPAM migration hook passed the whole `kubectl get ciliumnodes -o json`
to `jq` as one argument, and Linux allows 128 KiB for a single argument.
Cilium copies Node labels onto every CiliumNode (22 to 30 KiB per node
with Talos, NFD or GPU labels), so the list is over the limit within six
nodes. `jq` can't start (`Argument list too long`), the hook Job fails
before it moves any node, `cilium` HelmRelease loops on `UpgradeFailed`
and everything that depends on networking stays in `DependencyNotReady`.

The hook is rendered on every upgrade of cilium in `cluster-pool` mode,
so a cluster that already finished the migration hits the same on its
next values change. The failing step only logs the pods that still hold
an old Ingress IP, but it runs before the node loop, so on a cluster
that is not migrated yet it blocks the migration itself.

Now the list goes to `jq` on stdin together with the IP objects, same as
the other `jq` calls in the script get their input.

The new test in `hack/cilium-ipam-migration.bats` makes six CiliumNodes
over the limit (one 24 KiB annotation each instead of labels, and it
checks the list is over 128 KiB even printed compact). It fails on the
old script with status 126 and passes now. It also checks that each pod
is named with the node whose Ingress IP it holds and that a pod on an
unrelated address is not named, because a join that always names the
first node, or reports every IP object, passed the old tests.

I did not take two suggestions from the issue. Skipping the hook when
every CiliumNode is in the pool changes what one existing test pins (the
hook adds the pool to `excludeIps` in that case) and doesn't help
clusters that are not migrated yet. A value to disable the hook leaves
the default broken. `cozystack-operator` rewriting `disableHooks` and
helm-controller rolling the DaemonSet on every retry are other
components. The bug is in release-1.6 and release-1.7 as well.

### Downstream repositories

I walked the trigger map against the diff (one hook script and its bats
test): no package, values, schema, default or output name changed.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(cilium): the IPAM migration hook no longer fails with "Argument list too long" on clusters with more than a few nodes, which blocked the cilium upgrade and every package that depends on networking.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved IP address migration handling for environments with large
node lists, avoiding failures caused by operating system argument-size
limits.
* Migration continues to identify affected pods by node and distinguish
unrelated pods.
* **Tests**
* Added coverage for migrations involving large node lists, multiple IP
objects, and unrelated pods.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: ab13f9f
IvanHunters and others added 12 commits November 10, 2026 12:52
Update the tap command help and the community-index README for the new
behaviour: digest references, the automatic tag-to-digest pin, the
--certificate-identity / --certificate-oidc-issuer pull-time verify flags, and
the optional index entry digest. Extend the generated openapi definitions and
model-name helper for the new TapVerify type (code generation is otherwise
run in CI).

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit 5747f94)
Signed-off-by: Myasnikov Daniil <[email protected]>
A re-tap applies the OCIRepository under field owner cozypkg via server-side
apply; a field this owner set earlier and now omits is removed. Re-tapping to
move the pin without repeating --certificate-identity/--certificate-oidc-issuer
therefore silently dropped the signature-verification policy, and re-tapping a
private repo without --secret dropped the pull credential. The CLI now carries
an existing source's spec.verify and secretRef onto the applied object when the
invocation does not set them, mirroring the API connect path, so moving the pin
never silently downgrades trust or breaks a private pull.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit 0fd9703)
Signed-off-by: Myasnikov Daniil <[email protected]>
TapVerify advertised provider notation, but buildVerifySpec always emits a
matchOIDCIdentity block, which is meaningful only for keyless cosign. Setting
provider: notation produced a nonsensical verify block. The connect now rejects
any provider other than cosign and the type doc no longer advertises notation,
since only keyless cosign identity matching is settable here.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit 10be3d2)
Signed-off-by: Myasnikov Daniil <[email protected]>
The repeat-connect merge preserved spec.verify but still dropped spec.secretRef
when the connect omitted a secret, breaking a private tap on a pin move. It now
preserves an existing secretRef the same way, so the API and CLI connect paths
are symmetric: a repeat connect overrides verify and the secret only when it
carries its own, and never silently drops either.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit f8e3b0d)
Signed-off-by: Myasnikov Daniil <[email protected]>
The index-entry `digest` field, and the gate and README claims around it, assume
a published index that does not exist yet: decision 0002 in the marketplace
proposal defers it, and #4981 tracks that work. The gate also never compared a
recorded digest against the artifact it verified, so the field added surface
without the check that would make it trustworthy.

Drop it from this PR. Digest pinning still works from an explicit
oci://host/org/repo@sha256:... reference and from the digest a tag resolves to;
only the index-entry path is removed. The tag-based pinning and the verify-flag
wording stay in the index README.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit 96758b9)
Signed-off-by: Myasnikov Daniil <[email protected]>
…ifact

The materializer pinned whatever digest status.artifact reported, even when the
artifact still described the previous reference. On a re-connect to a new tag,
source-controller has not re-resolved yet, so the operator materialized and
pinned the old tag's content (issue #4979). Gate materialization and pinning on
the artifact reflecting the current spec: status.observedGeneration equals
metadata.generation, and for a tag-only source a revision that belongs to that
tag (<tag>@<digest>).

A tap connected before digest pinning existed already has its
materialized-revision equal to the artifact revision, so Reconcile short-circuited
and never pinned, letting it follow the tag one more time. Pin the current digest
before that early return.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit 7494d1f)
Signed-off-by: Myasnikov Daniil <[email protected]>
… identity match

With --certificate-identity and --certificate-oidc-issuer, `cozypkg tap` pulled by
tag without verifying and applied the artifact's PackageSources with
ForceOwnership; only the Flux source carried the verify policy. A tag moved by
someone without the signing identity could land unsigned content in the cluster.
Verify the signature at tap time and pin the verified digest, so the content
pulled and applied is what a trusted identity signed.

Flux evaluates a cosign matchOIDCIdentity issuer and subject as Go regexps, and
they were written unanchored. An exact identity such as [email protected] would
also accept [email protected], and the dots would match any character. Anchor
and quote both values at the API and the CLI.

Reject --tag together with a digest reference instead of silently ignoring the
tag.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit 0195e73)
Signed-off-by: Myasnikov Daniil <[email protected]>
…cover the generation gate

A verified cosign signature whose manifest digest could not be parsed fell back
to an unpinned tag pull, which reopens the tag-mutation window the verification
was meant to close. Return an error instead.

Add a test that isolates the status.observedGeneration == metadata.generation
conjunct of artifactReady; the existing tests exercised only the tag-prefix
conjunct, so a mutation of the generation check went uncaught.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
(cherry picked from commit 09b0688)
Signed-off-by: Myasnikov Daniil <[email protected]>
…table tag (#5082)

## What this PR does

A tapped (community-connected) repository followed a mutable OCI tag:
the Flux `OCIRepository` carried only `spec.ref.tag` with a 5-minute
interval, so whoever controlled the tag controlled what ran in the
management cluster within that interval, down to the HelmReleases of
already-running tenant application instances. Fixes #4979.

A tap is now pinned to a digest. `cozypkg tap` and the connect API
accept an explicit `oci://host/org/repo@sha256:...` reference; a tag
reference is resolved and pinned to its digest by the tap materializer
once the artifact for the current spec generation is ready, from the
digest Flux reports in `status.artifact.revision`, with the tag kept
informational. A tap connected before this change is pinned to its
current digest on the first reconcile after upgrade, and a re-connect to
a new tag waits for source-controller to re-resolve before pinning, so a
stale revision is never pinned. Moving the tag upstream no longer
changes anything in the cluster; an explicit re-connect moves the pin.

Connect also accepts an optional keyless cosign verification policy
(`--certificate-identity` / `--certificate-oidc-issuer` on the CLI,
`spec.verify` on the API), anchored to an exact identity so a
superstring cannot match, and preserves it, and the pull secret, across
repeat connects rather than silently dropping them. With those flags the
CLI verifies the signature at tap time and pins the verified digest
before pulling, so the content it applies is what a trusted identity
signed; it fails closed if the verified digest cannot be determined.

### Screenshots

Not applicable: no UI changes.

### Downstream repositories

I walked the trigger map against the diff. The change is confined to
`cozypkg`, the `Tap` aggregated API (a virtual, cluster-scoped resource
with no CRD), the tap materializer, and the community-index docs. It
does not touch `packages/apps`, `packages/core/platform/values.yaml`,
installer values, platform variants/components, release assets,
`ApplicationDefinition` semantics, the `hack/` make targets, or the
commit convention, so the provider, ansible, ccp and external-apps paths
are not reached.

One path is arguable and needs a human decision, so I have left every
box unticked rather than tick or dismiss it:

- `cozystack/website` — `cozypkg tap` gains flags and digest-pinning
behaviour. The marketplace tap flow is new and I am not sure it has a
reference page on the site yet, so whether a docs follow-up is warranted
(and where) is a maintainer call.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(marketplace): a tapped repository is pinned to a digest instead of following a mutable tag, so moving the tag upstream no longer changes what runs in the cluster; connect accepts oci://...@sha256:... and an optional keyless cosign verify policy (anchored to an exact identity and preserved across re-connects), and with verify flags the CLI verifies the signature and pins the verified digest at tap time.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

- **New Features**
- OCI repositories can be connected using a SHA-256 digest, with or
without a tag. Tag-based sources are pinned to the resolved artifact
digest.
- Optional Cosign keyless verification is available during connection
using certificate identity and OIDC issuer details.
- Reconnecting preserves existing verification and pull-credential
settings when replacement values aren’t provided.
- **Bug Fixes**
- Reconnecting with a new tag clears previous operator-managed pins so
the requested tag can resolve to its current artifact.
- **Documentation**
- Clarified tag pinning, signature verification, and where tapped app
charts run.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: 94ada11
… image

The Altinity and RabbitMQ strategy Pods ran platform-migrations, a 349 MB
image carrying kubectl, helm and etcd-migrate whose ENTRYPOINT is the
migration runner, and whose pins were stamped from packages/core/platform
so every migration change moved the backup clients too.

Build images/backup-http-client (alpine + curl + jq + ca-certificates,
non-root) from this package's Makefile and stamp both client keys from
it. The cross-package stamp in packages/core/platform/Makefile and the
platform-migrations-specific check in hack/image-pin-consistency.bats go
away; the general one-digest-per-repository check still covers the two
keys sharing one image.

Fixes #3420

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <[email protected]>
(cherry picked from commit 6474075)
Signed-off-by: Myasnikov Daniil <[email protected]>
docs/agents/image-refs.md forbade committing a bare tag without saying
how a new image's first pin can carry a digest it does not have yet.
Record the exception the backup client keys rely on: the bare tag is a
placeholder the producing `make image` replaces, and an unbuilt tree
pulls a tag that does not exist. Drop the securitygroup-controller
example, which is pinned by digest now.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <[email protected]>
(cherry picked from commit 770f515)
Signed-off-by: Myasnikov Daniil <[email protected]>
… image (#4756)

## What this PR does

The Altinity (ClickHouse) and RabbitMQ backup strategy Pods ran
`platform-migrations`: a 349 MB image carrying `kubectl`, `helm`, `git`,
`cozyhr` and `etcd-migrate`, whose `ENTRYPOINT` is `run-migrations.sh`.
Both pins were stamped from `packages/core/platform/Makefile`, so any
migration change re-digested the backup clients too, and only the
`command:` override kept the Pods from running platform migrations
(#3420).

- New
`packages/system/backupstrategy-controller/images/backup-http-client/`:
`alpine:3.24` (digest-pinned, the same base the tree already uses) +
`curl` + `jq` + `ca-certificates`, `USER 65532:65532`, no `ENTRYPOINT`.
Everything is installed at build time, so the Pods still never run `apk
add` and keep working air-gapped.
- New `image-backup-http-client` target in the package's own `Makefile`,
chained after `image-redis-backup` like the existing ones so only one
`yq -i` writes `values.yaml` at a time. It stamps `chBackupClientImage`
and `rabbitmqBackupClientImage` with the same ref.
- Removed the cross-package stamps from
`packages/core/platform/Makefile`, the WORKAROUND/TODO comments in the
chart and the RabbitMQ example, and the stale "live violation" line in
`docs/agents/image-refs.md`.
- `hack/image-pin-consistency.bats`: dropped the
`platform-migrations`-specific test, whose subject no longer exists. The
general check (no repository pinned at more than one digest, computed
from `hack/promote-retag.sh --dry-run`) still covers the two keys
sharing one image.

**Why one image for both strategies rather than `ch-backup-client` as
the issue named it.** The RabbitMQ strategy had the same reuse and the
same cross-package stamp; leaving it would have kept the stamp alive.
Its script needs `curl --aws-sigv4` + `ca-certificates`, a subset of the
ClickHouse one, so one image serves both. The two values keys stay
separate so either strategy can move to its own image later.

**Initial pin.** The image does not exist in ghcr until CI builds it, so
both keys are committed as `backup-http-client:v1.6.0` without a digest,
the same way `redis-backup` was introduced. PR CI rebuilds touched
packages and overlays the fresh refs for e2e; release preparation's
`make image` stamps tag+digest.

Image size: 6.9 MB vs 349 MB. Verified locally:
- `docker buildx build` for `linux/amd64,linux/arm64`; in the image as
uid 65532: curl 8.22 with `--aws-sigv4`, jq 1.8.2, `date`, `seq`, writes
to `/tmp` (the RabbitMQ script's only write), TLS verification against
the bundled CA store.
- `helm unittest` (23/23), `helm template` renders both strategies with
`backup-http-client` and no `platform-migrations` reference.
- `hack/cozytest.sh hack/image-pin-consistency.bats` and
`hack/build-matrix_test.bats` pass.

Not yet run end to end on a cluster; the e2e backup round-trips will
exercise it.

Fixes #3420

### Screenshots

N/A — no UI changes.

### Downstream repositories

Walked the trigger map against the diff: it touches
`packages/system/backupstrategy-controller` (new image, Makefile target,
values comments/pins, template comments),
`packages/core/platform/Makefile` (stamp removal only, no `values.yaml`
key change), `hack/image-pin-consistency.bats`,
`docs/agents/image-refs.md` and a comment in
`examples/backups/rabbitmq`. No app package, schema, CRD, variant,
asset, platform values key or `hack/package.mk` change. The website does
not list first-party image repositories (searched for
`platform-migrations` and `redis-backup`), so no mirror list needs the
new one.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(backupstrategy-controller): the ClickHouse (Altinity) and RabbitMQ backup strategy Pods now run a dedicated `backup-http-client` image (alpine + curl + jq + ca-certificates, non-root, ~7 MB) instead of the 349 MB `platform-migrations` image. Air-gapped mirrors need the new `ghcr.io/cozystack/cozystack/backup-http-client` repository.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Improvements**
* Altinity and RabbitMQ backups now use a dedicated backup HTTP client
image, configured separately from the platform migrations image.
* The client image includes the tools required for backup operations and
runs with a non-root identity.
* Both backup configurations receive the same digest-pinned image
reference when the image is built.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backported-from: 7c46fff
@myasnikovdaniil
myasnikovdaniil merged commit 3cd9d4c into release-1.7 Nov 11, 2026
61 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-batch/release-1.7-20261109 branch November 11, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/release Issues or PRs related to release tooling (changelog, backport, release pipeline) kind/bug Categorizes issue or PR as related to a bug size/XXL This PR changes 1000+ lines, ignoring generated files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

8 participants