Repository navigation
fix(backport): batch for release-1.6, 2026-11-09 - #5247
cozystack-ci[bot] wants to merge 24 commits into
Conversation
…t back A rebase re-applied the namespaced CiliumNetworkPolicy that the move to source-side clusterwide policies had deleted, so two policies now guard the same webhook port and the older one is the worse of the two. It is gated only on admissionWebhookPolicy.enabled, which defaults to true, and never checks that the Cilium API exists. On a cluster without Cilium every tenant ingress release therefore fails to install with `no matches for kind "CiliumNetworkPolicy"`. On a Cilium cluster it repeats the target-side deny the clusterwide policy already carries. It also renders for child tenants, whose controller runs with the admission webhook disabled and for which the clusterwide policy renders nothing, so there it denies a port nothing listens on and is the only thing that puts a cluster-spanning peer onto the controller endpoints. That peer is enumerated per pod identity, the coupling to namespace churn the clusterwide policy was written to contain. The clusterwide policy and the admissionWebhookPolicy value it reads are left as they are. The existing capability suite selects only the clusterwide template, so it could not have noticed this file coming back; a suite with no template filter now fails if any Cilium kind renders while the Cilium API is absent. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]> (cherry picked from commit 8c1d2db) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…t back (#5088) ## What this PR does Rebase behind [#3931](#3931) brought back namespaced `CiliumNetworkPolicy` for admission webhook, which [#3997](#3997) deleted in favour of clusterwide policy. It renders only on `admissionWebhookPolicy.enabled` (true by default) and never checks that Cilium API exists, so on cluster without Cilium every tenant with `ingress: true` fails to install with `no matches for kind "CiliumNetworkPolicy"`. On cluster with Cilium it repeats deny that clusterwide policy already has, and also renders for child tenants where webhook is disabled. This PR removes the template with its test and adds suite that renders whole package without Cilium API and fails if any Cilium resource comes out. Existing capability suite selects only clusterwide template, so it could not catch this file coming back. Clusterwide policy and `admissionWebhookPolicy` value are unchanged. Fixes #4706 On cluster without Cilium nothing in this chart restricts webhook port. Removed policy could not do it either because it needs Cilium CRD to install at all, and clusterwide policy already renders only where Cilium API is served, so no protection is lost, such cluster now just installs ingress instead of failing. Only `tenant-root` runs webhook, child tenants set `admissionWebhooks.enabled: false`. Restricting webhook port on non-Cilium clusters is separate design question, not part of this PR. Checked `helm template` with and without `cilium.io/v2/CiliumClusterwideNetworkPolicy` in api versions: without it no Cilium object renders, with it only clusterwide policy, same as before. New suite fails on tree before this change. ### Screenshots Not a UI change. ### Downstream repositories Only template and tests under `packages/system/ingress-nginx` changed, `admissionWebhookPolicy` value kept, nothing from trigger map in `docs/agents/contributing.md` matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ingress-nginx): remove the namespaced admission-webhook CiliumNetworkPolicy that was re-added by mistake. It was rendered without checking that the Cilium API exists, so on clusters without Cilium every tenant ingress failed to install with `no matches for kind "CiliumNetworkPolicy"`. The clusterwide admission-webhook policy is unchanged and still renders only where the Cilium API is served. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Configuration** * The ingress-nginx chart no longer generates an admission webhook network policy. Deployments that relied on this policy to restrict access to the webhook will need to configure equivalent traffic controls separately. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: b366286
|
Important Review skippedBot user detected. To trigger a single review, invoke the
|
|
This batch has been open since 2026-11-09, 3 days, through at least one scheduled run. Merge it, or close it and the next run starts a new one; until then each run adds what the rules take to it. These conflicts wait for a maintainer, as resolving a conflict describes; unresolved, each goes to the next batch: These change Updated by the backport batch on 2026-11-12. |
The hook fed the whole `kubectl get ciliumnodes -o json` to jq as one argument. Cilium copies every Node label onto its CiliumNode, 22 to 30 KiB per node on Talos or with NFD and GPU labels, so the list passes the 128 KiB Linux allows a single argument within six nodes. jq then fails to start with "Argument list too long", the hook Job fails before it moves any node, and the cilium HelmRelease loops on UpgradeFailed with the HelmReleases behind it stuck in DependencyNotReady. The hook renders on every upgrade of cilium in cluster-pool mode, so a cluster that already finished the migration hits the same failure on its next values change. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]> (cherry picked from commit a2d0e4c) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
The bootstrap Job wrote a ClusterRoleBinding per oidc.users entry into its script, and the script reaches the container as one `sh -c` argument. Linux caps a single argument at 128 KiB, which about 240 users cross at roughly 530 bytes each. The container then cannot start, the post-install and post-upgrade hook fails, and the tenant cluster's HelmRelease stays unready. The values schema puts no bound on oidc.users, so a shared cluster reaches that size with ordinary per-person entries. The bindings and the list of their names now reach the Job through a ConfigMap it mounts, so the script is the same size for any list. Names, labels and roles of the bindings are unchanged, so the bindings already in a tenant cluster are updated in place and none is orphaned. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]> (cherry picked from commit 7b7444f) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…#5212) ## What this PR does Fixes #4968. The cilium IPAM migration hook passed the whole `kubectl get ciliumnodes -o json` to `jq` as one argument, and Linux allows 128 KiB for a single argument. Cilium copies Node labels onto every CiliumNode (22 to 30 KiB per node with Talos, NFD or GPU labels), so the list is over the limit within six nodes. `jq` can't start (`Argument list too long`), the hook Job fails before it moves any node, `cilium` HelmRelease loops on `UpgradeFailed` and everything that depends on networking stays in `DependencyNotReady`. The hook is rendered on every upgrade of cilium in `cluster-pool` mode, so a cluster that already finished the migration hits the same on its next values change. The failing step only logs the pods that still hold an old Ingress IP, but it runs before the node loop, so on a cluster that is not migrated yet it blocks the migration itself. Now the list goes to `jq` on stdin together with the IP objects, same as the other `jq` calls in the script get their input. The new test in `hack/cilium-ipam-migration.bats` makes six CiliumNodes over the limit (one 24 KiB annotation each instead of labels, and it checks the list is over 128 KiB even printed compact). It fails on the old script with status 126 and passes now. It also checks that each pod is named with the node whose Ingress IP it holds and that a pod on an unrelated address is not named, because a join that always names the first node, or reports every IP object, passed the old tests. I did not take two suggestions from the issue. Skipping the hook when every CiliumNode is in the pool changes what one existing test pins (the hook adds the pool to `excludeIps` in that case) and doesn't help clusters that are not migrated yet. A value to disable the hook leaves the default broken. `cozystack-operator` rewriting `disableHooks` and helm-controller rolling the DaemonSet on every retry are other components. The bug is in release-1.6 and release-1.7 as well. ### Downstream repositories I walked the trigger map against the diff (one hook script and its bats test): no package, values, schema, default or output name changed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(cilium): the IPAM migration hook no longer fails with "Argument list too long" on clusters with more than a few nodes, which blocked the cilium upgrade and every package that depends on networking. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved IP address migration handling for environments with large node lists, avoiding failures caused by operating system argument-size limits. * Migration continues to identify affected pods by node and distinguish unrelated pods. * **Tests** * Added coverage for migrations involving large node lists, multiple IP objects, and unrelated pods. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: ab13f9f
The package pinned clusterName to cozy.local, so on a cluster installed with any other networking.clusterDomain the operator resolved its own MariaDB pods as <name>.<ns>.svc.cozy.local and got NXDOMAIN. Replication between replicas was never configured, and User, Database and Grant resources failed to reconcile, with the operator only retrying at a verbose log level. clusterName is a vendored subchart value, out of reach of the _cluster lookup that package templates use, so the platform now carries it under the subchart key in the shared values Secret, the same route KEDA and the OpenSearch operator already take. The literal in the package stays as the default for a render without that Secret. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]> (cherry picked from commit c70f83b) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
keys returns a map's keys in Go's randomized iteration order, so a release with several databases rendered DEFINED_DBS in a different order on each upgrade. The init-script Secret, and the checksum the init Job carries, then changed on upgrades that left the databases untouched, which makes a no-op upgrade look like a real change in the release history. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]> (cherry picked from commit dc94660) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
#5213) ## What this PR does Found while checking for other places with the failure from #4968: data that grows with configuration is passed to a process as one argument, and Linux allows 128 KiB for that. This one is in the `kubernetes` app and there is no issue about it yet. The OIDC bootstrap Job wrote a ClusterRoleBinding for every `oidc.users` entry into its `sh -c` script, 526 bytes per user. I rendered current main with `helm template` and users like `[email protected]`: 238 users give a 130,889 bytes script that starts, 239 give 131,415 bytes and `exec` fails with E2BIG. Then the container can't start, the post-install and post-upgrade hook fails after its retries and the HelmRelease of the tenant cluster stays unready. Schema has no limit for `oidc.users` and each person needs own entry, so a cluster shared by 240 people gets there. Now the bindings and the list of their names are rendered by the new `oidc-rbac-bindings.yaml` into a ConfigMap `<release>-oidc-bindings` that the Job mounts, so the script is 6,380 bytes for any number of users. The ConfigMap has its own 1 MiB limit, about 2,500 users at 417 bytes each. Names, labels and roles of the bindings stay the same. I compared the render of old and new chart for eight emails (ones YAML reads as a boolean or null, a number, a quote, an apostrophe, unicode and two plain) and the bindings are identical, so existing ones are updated in place and nothing is orphaned. The prune step reads the same ConfigMap. If the list is not mounted the Job stops before it deletes anything, because `grep` on a missing file exits 2, `! grep` is true and every binding of the release would be pruned. A release with no users applies nothing (`kubectl apply` fails on an empty file) and prunes all stale bindings, same as before. `hack/kubernetes-oidc-bindings.bats` renders 300 users, checks the script is under the limit without any email in it and that every user has one binding with the old name and role. Three more tests run the rendered script against a fake `kubectl` and check what it applies and deletes. The prune condition forced to true or to false passes all helm-unittest cases, so this part is tested in bats. The bug is in release-1.6 and release-1.7 as well. ### Downstream repositories I walked the trigger map against the diff (one chart template, its tests and a bats file): no package added or removed, no values, schema or default changed, no output Secret or Service renamed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubernetes): the OIDC bootstrap Job no longer fails to start for a tenant cluster with about 240 or more oidc.users, the bindings now come from a ConfigMap the Job mounts. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Kubernetes OIDC configuration now supports per-user role bindings, with administrator and viewer access assigned according to each user’s role. * OIDC role bindings are kept in sync with the configured user list, including removal of obsolete bindings. * Deployments with no OIDC users or with OIDC mode disabled continue to avoid creating user bindings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: e5bb3cd
The helm-unittest sweep reaches a package only through its make test target, so the suite added with the cluster-name fix would otherwise never run in CI. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]> (cherry picked from commit f21abe0) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The postgres init script lists the declared databases with `keys .Values.databases`, which returns them in Go's randomized map order. A release with two or more databases therefore renders a different `DEFINED_DBS` line on each upgrade, so the `<release>-init-script` Secret and the `checksum/config` annotation of the init Job change on upgrades that did not touch the databases. Sorting the keys makes the render a function of the values alone. Found while measuring a platform-wide values change on a production cluster (#5092): of 36 releases that ran Jobs again, 31 rendered a byte-identical manifest across the two Helm revisions and the other 5 were Postgres releases whose only difference was this ordering, e.g. `DEFINED_DBS="db_a db_b"` → `DEFINED_DBS="db_b db_a"`. This does not stop the init Job from running on every upgrade (it is a `post-upgrade` hook); the issue covers that part. Verification: the new `tests/init_script_deterministic_test.yaml` renders eight databases declared out of order and expects them sorted. Without the fix it failed 5 runs out of 5 with a different order each time; with it, `helm unittest packages/apps/postgres` passes (27 suites, 210 tests). ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(postgres): the init script renders the declared databases in a stable order, so an upgrade that leaves the databases unchanged no longer rewrites the init-script Secret. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Database initialization now lists configured databases in alphabetical order, providing consistent output across deployments. * **Tests** * Added coverage to verify the alphabetical ordering. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: 0c48626
…in (#5123) <!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Fixes #2348. `packages/system/mariadb-operator/values.yaml` pins `mariadb-operator.clusterName: cozy.local`, and the vendored chart turns it into the operator's `CLUSTER_NAME` and into a SAN of the webhook certificate. On a platform installed with any other `networking.clusterDomain`, the operator resolves its own pods as `<name>.<ns>.svc.cozy.local` and gets NXDOMAIN. Observed on a cluster with `clusterDomain: lab.local`: a 2-replica MariaDB never gets replication configured (`status.replication` stays `{}`, the secondary runs as a standalone semi-sync master), and User, Database and Grant fail with `lookup mariadb-<name>-primary.<ns>.svc.cozy.local ... no such host`, while the operator only retries at verbosity 1. `clusterName` is a value of the vendored subchart, which the `_cluster` lookup used by package templates cannot reach. The platform now carries `networking.clusterDomain` under the `mariadb-operator` subchart key in the shared `cozystack-values` Secret, the same route `keda.clusterDomain` and `opensearch-operator.manager.dnsBase` already take. The literal in the package stays as the fallback for a render without that Secret, so a cluster on `cozy.local` renders exactly as before. Tests cover both ends of the path: `packages/core/platform/tests/apps_mariadb_operator_cluster_name_test.yaml` pins that the Secret carries the domain under the anchored subchart key and tracks a non-default value (it fails with the `apps.yaml` change reverted), and `packages/system/mariadb-operator/tests/cluster_name_test.yaml` pins that a domain set on that key reaches the operator's `CLUSTER_NAME` and the webhook certificate, with `cozy.local` kept as the standalone default. `make helm-unit-tests` passes locally. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(mariadb-operator): the MariaDB operator now uses the platform's networking.clusterDomain instead of a hard-coded cozy.local, so replication and User/Database/Grant reconciliation work on clusters with a non-default cluster domain. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * MariaDB now uses the configured networking domain as its cluster name and webhook certificate address. If no custom domain is configured, it continues to use the default cluster name and address. * **Tests** * Added coverage for default and custom domain settings, including their effect on the MariaDB cluster name and webhook certificate address. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: d61db78
mariadb-operator reports a failed backing Job as Complete=True with reason JobFailed, and Complete=True alone was read as success. A backup whose dump or upload failed therefore ended Succeeded and left a Ready Backup pointing at an archive that does not exist, and a scheduled Plan kept producing such Backups; a failed restore ended Succeeded too. Success now requires reason JobComplete, and JobFailed fails the job whatever the status. Signed-off-by: Alexey Artamonov <[email protected]> Assisted-by: LLM (cherry picked from commit ccac0e1) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
Every first reconcile after the operator Backup or Restore is created sees no Complete condition, and the guards that keep that case from dereferencing nil could be dropped with the outcome tables staying green. The tables now also cover the deadline failure for a run that never reports a known reason, and read the job status back from the API, so a lost status write fails them instead of passing on the in-memory object. Assisted-by: LLM Signed-off-by: Alexey Artamonov <[email protected]> (cherry picked from commit 7e1fc71) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
The operator's JobFailed condition message is only "Failed", and with the operator's default restartPolicy OnFailure the Job controller deletes the pod once the backoff limit is spent, before the operator sets JobFailed. Pointing the reader at the Job's logs therefore sent them to logs that no longer exist. The Job, which the operator names after the Backup or Restore, keeps its Failed condition with the reason and message, so the BackupJob or RestoreJob now carries those instead; a Job that cannot be read is named in the message rather than dropped. Assisted-by: LLM Signed-off-by: Alexey Artamonov <[email protected]> (cherry picked from commit 9f51b9d) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…Secret Nothing reads the Secret: the kubeadm certs.d mount it fed is gone and the Talos worker config never took it. The value stayed in the Package spec as a knob that accepts mirrors and does nothing, which is worse than no knob for an operator in an air-gapped setup. Mirrors for tenant workers are a per-pool field of the KubernetesNodes chart. A Package that still sets registries keeps installing: the platform chart has no values schema, so the key is ignored. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <[email protected]> (cherry picked from commit e79946e) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
Hierarchical quota admission subtracts the parent pool's current usage from what a child tenant may still claim, and reads that usage from ResourceQuota.status.used. It looked every non-raw key up under limits.<key>, but the ResourceQuota pod evaluator reports usage under limits.* only for cpu, memory and ephemeral-storage. Extended resources and hugepages are counted under requests.* alone, so their usage always read as zero and a child could be granted budget the parent pool had already consumed. Read usage from requests.<key> for every resource the evaluator counts only on the request side. flatten applies no allocation ratio there, so that key holds the declared amount just as limits.<key> does for cpu, memory and ephemeral-storage. Fixes: e3f09ba ("fix(apiserver): subtract parent pool usage in hierarchical tenant quota admission") Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]> (cherry picked from commit 54695fc) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
The operator sets Complete=False with reason Failed and its own cause
in the message ("Error creating Job", "Backup not complete") when a
run fails before or outside its Job. Such a run waits out the
deadline, and the deadline failure kept only the status and reason,
so the BackupJob or RestoreJob ended without the one line that says
why.
Assisted-by: LLM
Signed-off-by: Alexey Artamonov <[email protected]>
(cherry picked from commit 69cea9c)
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
The air-gapped note said in-guest image pulls honour no per-tenant mirror until a Phase 2 that has landed, so an operator setting up an air-gapped cluster would read that mirroring is unavailable and file an issue. It now says the Secret and the platform registries value are removed, that a Package still setting the value keeps installing, and that worker mirrors are set per pool through talos.registryMirrors of the KubernetesNodes resource. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <[email protected]> (cherry picked from commit 41507e3) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…5184) ## What this PR does Tenant quota admission does not see how much of an extended resource or hugepages the parent pool already uses, so a child tenant can be granted a share the parent has already spent. Admission reads that usage from `limits.<key>`, but Kubernetes reports it only under `requests.<key>`, so the value is always zero. Now each key is read where the quota controller counts it. cpu, memory and ephemeral-storage still read `limits.*`. The new tests fail on main. ### Screenshots ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(api): tenant quota admission now counts the parent pool's usage of extended resources and hugepages, so a child tenant can no longer get a share the parent has already used. After upgrading, updates to existing tenants may be rejected if their quota for these resources exceeds the parent's remaining budget. This includes updates that leave the quota unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Quota checks now account for existing usage of extended resources, preventing child requests from exceeding the parent’s remaining quota. * Requests are allowed when current usage leaves enough quota available, and denied when usage has consumed the parent’s full quota. * Resource usage is matched against the appropriate quota category, including requests for extended resources and hugepages. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: 63abbc8
…ition The reason list read as everything the operator writes on Complete, yet mariadb-operator also sets Complete=False with reason Failed when a run fails before or outside its Job. A reader taking the list as exhaustive would treat Failed as an unknown reason, or make it terminal, while the operator retries those failures. A table row pins that the driver keeps waiting on it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <[email protected]> (cherry picked from commit 5f0da5a) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…Secret (#5190) Nothing reads the `patch-containerd` Secret: the kubeadm certs.d mount it fed is gone, and the v1.6.0 changelog already says the passthrough is removed. The `registries` value stayed in the Package spec and takes mirrors that do nothing, which is worse than no knob for someone setting up an air-gapped cluster. Removed both. A Package that still sets `registries` keeps installing, the platform chart has no values schema so the key is ignored. On upgrade the Secret in `cozy-system` is deleted by the release. Mirrors for tenant workers are the per-pool `talos.registryMirrors` of KubernetesNodes. I checked that nothing is left: `git grep` for `.Values.registries` and for `patch-containerd` outside the changelogs finds nothing but the kubernetes README sentence that says the Secret was removed. Full `make unit-tests` passes. Fixes #4006 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Configuration Changes** * The platform chart no longer supports the `registries` configuration option or generates containerd registry secrets. Mirror endpoint settings, including credentials and TLS verification options, are no longer applied through this chart. * Registry mirrors can be configured per worker pool using `talos.registryMirrors` on `KubernetesNodes`. Existing packages that set `registries` continue to install, but the setting is ignored. * **Documentation** * Updated guidance for configuring registry mirrors on worker pools. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: 91f03dd
…t's vminsert The monitoring-agents chart reads vmagent.remoteWrite.urls as a list and takes the tenant label from global.target, while the kubernetes app still passed a single remoteWrite.url and vmagent.externalLabels.tenant. Both keys were dropped in silence, so the vmagent of every tenant cluster fell back to the chart defaults: it wrote to vminsert services in cozy-monitoring that do not exist inside the tenant cluster, queued the failed writes on the node disk, and labelled every series tenant="cozy-monitoring". No tenant cluster metric reached the tenant's Monitoring. The tenant label has to name the tenant that owns the cluster, which is not always the tenant whose Monitoring receives the data: a tenant without its own Monitoring inherits a parent's. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <[email protected]> (cherry picked from commit 55d6c52) Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com>
…5207) <!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does When the Job behind a mariadb-operator `Backup` or `Restore` fails, the operator sets `Complete=True` with reason `JobFailed` (`SetCompleteWithJob` in mariadb-operator 25.10.2). The MariaDB driver read `Complete=True` alone as success, so a backup whose dump or upload failed ended `Succeeded` and produced a `Ready` Cozystack `Backup` pointing at an archive that does not exist, and a failed restore ended `Succeeded` too. A scheduled `Plan` kept producing such backups. The driver now treats `Complete=True` as success only with reason `JobComplete`, and fails the BackupJob or RestoreJob on `JobFailed` whatever the status, without creating a `Backup`. The failure message names the operator object and carries the Job's own `Failed` reason and message, since the operator's condition message is only `Failed` and the Job's pod, with its logs, is already deleted when the Job fails. A Job that is missing or cannot be read is named in the message instead. Any other `Complete` state keeps waiting under the existing deadline, as before; that includes `Complete=False` with reason `Failed`, which the operator sets with its own cause (`Error creating Job`, `Backup not complete`) for a failure outside the Job and then retries, and the deadline failure now carries that message. Regression tests drive the backup and restore reconcilers through every `Complete` status/reason combination, an absent `Complete` condition, and expired deadlines for an absent condition, an unknown reason and reason `Failed`. They assert the job phase and message as persisted in the API rather than on the in-memory object, a requeue while the run is in progress, and, for backups, that no `Backup` is created on failure. Separate tests cover the failure message for a Job that cannot be read and for a Job without a `Failed` condition. Backups that already exist are not touched. A `Ready` MariaDB `Backup` created by a failed run before this fix may have no archive of its own. Restoring from it can fail or select another dump from the shared prefix. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(mariadb): a MariaDB BackupJob or RestoreJob whose operator Job failed now ends Failed instead of Succeeded, and a failed backup no longer creates a Ready Backup without an archive. Backups created by failed runs before this fix are not removed; restoring from one can fail or select another dump from the shared prefix. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Backup and restore operations now report failed jobs as failures, including details from the underlying job when available. * Operations are marked successful only when the job completes successfully. Other outcomes continue to be monitored until the existing deadline, after which failures include available condition details. * Failed backups no longer create backup artifacts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: 456473d
…t's vminsert (#5024) <!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The vmagent that `packages/apps/kubernetes` deploys into a tenant Kubernetes cluster has not been shipping metrics anywhere, because the app and the `monitoring-agents` chart disagree on two values keys, and Helm drops a key nobody reads without a word. - **Remote write URL.** #697 changed the chart to read `vmagent.remoteWrite.urls` as a list, but the app still passed a single `vmagent.remoteWrite.url` (the cluster domain was later added to that ignored key in #2086). The chart then applied its own defaults, `vminsert-{shortterm,longterm}.cozy-monitoring.svc`, which do not exist inside a tenant cluster. vmagent fails with `no such host` and queues the failed writes on the node disk, up to `-remoteWrite.maxDiskUsagePerURL` (1 GiB) per URL, which is enough to push a small node into DiskPressure. - **Tenant label.** The chart writes `tenant: {{ .Values.global.target }}`, which only the root bundle sets, so every tenant cluster series was labelled `tenant="cozy-monitoring"` and the `vmagent.externalLabels.tenant` the app passes was ignored. The fix: - the app passes `remoteWrite.urls` with the tenant's shortterm vminsert; - the chart uses `vmagent.externalLabels.tenant` when set, and falls back to `global.target`. The label is deliberately not fixed by passing `global.target` from the app, because the tenant label must name the tenant that owns the cluster (`.Release.Namespace`), while the monitoring target can be a parent tenant. The root install sets neither key: `helm template` of `templates/vmagent.yaml` with `global.target=tenant-root` renders the same URLs and label values before and after; the only textual change is that the `tenant` label is now quoted, so a tenant name YAML would read as a boolean or number still renders as a string. Tests: `packages/apps/kubernetes/tests/monitoring_agents_test.yaml` pins the keys the app sends, and `packages/system/monitoring-agents/tests/vmagent_remote_write_test.yaml` pins how the chart reads them, for both the root install and a tenant cluster. Reverting either template change fails its suite. `helm unittest` passes on both packages (358 and 23 tests). Upgrade note for operators: on tenant clusters the `tenant` label of vmagent series moves from `cozy-monitoring` to the owning tenant's namespace, so dashboards or alerts that filtered on `tenant="cozy-monitoring"` for those series need updating. Before this change those series never reached the tenant's Monitoring in the first place. Left out on purpose: the shortterm store stays hardcoded, as it was in the app before; deriving it from the target Monitoring's `metricsStorages`, and bounding vmagent's on-disk buffer, are separate changes. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes, monitoring): the vmagent of tenant Kubernetes clusters now writes to the tenant's vminsert and labels series with the owning tenant instead of `cozy-monitoring`. Previously no tenant cluster metric reached the tenant's Monitoring. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **Bug Fixes** - Monitoring agents now configure remote-write endpoints using a URL list. - Tenant labels use explicitly configured values when available, with a fallback to the global target. - Tenant and cluster labels, remote-write URLs, and external labels are applied consistently across root and tenant-cluster deployments. - **Tests** - Added coverage for root and tenant-cluster monitoring configurations, including remote-write settings and metric labels. <!-- end of auto-generated comment: release notes by coderabbit.ai --> Backported-from: c117420
Backport batch report for release-1.6, 2026-11-12 (6217f143abd7) |
Backport batch report for release-1.6, 2026-11-12 (e0694d897023) |
What this PR does
Backports to
release-1.6frommain, picked by the rules of the backport policy in merge order. A merged pull request comes as its own commits, picked withgit cherry-pick -xso each keeps its author, message and sign-offs, and a merge commit that carries the original's merge message with aBackported-from:trailer naming it; a squash-merged one comes as one pick.backport-batch verifychecks that shape. A pick listed as changed differs from the commit it came from;git range-diff <merge>^1..<merge>^2 <batch merge>^1..<batch merge>^2shows how. A conflict or a failing check here is a backport task.fix(ingress-nginx): drop the admission-webhook policy a rebase brought back, 1 commit, changed in the pick:50a9d7dcbc95fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook, 1 commitfix(kubernetes): move the OIDC users out of the bootstrap Job's script, 1 commitfix(postgres): render the declared databases in a stable order, 1 commitfix(mariadb-operator): derive the cluster name from the platform domain, 2 commitsfix(api): read pool usage under the quota key the controller counts, 1 commitfix(platform): remove the dead registries value and patch-containerd Secret, 2 commitsfix(mariadb): fail backup and restore jobs whose operator Job failed, 5 commitsfix(kubernetes, monitoring): send tenant cluster metrics to the tenant's vminsert, 1 commitDeferred conflicts
These do not pick cleanly onto this branch. Each names the first of its commits that conflicted and the files it conflicted in, and nothing of it is on the branch. Each moves to the next batch, unless a resolution checked with
backport-batch foldis pushed here first, as resolving a conflict describes.fix(marketplace): do not install a connected repository's components on connectat3fa534f3cf6c:internal/operator/tapmaterializer_reconciler.go,internal/operator/tapmaterializer_reconciler_test.gofix(tenant): stop the gateway release waiting on TenantGateway listeners and certificatesat5265ee662c59:packages/apps/tenant/templates/gateway.yamlfix(controller): carry healthCheckExprs and waitStrategy to existing releasesat8d8179f20a2c:internal/controller/applicationdefinition_helmreconciler.go,internal/controller/applicationdefinition_helmreconciler_test.gofix(kubernetes): run Proxmox CCM v0.16.1 and pin the Proxmox images by digestat05a347e6f6f7:packages/apps/kubernetes/templates/ccm-proxmox/controller.yamlfix(kubernetes-nodes): plan autoscaler capacity on the cores the VM actually getsatf7913329b256:packages/apps/kubernetes-nodes/README.md,packages/apps/kubernetes-nodes/templates/nodegroup.yaml,packages/apps/kubernetes-nodes/tests/resources_cpu_test.yaml,packages/apps/kubernetes-nodes/tests/talos_worker_config_test.yaml,packages/system/kubernetes-nodes-rd/cozyrds/kubernetes-nodes.yamlfix(e2e): read the KubernetesNodes pool release in the HelmRelease guardatcac152048efe:hack/e2e-chainsaw/_lib/run-kubernetes.sh,hack/run-kubernetes-remediation_test.batsfix(e2e): validate the leak allowlist against every suite layoutat782e0ab64d67:docs/agents/e2e-testing.md,hack/chainsaw-leak-guard_test.bats,hack/e2e-chainsaw/README.md,hack/e2e-chainsaw/_lib/leak-guard.shfix(marketplace): pin a tapped repository to a digest instead of a mutable tagatb8a129e356d3:cmd/cozypkg/cmd/tap.go,cmd/cozypkg/cmd/tap_test.go,pkg/registry/core/tap/compute.go,pkg/registry/core/tap/rest_test.gofix(dashboard): link a tenant row to its detail pageat7f55ff5d85fd:packages/system/dashboard/images/console/apps/console/src/routes/TenantsPage.test.tsxfix(clickhouse): restore the archive a Backup was taken asat47b497194eab:internal/backupcontroller/altinitystrategy_controller.gofix(backups): re-render the default backup objects when the S3 endpoint changesatcd91138f3327:hack/backupstrategy-crd-gate.bats,packages/system/backupstrategy-controller/templates/strategy-kafka-default.yaml,packages/system/backupstrategy-controller/templates/strategy-mongodb-default.yaml,packages/system/backupstrategy-controller/templates/strategy-rabbitmq-default.yaml,packages/system/backupstrategy-controller/templates/strategy-redis-default.yamlfix(securitygroups): project a membership-only group to a policy Cilium acceptsatdde255b7af10:pkg/registry/sdn/securitygroup/rest_test.gofix(redis): run one Sentinel when Redis runs a single podat48779cc03b23:api/apps/v1alpha1/redis/types.go,packages/apps/redis/README.md,packages/apps/redis/templates/redisfailover.yaml,packages/apps/redis/values.schema.json,packages/apps/redis/values.yaml, 1 morefix(lineage, kubernetes): resolve CAPK node VMs to their clusterat93faa7a8620d:pkg/lineage/lineage.gofix(postgres): fail a backup job as soon as its cnpg Backup failsatfe6968be7f25:internal/backupcontroller/cnpgstrategy_controller.go,internal/backupcontroller/cnpgstrategy_controller_test.gofix(postgres): refuse system databases and never drop thematf003851338f8:packages/apps/postgres/templates/init-script.yaml,packages/system/postgres-rd/cozyrds/postgres.yamlWaiting for a maintainer
These change
.github/workflows/, which the token this batch runs with may not push, so it leaves them out. The policy still takes each, andbackport-batch checklists it until it is on the line. A maintainer brings one over as changes to workflows describes.fix(release): keep release bodies within GitHub's size limit:.github/workflows/pull-requests-release.yaml,.github/workflows/update-releasenotes.yamlfix(e2e): keep the Proxmox tenant kubeconfig per run and remove it:.github/workflows/e2e-proxmox.yamlMaintainer overrides
fix(workloadmonitor): stop doubling the bucket- prefix in bucket Workload names:backport/skip, left outfix(kubernetes-nodes): refuse a partial resources spec on every pool:backport/skip, left outfix(tests): keep two unit tests green after the release image restamp:backport/skip, left outSkipped
release-1.6started under the policy on 2026-10-26: 562, too many to list here; the run's output lists themfixorsecurity, or the scope isdeps: 20targetVersionor a CRD, or makes a sizeable API change: 1release-1.7yet: 1Rule 2, the title type is not `fix` or `security`, or the scope is `deps`
feat(keycloak): load additional CA files from Secret or ConfigMap keys,type featfeat(kubernetes-nodes): trust a factory CA in the Proxmox template script,type featci(e2e): end the Proxmox lane's SSH key with a newline,type citest(kubernetes): pin RetryOnFailure on the Proxmox csi release,type testtest(kubernetes): name the Proxmox fixture's node pool <cluster>-<pool>,type testdocs(agents): steer agents away from Helm lookup,type docsfeat(release): attach an SBOM to every release image,type featci(api-review-gate): publish the gate as a commit status,type cidocs: move the repo-only guides onto the website,type docsfeat(monitoring-agents): export PVC labels from kube-state-metrics,type featfeat(monitoring-agents): keep application identity in container log streams,type featfeat(api): expose HelmRelease failure state on Application status,type featfeat(operator): let the platform source trust a private registry CA,type featfeat(release): make backports automatic with backport-batch,type featfeat(api): add the reservation evaluator for tenant quotas,type featfeat(release): keep a line's backport record and reports across batch pull requests,type featci(e2e): skip final cleanup on disposable PR runners,type cifeat(backups): record the RDB snapshot location and size for Redis backups,type featfeat(observability-proxy): serve tenant namespace logs through an authorizing proxy,type feattest(release): isolate report deduplication from workflow attempts,type testRule 3, a breaking change
fix(kubernetes, opensearch, foundationdb)!: restrict managed application images,! in the titlefix(keycloak)!: make the KMS proxy keep its keys across restarts,! in the titlefix(monitoring-agents)!: isolate merged log payload under log_processed,! in the titleRule 4, touches platform migrations, `targetVersion` or a CRD, or makes a sizeable API change
fix(backups): report sizeBytes for MariaDB and CNPG backup artifacts,a CRDRule 6, not on `release-1.7` yet
fix(seaweedfs): bump the COSI driver to v0.4.0 for the lost-grant fix,not on release-1.7 yetScreenshots
See the original pull requests.
Downstream repositories
Release note