Repository navigation
feat(kubernetes): give each tenant Kubernetes cluster its own etcd - #4817
Timofei Larkin (lllamnyp) wants to merge 3 commits into
Conversation
The kubernetes chart is about to render an etcd of its own per cluster, and it needs the same EtcdCluster, PKI, DataStore, defrag, VPA and scrape shapes the etcd chart renders. It cannot take the etcd chart as a subchart: the chart refuses any release not named "etcd", and the names it renders are pinned by the immutable spec.tls of adopted clusters. The shapes therefore move into cozy-lib, parameterised by the cluster name, and the etcd chart calls them with "etcd". Its rendered output is byte-identical before and after. Assisted-by: LLM Signed-off-by: Timofei Larkin <[email protected]>
A tenant Kubernetes cluster could only start once an ancestor Tenant had an etcd module turned on, which nobody creating a cluster may do, and every cluster below that tenant then shared one etcd, one CA and one disk. Anything that orders clusters on a tenant's behalf had to flip the tenant's etcd on first. A new cluster now renders its own EtcdCluster and DataStore, from the cozy-lib shapes the etcd chart uses, and is created and deleted with them. The awaiting-etcd state goes, since a datastore always exists. An existing cluster stays on the DataStore its live KamajiControlPlane names. Changing it starts a Kamaji datastore migration, which freezes the tenant's writes and leaves every worker kubelet needing a restart, so no upgrade may do it. _namespace.etcd cannot decide this: tenant-root publishes one to every tenant whether or not an etcd runs behind it. The pre-delete hook now fails while the TenantControlPlane remains, because Kamaji cannot finalise one whose etcd Helm has already removed. A Cilium deny policy keeps the rest of the tenant namespace off the etcd ports, and the platform waits for etcd-operator before offering the application. Assisted-by: LLM Signed-off-by: Timofei Larkin <[email protected]>
Existing tenant clusters keep the shared tenant etcd they run on, and moving one is a per-cluster operator decision that the platform never takes on an upgrade. Retiring Tenant.spec.etcd has to wait until no cluster uses it, so the upgrade lists the clusters that still do. It writes nothing beyond the version stamp. Assisted-by: LLM Signed-off-by: Timofei Larkin <[email protected]>
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: true
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
github-actions
Bot
added
area/kubernetes
Aleksei Sviridkin (lexfrei)
left a comment
Aleksei Sviridkin (lexfrei)
left a comment
There was a problem hiding this comment.
Timofei Larkin (@lllamnyp) NOT LGTM. The design holds up, but the branch conflicts with main, two existing bats suites fail because of this change, long cluster names cannot install, and a tenant that keeps its shared etcd gets its quorum alert diluted.
Business context: a tenant Kubernetes cluster gets its own etcd and Kamaji DataStore instead of waiting for an ancestor Tenant to turn on a shared one, while existing clusters stay on the shared datastore they already use.
I tried hardest to break the parts that touch existing control planes, and they hold:
- Upgrade.
kubernetes.etcd.sharedDataStorekeys on the liveKamajiControlPlane, so a cluster already ontenant-rootkeeps thatdataStoreNameand renders no etcd of its own. When I forced that helper to return empty, five cases inetcd_test.yamlfailed, so the stickiness is pinned. Migration 60 only reads and stamps, and its bats pass. - Kamaji wiring. The TenantControlPlane webhook only requires the DataStore to exist, and the release creates it next to the
KamajiControlPlane, so the provider's retry covers the ordering. - Deletion. Member PVCs are controller-owned by the
EtcdMembers in etcd-operator, so they go with theEtcdCluster. The five cert-manager Secrets go through the post-delete Job. When I disabled the hook gate,etcd_test.yamlcaught it.
On the PR head helm unittest passes for kubernetes (359), extra/etcd (22), platform (258), tenant (76), computeplane (13) and cozy-lib-tests (62), and select-install_test.bats passes.
Blockers
B1: conflicts with main
packages/apps/kubernetes/templates/delete.yaml, packages/extra/etcd/templates/etcd-defrag.yaml and packages/system/kubernetes-rd/cozyrds/kubernetes.yaml conflict, and GitHub reports CONFLICTING. One of them needs more than a textual fix. fe59a6b added --compaction=false and --etcd-storage-quota-bytes to the etcd chart's defrag CronJob, and that CronJob now lives in cozy-lib.etcd.defrag. The flags have to go into the cozy-lib template so both charts get them, with the quota computed from each caller's size (4Gi for a cluster's own etcd). Otherwise the extraction silently reverts that fix. The defrag_test.yaml that main added will catch the etcd chart side, but nothing checks the per-cluster side. 904bc38 also changed the same pre-delete script this PR edits.
B2: two existing bats suites fail on this head
Unit & controller tests is red (run of 2026-10-08T10:30Z, attempt 1), so E2E (in-tree) was skipped. That means none of the "Needs a live cluster" items has run, the upgrade case included. I reproduced both failures locally:
hack/kubernetes-pre-delete-hook.batscases 5, 6, 8 and 10 still expect exit 0 when the TenantControlPlane outlives step 4 or its probe fails. Case 10 says "an all-failures run exited 1, which would wedge the release". The newexit 1is the opposite contract, so those cases have to be split into a shared-datastore case (still exits 0) and an own-etcd case (fails closed). This file is not in the test plan.hack/check-render-matrix.bats"fixture states enable OIDC groups and shared-service resources" expects thefreshfixture, which has_namespace.etcd: "", to render nokubernetes-check-metrics-server. With the etcd gates gone, it does render. The test and the fixture's meaning need updating, not just the expected count.
B3: the defrag CronJob name exceeds 52 characters for long cluster names
cozy-lib.etcd.defrag names the CronJob <etcd name>-defrag, which for this chart is kubernetes-<name>-etcd-defrag. The app name allows 32 characters, so the CronJob name reaches 55. The apiserver rejects CronJob names longer than 52, so any cluster named with 30 to 32 characters fails to install. I rendered kubernetes-abcdefghijklmnopqrstuvwxyz0123 and got the 53-character kubernetes-abcdefghijklmnopqrstuvwxyz0123-etcd-defrag. The 56-character guard in kubernetes.etcd.name does not cover this. Either shorten the CronJob name (truncate plus hash) or lower the guard, and add a test at the maximum name length.
B4: a tenant's shared etcd scrapes the cluster etcds in its namespace
packages/extra/etcd/templates/podscrape.yaml selects app.kubernetes.io/name: etcd, and etcd-operator puts that label on every member (clusterLabels in controllers/helpers.go). In a tenant with etcd: true that creates a cluster in the same namespace, etcd-pod-scrape now also scrapes kubernetes-<name>-etcd. Those members are then scraped twice, and the shared etcd's job counts six members instead of three. etcdInsufficientMembers sums up by job, so with two of the three shared members down it sees 4 of 6 up and stays quiet. Before this PR nothing else in that namespace carried the label. Selecting on app.kubernetes.io/instance: etcd (or etcd-operator.cozystack.io/cluster: etcd) fixes it.
Decisions to confirm, not blocking
- New clusters lose platform backup. Before, a new cluster's control plane lived in the tenant etcd, which an
EtcdBackupJob covers. After this PR it lives in an etcd no strategy routes to. The README and the release note say so. I'd still like a maintainer to accept that explicitly before this leaves draft, because it is a regression for anyone who backs up tenant etcd today. - The new
exit 1wedges the uninstall whenever the TenantControlPlane cannot finalize, including when the cluster's own etcd never became Available, for example because of the quota case in your test plan. Whether Kamaji can finalize a TenantControlPlane whose datastore was never reachable decides whether that deletion ever finishes without manual work. That case belongs in the live test list. kubernetesEtcd.storageClassLocaldefaults to empty, so on a default install three members land on the namespace default class. That is usually the replicated one, which is the double replication the values comment says the local class avoids.
| apiVersion: batch/v1 | ||
| kind: CronJob | ||
| metadata: | ||
| name: {{ $name }}-defrag |
There was a problem hiding this comment.
B3: for this chart this is kubernetes-<name>-etcd-defrag, which is 53 to 55 characters when the app name is 30 to 32. The apiserver caps CronJob names at 52.
| selector: | ||
| matchLabels: | ||
| app.kubernetes.io/name: etcd | ||
| {{ include "cozy-lib.etcd.podScrape" (dict "name" "etcd" "selector" (dict "app.kubernetes.io/name" "etcd")) }} |
There was a problem hiding this comment.
B4: every etcd-operator member carries app.kubernetes.io/name: etcd, so this now also matches kubernetes-<name>-etcd pods in the same namespace. app.kubernetes.io/instance: etcd keeps the job to this cluster.
| TenantControlPlane whose datastore no longer answers: its | ||
| finalizer then stays until removed by hand. Failing here | ||
| keeps the etcd up and makes Flux retry the uninstall. */}} | ||
| if [ "$tcp_lookup" != 0 ] || [ -n "$tcp" ]; then |
There was a problem hiding this comment.
B2: hack/kubernetes-pre-delete-hook.bats cases 5, 6, 8 and 10 still expect exit 0 on this path and fail.
Marian Koreniuk (themoriarti)
left a comment
Marian Koreniuk (themoriarti)
left a comment
There was a problem hiding this comment.
Timofei Larkin (@lllamnyp) This is a comment, not a vote. lexfrei's CHANGES_REQUESTED already covers the merge blockers, and I re-checked those on d211567 below. On top of them I found five things and a few nits. I'm reviewing as CODEOWNER of the Proxmox paths. Dropping the _namespace.etcd gates from the Proxmox CCM and CSI templates is right, and so is the Proxmox e2e tenant moving to etcd: false.
Re-checked on this head
- B1:
git merge-treeagainst main still conflicts indelete.yaml,etcd-defrag.yamlandcozyrds/kubernetes.yaml. Migration slot 60 is still free on main, because #4638 landed without it, so no renumbering is needed today. - B2: the failing job lists
check-render-matrix#493 and pre-delete-hook #980, #981, #983 and #985. - B4: etcd-operator v0.5.6 stamps
app.kubernetes.io/name: etcdon every member (clusterLabelsincontrollers/helpers.go). helm unittestonpackages/apps/kubernetes: 359 passed.
Findings
1. Remove the uninstall wedge instead of failing on it
This extends lexfrei's decision 2. On Kamaji 26.9.4-edge, a TenantControlPlane that still carries finalizer.kamaji.clastix.io finishes deleting only when all three of these hold:
- its DataStore exists; otherwise Kamaji requeues every second;
- the DataStore is Ready;
- Kamaji can open a connection.
Setup.Deletethen revokes the grants and deletes the prefix and the user.
Any error there requeues (controllers/tenantcontrolplane_controller.go lines 152-215, internal/resources/datastore/datastore_setup.go:145). So a cluster whose own etcd never came up, or has lost quorum, has a TCP that can never be deleted. Examples are the quota case in your test plan and a Pending PVC. With the new exit 1, Flux then retries the uninstall indefinitely until someone strips the finalizer by hand.
For a cluster that owns its etcd, everything that finalizer cleans lives inside the etcd this same uninstall deletes a moment later. So on the own-etcd branch, the hook can do this instead:
- Remove
finalizer.kamaji.clastix.iofrom the TCP, once the Step 4 wait expires or before it. - Wait for the TCP to go.
- Let Step 4b clear the datastore-secret finalizer as it does today.
Nothing else wedges after that:
- Kamaji's DataStore controller drops its own finalizer without touching etcd once no TCP references the DataStore.
- The soot finalizer is cleared without contacting the tenant API.
The shared branch is unchanged, so a shared etcd's state is never skipped. The hook's Role needs patch on tenantcontrolplanes, scoped by resourceName to this release.
2. Going from 3 members to 1 leaves a single member on node-local storage
kubernetes.etcd.storageClass returns the live class whatever the replica count. The test covers 1 → 3, which stays on replicated: wasteful but safe. The reverse is untested.
I tried an ad-hoc case: a live EtcdCluster with storageClassName: local, then etcd.replicas: 1. It renders replicas: 1 on local. That is one copy with no Raft peer and no volume replication. It is exactly the shape the kubernetesEtcd values comment says the replicated class exists to avoid, and per decision 1 no platform backup covers it.
Suggested fix:
- Refuse that render when the live class equals
etcd-storage-class-local. Whenetcd-storage-class-replicatedis set, refuse whenever the live class differs from it. - Add the case to
etcd_test.yaml. - Spell out the consequence in the
etcd.replicasdescription.
3. No CI lane exercises the path every existing cluster is on
e2e-install-cozystack.bats now leaves tenant-root's etcd off, and the Proxmox tenant moves to etcd: false. As a result, no e2e lane creates a cluster on a shared DataStore.
After the upgrade, every existing cluster renders the sharedDataStore branch:
- the KCP lookup;
- the shared DataStore name;
- no etcd objects of its own;
- the ungated delete hook.
Only helm-unittest covers that branch, with stubbed lookups, and the upgrade test is in the unchecked "needs a live cluster" list.
Before this leaves draft, I'd do two things:
- Keep one cluster on a shared etcd in CI. That could be a tenant with
etcd: truein one of the kubernetes lanes, or an upgrade lane from the previous chart. - Run the upgrade from
mainonce on a live cluster.
4. The documented limits are only the starting values
cozy-lib.etcd.vpa renders for each cluster's etcd wherever VPA is served. It uses updateMode: Initial, maxAllowed 5000m/8Gi before allocation ratios, and the default RequestsAndLimits. So VPA re-sizes every new member pod, limits included.
The release note and NOTES.txt ("1000m CPU / 512Mi memory limits ... counted against this tenant's resource quota") describe only the first pod. During a recovery, a replacement member that VPA sized above the tenant's remaining quota is refused admission. The mechanism is the same as the tenant etcd's today, but now it is multiplied per cluster.
The chart already pins resources for this etcd. Either drop the VPA on this path, or say in the note that the numbers can change.
5. The deny policy covers pods, not entities
The ingressDeny rules are fromEndpoints only. The tenant's allow-external-communication admits fromEntities: [world, cluster] to every endpoint in the namespace. So traffic Cilium classifies as world, remote-node or host still reaches 2379/2380.
"Worker VMs included" holds for KubeVirt workers, which are pods. It does not hold for Proxmox workers, which run outside the cluster, wherever their network can route to pod IPs. Client certificates still guard etcd, so this is a defence-in-depth gap rather than an open door.
Adding fromEntities: [world] to the deny does not affect member probes, which use 2381 (etcdmember_controller.go). It does need the live check you already list, because Cilium can briefly see a fresh pod IP as world before ipcache syncs. Alternatively, narrow the comment.
Nits
- New clusters still get
policy.cozystack.io/allow-to-etcd: "true", which admits them to the tenant's shared etcd through<tenant>-ingress-etcd. They never use it. Rendering it only on the shared branch costs no rollout, since only new clusters change. - After rebasing on 904bc38, the five etcd Secrets could be deleted the way Step 7 already deletes the Talos Certificates and Secrets: inside the pre-delete hook, once the TCP is confirmed gone. That drops the separate post-delete ServiceAccount/Role/RoleBinding/Job. It also drops the hook objects a failed post-delete leaves behind, since that delete policy has no
hook-failed. etcd-cleanup.yamlhardcodesdocker.io/clastix/kubectl:v1.32, asdelete.yamldoes. The other Jobs in this chart takeimages/kubectl.tagwith.Values.images.kubectl, so mirrors and air-gapped installs cannot override this one.
| keeps the etcd up and makes Flux retry the uninstall. */}} | ||
| if [ "$tcp_lookup" != 0 ] || [ -n "$tcp" ]; then | ||
| echo "TenantControlPlane {{ .Release.Name }} is not confirmed gone; failing so that its etcd is not deleted while Kamaji still needs it, and the uninstall is retried" >&2 | ||
| exit 1 |
There was a problem hiding this comment.
Finding 1: for an own etcd, everything finalizer.kamaji.clastix.io cleans lives inside the etcd this uninstall deletes next: the grants, the prefix and the user. Kamaji needs a Ready DataStore and a live connection to finish (tenantcontrolplane_controller.go 152-215 in 26.9.4-edge). So when that etcd never came up or has lost quorum, failing here makes Flux retry forever. Stripping the finalizer on this branch lets such a cluster be deleted at all. The Role needs patch on this one TCP.
| {{- define "kubernetes.etcd.storageClass" -}} | ||
| {{- $live := lookup "etcd-operator.cozystack.io/v1alpha2" "EtcdCluster" .Release.Namespace (include "kubernetes.etcd.name" .) -}} | ||
| {{- if $live -}} | ||
| {{- dig "spec" "storage" "storageClassName" "" $live -}} |
There was a problem hiding this comment.
Finding 2: the live class also wins when going from 3 members to 1. With a live local EtcdCluster and replicas: 1, this renders one member on node-local storage. I'd refuse that render and test it next to the 1 → 3 case.
| {{ include "cozy-lib.etcd.defrag" (dict "name" $name "root" $ "podLabels" $clientLabels) }} | ||
| {{- if .Capabilities.APIVersions.Has "autoscaling.k8s.io/v1/VerticalPodAutoscaler" }} | ||
| --- | ||
| {{ include "cozy-lib.etcd.vpa" $etcd }} |
There was a problem hiding this comment.
Finding 4: with updateMode: Initial and the default RequestsAndLimits, VPA re-sizes each new member pod, up to 5000m/8Gi before ratios. So the 1000m/512Mi in NOTES.txt and the release note is only the first pod's size. The chart pins resources for this etcd, so this VPA may be better left out here.
| endpointSelector: | ||
| matchLabels: | ||
| etcd-operator.cozystack.io/cluster: {{ $name }} | ||
| ingressDeny: |
There was a problem hiding this comment.
Finding 5: fromEndpoints matches pods only. allow-external-communication admits the world and cluster entities to every endpoint in the namespace, so non-pod sources are not denied. That includes Proxmox workers if they route to pod IPs. Member probes are on 2381, so a fromEntities: [world] deny on 2379/2380 does not touch them.
| # The root etcd stays off as well: every tenant Kubernetes cluster brings its | ||
| # own, and leaving the shared one off is what proves a cluster comes up with | ||
| # no tenant etcd anywhere above it. | ||
| kubectl patch tenants/root -n tenant-root --type merge -p '{"spec":{"host":"example.org","ingress":true,"monitoring":false,"etcd":false,"isolated":true, "seaweedfs": true}}' |
There was a problem hiding this comment.
Finding 3: with this change and the Proxmox tenant on etcd: false, no lane creates a cluster on a shared DataStore. That is the branch every existing cluster renders after the upgrade.
| podAdditionalMetadata: | ||
| labels: | ||
| policy.cozystack.io/allow-to-etcd: "true" | ||
| {{- if not $sharedDataStore }} |
There was a problem hiding this comment.
Nit: the line above still puts policy.cozystack.io/allow-to-etcd: "true" on new clusters, which admits them to the tenant's shared etcd they never use. Moving it under the shared branch changes only new clusters.
…4986) ## What this PR does The tenant etcd chart spreads members with `maxSkew: 1` on `kubernetes.io/hostname` and `whenUnsatisfiable: ScheduleAnyway`, the default since #119. `ScheduleAnyway` only adds to the score of a node and other scores can outweigh it, so two members of one etcd can still start on the same node, where a node-local volume then keeps them. This PR makes the default `DoNotSchedule` with `nodeTaintsPolicy: Ignore`. The branch for `_cluster.scheduling.globalAppTopologySpreadConstraints` is untouched, so a platform that sets its own policy renders exactly as before, and the soft pod anti-affinity from #3962 stays. Seen on a three-node cluster running v1.6.4, whose `etcd-cluster.yaml` and etcd-operator v0.5.5 are the same as in v1.6.5. A new tenant etcd got the Pods of its first two members on one node, created 26 seconds apart, and the third on another node 18 seconds later. The Pods carry the `ScheduleAnyway` constraint and were scheduled by `linstor-scheduler`, whose profile adds the LINSTOR extender's prioritize score to the same sum with weight 5. The data volumes are on a node-local StorageClass with `WaitForFirstConsumer` and `allowRemoteVolumeAccess: "false"`, so the two members cannot leave that node, and losing it takes the quorum. A second tenant etcd created there the next day came up the same way. On a test cluster four tenant etcds created within one week each had two members on one node, and we moved one member of each by hand. `DoNotSchedule` does not stop an etcd with more members than nodes from starting when every node can run its Pods, unlike a required anti-affinity, because the skew is counted between the nodes that exist. kube-scheduler rejects a node only when the matching Pods on it plus the new one exceed the node with the fewest by more than `maxSkew` ([filtering.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/filtering.go#L334-L352)). One node takes all three members, two nodes take two and one, and three or more nodes take one each. A member whose only allowed node cannot take it, for example because the storage pool there is full, now waits Pending instead of joining another member. `nodeTaintsPolicy: Ignore` is the Kubernetes default and is written out on purpose. A cordoned node carries `node.kubernetes.io/unschedulable`, and `Honor` would take it out of the count ([common.go](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/scheduler/framework/plugins/podtopologyspread/common.go#L50-L54)), so a member created during a drain would start next to another member and stay there on a node-local volume. With `Ignore` it waits Pending until the node is back, while the other two keep quorum. The cost is that a node no member can run on, such as a tainted control plane, counts as an empty node too, so on a cluster with such a node each schedulable node takes at most one member and the members beyond that do not start. #4947 makes the same choice for the SeaweedFS masters and checked both policies in kwok. A platform that wants the old behaviour can set `scheduling.globalAppTopologySpreadConstraints` to `ScheduleAnyway`, which the other charts that read it pick up as well, and a new test case pins that the chart renders such a policy as written. On upgrade nothing is recreated. The etcd-operator adopts a changed `spec.topologySpreadConstraints` into `status.observed` once the cluster has reached its current target, or its progress deadline has passed ([etcdcluster_controller.go#L203-L214](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L203-L214)). It copies the value from there into the spec of an `EtcdMember` only when it creates one, at bootstrap ([#L446](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L446)) and on scale-up ([#L851](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdcluster_controller.go#L851)), and nothing in the operator writes it into an existing `EtcdMember`. The member controller creates a Pod only when it is missing and builds it from that member's own copy ([etcdmember_controller.go#L499-L512](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L499-L512), [#L545-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L545-L549), [#L860](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L860)). So running members keep `ScheduleAnyway`, and so does a member whose Pod is evicted, deleted or replaced after a terminal phase ([#L164-L186](https://github.com/cozystack/etcd-operator/blob/v0.5.6/controllers/etcdmember_controller.go#L164-L186)), since the new Pod is built from the same copy. Only members created after the upgrade get `DoNotSchedule`, which means a scale-up, a crash-loop replacement or a member whose `EtcdMember` was deleted. No member is rolled, neither one at a time nor all at once. This matches the field docs ([etcdcluster_types.go#L542-L549](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdcluster_types.go#L542-L549), [etcdmember_types.go#L112-L116](https://github.com/cozystack/etcd-operator/blob/v0.5.6/api/v1alpha2/etcdmember_types.go#L112-L116)). The code paths are the same in v0.5.8, and in v0.5.5, which v1.6.4 and v1.6.5 ship, except that v0.5.5 does not yet replace a Pod left in a terminal phase. Members that already share a node therefore stay there. One of them moves only when a new `EtcdMember` takes its place, and the constraint then sends the replacement to the node without a member. That is how we moved the members on the test cluster, with `DoNotSchedule` set on the `EtcdCluster` by hand. The operator's docs say a scheduling change can be applied by deleting one Pod at a time ([concepts.md#L272](https://github.com/cozystack/etcd-operator/blob/v0.5.6/docs/concepts.md#L272)), but the recreated Pod gets the member's old copy, so that does not move it. The release note now declares two cases from review. In the first, an etcd created while the cluster had fewer nodes than members has two or three members on one node, and once a node is added a recreated Pod of one of them stays Pending, since `maxSkew` rejects its own node and its node-local volume rules out the others. Deleting that member's `EtcdMember` while the other members keep quorum replaces it on a node without a member. In v0.5.5 the member finalizer removes the member from etcd through the remaining members before the object goes ([`handleDeletion`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L195-L231), [`removeMemberFromEtcd`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L243-L310)), its Pod and its `data-<member>` PVC are owned by it and deleted with it ([#L403](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L403), [#L490](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdmember_controller.go#L490)), and the cluster controller waits for the deletion, then adds a member under a new name with the constraints from `status.observed` ([etcdcluster_controller.go#L254-L312](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L254-L312), [`scaleUp`](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L820-L851)). Its new PVC binds where its Pod lands, and under `DoNotSchedule` only a node without a member passes. The operator documents the same steps for a broken member, along with the limit that `MemberRemove` cannot run without quorum ([operations.md#L208-L231](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L208-L231)), and v0.5.6, which main and `release-1.6` ship now, has the same path. The upstream chart of the operator installs a policy that denies `EtcdMember` deletion without the `etcd-operator.cozystack.io/allow-deletion=true` annotation ([member-deletion-policy.yaml](https://github.com/cozystack/etcd-operator/blob/v0.5.5/charts/etcd-operator/templates/member-deletion-policy.yaml)), and the `etcd-operator` package here does not install it. On a three-node cluster running v0.5.5 a tenant etcd had two members on one node with `DoNotSchedule` in effect. We deleted the `EtcdMember` of one of the two while the other members kept quorum. The replacement's Pod and its new node-local PVC were scheduled on the node without a member, the member joined, and the `EtcdCluster` stayed `Available` with quorum the whole time, polled every 5 seconds. In the second case `nodeTaintsPolicy: Ignore` counts a node the Pods cannot tolerate as one with zero members, so on a cluster with such a node and fewer schedulable nodes than members the members beyond the schedulable nodes do not start. The way out is the soft global policy in the platform `Package`, set before the upgrade. ```yaml apiVersion: cozystack.io/v1alpha1 kind: Package metadata: name: cozystack.cozystack-platform spec: components: platform: values: scheduling: globalAppTopologySpreadConstraints: | topologySpreadConstraints: - maxSkew: 1 topologyKey: kubernetes.io/hostname whenUnsatisfiable: ScheduleAnyway ``` The platform chart writes `scheduling` into `_cluster` of the `cozystack-values` Secret ([apps.yaml#L292-L295](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/core/platform/templates/apps.yaml#L292-L295)), the tenant chart copies `_cluster` into the Secret of each tenant namespace ([namespace.yaml#L109-L120](https://github.com/cozystack/cozystack/blob/55aad6f91456cb545601a83aaecf39118e943b04/packages/apps/tenant/templates/namespace.yaml#L109-L120)), and this chart renders the value as written, which the new test case pins. Rendering the platform and tenant charts with the value above, on main and on `release-1.6`, gives the same string in both Secrets. The operator adopts a changed spec into `status.observed` only once all members are Ready or the progress deadline has passed ([etcdcluster_controller.go#L196-L213](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L196-L213)), and it clears the deadline when the cluster converges ([#L1448-L1456](https://github.com/cozystack/etcd-operator/blob/v0.5.5/controllers/etcdcluster_controller.go#L1448-L1456)). A new etcd stuck short of its members adopts the policy once its first deadline passes, and deleting the `EtcdMember` of the Pending member then brings a replacement that starts. An etcd that had converged before one of its replacements went Pending adopts the policy only after the deadline is forced as the operator docs describe ([operations.md#L197-L206](https://github.com/cozystack/etcd-operator/blob/v0.5.5/docs/operations.md#L197-L206)), which is why the release note says to set it before the upgrade. #4817 moves this block into `_etcd.tpl` in cozy-lib without changing it, so whichever of the two lands second has to carry the two lines over. `tests/topology-spread_test.yaml` now pins the whole default constraint, and a new case pins that a `ScheduleAnyway` policy from `globalAppTopologySpreadConstraints` renders as written, without `nodeTaintsPolicy`. The existing zone override case is unchanged. Against the template on main the suite fails only the default case, 1 failed and 25 passed. Switching the default to `Honor`, to `ScheduleAnyway` or to `maxSkew: 2` fails the default case, and adding `nodeTaintsPolicy` to the override branch fails the new case. ```console $ docker run --rm -v "$PWD":/apps -w /apps/packages/extra/etcd helmunittest/helm-unittest:4.2.3-1.1.2 . Charts: 1 passed, 1 total Test Suites: 9 passed, 9 total Tests: 26 passed, 26 total ``` On main the same command gives 25 tests in 9 suites, with helm-unittest 1.1.2 as CI pins it. With default values `helm template` differs from main in the two lines and in the comment above the affinity block, and with a global policy set only in that comment. The docs commit changes the `affinity` description, from which `make generate` rewrites the README, `values.schema.json` and the `ApplicationDefinition` with nothing but that text, and the suite stays at 26 tests. I have not rolled this chart change out on a cluster. The spread block and the test file are the same on release-1.6 and release-1.7. On release-1.6 the template has no affinity block yet, so the fix commit cherry-picks without conflicts and the docs commit has nothing to change there. That backport is a separate PR. ### Screenshots Not a UI change. ### Downstream repositories One changed and one added line in `packages/extra/etcd/templates/etcd-cluster.yaml` and its test suite, and new text for the `affinity` description and the comment above the affinity block. The description reaches `values.schema.json`, the README and the `ApplicationDefinition` as text only, with no key, type or default change, and the website rebuilds the etcd reference page from the README on release. The website mentions `globalAppTopologySpreadConstraints` only as a platform setting, which stays as it is, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(etcd): tenant etcd members now spread over nodes with whenUnsatisfiable DoNotSchedule by default, so two members of one etcd no longer start on the same node while another node has none. Members that exist are not moved, only members created after the upgrade get the new constraint. An etcd created while the cluster had fewer nodes than members puts two or three members on one node. Once a node is added, such a member whose Pod is recreated (drain, eviction, graceful node shutdown) stays Pending, because maxSkew rejects its own node and its node-local volume rules out the others. With two nodes grown to three the etcd runs two of three members and its PodDisruptionBudget blocks draining the other nodes. With one node grown to three a reboot of that node brings back one member of three and quorum is lost. A stranded member is replaced by deleting its EtcdMember while the other members keep quorum. Its Pod and PVC go with it, and the replacement gets a new Pod and PVC on the node without a member. Right after adding nodes, doing this for one co-located member at a time while all members are Ready avoids the case. With nodeTaintsPolicy Ignore a node the etcd Pods cannot tolerate, such as a tainted control plane, counts as a node with zero members, so no node can take a second member. On a cluster with such a node and fewer schedulable nodes than members, a new etcd starts only as many members as there are schedulable nodes, one of three on a single schedulable node. An existing etcd there keeps its members, but a member it replaces stays Pending the same way. Either case is avoided by setting scheduling.globalAppTopologySpreadConstraints in spec.components.platform.values of the cozystack.cozystack-platform Package to a topologySpreadConstraints entry with maxSkew 1, topologyKey kubernetes.io/hostname and whenUnsatisfiable ScheduleAnyway before the upgrade. Every app chart that reads this value applies it, two members of a new etcd can share a node again, and the operator gives it only to members created after it adopts the changed EtcdCluster spec into status.observed. Once status.observed.topologySpreadConstraints of an EtcdCluster shows ScheduleAnyway, a member of it that stays Pending is replaced by deleting its EtcdMember. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **Bug Fixes** - Default etcd placement now requires even spreading across hosts and ignores node taints when evaluating placement. - Configurable cluster-wide spreading continues to use a flexible scheduling policy when selected. - **Documentation** - Clarified how affinity and topology spreading work together, when required anti-affinity or node-local storage can leave members pending, and that affinity changes take effect as members are replaced. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
What this PR does
A tenant Kubernetes cluster could not start until an ancestor
Tenanthadspec.etcd: true. The chart read the datastore name from_namespace.etcdand, with none, rendered only a<release>-awaiting-etcdConfigMap. Every cluster below that tenant then shared one etcd, one CA and one disk, separated by key prefix only. Whoever creates a cluster usually may not edit theTenant, so anything ordering clusters on a tenant's behalf (the portal's controller, for one) had to flipetcd: truefirst.This PR makes the
Kubernetesapplication carry its own etcd. It implements phases 1 and 2 of the per-cluster etcd design proposal, cozystack/community#25, as three commits that can be split later:refactor(etcd): the etcd chart'sEtcdCluster, cert-manager PKI,DataStore,WorkloadMonitor, defragCronJob, VPA andVMPodScrapemove into named templates incozy-lib(packages/library/cozy-lib/templates/_etcd.tpl), parameterised by the cluster name.packages/extra/etcdcalls them with"etcd", and its rendered output is byte-identical before and after (see Verification).feat(kubernetes): a new cluster renders its ownEtcdClusterandDataStorefrom those templates, and a cluster that already runs on a shared tenant etcd keeps it.feat(platform): migration 60 → 61 reports which clusters stay on a shared etcd and changes nothing.Phases 3 and 4 (deprecating, then removing
Tenant.spec.etcd,_namespace.etcdandcozy-lib.ns-etcd) are left for later.Design
Names. For release
kubernetes-<name>in namespace<ns>:EtcdCluster, its headless Service, members<…>-xxxxx,<…>-clientServicekubernetes-<name>-etcdkubernetes-<name>-etcd-{ca,client,server,peer,peer-ca}-tlsDataStore(cluster-scoped)<ns>.kubernetes-<name>kubernetes-<name>-etcd.<ns>.svc:2379Values. One new field,
etcd.replicas:3(default) or1. The schema bounds it to 1..3 and the chart refuses 2 at render time, becausecozyvalues-gencannot express an integer enum (it emits string members for one). Size (4Gi), resources (1000m / 512Mi limits) and the etcd version (3.6.11, the etcd chart's default) are chart-internal.StorageClass. Two new platform values,
kubernetesEtcd.storageClassLocalandkubernetesEtcd.storageClassReplicated, reach the chart as_cluster.etcd-storage-class-local(used with 3 members) and_cluster.etcd-storage-class-replicated(used with 1 member). Empty keys fall back to the namespace default StorageClass.spec.storage.storageClassNameis immutable on anEtcdCluster, so the chart reads the live object's class on every render after the first and never re-derives it; scaling 1 → 3 keeps the class the cluster started with.New clusters vs existing ones. The chart decides from the live
KamajiControlPlane, looked up by release name:KamajiControlPlane, or one whosespec.dataStoreNameis the cluster's own name: own etcd;DataStore: thatDataStoreis kept, and the chart renders no etcd of its own.Clusters stuck in the old awaiting-etcd state have no
KamajiControlPlane, so they get their own etcd on their next reconcile._namespace.etcdis deliberately not the signal.packages/system/cozystack-basics/templates/cozystack-values-secret.yamlhardcodesetcd: tenant-root, andpackages/apps/tenant/templates/namespace.yamlpasses it down, so every tenant publishes a value whether or not an etcd runs behind it. Honouring it for new clusters would change nothing for anyone. Following it for existing clusters is also wrong: today, switching onspec.etcdfor a mid-level tenant silently moves every cluster below it to a new datastore, which is a Kamaji datastore migration. With this PR, the live binding sticks.An existing cluster renders exactly what it rendered before, apart from randomly generated Talos tokens, so the upgrade rolls no control-plane pods. The new pod label goes on new clusters only.
What goes away. The
awaiting-etcdbeacon goes, along with the_namespace.etcdgate on the ~25 addon and satellite templates (helmreleases/*,csi/deploy.yaml,kccm/manager.yaml,cluster-autoscaler/deployment.yaml,ingress.yaml, the Proxmox CCM and CSI). A datastore now always exists, so these gates were always true.Deletion. The existing pre-delete hook deletes the
TenantControlPlaneand waits up to 120s. Before this PR it exited 0 even when theTenantControlPlanewas still there. On a cluster that owns its etcd, Helm deletes the etcd right after the hook, and Kamaji then cannot finish its finalizer, because it needs the datastore to drop the tenant's role, prefix and user. So on such a cluster the hook now fails while theTenantControlPlaneremains, keeping the etcd up and making Flux retry the uninstall. After it:EtcdCluster,DataStoreand Certificates;EtcdMembers;The hook never touches a shared etcd, and the Secret cleanup Job is not rendered for a cluster on one.
Deviations from cozystack/community#25
<namespace>.<release>, not<namespace>-<release>. Tenant names are alphanumeric and nest with dashes, so the dashed form collides: Kubernetesfoointenant-agivestenant-a-kubernetes-foo, which is also the namespace, and therefore the sharedDataStorename, of tenantfoounder tenantkubernetesundertenant-a. A namespace cannot contain a dot. Kamaji copies theDataStorename into the pod labelcomponent.kamaji.clastix.io/datastore, so the name is capped at 63 characters, truncated with an 8-character hash past that.kubernetes.etcdName. The app name is now capped at 32 characters (@maxLength 32, added with the nodes split), sokubernetes-<name>-etcdis at most 48 characters. That is inside the 56 the operator needs: the<name>-clientService and the generated member names must fit in 63. The helper fails the render past 56 rather than rename anything._namespace.etcd" means keeping a cluster on theDataStoreits live control plane names, for the reason above. It does not mean following_namespace.etcdfor new clusters.etcd.replicasvalidation is a schema range of 1..3 plus a render-time refusal of 2, rather than a schema enum.KamajiControlPlanecondition as the render, as Release 0.1.0 #25 asks, but it fails the uninstall rather than ordering deletions inside the hook. Helm deletes release resources only after a successful pre-delete hook, so that ordering already holds once the hook refuses to pass early.secretRefshape (chart-issued certificates) rather than the operator'scertManagermode. Kamaji'sDataStoreneeds the CA private key to sign each tenant's client certificate, andsecretRefkeeps that key in a Secret the chart controls. The server certificate also carries the operator's<name>-clientService names, which the etcd chart's does not.PrometheusRulestays in the etcd chart only: its expressions match every etcd job, so one copy per cluster would duplicate every alert.VMPodScrape, VPA and theCiliumNetworkPolicyrender only where their APIs are served.talosconfigis issued. The numbered migration only reports, as Release 0.1.0 #25 suggests.Gaps #25 did not cover
Etcdbackup strategy (internal/backupcontroller/etcdstrategy_controller.go,packages/system/backupstrategy-controller/templates/strategy-etcd-default.yaml) is keyed on theapps.cozystack.io/Etcdapplication and on the fixedEtcdClusternameetcd, andcozy-defaultroutes noKuberneteskind. A cluster's own etcd is therefore not backed up by the platform. A cluster still on a shared tenant etcd keeps exactly the coverage it had. The guest-side Velero addon covers workloads, not control-plane state. An operator can snapshot a cluster's etcd by hand with anEtcdSnapshotagainstkubernetes-<name>-etcd; the server certificate covers the-clientService the snapshot Job dials. Backup alone could routeKubernetesto the existing driver with a name lookup. Restore is the hard part: it means recreating theEtcdClusterunder a live KamajiTenantControlPlane, and needs its own design before anything is wired. The README states the gap.CiliumNetworkPolicynamed after the etcd selects its member pods withingressDenyon 2379/2380 against two sets of pods. The first is pods in the tenant namespace that are neither members nor carrypolicy.cozystack.io/allow-to-etcd-cluster: <etcd name>, a label set on the control-plane pods and the defrag Job. The second is pods in any other namespace exceptkube-systemand namespaces labelledcozystack.io/system: "true", which is where Kamaji, its migrate Job and etcd-operator run. Deny rules are needed because the tenant's own policies allow all traffic within a namespace and from ancestor tenants, and allow rules only add. Metrics on 2381 stay open. The pods also carrycozystack.io/service: kubernetes-etcdrather thancozystack.io/service: etcd, so the tenant-wide<tenant>-ingress-etcdpolicy does not admit child tenants' control planes to them. The policy renders only where Cilium servescilium.io/v2/CiliumNetworkPolicy.cozystack.kubernetes-applicationnowdependsOncozystack.etcd-operator. Both come with the isp-full variants, the only ones the iaas bundle allows. The selective-install fixture is unchanged (hack/select-install.sh --disabled postgresproduces the same set). The same edge makes an etcd-operator change select thekubernetes-latest,kubernetes-previousandcomputeplanee2e suites besidesetcd, andhack/select-e2e_test.batsnow expects that.hack/e2e-install-cozystack.batsno longer turns on tenant-root's etcd, which only Kubernetes clusters used. A cluster that fell back on a shared etcd would therefore not come up.run_kubernetes_testasserts the control plane namestenant-test.kubernetes-<name>and thatkubernetes-<name>-etcdreachesAvailable.cozy_cleanupwaits until theEtcdCluster,DataStore, member PVCs and Secrets are gone.etcd: trueand the comment calling it mandatory.hack/admin-kubeconfig-invariant.batsdrops its two awaiting-etcd cases, and the now-unusedtests/values-ci-no-etcd.yamlgoes with them.make -C packages/apps/kubernetes generateproducedvalues.schema.json, the README table,api/apps/v1alpha1/kubernetes/types.goandpackages/system/kubernetes-rd/cozyrds/kubernetes.yaml. Rootmake generateregeneratedzz_generated.deepcopy.go, andpkg/generated/openapidid not change.make generatein every touched package leaves the tree clean.KamajiControlPlanewhoseDataStoreits own release did not render, judged by Helm's release annotations on theDataStore, with thatDataStore's endpoints. It writes nothing but the version stamp. Slot check on 2026-10-08:mainships through 59 withtargetVersion: 60;release-1.6ships through 53 (target 54);release-1.7.0-alpha.3ships through 58 (target 59);packages/apps/kubernetes/README.mdclaimed a dedicated etcd per cluster while clusters shared the tenant's. It now describes what this PR ships, including the legacy case and the backup gap. The ComputePlane README no longer lists a tenant etcd as a prerequisite.Migrating an existing cluster (designed, not implemented)
Nothing here moves a cluster, and nothing should until the runbook below has an e2e. It combines #25's runbook with SerjioTT's Talos measurements of 2026-09-03 (about 20s copy, 43s from patch to Ready, reads uninterrupted) and with what Kamaji 26.9.4-edge's code does.
dataStoreName. The likely shape is a chart-internal value, or an annotation the chart reads, that the operator sets. Wait forAvailable=True.nc -zv <target> 2379from a pod carrying the control plane's labels in the cluster namespace. For the per-cluster target this holds by construction (same namespace, and the deny policy admits the control plane), but the check stays because rollback goes the other way.KamajiControlPlane.spec.dataStoreNameat<ns>.<release>, with the CAPI provider re-projecting it onto theTenantControlPlane.kamaji.clastix.io/migration-timeoutcan go on theKamajiControlPlane, because the provider copies annotations to theTenantControlPlane. Before switching, delete any leftovermigrate-<TCP UID>Job in Kamaji's namespace:kamaji-freezeValidatingWebhookConfiguration in the guest, and only its soot controller removes it, only when theTenantControlPlanestatus is exactlyReady. A pausedTenantControlPlane, or one sitting inWriteLimited,SleepingorUpgrading, keeps the freeze. Node leases are exempt and reads are untouched, so nodes andkubectl getlook healthy. The success criterion is a canary write through the guest API; ifkamaji-freezeis still in the guest after theTenantControlPlaneisReady, delete it.talos-caIssuer but issues no client credentials from it, so the fallback isvirtctl restartper VM.talosconfigfrom that CA alongside the admin kubeconfig is the follow-up that makes this step cheap. It is not in this PR: it is a separate credential with its own exposure question.Test plan
Done here, no cluster involved:
helm templateofpackages/extra/etcdbefore and after the extraction is byte-identical: 12 renders (default; 1 member + local SC + 10Gi; resources with CPU and memory allocation ratios + another version; a global topology spread; a hard affinity with 5 members and 512Mi; leftoverbackup.*values with 0 members), each intenant-rootandtenant-abc, all identical by sha256.helm templateofpackages/apps/kuberneteswith Cilium served. With_namespace.etcdset totenant-root, or empty, both with 3 members:EtcdCluster/kubernetes-demo-etcdreplicas=3 on the namespace default StorageClass, andDataStore/tenant-demo.kubernetes-demopointing atkubernetes-demo-etcd.tenant-demo.svc:2379. TheKamajiControlPlanenames thatDataStore, and the policy renders.replicatedand 3 members uselocal. 2 members is refused at render time; 5 is refused by the schema.KamajiControlPlanelookup stubbed to returntenant-root) renders identically tomainwithvalues-ci.yaml, apart from random Talos tokens. A new cluster's render differs frommainonly by the old shareddataStoreNameline and the added etcd objects.packages/apps/kubernetes: 359 tests, including a new 19-casetests/etcd_test.yaml. It covers names, the second cluster in one namespace, the 63-characterDataStorecap, StorageClass choice and stickiness, refusal of 2, the policy, the Secret cleanup and the delete-hook gate. For an existing cluster it covers staying on the sharedDataStore, ignoring a changed_namespace.etcd, no own etcd, no Secret cleanup and no gate.packages/extra/etcd: 22.packages/core/platform: 258, including a new wiring test for the StorageClass keys.cozy-lib-tests,packages/apps/tenantandpackages/extra/computeplane: green.hack/migration-60-shared-etcd-report.bats(new, 4 cases);hack/run-kubernetes*_test.bats,admin-kubeconfig-invariant,check-hook-request-timeout,app-cleanup-hooks_test,bats-no-negated-assert,bats-no-exit-trap,select-install_test,select-e2e_test,helmrelease-readiness,proxmox-e2e-fixture_test,talos-image-cache_test,linstor-storage-mode_testandcozystack-version-stamp.go build ./...,go test ./pkg/registry/...and theapi/apps/v1alpha1module.make generatein every touched package and at the root leaves no diff;hack/check-commit-trailers.sh origin/main..HEADpasses.Needs a live cluster:
kubernetes-latest,kubernetes-previousandcomputeplane, now without tenant-root's etcd. The Proxmox lane on its self-hosted runner.mainwith an existing cluster on tenant-root's etcd. It must stay ontenant-rootwith no control-plane rollout, and migration 60 must list it.nc -zv kubernetes-<name>-etcd.<ns>.svc 2379must be refused. The control plane, the defrag Job, anEtcdSnapshotJob, Kamaji and etcd-operator must still connect.TenantControlPlaneis held past the 120s wait. The uninstall must fail and retry rather than leave aTenantControlPlanestuck on its finalizer.ResourceQuotacannot fit 3 × 1000m/512Mi limits and 3 × 4Gi requests must surface the quota error on the etcd member pods.Follow-ups, not opened
These are listed here at the maintainer's request rather than filed:
content/en/docs/next/operations/configuration/platform-package.mdgainskubernetesEtcd.storageClassLocalandkubernetesEtcd.storageClassReplicated;content/en/docs/next/kubernetes/relocate-etcd.mddescribes the shared tenant etcd as what a cluster uses;etcd: truefor a cluster, which is now an obsolete workaround;internal/provider/kubernetes_schema.goandkubernetes_model.goneedetcd.replicas, with the default of 3, a 1..3 range and the expand/flatten pair.talosconfig;Screenshots
Not a UI change.
Downstream repositories
Follow-ups are listed above under "Follow-ups, not opened". By the maintainer's decision they are not filed from this draft, so no box is ticked.
Release note