Default tenant Kubernetes control plane does not fit a tenant with resourceQuotas.cpu 24 and deadlocks on creation #5070

What we see

On v1.6.5 a tenant was created with resourceQuotas set to cpu: 24, memory: 24Gi, storage: 50Gi and services.loadbalancers: 1, and with etcd: true. A Kubernetes app was then created in it with chart defaults plus addons.ingressNginx.enabled: true, so version v1.35, two control plane replicas and the default md0 node group. Apart from the tenant etcd nothing else ran in the namespace at that point. 16 hours later the cluster still has no nodes.

The tenant chart rendered tenant-quota with limits.cpu: 24 and requests.cpu: 2400m. requests.cpu is cpu divided by the platform cpu-allocation-ratio, 10 by default (quota.yaml L1-L9, _resources.tpl L99-L125).

The control plane Deployment owned by the TenantControlPlane got two ReplicaSets in the same second. The first one has 4 containers, the second one 5. The first ReplicaSet could not create its second pod, and the Deployment condition ReplicaFailure says this.

pods "kubernetes-<name>-<hash>-<id>" is forbidden: exceeded quota: tenant-quota, requested: requests.cpu=310m, used: requests.cpu=2140m, limited: requests.cpu=2400m

Since then the new ReplicaSet retries its second pod about every 16 minutes and is refused the same way with requested: requests.cpu=360m. About 10 minutes after creation Progressing went False with ProgressDeadlineExceeded. The old ReplicaSet has 1 pod, the new one 1 of 2, and nothing has changed since.

The rest follows from that.

• The TenantControlPlane status is Provisioning. The KamajiControlPlane has initialized: false and ready: false.
• The md0 MachineDeployment is ScalingUp from 0 to 1, but its MachineSet creates no Machine and reports KamajiControlPlane <namespace>/kubernetes-<name> is provisioning ("ControlPlaneIsStable" preflight check failed). There is no VM and no node.
• The cilium addon HelmRelease is UpgradeFailed with failed early due to stalled resources: [Deployment/cozy-cilium/cilium-operator status: 'Failed']. coredns, csi, ingress-nginx and metrics-server are DependencyNotReady.

None of these mention the quota. We found it only in the Deployment condition above.

Where the requests come from

With chart defaults these pods run in the tenant namespace. The control plane containers are sized by presets through cozy-lib, so their requests follow cpu-allocation-ratio like the quota does. kcsi-controller, kccm and cluster-autoscaler have fixed values that do not (csi/deploy.yaml L69-L75 and the same block on all 7 containers, kccm/manager.yaml L41-L47, cluster-autoscaler/deployment.yaml L42-L48, unchanged on main).

At rest there are two control plane pods of 360m, so this comes to 2145m. A healthy tenant on another cluster with the same defaults, one md0 worker and one small Postgres has 2295m of requests.cpu in use, which is exactly 2145m plus 100m for the worker virt-launcher and 50m for Postgres. So under requests.cpu: 2400m the default cluster leaves 255m with no workers and 155m with one. The nine containers with a fixed 125m take 1125m of the 2400m, almost half.

Why it does not recover

The chart sets no spec.deployment.strategy on the KamajiControlPlane (cluster.yaml L426-L486, same on main). capi-kamaji v0.19.0 copies it to the TenantControlPlane as is (kamajicontrolplane_controller_tcp.go L236), and Kamaji falls back to maxSurge: 100% and maxUnavailable: 0, both as the CRD default and in its Deployment builder (tenantcontrolplane_types.go L255-L258, deployment.go L140-L175, the same in 26.9.4-edge). With maxUnavailable: 0 the Deployment keeps the last old pod until two new pods are available. So every control plane rollout needs quota for at least one more control plane pod, 360m here.

A new cluster starts with such a rollout. Kamaji creates the Deployment without konnectivity-server and its konnectivity handler adds the container in a second step (deployment_resource.go L85-L101), which gives the 4 and 5 container ReplicaSets above. The second new pod does not fit, so the old pod is never removed and the rollout never finishes.

clastix/kamaji#1372 renders konnectivity-server in the first revision. It was merged to Kamaji master on 2026-10-21 and is not in a tag yet, the latest tag is 26.10.4-edge. With it, creating the default cluster would need 2145m and fit. The next rollout would still stop the same way, because 360m does not fit into 255m. A Kubernetes version upgrade is such a rollout, and per clastix/kamaji#663 so is a change of the control plane replica count while konnectivity is on.

Ideas

Would any of these make sense?

1. Lower the fixed requests of kcsi-controller, kccm and cluster-autoscaler, pass them through cozy-lib so they follow cpu-allocation-ratio, or make them values. Upstream kubevirt/csi-driver requests at most 10m of cpu and 20Mi or 50Mi of memory per controller container and sets no limits (controller-infra, controller-tenant). The chart had 10m and 50Mi too until bugfix/fix-vpa #698 set 125m and 128Mi with 512m and 512Mi limits, and [kubernetes] Add CPU/memory limits to csi-resizer sidecar #2614 brought csi-resizer in line with that. We have no usage numbers for these containers yet, so we can't say which value is right.
2. Is the blue/green control plane rollout something Cozystack wants for tenant clusters? KamajiControlPlane has spec.deployment.strategy. maxUnavailable: 1 would not need room for an extra pod, at the cost of one control plane replica during a rollout and a longer window with old and new apiservers side by side, which the Kamaji default keeps short. If the default stays, the quota has to leave room for one more control plane pod.
3. A note in the tenant docs. The tenant README describes resourceQuotas but does not say that requests.cpu is cpu divided by cpu-allocation-ratio, or that a default Kubernetes app requests 1845m by itself, plus 300m for the tenant etcd, 100m per u1.medium worker and 360m during a rollout.
4. Show the quota error in the Kubernetes app status. Today it is only on the control plane Deployment. Proposal: improve error diagnostics surface across controller, UI, and chart authoring #2733 tracks this kind of surfacing in general.

The reservation design in cozystack/community#48, with phase 1 in #4983, counts these components as platform overhead and deletes tenant-quota in phase 5, which would remove this problem. Until then tenant-quota counts them. One detail for that design, its overhead table lists a 4-container CSI deployment, while kcsi-controller has 7 containers on v1.6.5 and on main. #4817 would add the etcd members of each new cluster to this footprint, and its release note says they count against the tenant quota.

Side observation

While it was stuck, the API audit log showed about 12.6k update requests per hour from the capi-kamaji controller on the KamajiControlPlane and about 6.3k on the TenantControlPlane, with resourceVersion unchanged over 20 s, so they look like no-op updates. Our log store seemed to keep each audit event twice, so the real rate may be half that.

Environment

Cozystack v1.6.5 with Kamaji 26.3.6-edge and capi-kamaji v0.19.0 as shipped, management cluster Kubernetes v1.34.3 on 3 nodes, tenant cluster v1.35. The control plane defaults, the fixed requests and the missing strategy are the same in v1.7.0-rc.2 and on main at 215e582, both with Kamaji 26.9.4-edge. main moved worker pools to KubernetesNodes in #3315, which does not touch the control plane side.