Skip to content

Commit 53c612f

Browse files
committed
feat(kubernetes): deploy the Proxmox cloud-controller-manager with the cluster
On this substrate the worker kubelet runs with --cloud-provider=external, which taints every node node.cloudprovider.kubernetes.io/uninitialized until a cloud provider clears it. Without a cloud-controller-manager the taint is permanent: nothing but tolerating DaemonSets schedules, and the CSI node plugin — which reads topology.kubernetes.io/region and zone off its own Node and exits fatally without them — never starts. It runs in the management cluster for the same reason as the CSI controller beside it: it authenticates to the hypervisor, and the tenant is cluster-admin of its own cluster. Its whole job is to write on the tenant's Nodes, and it does that over the tenant's admin kubeconfig from out of cluster, which also means there is no tenant-side RBAC left to install. capmox mode, not the default one. The default resolves a node to its VM by name and writes providerID as proxmox://<node>/<vmid>, while capmox reads the CAPI Machine and writes proxmox://<SMBIOS UUID> — the form capmox itself puts on the Machine. With the default the Node and its Machine carry two different identifiers for the same VM. The tolerations are spelled out rather than a blanket `operator: Exists`. That one also matches node.kubernetes.io/unreachable:NoExecute with no tolerationSeconds, which stops admission adding the bounded 300s one and lets the Pod outlive the Node it runs on. `proxmox.insecure` defaults to false: the controllers verify the hypervisor's certificate unless an operator turns that off, and both run here, so the value describes what the management cluster trusts, not the tenant. The README gains a section on what the two tokens can reach — the controller acts on whatever VMID and volume handle the tenant's objects name — and on the per-tenant Proxmox scope that contains it: a storage of the tenant's own, a token that reaches only its VMs, and VM.Audit on every one of them for the CCM, which deletes a NotReady Node whose VM it cannot see. The kubelet flag is asserted on both substrates: present on proxmox, absent on kubevirt, where nothing would ever clear the taint it causes. Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM
1 parent f24deb3 commit 53c612f

19 files changed

Lines changed: 509 additions & 43 deletions

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 40 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎api/apps/v1alpha1/kubernetes/zz_generated.deepcopy.go‎

Lines changed: 52 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -338,6 +338,22 @@ spec:
338338
{{- end }}
339339
kubelet:
340340
image: ghcr.io/siderolabs/kubelet:{{ .kubeletVersion }}
341+
{{- /* Without this the kubelet never applies the
342+
node.cloudprovider.kubernetes.io/uninitialized
343+
taint, the Proxmox cloud-controller-manager
344+
never gets to initialise the Node, and the
345+
topology.kubernetes.io/{region,zone} labels are
346+
never set. The CSI node plugin then exits
347+
fatally on "Failed to get region or zone for
348+
node", so the tenant ends up with a
349+
StorageClass and no way to attach a volume.
350+
The kubevirt path has no cloud provider and
351+
must not carry the flag: it would taint every
352+
worker with nothing to remove the taint. */}}
353+
{{- if eq ($.Values.substrate | default "kubevirt") "proxmox" }}
354+
extraArgs:
355+
cloud-provider: external
356+
{{- end }}
341357
extraConfig:
342358
systemReserved:
343359
cpu: "{{ .systemReservedCpu }}"

‎packages/apps/kubernetes-nodes/tests/talos_worker_config_test.yaml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,3 +73,32 @@ tests:
7373
- matchRegex:
7474
path: spec.template.spec.containers[0].command[2]
7575
pattern: 'kubeReserved:\s*\n\s+cpu: "100m"\s*\n\s+memory: "512Mi"'
76+
77+
# On proxmox the worker kubelet runs with an external cloud provider: the
78+
# node is tainted uninitialized until the cloud-controller-manager clears it
79+
# and applies the topology labels the CSI node plugin needs. On kubevirt no
80+
# controller would ever clear that taint, so the flag must not be there.
81+
- it: runs the kubelet with an external cloud provider on proxmox
82+
set:
83+
substrate: proxmox
84+
proxmox:
85+
templateTags: [capmox-template, talos]
86+
network:
87+
bridge: vmbr0
88+
dnsServers: ["8.8.8.8"]
89+
documentSelector:
90+
path: kind
91+
value: Job
92+
asserts:
93+
- matchRegex:
94+
path: spec.template.spec.containers[0].command[2]
95+
pattern: 'extraArgs:\s*\n\s+cloud-provider: external'
96+
97+
- it: leaves the kubevirt kubelet without a cloud provider, so no node is tainted for nothing
98+
documentSelector:
99+
path: kind
100+
value: Job
101+
asserts:
102+
- notMatchRegex:
103+
path: spec.template.spec.containers[0].command[2]
104+
pattern: 'cloud-provider: external'

0 commit comments

Comments
 (0)