Skip to content

Commit 6e8368f

Browse files
committed
feat(kubernetes): add the Proxmox CSI driver for tenant clusters
Packages the Proxmox CSI driver and cloud-controller-manager, and wires the driver into the tenant through its own HelmRelease. Without them a Proxmox-backed tenant has no way to provision a PersistentVolume at all. The two arrive together because they are not independent: the CSI node plugin reads topology.kubernetes.io/region and zone off its own Node and exits fatally when they are absent — "Failed to get region or zone for node …, see documentation about topology labels" — and those labels are applied by the cloud-controller-manager. Shipping the driver alone leaves a tenant with a StorageClass, a healthy CSI controller, and a node plugin in CrashLoopBackOff. The CCM in turn needs the kubelet started with --cloud-provider=external, which is what applies the node.cloudprovider.kubernetes.io/uninitialized taint it waits on. That flag is now set in the worker machineconfig on the proxmox path only: on kubevirt there is no cloud provider, and the flag would taint every worker with nothing to remove the taint. Both packages drop the upstream values.talos.yaml presets, which pin their workloads onto node-role.kubernetes.io/control-plane. A Kamaji tenant has no control-plane nodes — its control plane is a Deployment in the management cluster — so those presets schedule nowhere. The CSI keeps config.features.provider: proxmox, documented upstream as the setting for clusters provisioned by capmox. The CSI release is a separate template rather than a branch inside helmreleases/csi.yaml: the two drivers share only their role, and keeping them apart leaves the KubeVirt path rendering what it rendered before. Credentials reach the tenant through Flux valuesFrom, so the hypervisor token is never written into a rendered HelmRelease, and they are a distinct token from capmox's on purpose — attaching disks is a different blast radius from creating VMs. Verified on a live tenant: the driver comes up, a PVC binds, and the volume appears on the hypervisor as a real disk attached to the worker VM (scsi1: vm-disks1:vm-9999-pvc-…), then detaches cleanly on delete. Signed-off-by: Marian Koreniuk <[email protected]> Assisted-by: LLM
1 parent a860df1 commit 6e8368f

53 files changed

Lines changed: 2615 additions & 12 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎api/apps/v1alpha1/kubernetes/types.go‎

Lines changed: 28 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎packages/apps/kubernetes-nodes/templates/talos-reconcile-job.yaml‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -340,6 +340,22 @@ spec:
340340
{{- end }}
341341
kubelet:
342342
image: ghcr.io/siderolabs/kubelet:{{ .kubeletVersion }}
343+
{{- /* Without this the kubelet never applies the
344+
node.cloudprovider.kubernetes.io/uninitialized
345+
taint, the Proxmox cloud-controller-manager
346+
never gets to initialise the Node, and the
347+
topology.kubernetes.io/{region,zone} labels are
348+
never set. The CSI node plugin then exits
349+
fatally on "Failed to get region or zone for
350+
node", so the tenant ends up with a
351+
StorageClass and no way to attach a volume.
352+
The kubevirt path has no cloud provider and
353+
must not carry the flag: it would taint every
354+
worker with nothing to remove the taint. */}}
355+
{{- if eq ($.Values.substrate | default "kubevirt") "proxmox" }}
356+
extraArgs:
357+
cloud-provider: external
358+
{{- end }}
343359
extraConfig:
344360
systemReserved:
345361
cpu: "{{ .systemReservedCpu }}"

‎packages/apps/kubernetes/README.md‎

Lines changed: 19 additions & 10 deletions
Large diffs are not rendered by default.
Lines changed: 106 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,106 @@
1+
{{- /* The Proxmox counterpart of helmreleases/csi.yaml. Kept as its own file
2+
rather than a branch inside that one: the two drivers share nothing but
3+
the role. kubevirt-csi reaches DataVolumes in this cluster over the
4+
management API; proxmox-csi reaches the hypervisor over its own API with
5+
its own credentials, lands in its own namespace, and takes an entirely
6+
different values shape. Splitting them keeps the KubeVirt path rendering
7+
byte-for-byte what it rendered before Proxmox existed. */}}
8+
{{- if eq (.Values.substrate | default "kubevirt") "proxmox" }}
9+
{{- if and .Values._namespace.etcd .Values.proxmox.csi.enabled }}
10+
{{- if not .Values.proxmox.csi.credentialsSecretName }}
11+
{{- fail "proxmox.csi.credentialsSecretName is required when proxmox.csi.enabled: the driver authenticates to the hypervisor and the chart will not invent credentials" }}
12+
{{- end }}
13+
apiVersion: helm.toolkit.fluxcd.io/v2
14+
kind: HelmRelease
15+
metadata:
16+
name: {{ .Release.Name }}-csi
17+
labels:
18+
cozystack.io/repository: system
19+
cozystack.io/target-cluster-name: {{ .Release.Name }}
20+
sharding.fluxcd.io/key: tenants
21+
spec:
22+
interval: 5m
23+
timeout: 10m
24+
releaseName: csi
25+
chartRef:
26+
kind: ExternalArtifact
27+
{{- /* The artifact keeps the kubevirt- prefix because that is the variant
28+
of kubernetes-application it is published under, not a statement
29+
about the substrate. Naming it otherwise would mean a second variant
30+
and a rename of all eighteen tenant addon artifacts. */}}
31+
name: cozystack-kubernetes-application-kubevirt-kubernetes-proxmox-csi
32+
namespace: cozy-system
33+
kubeConfig:
34+
secretRef:
35+
name: {{ .Release.Name }}-admin-kubeconfig
36+
key: super-admin.svc
37+
targetNamespace: csi-proxmox
38+
storageNamespace: csi-proxmox
39+
{{- /* Same reasoning as the kubevirt release: install remediation is an
40+
uninstall, which would take the tenant's StorageClasses down between
41+
attempts. */}}
42+
install:
43+
createNamespace: true
44+
strategy:
45+
name: RetryOnFailure
46+
retryInterval: 30s
47+
remediation:
48+
retries: -1
49+
upgrade:
50+
strategy:
51+
name: RetryOnFailure
52+
retryInterval: 30s
53+
remediation:
54+
retries: -1
55+
{{- /* The credentials arrive by reference, not by value: Flux reads the
56+
Secret in this namespace and injects each field into the tenant
57+
release, so no token is ever written into a rendered HelmRelease. */}}
58+
valuesFrom:
59+
- kind: Secret
60+
name: {{ .Values.proxmox.csi.credentialsSecretName | quote }}
61+
valuesKey: url
62+
targetPath: proxmox-csi-plugin.config.clusters[0].url
63+
- kind: Secret
64+
name: {{ .Values.proxmox.csi.credentialsSecretName | quote }}
65+
valuesKey: token_id
66+
targetPath: proxmox-csi-plugin.config.clusters[0].token_id
67+
- kind: Secret
68+
name: {{ .Values.proxmox.csi.credentialsSecretName | quote }}
69+
valuesKey: token_secret
70+
targetPath: proxmox-csi-plugin.config.clusters[0].token_secret
71+
- kind: Secret
72+
name: {{ .Values.proxmox.csi.credentialsSecretName | quote }}
73+
valuesKey: region
74+
targetPath: proxmox-csi-plugin.config.clusters[0].region
75+
values:
76+
proxmox-csi-plugin:
77+
config:
78+
features:
79+
{{- /* Documented by the upstream chart as the setting for clusters
80+
provisioned by capmox: the driver then resolves a node's
81+
hypervisor placement from the Machine's providerID instead of
82+
the topology labels a cloud-controller-manager would apply.
83+
Without it the tenant needs a CCM before it can attach a single
84+
volume. */}}
85+
provider: proxmox
86+
clusters:
87+
- insecure: true
88+
storageClass:
89+
{{- range .Values.proxmox.csi.storageClasses }}
90+
- name: {{ .name | quote }}
91+
storage: {{ .storage | quote }}
92+
{{- with .fstype }}
93+
fstype: {{ . | quote }}
94+
{{- end }}
95+
{{- with .reclaimPolicy }}
96+
reclaimPolicy: {{ . | quote }}
97+
{{- end }}
98+
{{- if .ssd }}
99+
ssd: true
100+
{{- end }}
101+
{{- end }}
102+
dependsOn:
103+
- name: {{ .Release.Name }}-cilium
104+
namespace: {{ .Release.Namespace }}
105+
{{- end }}
106+
{{- end }}

‎packages/apps/kubernetes/values.schema.json‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,7 @@
4343
"type": "object",
4444
"default": {},
4545
"required": [
46+
"csi",
4647
"ipv4Config"
4748
],
4849
"properties": {
@@ -54,6 +55,61 @@
5455
"type": "string"
5556
}
5657
},
58+
"csi": {
59+
"description": "Proxmox CSI driver settings.",
60+
"type": "object",
61+
"default": {},
62+
"required": [
63+
"credentialsSecretName",
64+
"enabled"
65+
],
66+
"properties": {
67+
"credentialsSecretName": {
68+
"description": "Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. Flux injects them into the tenant release, so the token is never written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.",
69+
"type": "string",
70+
"default": ""
71+
},
72+
"enabled": {
73+
"description": "Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.",
74+
"type": "boolean",
75+
"default": true
76+
},
77+
"storageClasses": {
78+
"description": "StorageClasses to create in the tenant.",
79+
"type": "array",
80+
"default": [],
81+
"items": {
82+
"type": "object",
83+
"required": [
84+
"name",
85+
"storage"
86+
],
87+
"properties": {
88+
"fstype": {
89+
"description": "Filesystem the driver formats volumes with: ext4 or xfs.",
90+
"type": "string"
91+
},
92+
"name": {
93+
"description": "StorageClass name inside the tenant cluster.",
94+
"type": "string"
95+
},
96+
"reclaimPolicy": {
97+
"description": "Delete or Retain.",
98+
"type": "string"
99+
},
100+
"ssd": {
101+
"description": "Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.",
102+
"type": "boolean"
103+
},
104+
"storage": {
105+
"description": "Proxmox storage id the volumes are created on (as it appears in `pvesm status`).",
106+
"type": "string"
107+
}
108+
}
109+
}
110+
}
111+
}
112+
},
57113
"dnsServers": {
58114
"description": "Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.",
59115
"type": "array",

‎packages/apps/kubernetes/values.yaml‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,10 +95,29 @@ substrate: kubevirt
9595
## @field {[]string} dnsServers - Nameservers written into each worker's network config. Required by the ProxmoxCluster schema.
9696
## @field {ProxmoxIPv4} ipv4Config - Address pool for workers. capmox assigns static addresses and has no DHCP mode, so this is required rather than optional.
9797

98+
## @typedef {struct} ProxmoxStorageClass - A StorageClass backed by a Proxmox storage.
99+
## @field {string} name - StorageClass name inside the tenant cluster.
100+
## @field {string} storage - Proxmox storage id the volumes are created on (as it appears in `pvesm status`).
101+
## @field {string} [fstype] - Filesystem the driver formats volumes with: ext4 or xfs.
102+
## @field {bool} [ssd] - Mark volumes as SSD-backed, which changes the discard/cache defaults Proxmox applies.
103+
## @field {string} [reclaimPolicy] - Delete or Retain.
104+
105+
## @typedef {struct} ProxmoxCSI - Proxmox CSI driver, deployed into the tenant cluster.
106+
## @field {bool} enabled - Install the driver. Without it a Proxmox-backed tenant has no way to provision PersistentVolumes at all.
107+
## @field {string} credentialsSecretName - Secret in THIS namespace holding the driver's Proxmox API credentials under the keys `url`, `token_id`, `token_secret` and `region`. Flux injects them into the tenant release, so the token is never written into a rendered manifest. Distinct from the capmox credentials on purpose: the driver attaches and detaches disks, which is a different blast radius from creating VMs.
108+
## @field {[]ProxmoxStorageClass} storageClasses - StorageClasses to create in the tenant.
109+
110+
## @typedef {struct} Proxmox - Proxmox substrate settings for the cluster's infrastructure object. Ignored unless `substrate` is `proxmox`.
111+
## @field {ProxmoxCSI} csi - Proxmox CSI driver settings.
112+
98113
## @param {Proxmox} proxmox - Proxmox substrate settings.
99114
proxmox:
100115
allowedNodes: []
101116
dnsServers: []
117+
csi:
118+
enabled: true
119+
credentialsSecretName: ""
120+
storageClasses: []
102121
ipv4Config:
103122
addresses: []
104123
gateway: ""

‎packages/core/platform/sources/kubernetes-application.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,12 @@ spec:
3838
path: system/victoria-metrics-operator
3939
- name: kubernetes-kubevirt-csi-node
4040
path: system/kubevirt-csi-node
41+
# The storage driver for the proxmox substrate. Published alongside the
42+
# KubeVirt one rather than in a variant of its own: a second variant
43+
# would rename every tenant addon artifact, and only this component
44+
# differs between the two substrates.
45+
- name: kubernetes-proxmox-csi
46+
path: system/proxmox-csi
4147
- name: kubernetes-vertical-pod-autoscaler-crds
4248
path: system/vertical-pod-autoscaler-crds
4349
- name: kubernetes-coredns

‎packages/system/kubernetes-rd/cozyrds/kubernetes.yaml‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
apiVersion: v2
2+
name: cozy-proxmox-ccm
3+
version: 0.0.0 # Placeholder, the actual version will be automatically set during the build process
4+
dependencies:
5+
- name: proxmox-cloud-controller-manager
6+
version: "*"
7+
repository: ""
Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
export NAME=proxmox-ccm
2+
export NAMESPACE=kube-system
3+
4+
CHART_VERSION=0.2.30
5+
6+
include ../../../hack/package.mk
7+
8+
update:
9+
rm -rf charts
10+
mkdir -p charts
11+
helm pull oci://ghcr.io/sergelogvinov/charts/proxmox-cloud-controller-manager \
12+
--version $(CHART_VERSION) --untar --untardir charts

0 commit comments

Comments
 (0)