Repository navigation
Description
Problem
The standard StatefulSet deployment (replicas=1) binds ALL volumes to a single node, making multi-node test scenarios impossible. This has been a recurring request:
- Use hostpath.csi.k8s.io across multiple node pools #211 ("Use hostpath.csi.k8s.io across multiple node pools")
- Hostpath across multiple nodes #49 ("Hostpath across multiple nodes")
- add distributed snapshotter #392 (distributed snapshotting attempt, went stale and was closed by the triage bot)
The existing deploy/kubernetes-distributed/ DaemonSet deployment partially addresses this: it runs the driver on every node with csi-provisioner --node-deployment=true, enabling per-node volume provisioning with topology and capacity tracking. However, it only includes the provisioner sidecar, so snapshots and volume expansion are not supported in the distributed deployment.
Proposal
Enhance the distributed deployment to include the csi-snapshotter and csi-resizer sidecars, using their respective --node-deployment modes so each instance only handles operations for volumes local to its node.
Snapshotter: upstream support exists
The external-snapshotter already supports distributed snapshotting (kubernetes-csi/external-snapshotter#585, merged Dec 2021). It requires two coordinating flags on two separate components:
- The common snapshot-controller (a central Deployment) must be deployed with
--enable-distributed-snapshotting=true. This makes it check PV node affinity and label each VolumeSnapshotContent withsnapshot.storage.kubernetes.io/managed-by=<nodeName>. - The csi-snapshotter sidecar (per-node, in the DaemonSet) must be deployed with
--node-deployment=true. This makes it only reconcile VolumeSnapshotContent objects whose label matches its node.
Without the snapshot-controller flag, the labeling never activates and the per-node sidecars would see nothing to process.
The integration still needs to be built and tested against current sidecar versions (the standard deployment uses csi-snapshotter:v8.4.0). The distributed deploy.sh would need to deploy (or patch) the common snapshot-controller with the non-default flag, add a VolumeSnapshotClass, and handle RBAC bindings. Notably, the DaemonSet currently uses serviceAccountName: csi-provisioner, so upstream snapshotter RBAC (which binds to a csi-snapshotter service account) would need to be adapted to bind to csi-provisioner instead.
Resizer: upstream work needed
The external-resizer does not yet support --node-deployment. A PR existed (kubernetes-csi/external-resizer#195) but was closed by the triage bot after going rotten. The tracking issue kubernetes-csi/external-resizer#142 is still open. A fresh PR on external-resizer would be needed to enable distributed resize support.
Changes required in this repo
Deployment-level only, no Go code changes to the driver:
| File | Change |
|---|---|
deploy/kubernetes-distributed/hostpath/csi-hostpath-plugin.yaml |
Add csi-snapshotter container (and csi-resizer once upstream supports it) |
deploy/kubernetes-distributed/hostpath/csi-hostpath-snapshotclass.yaml (new) |
VolumeSnapshotClass |
deploy/kubernetes-distributed/test-driver.yaml |
Add snapshotDataSource capability and SnapshotClass: FromName: true |
deploy/kubernetes-distributed/deploy.sh |
Fetch and adapt snapshotter RBAC (bind to csi-provisioner SA), deploy common snapshot-controller with --enable-distributed-snapshotting=true, install snapshot CRDs if needed |
Known limitations
These are inherent to a node-local hostpath driver and should be documented:
- Cross-node snapshot restore: creating a volume from a snapshot only works if the new volume lands on the same node as the snapshot.
loadFromSnapshotoperates on local filesystem paths. - Cross-node volume cloning: same constraint,
loadFromVolumerunscp -aon local paths. Immediatebinding mode: the distributed provisioner uses--immediate-topology=false, so onlyWaitForFirstConsumerworks.
CI
The existing CI already uses a 3-node Kind cluster (1 control-plane + 2 workers). The distributed deployment is never exercised in CI today. Setting CSI_PROW_DEPLOYMENT=kubernetes-distributed would select it, though the prow framework's snapshot-controller deployment would also need to be configured with --enable-distributed-snapshotting=true (either via the distributed deploy.sh or by patching the prow-managed manifests).
Next steps
I'd like to start by adding the snapshotter sidecar to the distributed DaemonSet, since the upstream support is already there. Meanwhile, I'm planning to open a fresh PR on kubernetes-csi/external-resizer for --node-deployment (the previous attempt in #195 went rotten, tracking issue kubernetes-csi/external-resizer#142 is still open). The resizer can be added to the distributed deployment once that lands.
Activity
mnencia commentedon Apr 12, 2026
The snapshotter part is up in #653. For the resizer, I have a branch ready at
dev/651-with-resizerthat adds the csi-resizer sidecar with--node-deployment=true, built on top of #653. It depends on kubernetes-csi/external-resizer#573 being merged first. Tested on a 3-node Kind cluster with provisioning, snapshots, and volume expansion all working.mnencia commentedon Apr 27, 2026
#653 (snapshotter sidecar) covers per-node snapshotting, but cross-node restore still deadlocks: hostpath snapshot data lives on the node that took the snapshot, and kube-scheduler may pick a different node. I have a two-part fix on top of #653 and want maintainer input on scope before opening more PRs.
What deadlocks today
Snapshots are stored as
.snapfiles on the node where they were taken (getSnapshotPathjoinsStateDirlocally). When a PVC under WaitForFirstConsumer restores from such a snapshot, kube-scheduler may pick a node that does not own the source data.CreateVolumecallsloadFromSnapshot, the local lookup fails withcodes.NotFound, external-provisioner classifies that asProvisioningFinished(no retry), and the PVC sits withvolume.kubernetes.io/selected-nodepointing at the wrong node. Same shape for clone sources vialoadFromVolume.Part 1: rescheduling fix in the driver (safety net)
Branch:
dev/651-followup-reschedule, one commit on top of #653.Adds a
--node-deploymentflag to the driver. When set,loadFromSnapshotandloadFromVolumetranslate the localcodes.NotFoundintocodes.ResourceExhausted. external-provisioner seesmayReschedule = supportsTopology() && len(SelectedNodeName) > 0and callsrescheduleProvisioning, which clearsselected-nodeand lets the scheduler pick again. Centralized deployments default to false, so behaviour for missing sources is unchanged.This converges in small clusters when the source node is schedulable. It does not fix:
Part 2: csi-topology-coordinator (steady-state path)
Branch:
dev/csi-topology-coordinator, one commit on top of Part 1.A small Deployment that watches PVCs in WaitForFirstConsumer with a snapshot or clone data source, resolves the owning node, and patches
volume.kubernetes.io/selected-nodebefore scheduling. For snapshots, the owning node is read from the VSC labelsnapshot.storage.kubernetes.io/managed-by, whichsnapshot-controller --enable-distributed-snapshottingsets to the node owning the source PV. For clones, it comes from the source PV'snodeAffinitydirectly. external-provisioner respects a pre-setselected-nodeannotation when present, so provisioning runs on the source node from the first scheduling pass.The CSI driver DaemonSet needs no Kubernetes API access. Putting this logic in the per-node snapshotter or provisioner would require granting every node-local sidecar cluster-wide PVC patch permissions; a separate centralized component keeps that permission off the DaemonSet. Coordinator RBAC: PVC list+watch+patch, get-only on PV/VS/VSC, lease scoped via
resourceNames.Without Part 1, the coordinator has to be perfect (one race against the scheduler and the PVC stalls). Without Part 2, Part 1's rescheduling loop only converges under benign conditions. Together: race-tolerant correctness plus right-node-first-try in the common case.
What I tested
Demonstrated end-to-end on a 3-node Kind cluster: 3 PVCs distributed across workers, snapshotted, deleted, restored with no pod node-affinity, all three restore pods scheduled on the source-snapshot node, data integrity verified.
Not yet covered: coordinator-down behaviour, scheduler-vs-coordinator race tests, cordoned source node, pod anti-affinity to the source node. cloudnative-pg/cloudnative-pg#10438 wires this branch into the CNPG operator E2E suite (full matrix green today against this full stack), but that is not a regression test for the driver-side scenarios above.
Full stack
Branch
dev/651-with-resizercarries snapshotter + rescheduling fix + coordinator + resizer-with---node-deployment(resizer pending its own upstream PR).Two questions
Does the coordinator belong in this repo? The package is driver-agnostic (driver name and topology key are flags). Landing it here first lets distributed-mode E2E exercise it without cross-repo coordination. Long-term it might fit better in its own kubernetes-csi repo, or absorbed into snapshot-controller or external-provisioner. If it does not belong here, I will move it to a separate repo.
Image publishing. The manifest currently pins
ghcr.io/mnencia/csi-topology-coordinator:latestas a placeholder. If the controller stays in this repo, what is the path toregistry.k8s.io/sig-storage/...for a new sidecar image? sig-storagecloud-buildplus image-promoter, or something else?The next PR is gated on these answers.
fix: deploy CSI hostpath as an opt-in single-node driver (#94)
k8s-triage-robot commentedon Oct 17, 2026
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
lifecycle/staleis appliedlifecycle/stalewas applied,lifecycle/rottenis appliedlifecycle/rottenwas applied, the issue is closedYou can:
/remove-lifecycle stale/closePlease send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
mnencia commentedon Oct 17, 2026
/remove-lifecycle stale