Skip to content

Enhance distributed deployment with snapshot and resize support for multi-node testing #651

Open

Description

@mnencia

Problem

The standard StatefulSet deployment (replicas=1) binds ALL volumes to a single node, making multi-node test scenarios impossible. This has been a recurring request:

The existing deploy/kubernetes-distributed/ DaemonSet deployment partially addresses this: it runs the driver on every node with csi-provisioner --node-deployment=true, enabling per-node volume provisioning with topology and capacity tracking. However, it only includes the provisioner sidecar, so snapshots and volume expansion are not supported in the distributed deployment.

Proposal

Enhance the distributed deployment to include the csi-snapshotter and csi-resizer sidecars, using their respective --node-deployment modes so each instance only handles operations for volumes local to its node.

Snapshotter: upstream support exists

The external-snapshotter already supports distributed snapshotting (kubernetes-csi/external-snapshotter#585, merged Dec 2021). It requires two coordinating flags on two separate components:

  1. The common snapshot-controller (a central Deployment) must be deployed with --enable-distributed-snapshotting=true. This makes it check PV node affinity and label each VolumeSnapshotContent with snapshot.storage.kubernetes.io/managed-by=<nodeName>.
  2. The csi-snapshotter sidecar (per-node, in the DaemonSet) must be deployed with --node-deployment=true. This makes it only reconcile VolumeSnapshotContent objects whose label matches its node.

Without the snapshot-controller flag, the labeling never activates and the per-node sidecars would see nothing to process.

The integration still needs to be built and tested against current sidecar versions (the standard deployment uses csi-snapshotter:v8.4.0). The distributed deploy.sh would need to deploy (or patch) the common snapshot-controller with the non-default flag, add a VolumeSnapshotClass, and handle RBAC bindings. Notably, the DaemonSet currently uses serviceAccountName: csi-provisioner, so upstream snapshotter RBAC (which binds to a csi-snapshotter service account) would need to be adapted to bind to csi-provisioner instead.

Resizer: upstream work needed

The external-resizer does not yet support --node-deployment. A PR existed (kubernetes-csi/external-resizer#195) but was closed by the triage bot after going rotten. The tracking issue kubernetes-csi/external-resizer#142 is still open. A fresh PR on external-resizer would be needed to enable distributed resize support.

Changes required in this repo

Deployment-level only, no Go code changes to the driver:

File Change
deploy/kubernetes-distributed/hostpath/csi-hostpath-plugin.yaml Add csi-snapshotter container (and csi-resizer once upstream supports it)
deploy/kubernetes-distributed/hostpath/csi-hostpath-snapshotclass.yaml (new) VolumeSnapshotClass
deploy/kubernetes-distributed/test-driver.yaml Add snapshotDataSource capability and SnapshotClass: FromName: true
deploy/kubernetes-distributed/deploy.sh Fetch and adapt snapshotter RBAC (bind to csi-provisioner SA), deploy common snapshot-controller with --enable-distributed-snapshotting=true, install snapshot CRDs if needed

Known limitations

These are inherent to a node-local hostpath driver and should be documented:

  1. Cross-node snapshot restore: creating a volume from a snapshot only works if the new volume lands on the same node as the snapshot. loadFromSnapshot operates on local filesystem paths.
  2. Cross-node volume cloning: same constraint, loadFromVolume runs cp -a on local paths.
  3. Immediate binding mode: the distributed provisioner uses --immediate-topology=false, so only WaitForFirstConsumer works.

CI

The existing CI already uses a 3-node Kind cluster (1 control-plane + 2 workers). The distributed deployment is never exercised in CI today. Setting CSI_PROW_DEPLOYMENT=kubernetes-distributed would select it, though the prow framework's snapshot-controller deployment would also need to be configured with --enable-distributed-snapshotting=true (either via the distributed deploy.sh or by patching the prow-managed manifests).

Next steps

I'd like to start by adding the snapshotter sidecar to the distributed DaemonSet, since the upstream support is already there. Meanwhile, I'm planning to open a fresh PR on kubernetes-csi/external-resizer for --node-deployment (the previous attempt in #195 went rotten, tracking issue kubernetes-csi/external-resizer#142 is still open). The resizer can be added to the distributed deployment once that lands.

Activity

  1. mnencia commented on Apr 12, 2026

    @mnencia
    Author

    The snapshotter part is up in #653. For the resizer, I have a branch ready at dev/651-with-resizer that adds the csi-resizer sidecar with --node-deployment=true, built on top of #653. It depends on kubernetes-csi/external-resizer#573 being merged first. Tested on a 3-node Kind cluster with provisioning, snapshots, and volume expansion all working.

  2. mnencia commented on Apr 27, 2026

    @mnencia
    Author

    #653 (snapshotter sidecar) covers per-node snapshotting, but cross-node restore still deadlocks: hostpath snapshot data lives on the node that took the snapshot, and kube-scheduler may pick a different node. I have a two-part fix on top of #653 and want maintainer input on scope before opening more PRs.

    What deadlocks today

    Snapshots are stored as .snap files on the node where they were taken (getSnapshotPath joins StateDir locally). When a PVC under WaitForFirstConsumer restores from such a snapshot, kube-scheduler may pick a node that does not own the source data. CreateVolume calls loadFromSnapshot, the local lookup fails with codes.NotFound, external-provisioner classifies that as ProvisioningFinished (no retry), and the PVC sits with volume.kubernetes.io/selected-node pointing at the wrong node. Same shape for clone sources via loadFromVolume.

    Part 1: rescheduling fix in the driver (safety net)

    Branch: dev/651-followup-reschedule, one commit on top of #653.

    Adds a --node-deployment flag to the driver. When set, loadFromSnapshot and loadFromVolume translate the local codes.NotFound into codes.ResourceExhausted. external-provisioner sees mayReschedule = supportsTopology() && len(SelectedNodeName) > 0 and calls rescheduleProvisioning, which clears selected-node and lets the scheduler pick again. Centralized deployments default to false, so behaviour for missing sources is unchanged.

    This converges in small clusters when the source node is schedulable. It does not fix:

    • source node cordoned, drained, or NotReady (PVC stays pending)
    • consuming pod has node affinity or anti-affinity that excludes the source node (deadlock)
    • source data genuinely lost from every node (loops until retry budget exhausts)
    • exponential backoff blowing past usable bind timeouts on large clusters

    Part 2: csi-topology-coordinator (steady-state path)

    Branch: dev/csi-topology-coordinator, one commit on top of Part 1.

    A small Deployment that watches PVCs in WaitForFirstConsumer with a snapshot or clone data source, resolves the owning node, and patches volume.kubernetes.io/selected-node before scheduling. For snapshots, the owning node is read from the VSC label snapshot.storage.kubernetes.io/managed-by, which snapshot-controller --enable-distributed-snapshotting sets to the node owning the source PV. For clones, it comes from the source PV's nodeAffinity directly. external-provisioner respects a pre-set selected-node annotation when present, so provisioning runs on the source node from the first scheduling pass.

    The CSI driver DaemonSet needs no Kubernetes API access. Putting this logic in the per-node snapshotter or provisioner would require granting every node-local sidecar cluster-wide PVC patch permissions; a separate centralized component keeps that permission off the DaemonSet. Coordinator RBAC: PVC list+watch+patch, get-only on PV/VS/VSC, lease scoped via resourceNames.

    Without Part 1, the coordinator has to be perfect (one race against the scheduler and the PVC stalls). Without Part 2, Part 1's rescheduling loop only converges under benign conditions. Together: race-tolerant correctness plus right-node-first-try in the common case.

    What I tested

    Demonstrated end-to-end on a 3-node Kind cluster: 3 PVCs distributed across workers, snapshotted, deleted, restored with no pod node-affinity, all three restore pods scheduled on the source-snapshot node, data integrity verified.

    Not yet covered: coordinator-down behaviour, scheduler-vs-coordinator race tests, cordoned source node, pod anti-affinity to the source node. cloudnative-pg/cloudnative-pg#10438 wires this branch into the CNPG operator E2E suite (full matrix green today against this full stack), but that is not a regression test for the driver-side scenarios above.

    Full stack

    Branch dev/651-with-resizer carries snapshotter + rescheduling fix + coordinator + resizer-with---node-deployment (resizer pending its own upstream PR).

    Two questions

    1. Does the coordinator belong in this repo? The package is driver-agnostic (driver name and topology key are flags). Landing it here first lets distributed-mode E2E exercise it without cross-repo coordination. Long-term it might fit better in its own kubernetes-csi repo, or absorbed into snapshot-controller or external-provisioner. If it does not belong here, I will move it to a separate repo.

    2. Image publishing. The manifest currently pins ghcr.io/mnencia/csi-topology-coordinator:latest as a placeholder. If the controller stays in this repo, what is the path to registry.k8s.io/sig-storage/... for a new sidecar image? sig-storage cloud-build plus image-promoter, or something else?

    The next PR is gated on these answers.

  3. k8s-triage-robot commented on Oct 17, 2026

    @k8s-triage-robot

    The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

    This bot triages un-triaged issues according to the following rules:

    • After 90d of inactivity, lifecycle/stale is applied
    • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
    • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

    You can:

    • Mark this issue as fresh with /remove-lifecycle stale
    • Close this issue with /close
    • Offer to help out with Issue Triage

    Please send feedback to sig-contributor-experience at kubernetes/community.

    /lifecycle stale

  4. added
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Oct 17, 2026
  5. mnencia commented on Oct 17, 2026

    @mnencia
    Author

    /remove-lifecycle stale

  6. removed
    lifecycle/staleDenotes an issue or PR has remained open with no activity and has become stale.
    on Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions