Skip to content

[BUG] Support 3rd party RWO CSI drivers to be able to create non-root volumes from the UI #8199

Closed

Description

@rajha-korithrien

Describe the Bug

3rd party CSI provisioners are not always RWX some are RWO.
This is taken care of correctly when using lvm.driver.harvesterhci.io as the provisioner, but when using any other local CSI compliant provisioner (i.e. topolvm, openebs local-pv, etc.) the PVCs created are always set to RWX.

There needs to be some mechanism put in place for when using 3rd party CSI drivers to indicate to Harvester to create PVC as RWO.

To Reproduce

We provide an example using topolvm, but this problem will exist for any non RWX capable CSI provisioner.

  1. In you local context, add the helm repos needed to install your chosen 3rd party csi driver (we show topolvm):
helm repo add jetstack https://charts.jetstack.io
helm repo add topolvm https://topolvm.github.io/topolvm
  1. Deploy Harvester 1.5
  • Ensure there is an unused local storage location available to the bare metal host (i.e. not being used by Harvester/Longhorn). In our case this is at /dev/sdb.
  • create a local volume group by using pvcreate, vgcreate, lvcreate as root on the harvester node
h2:~ # pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.
h2:~ # vgcreate topolvm-test /dev/sdb
  Volume group "topolvm-test" successfully created
h2:~ # lvcreate --size 238G --type thin-pool --thinpool thinpool topolvm-test
  Thin pool volume with chunk size 128.00 KiB can address at most 31.75 TiB of data.
  Logical volume "thinpool" created.
h2:~ #
  1. Choose a 3rd party CSI driver that only supports RWO volumes. In this example we show using topolvm- Install cert-manage to you can install topolvm (into the base Harvester cluster)
helm -n cert-manager --create-namespace install cert-manager --set crds.enabled=true jetstack/cert-manager
  • For topolvm first create an answers file with the following content):
lvmd:
  deviceClasses:
    - name: local-ssd
      volume-group: topolvm-test
      default: false
      spare-gb: 1
      type: thin
      thin-pool:
        name: thinpool
        overprovision-ratio: 1.0
storageClasses:
  - name: local-ssd
    storageClass:
      reclaimPolicy: Delete
      annotations: {}
      isDefaultClass: false
      volumeBindingMode: Immediate
      allowVolumeExpansion: true
      additionalParameters:
        "topolvm.io/device-class": "local-ssd"
      mountOptions: []
  • Install topolvm
helm -n topolvm --create-namespace install -f topolvm-answers.yaml topolvm topolvm/topolvm
  1. Use the Harvester UI to create a Volume (notice that we now have a "local-ssd" storageClass available)
  • Ensure that you choose the "local-ssd" Storage Class
  • Notice that you can specify a Volume Mode (as Block) and a Size, but you can not specify an accessMode
  1. Observe that the PVC is never bound to a PV because the chosen provisioner returns an error:
rpc error: code = InvalidArgument desc = unsupported access mode: MULTI_NODE_MULTI_WRITER

Expected Behavior

It is expected that the UI should allow the user to choose an accessMode that is supported by the underlying 3rd party CSI provisioner. Or perhaps annotations must be placed on storageClass to indicate the supported accessMode.

This is necessary to allow full support of any 3rd CSI compliant provisioner.

Support Bundle for Troubleshooting

Environment

  • Harvester version: 1.5
  • Impacted VM: N/A the issues is about volumes, doesn't matter if they are created along with a VM or not
  • Impacted volume (PV): pvc-c8befe7e-0e4c-4199-a703-56c1145fbb36 from PVC p-4pffp
  • Underlying Infrastructure (e.g., Baremetal with Dell PowerEdge R630): baremetal
  • Rancher version: 2.11.1

Additional context

The example provided uses TopoLVM, but any 3rd party CSI driver that only supports RWO will encounter this problem. We have also tested with OpenEBS local-pv

Workaround and Mitigation

It is possible to manually create a PVC yourself for example create a file with contents:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    volume.beta.kubernetes.io/storage-provisioner: topolvm.io
    volume.kubernetes.io/storage-provisioner: topolvm.io
  finalizers:
  - kubernetes.io/pvc-protection
  - topolvm.io/pvc
  - wrangler.cattle.io/persistentvolumeclaim-controller
  name: topolvm-test-vm-disk-1-cyc2m
  namespace: topolvm-testing
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
  storageClassName: local-ssd
  volumeMode: Block

Then apply it to the cluster:

kubectl  -n topolvm-testing apply -f topolvm-pvc.yaml

At this point the volume can be attached to VMs

Activity

  1. added
    kind/bugIssues that are defects reported by users or that we know have reached a real release
    require/reproduceRequire adding a `reproduce` label.
    require/severityRequire adding a `severity` label.
    on May 5, 2025
  2. rajha-korithrien commented on May 5, 2025

    @rajha-korithrien
    Author

    Tried to add a support bundle but was told failed to upload supportbundle_c686f2fc-eb5a-4c3a-9d1e-1bb5778d4ad5_2025-05-05T21-27-58Z.zip

  3. ibrokethecloud commented on May 5, 2025

    @ibrokethecloud
    Contributor

    you should be able to edit the storageprofile for your storage class to reflect the correct access method

    https://docs.harvesterhci.io/v1.5/advanced/csidriver/#storage-profiles

  4. rajha-korithrien commented on May 6, 2025

    @rajha-korithrien
    Author

    Thanks for pointing me to "storageProfiles" I have added one, but the behavior remains unchanged:

    Here is my storageClass:

    kubectl get sc local-ssd -o yaml
    allowVolumeExpansion: true
    allowedTopologies:
    - matchLabelExpressions:
      - key: topology.topolvm.io/node
        values:
        - h6
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"allowVolumeExpansion":true,"allowedTopologies":[{"matchLabelExpressions":[{"key":"topology.topolvm.io/node","values":["h6"]}]}],"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{"meta.helm.sh/release-name":"topolvm","meta.helm.sh/release-namespace":"topolvm"},"creationTimestamp":"2025-05-05T23:20:13Z","labels":{"app.kubernetes.io/instance":"topolvm","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"topolvm","app.kubernetes.io/version":"0.36.4","helm.sh/chart":"topolvm-15.5.4"},"name":"local-ssd"},"parameters":{"topolvm.io/device-class":"local-ssd"},"provisioner":"topolvm.io","reclaimPolicy":"Delete","volumeBindingMode":"Immediate"}
        meta.helm.sh/release-name: topolvm
        meta.helm.sh/release-namespace: topolvm
      creationTimestamp: "2025-05-05T23:30:31Z"
      labels:
        app.kubernetes.io/instance: topolvm
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: topolvm
        app.kubernetes.io/version: 0.36.4
        helm.sh/chart: topolvm-15.5.4
      name: local-ssd
      resourceVersion: "6690188"
      uid: efdb4280-71ba-45c8-b953-145ccf43b9b8
    parameters:
      topolvm.io/device-class: local-ssd
    provisioner: topolvm.io
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    

    Here is my storageProfile:

    kubectl get storageprofile local-ssd -o yaml
    apiVersion: cdi.kubevirt.io/v1beta1
    kind: StorageProfile
    metadata:
      creationTimestamp: "2025-05-05T23:30:31Z"
      generation: 3
      labels:
        app: containerized-data-importer
        app.kubernetes.io/component: storage
        app.kubernetes.io/managed-by: cdi-controller
        cdi.kubevirt.io: ""
      name: local-ssd
      ownerReferences:
      - apiVersion: cdi.kubevirt.io/v1beta1
        blockOwnerDeletion: true
        controller: true
        kind: CDI
        name: cdi
        uid: 9b68cf66-bb4c-4e9d-aab3-a883efda89d3
      resourceVersion: "7707018"
      uid: e5003b05-9fc9-4fec-9252-f52d43959303
    spec:
      claimPropertySets:
      - accessModes:
        - ReadWriteOnce
        volumeMode: Block
      cloneStrategy: copy
      dataImportCronSourceFormat: pvc
      snapshotClass: local-ssd
    status:
      claimPropertySets:
      - accessModes:
        - ReadWriteOnce
        volumeMode: Block
      cloneStrategy: copy
      dataImportCronSourceFormat: pvc
      provisioner: topolvm.io
      snapshotClass: local-ssd
      storageClass: local-ssd
    

    And my associated volumesnapshotclass:

    kubectl  get volumesnapshotclass local-ssd -o yaml
    apiVersion: snapshot.storage.k8s.io/v1
    deletionPolicy: Delete
    driver: topolvm.io
    kind: VolumeSnapshotClass
    metadata:
      annotations:
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"snapshot.storage.k8s.io/v1","deletionPolicy":"Delete","driver":"topolvm.io","kind":"VolumeSnapshotClass","metadata":{"annotations":{},"labels":{"velero.io/csi-volumesnapshot-class":"true"},"name":"local-ssd"}}
      creationTimestamp: "2025-05-05T20:09:00Z"
      generation: 1
      labels:
        velero.io/csi-volumesnapshot-class: "true"
      name: local-ssd
      resourceVersion: "6521185"
      uid: 367a5a69-9c6d-47eb-9244-bcf682ce12df
    

    The PVC created by the UI still specifies RWX:

    kubectl -n topolvm-testing get pvc some-test -o yaml
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      annotations:
        volume.beta.kubernetes.io/storage-provisioner: topolvm.io
        volume.kubernetes.io/storage-provisioner: topolvm.io
      creationTimestamp: "2025-05-06T19:50:11Z"
      finalizers:
      - kubernetes.io/pvc-protection
      - topolvm.io/pvc
      - wrangler.cattle.io/persistentvolumeclaim-controller
      name: some-test
      namespace: topolvm-testing
      resourceVersion: "7707742"
      uid: 9ca21d8f-5c48-4b1a-bc5d-b4bda575178a
    spec:
      accessModes:
      - ReadWriteMany
      resources:
        requests:
          storage: 7Gi
      storageClassName: local-ssd
      volumeMode: Block
    status:
      phase: Pending
    

    Thanks for taking the time to help!

  5. rajha-korithrien commented on May 7, 2025

    @rajha-korithrien
    Author

    It seems that when a new StorageClass is made, a new StorageProfile is automatically made by the cdi-controller. This automatically generated StorageProfile has an empty spec. It also has a status that shows cloneStrategy as being copy. Interestingly the status of this automatically created StorageProfile has a claimPropertySets filled out with an accessModes entry of ReadWriteOnce:

    spec: {}
    status:
      claimPropertySets:
        - accessModes:
            - ReadWriteOnce
          volumeMode: Block
      cloneStrategy: copy
      dataImportCronSourceFormat: pvc
      provisioner: topolvm.io
      snapshotClass: local-ssd
      storageClass: local-ssd
    

    However, with this default StorageProfile for local-ssd all my resulting PVCs created have an accessMode of ReadWriteMany

    To try to fix this, I fill in the spec with:

    spec:
      claimPropertySets:
        - accessModes:
            - ReadWriteOnce
          volumeMode: Block
      cloneStrategy: snapshot
    

    And the StorageProfile changes its status to show a snapshot cloneStrategy but the PVCs created after changing this spec are still ReadWriteMany

  6. rajha-korithrien commented on May 7, 2025

    @rajha-korithrien
    Author

    The documentation for StorageProfiles suggests that they rely on a mutating web hook: https://github.com/kubevirt/containerized-data-importer/blob/main/doc/pvc-mutating-webhook-rendering.md

    This web hook is only triggered when the PVC has the appropriate label applied:

    labels:
        cdi.kubevirt.io/applyStorageProfile: "true"
    

    When using the Harvester UI to create a new empty volume, this label is not applied. So I have doubts a StorageProfile can solve this issue of the UI creating ReadWriteMany PVCs by default.

  7. rajha-korithrien commented on May 23, 2025

    @rajha-korithrien
    Author

    I have been unable to make this work, is there another hint for how StorageProfiles are supposed to work?
    Additionally what is the mechanism for doing this in Harvester 1.4.2 (or 1.4.3 when it comes out)?

  8. WebberHuang1118 commented on May 26, 2025

    @WebberHuang1118
    Member

    Hi @rajha-korithrien

    When creating a volume from the Harvester GUI, you can change the PVC access mode by selecting Edit YAML.

  9. rajha-korithrien commented on May 27, 2025

    @rajha-korithrien
    Author

    @WebberHuang1118 Thanks for the hint, I agree that I can manually edit the yaml in the UI to create the volume, but this is basically equivalent to using kubectl to make the volume. I think that it should be possible to create RWO volumes without resorting to editing the YAML. Other relevant properties of a PVC are exposed via the UI.

  10. WebberHuang1118 commented on May 28, 2025

    @WebberHuang1118
    Member

    @rajha-korithrien I agree — this seems more like a GUI issue related to third-party storage integration. cc @innobead @Vicente-Cheng @a110605

  11. 18 remaining items

  12. rajha-korithrien commented on Nov 3, 2025

    @rajha-korithrien
    Author

    I have just confirmed this issue still exists in Harvester 1.6.1. When creating a volume manually using the UI, there is no means via the UI to set the accessMode, and it defaults to ReadWriteMany which is not certain to be correct depending on the StorageClass being selected by the user.

    Image

    To demonstrate the issue:
    a) have a StorageClass that doesn't support RWX mode
    b) then in the UI select Volumes on the left
    c) press the Create button in the upper right
    d) fill in the StorageClass from (a) and select the size
    e) press the Create button in the lower left

    The volume will never create because it has by default been populated with ReadWriteMany.
    It is true that you can use the Edit as YAML feature to create the volume, but given that Volume Mode exists in the UI, why can't Access Mode also exist in the UI?

  13. rajha-korithrien commented on Nov 3, 2025

    @rajha-korithrien
    Author

    @ihcsim I have followed your suggestion of adding annotations to my TopoLVM StorageClass:

    allowVolumeExpansion: true
    allowedTopologies:
      - matchLabelExpressions:
          - key: topology.topolvm.io/node
            values:
              - ds9-1
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      annotations:
        cdi.harvesterhci.io/storageProfileVolumeModeAccessModes: '{"Block":["ReadWriteOnce"]}'
        meta.helm.sh/release-name: topolvm
        meta.helm.sh/release-namespace: topolvm
      creationTimestamp: '2025-07-14T19:49:27Z'
      labels:
        app.kubernetes.io/instance: topolvm
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: topolvm
        app.kubernetes.io/version: 0.36.5
        helm.sh/chart: topolvm-15.5.6
      ...
      name: ds9-1-worker-local-nvme1
      resourceVersion: '324828772'
      uid: 7a6ea07a-bd41-4fc6-9269-175a93a72990
    parameters:
      topolvm.io/device-class: worker-local-nvme1
    provisioner: topolvm.io
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    

    Adding cdi.harvesterhci.io/storageProfileVolumeModeAccessModes: '{"Block":["ReadWriteOnce"]}' to the StorageClass doesn't allow a Volume to be created via the UI as I outlined in my previous post.

  14. ihcsim commented on Nov 3, 2025

    @ihcsim
    Contributor

    Did you check the 'Storage Classes' page under the 'Advanced' section?

  15. rajha-korithrien commented on Nov 3, 2025

    @rajha-korithrien
    Author

    @ihcsim Thanks very much for getting back with me. To answer your question

    Below is what I have for my StorageClass in question ds9-1-worder-local-nvme1 using the Advanced --> Storage Classes --> MyStorageClass --> CDI Settings section of the UI as you suggest:

    Image

    It indicates that ReadWriteOnce is checked under Access Modes. But when I attempt to use the UI to create a volume, it never provisions:

    Image

    The error given by the provisioner (in the case TopoLVM) is:

    failed to provision volume with StorageClass "ds9-1-worker-local-nvme1": rpc error: code = InvalidArgument desc = unsupported access mode: MULTI_NODE_MULTI_WRITER
    

    and indeed the yaml that defines the volume claim shows accessModes with only ReadWriteMany:

    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      annotations:
        volume.beta.kubernetes.io/storage-provisioner: topolvm.io
        volume.kubernetes.io/storage-provisioner: topolvm.io
      creationTimestamp: '2025-11-03T15:13:05Z'
      finalizers:
        - kubernetes.io/pvc-protection
        - topolvm.io/pvc
        - wrangler.cattle.io/persistentvolumeclaim-controller
    ...
      name: test-again
      namespace: default
      resourceVersion: '324829198'
      uid: f6132466-1f6d-4423-bd8c-75fedd5f3b33
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 10Gi
      storageClassName: ds9-1-worker-local-nvme1
      volumeMode: Block
    
  16. ihcsim commented on Nov 3, 2025

    @ihcsim
    Contributor

    Strange.. what does the storage class yaml look like?

  17. rajha-korithrien commented on Nov 3, 2025

    @rajha-korithrien
    Author

    Here is my StorageClass yaml (unedited, so it has the managedFields section, my previous post cut that out):

    allowVolumeExpansion: true
    allowedTopologies:
      - matchLabelExpressions:
          - key: topology.topolvm.io/node
            values:
              - ds9-1
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      annotations:
        cdi.harvesterhci.io/storageProfileVolumeModeAccessModes: '{"Block":["ReadWriteOnce"]}'
        meta.helm.sh/release-name: topolvm
        meta.helm.sh/release-namespace: topolvm
      creationTimestamp: '2025-07-14T19:49:27Z'
      labels:
        app.kubernetes.io/instance: topolvm
        app.kubernetes.io/managed-by: Helm
        app.kubernetes.io/name: topolvm
        app.kubernetes.io/version: 0.36.5
        helm.sh/chart: topolvm-15.5.6
      managedFields:
        - apiVersion: storage.k8s.io/v1
          fieldsType: FieldsV1
          fieldsV1:
            f:allowVolumeExpansion: {}
            f:metadata:
              f:annotations:
                .: {}
                f:meta.helm.sh/release-name: {}
                f:meta.helm.sh/release-namespace: {}
              f:labels:
                .: {}
                f:app.kubernetes.io/instance: {}
                f:app.kubernetes.io/managed-by: {}
                f:app.kubernetes.io/name: {}
                f:app.kubernetes.io/version: {}
                f:helm.sh/chart: {}
            f:parameters:
              .: {}
              f:topolvm.io/device-class: {}
            f:provisioner: {}
            f:reclaimPolicy: {}
            f:volumeBindingMode: {}
          manager: helm
          operation: Update
          time: '2025-07-14T19:49:27Z'
        - apiVersion: storage.k8s.io/v1
          fieldsType: FieldsV1
          fieldsV1:
            f:allowedTopologies: {}
          manager: kubectl-edit
          operation: Update
          time: '2025-07-14T22:44:32Z'
        - apiVersion: storage.k8s.io/v1
          fieldsType: FieldsV1
          fieldsV1:
            f:metadata:
              f:annotations:
                f:cdi.harvesterhci.io/storageProfileVolumeModeAccessModes: {}
          manager: harvester
          operation: Update
          time: '2025-11-03T15:12:35Z'
      name: ds9-1-worker-local-nvme1
      resourceVersion: '324828772'
      uid: 7a6ea07a-bd41-4fc6-9269-175a93a72990
    parameters:
      topolvm.io/device-class: worker-local-nvme1
    provisioner: topolvm.io
    reclaimPolicy: Delete
    volumeBindingMode: Immediate
    
  18. ihcsim commented on Nov 3, 2025

    @ihcsim
    Contributor

    In 1.6, the CDI annotations don't work with non-root volumes; only with virtual machine images and root disk volumes. See #8199 (comment) where I described the relationship between PVC, storage profile and data volume. If you create a virtual machine image using your storage class, spin up a virtual machine with that image, then you will see the root disk PVC uses the access modes defined by the CDI annotations.

    For 1.6 non-root volumes, you will have to use the CDI PVC rendering by enabling the feature through the CDI custom resource and pre-label your PVCs

  19. changed the title [-][BUG] Support 3rd party RWO CSI drivers to be able to create volumes from the UI[/-] [+][BUG] Support 3rd party RWO CSI drivers to be able to create non-root volumes from the UI[/+] on Nov 3, 2025
  20. rajha-korithrien commented on Nov 3, 2025

    @rajha-korithrien
    Author

    Thanks for the help! I believe my point still stands, it is currently impossible to create RWO volumes (well, non-root volumes) from the UI without hand editing the yaml. I either have to:

    a) Edit the yaml and change ReadWriteMany to ReadWriteOnce - which is a straight forward work around, but the UI already allows setting volumeMode so why not accessMode?

    or

    b) Edit the yaml to put the label cdi.kubevirt.io/applyStorageProfile: "true" - which is also straight forward but more difficult to setup because one must enable the CDI WebhookPvcRendering feature gate

    So certainly there are workarounds, but isn't a better user experience obtained by adding the capability to the UI such that no yaml editing is required?

  21. ihcsim commented on Nov 4, 2025

    @ihcsim
    Contributor

    @rajha-korithrien See #9412 for enhancement request. Also, I am not sure why access modes aren't configurable from the volume page. I suspect it has something to do with the pre-3rd party storage support assumption that all PVCs have to be RWX for the virtual machines to be live-migratable, due to KubeVirt constraint.

  22. rajha-korithrien commented on Nov 4, 2025

    @rajha-korithrien
    Author

    @ihcsim Thanks for the enhancement request!
    I agree that the hard coded RWX assumption is a left over from when Longhorn was the only supported storage mechanism, and Longhorn is designed to be RWX, which fit nicely into how VMs are migrated.

    What is the protocol for this ticket, do I close it in favor of following the feature request?

    Thanks again!

  23. ihcsim commented on Nov 4, 2025

    @ihcsim
    Contributor

    @rajha-korithrien Sounds good - let's do the follow-up in #9412.

  24. moved this from In Progress to Closed in Community Issue Reviewon Nov 4, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugIssues that are defects reported by users or that we know have reached a real releaserequire/reproduceRequire adding a `reproduce` label.require/severityRequire adding a `severity` label.

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions