Repository navigation
Description
Describe the Bug
3rd party CSI provisioners are not always RWX some are RWO.
This is taken care of correctly when using lvm.driver.harvesterhci.io as the provisioner, but when using any other local CSI compliant provisioner (i.e. topolvm, openebs local-pv, etc.) the PVCs created are always set to RWX.
There needs to be some mechanism put in place for when using 3rd party CSI drivers to indicate to Harvester to create PVC as RWO.
To Reproduce
We provide an example using topolvm, but this problem will exist for any non RWX capable CSI provisioner.
- In you local context, add the helm repos needed to install your chosen 3rd party csi driver (we show topolvm):
helm repo add jetstack https://charts.jetstack.io
helm repo add topolvm https://topolvm.github.io/topolvm
- Deploy Harvester 1.5
- Ensure there is an unused local storage location available to the bare metal host (i.e. not being used by Harvester/Longhorn). In our case this is at /dev/sdb.
- create a local volume group by using pvcreate, vgcreate, lvcreate as root on the harvester node
h2:~ # pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created.
h2:~ # vgcreate topolvm-test /dev/sdb
Volume group "topolvm-test" successfully created
h2:~ # lvcreate --size 238G --type thin-pool --thinpool thinpool topolvm-test
Thin pool volume with chunk size 128.00 KiB can address at most 31.75 TiB of data.
Logical volume "thinpool" created.
h2:~ #
- Choose a 3rd party CSI driver that only supports RWO volumes. In this example we show using topolvm- Install cert-manage to you can install topolvm (into the base Harvester cluster)
helm -n cert-manager --create-namespace install cert-manager --set crds.enabled=true jetstack/cert-manager
- For topolvm first create an answers file with the following content):
lvmd:
deviceClasses:
- name: local-ssd
volume-group: topolvm-test
default: false
spare-gb: 1
type: thin
thin-pool:
name: thinpool
overprovision-ratio: 1.0
storageClasses:
- name: local-ssd
storageClass:
reclaimPolicy: Delete
annotations: {}
isDefaultClass: false
volumeBindingMode: Immediate
allowVolumeExpansion: true
additionalParameters:
"topolvm.io/device-class": "local-ssd"
mountOptions: []
- Install topolvm
helm -n topolvm --create-namespace install -f topolvm-answers.yaml topolvm topolvm/topolvm
- Use the Harvester UI to create a Volume (notice that we now have a "local-ssd" storageClass available)
- Ensure that you choose the "local-ssd" Storage Class
- Notice that you can specify a Volume Mode (as Block) and a Size, but you can not specify an
accessMode
- Observe that the PVC is never bound to a PV because the chosen provisioner returns an error:
rpc error: code = InvalidArgument desc = unsupported access mode: MULTI_NODE_MULTI_WRITER
Expected Behavior
It is expected that the UI should allow the user to choose an accessMode that is supported by the underlying 3rd party CSI provisioner. Or perhaps annotations must be placed on storageClass to indicate the supported accessMode.
This is necessary to allow full support of any 3rd CSI compliant provisioner.
Support Bundle for Troubleshooting
Environment
- Harvester version: 1.5
- Impacted VM: N/A the issues is about volumes, doesn't matter if they are created along with a VM or not
- Impacted volume (PV): pvc-c8befe7e-0e4c-4199-a703-56c1145fbb36 from PVC p-4pffp
- Underlying Infrastructure (e.g., Baremetal with Dell PowerEdge R630): baremetal
- Rancher version: 2.11.1
Additional context
The example provided uses TopoLVM, but any 3rd party CSI driver that only supports RWO will encounter this problem. We have also tested with OpenEBS local-pv
Workaround and Mitigation
It is possible to manually create a PVC yourself for example create a file with contents:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
volume.beta.kubernetes.io/storage-provisioner: topolvm.io
volume.kubernetes.io/storage-provisioner: topolvm.io
finalizers:
- kubernetes.io/pvc-protection
- topolvm.io/pvc
- wrangler.cattle.io/persistentvolumeclaim-controller
name: topolvm-test-vm-disk-1-cyc2m
namespace: topolvm-testing
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 10Gi
storageClassName: local-ssd
volumeMode: Block
Then apply it to the cluster:
kubectl -n topolvm-testing apply -f topolvm-pvc.yaml
At this point the volume can be attached to VMs
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsClosed
Activity
rajha-korithrien commentedon May 5, 2025
Tried to add a support bundle but was told
failed to upload supportbundle_c686f2fc-eb5a-4c3a-9d1e-1bb5778d4ad5_2025-05-05T21-27-58Z.zipibrokethecloud commentedon May 5, 2025
you should be able to edit the storageprofile for your storage class to reflect the correct access method
https://docs.harvesterhci.io/v1.5/advanced/csidriver/#storage-profiles
rajha-korithrien commentedon May 6, 2025
Thanks for pointing me to "storageProfiles" I have added one, but the behavior remains unchanged:
Here is my
storageClass:Here is my
storageProfile:And my associated
volumesnapshotclass:The PVC created by the UI still specifies
RWX:Thanks for taking the time to help!
rajha-korithrien commentedon May 7, 2025
It seems that when a new
StorageClassis made, a newStorageProfileis automatically made by thecdi-controller. This automatically generatedStorageProfilehas an empty spec. It also has astatusthat showscloneStrategyas beingcopy. Interestingly thestatusof this automatically createdStorageProfilehas aclaimPropertySetsfilled out with anaccessModesentry ofReadWriteOnce:However, with this default StorageProfile for
local-ssdall my resulting PVCs created have anaccessModeofReadWriteManyTo try to fix this, I fill in the spec with:
And the StorageProfile changes its
statusto show a snapshotcloneStrategybut the PVCs created after changing thisspecare stillReadWriteManyrajha-korithrien commentedon May 7, 2025
The documentation for StorageProfiles suggests that they rely on a mutating web hook: https://github.com/kubevirt/containerized-data-importer/blob/main/doc/pvc-mutating-webhook-rendering.md
This web hook is only triggered when the PVC has the appropriate label applied:
When using the Harvester UI to create a new empty volume, this label is not applied. So I have doubts a StorageProfile can solve this issue of the UI creating ReadWriteMany PVCs by default.
rajha-korithrien commentedon May 23, 2025
I have been unable to make this work, is there another hint for how StorageProfiles are supposed to work?
Additionally what is the mechanism for doing this in Harvester 1.4.2 (or 1.4.3 when it comes out)?
WebberHuang1118 commentedon May 26, 2025
Hi @rajha-korithrien
When creating a volume from the Harvester GUI, you can change the PVC access mode by selecting
Edit YAML.rajha-korithrien commentedon May 27, 2025
@WebberHuang1118 Thanks for the hint, I agree that I can manually edit the yaml in the UI to create the volume, but this is basically equivalent to using kubectl to make the volume. I think that it should be possible to create
RWOvolumes without resorting to editing the YAML. Other relevant properties of a PVC are exposed via the UI.WebberHuang1118 commentedon May 28, 2025
@rajha-korithrien I agree — this seems more like a GUI issue related to third-party storage integration. cc @innobead @Vicente-Cheng @a110605
18 remaining items
rajha-korithrien commentedon Nov 3, 2025
I have just confirmed this issue still exists in Harvester 1.6.1. When creating a volume manually using the UI, there is no means via the UI to set the
accessMode, and it defaults toReadWriteManywhich is not certain to be correct depending on the StorageClass being selected by the user.To demonstrate the issue:
a) have a StorageClass that doesn't support RWX mode
b) then in the UI select Volumes on the left
c) press the Create button in the upper right
d) fill in the StorageClass from (a) and select the size
e) press the Create button in the lower left
The volume will never create because it has by default been populated with
ReadWriteMany.It is true that you can use the
Edit as YAMLfeature to create the volume, but given thatVolume Modeexists in the UI, why can'tAccess Modealso exist in the UI?rajha-korithrien commentedon Nov 3, 2025
@ihcsim I have followed your suggestion of adding annotations to my TopoLVM StorageClass:
Adding
cdi.harvesterhci.io/storageProfileVolumeModeAccessModes: '{"Block":["ReadWriteOnce"]}'to the StorageClass doesn't allow a Volume to be created via the UI as I outlined in my previous post.ihcsim commentedon Nov 3, 2025
Did you check the 'Storage Classes' page under the 'Advanced' section?
rajha-korithrien commentedon Nov 3, 2025
@ihcsim Thanks very much for getting back with me. To answer your question
Below is what I have for my StorageClass in question
ds9-1-worder-local-nvme1using the Advanced --> Storage Classes --> MyStorageClass --> CDI Settings section of the UI as you suggest:It indicates that
ReadWriteOnceis checked under Access Modes. But when I attempt to use the UI to create a volume, it never provisions:The error given by the provisioner (in the case TopoLVM) is:
and indeed the yaml that defines the volume claim shows
accessModeswith onlyReadWriteMany:ihcsim commentedon Nov 3, 2025
Strange.. what does the storage class yaml look like?
rajha-korithrien commentedon Nov 3, 2025
Here is my StorageClass yaml (unedited, so it has the managedFields section, my previous post cut that out):
ihcsim commentedon Nov 3, 2025
In 1.6, the CDI annotations don't work with non-root volumes; only with virtual machine images and root disk volumes. See #8199 (comment) where I described the relationship between PVC, storage profile and data volume. If you create a virtual machine image using your storage class, spin up a virtual machine with that image, then you will see the root disk PVC uses the access modes defined by the CDI annotations.
For 1.6 non-root volumes, you will have to use the CDI PVC rendering by enabling the feature through the
CDIcustom resource and pre-label your PVCs[-][BUG] Support 3rd party RWO CSI drivers to be able to create volumes from the UI[/-][+][BUG] Support 3rd party RWO CSI drivers to be able to create non-root volumes from the UI[/+]rajha-korithrien commentedon Nov 3, 2025
Thanks for the help! I believe my point still stands, it is currently impossible to create RWO volumes (well, non-root volumes) from the UI without hand editing the yaml. I either have to:
a) Edit the yaml and change
ReadWriteManytoReadWriteOnce- which is a straight forward work around, but the UI already allows settingvolumeModeso why notaccessMode?or
b) Edit the yaml to put the label
cdi.kubevirt.io/applyStorageProfile: "true"- which is also straight forward but more difficult to setup because one must enable the CDIWebhookPvcRenderingfeature gateSo certainly there are workarounds, but isn't a better user experience obtained by adding the capability to the UI such that no yaml editing is required?
ihcsim commentedon Nov 4, 2025
@rajha-korithrien See #9412 for enhancement request. Also, I am not sure why access modes aren't configurable from the volume page. I suspect it has something to do with the pre-3rd party storage support assumption that all PVCs have to be RWX for the virtual machines to be live-migratable, due to KubeVirt constraint.
rajha-korithrien commentedon Nov 4, 2025
@ihcsim Thanks for the enhancement request!
I agree that the hard coded RWX assumption is a left over from when Longhorn was the only supported storage mechanism, and Longhorn is designed to be RWX, which fit nicely into how VMs are migrated.
What is the protocol for this ticket, do I close it in favor of following the feature request?
Thanks again!
ihcsim commentedon Nov 4, 2025
@rajha-korithrien Sounds good - let's do the follow-up in #9412.