Anti-affinity and allowRemoteVolumeAccess broken at some point

Operator v2.5.1 values:

---
installCRDs: true
imageConfigOverride:
- base: quay.io/piraeusdatastore
  components:
    linstor-satellite:
      image: piraeus-server
      tag: v1.26.1
    drbd-module-loader:
      image: drbd9-almalinux9
      tag: v9.2.10

Hello, I am a bit desperate to figure out what is going on, so shooting this here in case anyone has any insight.

We have been using pirearus operator back during v1, and have migrated to v2. Things have been going good till we noticed that the anti-affinity and allowRemoteVolumeAccess have seemed to have stopped working.

We have 4 external linstor nodes, not managed by piraeus operator. Things have been working great until this was discovered.

This is the storage class we have been using

---
apiVersion: storage.k8s.io/v1
kind: StorageClass
provisioner: linstor.csi.linbit.com
metadata:
  name: linstor-replica-one-local
volumeBindingMode: WaitForFirstConsumer 
allowVolumeExpansion: true
parameters:
  linstor.csi.linbit.com/placementCount: "1"
  linstor.csi.linbit.com/disklessStoragePool: "DfltDisklessStorPool"
  linstor.csi.linbit.com/disklessOnRemaining: "true"
  linstor.csi.linbit.com/allowRemoteVolumeAccess: |
    - fromSame:
      - xcp-ng/node

Now all PV's are going to xen13 no matter what k8s node the pod is running on which is using a PV.

┊ xcp-sr-linstor_group_thin_device ┊ ovbh-pprod-xen10                         ┊ LVM_THIN ┊ linstor_group/thin_device ┊     2.42 TiB ┊      3.49 TiB ┊ True         ┊ Ok    ┊ ovbh-pprod-xen10;xcp-sr-linstor_group_thin_device             ┊
┊ xcp-sr-linstor_group_thin_device ┊ ovbh-pprod-xen11                         ┊ LVM_THIN ┊ linstor_group/thin_device ┊     2.44 TiB ┊      3.49 TiB ┊ True         ┊ Ok    ┊ ovbh-pprod-xen11;xcp-sr-linstor_group_thin_device             ┊
┊ xcp-sr-linstor_group_thin_device ┊ ovbh-pprod-xen12                         ┊ LVM_THIN ┊ linstor_group/thin_device ┊     2.83 TiB ┊      3.49 TiB ┊ True         ┊ Ok    ┊ ovbh-pprod-xen12;xcp-sr-linstor_group_thin_device             ┊
┊ xcp-sr-linstor_group_thin_device ┊ ovbh-pprod-xen13                         ┊ LVM_THIN ┊ linstor_group/thin_device ┊     4.05 TiB ┊      4.99 TiB ┊ True         ┊ Ok    ┊ ovbh-pprod-xen13;xcp-sr-linstor_group_thin_device             ┊

When I try linstor.csi.linbit.com/allowRemoteVolumeAccess: 'false' I get the following in k8s.
0/7 nodes are available: 1 node(s) were unschedulable, 3 node(s) did not have enough free storage, 3 node(s) had untolerated taint {[node-role.kubernetes.io/control-plane](http://node-role.kubernetes.io/control-plane): }. preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling..

We obviously have not had any issues with our replica-three storage class.

I am not seeing any errors in piraeus operator pods yet. I have not found any issues with out linstor cluster yet.

linstor.csi.linbit.com/allowRemoteVolumeAccess: |
   - fromSame:
     - xcp-ng/node

I suspect the issue is that with the new operator this label is no longer automatically synced to LINSTOR (or at least, not to where LINSTOR CSI now expects).

You need to add a satellite configuration which synces the labels in the Aux/topology namespace:

spec:
  properties:
    - name: Aux/topology/xcp-ng/node
      valueFrom:
        nodeFieldRef: metadata.labels['xcp-ng/node']

Or is that property managed somehow externally? Then you need to prefix it manually with this Aux/topology/ key.

Sadly no.

The PV is on xen13, so it should only be allowed on worker04, but it started up on worker02.
Also, my memory may be wrong, but I remember the PV yaml stating somewhere what host it was on.

[12:38 ovbh-pprod-xen12 ~]# linstor v l | grep -e 'pvc-89ba17be-bca8-444f-becd-3ce819d5020f'
| ovbh-pprod-xen13                         | pvc-89ba17be-bca8-444f-becd-3ce819d5020f        | xcp-sr-linstor_group_thin_device |     0 |    1035 | /dev/drbd1035 |  430.18 MiB | Unused | UpToDate |
| ovbh-vtest-k8s02-worker02 | pvc-89ba17be-bca8-444f-becd-3ce819d5020f        | DfltDisklessStorPool             |     0 |    1035 | /dev/drbd1035 |             | Unused | Diskless |

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
    volume.kubernetes.io/selected-node: ovbh-vtest-k8s02-worker02
    volume.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
  creationTimestamp: '2024-06-25T16:37:48Z'
  finalizers:
    - kubernetes.io/pvc-protection
  name: test
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: linstor-replica-one-local
  volumeMode: Filesystem
  volumeName: pvc-89ba17be-bca8-444f-becd-3ce819d5020f
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 50Gi
  phase: Bound
---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
    volume.kubernetes.io/provisioner-deletion-secret-name: ''
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ''
  creationTimestamp: '2024-06-25T16:37:52Z'
  finalizers:
    - kubernetes.io/pv-protection
    - external-attacher/linstor-csi-linbit-com
  name: pvc-89ba17be-bca8-444f-becd-3ce819d5020f
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 50Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: test
    namespace: default
    resourceVersion: '771983962'
    uid: 89ba17be-bca8-444f-becd-3ce819d5020f
  csi:
    driver: linstor.csi.linbit.com
    fsType: ext4
    volumeAttributes:
      linstor.csi.linbit.com/mount-options: ''
      linstor.csi.linbit.com/post-mount-xfs-opts: ''
      linstor.csi.linbit.com/remote-access-policy: |
        - fromSame:
            - xcp-ng/node
      linstor.csi.linbit.com/uses-volume-context: 'true'
      storage.kubernetes.io/csiProvisionerIdentity: 1719263239344-2902-linstor.csi.linbit.com
    volumeHandle: pvc-89ba17be-bca8-444f-becd-3ce819d5020f
  persistentVolumeReclaimPolicy: Delete
  storageClassName: linstor-replica-one-local
  volumeMode: Filesystem
status:
  phase: Bound

Actually that got me progress. I added the key with topology to the hosts

[12:54 ovbh-pprod-xen12 ~]# linstor n list-properties ovbh-pprod-xen12
╭─────────────────────────────────────────────╮
┊ Key                      ┊ Value            ┊
╞═════════════════════════════════════════════╡
┊ Aux/topology/xcp-ng/node ┊ ovbh-pprod-xen12 ┊
┊ Aux/xcp-ng/node          ┊ ovbh-pprod-xen12 ┊
┊ CurStltConnName          ┊ default          ┊
┊ NodeUname                ┊ ovbh-pprod-xen12 ┊
┊ PrefNic                  ┊ stornet          ┊
╰─────────────────────────────────────────────╯

PV that gets deployed to xen13, only allows pods from worker 4 now, nice!

I was also able to move a pv from xen13 to xen12, and update the pvc to volume.kubernetes.io/selected-node: ovbh-vtest-k8s02-worker03 and it started up on worker03.

I am having problem with creation of new pv when worker04 is cordoned. I am seeing the host assignment in the PV again, which is nice.

0/7 nodes are available: 1 node(s) were unschedulable, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 3 node(s) had volume node affinity conflict. preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling.

[13:18 ovbh-pprod-xen12 ~]# linstor v l | grep -e 'pvc-c3fafcd5-5b33-44bf-8706-2318a5cd9062'
| ovbh-pprod-xen13                         | pvc-c3fafcd5-5b33-44bf-8706-2318a5cd9062        | xcp-sr-linstor_group_thin_device |     0 |    1035 | /dev/drbd1035 |  430.18 MiB | Unused | UpToDate |

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
    volume.kubernetes.io/selected-node: ovbh-vtest-k8s02-worker02
    volume.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
  creationTimestamp: '2024-06-25T17:17:09Z'
  finalizers:
    - kubernetes.io/pvc-protection
  name: test
  namespace: default
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: linstor-replica-one-local
  volumeMode: Filesystem
  volumeName: pvc-c3fafcd5-5b33-44bf-8706-2318a5cd9062
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 50Gi
  phase: Bound

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
    volume.kubernetes.io/provisioner-deletion-secret-name: ''
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ''
  creationTimestamp: '2024-06-25T17:17:13Z'
  name: pvc-c3fafcd5-5b33-44bf-8706-2318a5cd9062
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 50Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: test
    namespace: default
    resourceVersion: '772016778'
    uid: c3fafcd5-5b33-44bf-8706-2318a5cd9062
  csi:
    driver: linstor.csi.linbit.com
    fsType: ext4
    volumeAttributes:
      linstor.csi.linbit.com/mount-options: ''
      linstor.csi.linbit.com/post-mount-xfs-opts: ''
      linstor.csi.linbit.com/remote-access-policy: |
        - fromSame:
            - xcp-ng/node
      linstor.csi.linbit.com/uses-volume-context: 'true'
      storage.kubernetes.io/csiProvisionerIdentity: 1719334484030-2173-linstor.csi.linbit.com
    volumeHandle: pvc-c3fafcd5-5b33-44bf-8706-2318a5cd9062
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: xcp-ng/node
              operator: In
              values:
                - ovbh-pprod-xen13
  persistentVolumeReclaimPolicy: Delete
  storageClassName: linstor-replica-one-local
  volumeMode: Filesystem
status:
  phase: Bound

This is the storage class, so no idea why it is not waiting for the pod to start somewhere before creating the pv.

allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: linstor-replica-one-local
parameters:
  linstor.csi.linbit.com/allowRemoteVolumeAccess: |
    - fromSame:
      - xcp-ng/node
  linstor.csi.linbit.com/disklessOnRemaining: 'true'
  linstor.csi.linbit.com/disklessStoragePool: DfltDisklessStorPool
  linstor.csi.linbit.com/placementCount: '1'
provisioner: linstor.csi.linbit.com
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

Deleted the PV and redid test, caught this message before it went away
Running PreBind plugin "VolumeBinding": binding volumes: pv "pvc-8db6569b-8b5a-487d-8708-28f30fe5977b" node affinity doesn't match node "ovbh-vtest-k8s02-worker02": no matching NodeSelectorTerms

got all the relevant events
pvc-5aab1e01-6ded-4cfa-8101-ab7d613971d8.17dc4e167f164d6e.yaml.txt
test.17dc4f7a159814d0.yaml.txt
test.17dc4f7a155db34b.yaml.txt
test.17dc4f7b0c5f4667.yaml.txt
test-f98b6b876-ckz4f.17dc4f7796f73df6.yaml.txt
test-f98b6b876-ckz4f.17dc4ed51d712b91.yaml.txt
test.17dc4f7796c735a8.yaml.txt
test-f98b6b876-ckz4f.17dc4f7b3ed14fab.yaml.txt

@WanzenBug

Ok I figured out that part of the problem.

linstor controller set-property Autoplacer/Weights/MaxFreeSpace 0.5
linstor controller set-property Autoplacer/Weights/MinRscCount 1.0

Added the above due to us updating to 1.26.1, which means this needs to be considered: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#_automatically_maintaining_resource_group_placement_count

our xen13 has much more space, so that explains what was going on.

Thanks for the help @WanzenBug!

Welp, celebrated too soon. It can now deploy a pv to xen12 in addition to xen13, but for this test redis I have setup with a pv on xen13 and xen12 already, it is not working right.

0/7 nodes are available: 1 node(s) didn't match pod anti-affinity rules, 1 node(s) were unschedulable, 2 node(s) had volume node affinity conflict, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/7 nodes are available: 1 No preemption victims found for incoming pod, 6 Preemption is not helpful for scheduling..

I am reading things here: https://linbit.com/drbd-user-guide/linstor-guide-1_0-en/#_automatically_maintaining_resource_group_placement_count, and I just want the old behavior back where it will allow the putting of pv's wherever they need to go.

linstor controller set-property BalanceResourcesEnabled "false" is not reverting.

@WanzenBug

I have a question @WanzenBug, to make sure I am not wrong about something.

How is linstor supposed to know what external linstor satellite is supposed to match with the satellite worker?

WaitForFirstConsumer should be what is doing that, at least I think. But now I am wondering if that is only in respect to the k8s worker, and no the linstor satellite where the data actually lives.

The thing is that external linstor cluster doesn't know which k8s worker is on which host, and they are all external. Would need to match a label on both sides but I don't think it does it?

How is linstor supposed to know what external linstor satellite is supposed to match with the satellite worker?

Since you used

  linstor.csi.linbit.com/allowRemoteVolumeAccess: |
    - fromSame:
      - xcp-ng/node

the CSI plugin wil look for all nodes with the same value for the Aux/topology/xcp-ng/node property. So all your "external" satellites need those as well. Unless I'm misunderstanding your setup?

The external satellites, xen10 xen11 xen12 xen13 all have those properties, like I showed above.

So at this point the piraeus operator and the satellites on k8s are now host aware. My question is, are the external satellites aware of what k8s satellite is running on it?

We have four hosts, with a k8s worker on each. The way we want replica-one pv's deployed: the pod needs to run on the k8s node, which corresponds to the host with the pv. So if the pv is on xen13, the pod needs to run on worker04. That is all working as expected.

What I am struggling with is the placement of the pv. Are the external satellites aware of what k8s satellite is running on it? Is xen13 aware that worker04 is the one that is running on it, and not worker01 worker02 and/or worker03?

When I have a new pv with WaitForFirstConsumer and the pod gets deployed to worker02, the pv should get deployed to xen11 as that is the host that worker02 is running on, but it is not.

I am wondering if it would be possible to create the opposite of the affinity controller. An anti-affinity controller.

Instead of updating the PVC to reflect where the PV lives (after migration), we want to move the PV to where it should be based on the PVC.

Monitoring for newly created PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
    volume.kubernetes.io/selected-node: ovbh-vtest-k8s02-worker01
    volume.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
  creationTimestamp: '2024-06-26T17:09:58Z'
  finalizers:
    - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/component: node
    app.kubernetes.io/instance: preprod-notifications-redis
    app.kubernetes.io/name: redis
  name: redis-data-preprod-notifications-redis-node-2
  namespace: notifications3
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 8Gi
  storageClassName: linstor-replica-one-local
  volumeMode: Filesystem
  volumeName: pvc-ef32daaa-1c1b-4645-a6d5-7c0184219aee
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 8Gi
  phase: Bound

It then checks the PV

apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
    volume.kubernetes.io/provisioner-deletion-secret-name: ''
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ''
  creationTimestamp: '2024-06-26T17:10:03Z'
  finalizers:
    - kubernetes.io/pv-protection
  name: pvc-ef32daaa-1c1b-4645-a6d5-7c0184219aee
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 8Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: redis-data-preprod-notifications-redis-node-2
    namespace: notifications3
    resourceVersion: '773082123'
    uid: ef32daaa-1c1b-4645-a6d5-7c0184219aee
  csi:
    driver: linstor.csi.linbit.com
    fsType: ext4
    volumeAttributes:
      linstor.csi.linbit.com/mount-options: ''
      linstor.csi.linbit.com/post-mount-xfs-opts: ''
      linstor.csi.linbit.com/remote-access-policy: |
        - fromSame:
            - xcp-ng/node
      linstor.csi.linbit.com/uses-volume-context: 'true'
      storage.kubernetes.io/csiProvisionerIdentity: 1719334484030-2173-linstor.csi.linbit.com
    volumeHandle: pvc-ef32daaa-1c1b-4645-a6d5-7c0184219aee
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: xcp-ng/node
              operator: In
              values:
                - ovbh-pprod-xen12
  persistentVolumeReclaimPolicy: Delete
  storageClassName: linstor-replica-one-local
  volumeMode: Filesystem
status:
  phase: Bound

And seeing that it is on the wrong host (xen12) it moves it to the host that corresponds with volume.kubernetes.io/selected-node: ovbh-vtest-k8s02-worker01 using linstor commands.

linstor r c ovbh-pprod-xen10 pvc-ef32daaa-1c1b-4645-a6d5-7c0184219aee
command for check status to ensure it is fully up to date
linstor r d ovbh-pprod-xen13 pvc-ef32daaa-1c1b-4645-a6d5-7c0184219aee

Ok so, it is looking like the functionality we were used to is just broken/gone. I do not understand what happened.

In the mean time I wrote a script to automate/just do this. Obviously I can't write an anti-affinity controller, but I can bodge it and get moving again. I have a lot going on and I wanted to stop going back and forth between rancher/kubectl outputs and linstor outputs.

Putting here in case someone has similar setup and they find this useful.

#!/bin/bash

# Ensure the script is being run in an interactive shell
if [[ ! -t 0 ]]; then
    echo "Restarting script in interactive shell..."
    exec /bin/bash -i "$0" "$@"
fi

# Function to process a single PV
process_pv() {
    local PV_NAME=$1
    local QUIET=$2
    local DEBUG=$3
    local MIGRATE=$4

    # Get the PV info from the Kubernetes cluster
    PV_INFO=$(kubectl get pv $PV_NAME -o json)

    # Debug logging
    if [ "$DEBUG" == true ]; then
        echo "Debug: PV_INFO for $PV_NAME:"
        echo "$PV_INFO"
    fi

    # Check if the command succeeded
    if [ $? -ne 0 ]; then
        echo "Failed to get PV info for $PV_NAME. Please check if the PV name is correct."
        if [ "$MIGRATE" == true ]; then
            echo "Exiting..."
            exit 1
        else
            return
        fi
    fi

    # Check if the PV is using the correct storage class
    STORAGE_CLASS=$(echo $PV_INFO | jq -r '.spec.storageClassName // empty')
    if [ "$STORAGE_CLASS" != "linstor-replica-one-local" ]; then
        if [ "$DEBUG" == true ]; then
            echo "Debug: Skipping PV $PV_NAME with storage class $STORAGE_CLASS"
        fi
        return
    fi

    # Check if the PV is managed by LINSTOR
    PROVISIONER=$(echo $PV_INFO | jq -r '.spec.csi.driver // empty')
    if [ "$PROVISIONER" != "linstor.csi.linbit.com" ]; then
        if [ "$DEBUG" == true ]; then
            echo "Debug: Skipping non-LINSTOR PV $PV_NAME"
        fi
        return
    fi

    # Extract the PVC name and namespace from the PV info
    PVC_NAME=$(echo $PV_INFO | jq -r '.spec.claimRef.name // empty')
    PVC_NAMESPACE=$(echo $PV_INFO | jq -r '.spec.claimRef.namespace // empty')

    # Debug logging
    if [ "$DEBUG" == true ]; then
        echo "Debug: PVC_NAME: $PVC_NAME, PVC_NAMESPACE: $PVC_NAMESPACE"
    fi

    # Extract the node name from the PV info
    NODE_NAME=$(echo $PV_INFO | jq -r '.spec.nodeAffinity.required.nodeSelectorTerms[]?.matchExpressions[]? | select(.key=="xcp-ng/node") | .values[] // empty')

    # Debug logging
    if [ "$DEBUG" == true ]; then
        echo "Debug: NODE_NAME from PV: $NODE_NAME"
    fi

    # Get the PVC info from the Kubernetes cluster
    PVC_INFO=$(kubectl get pvc $PVC_NAME -n $PVC_NAMESPACE -o json)

    # Debug logging
    if [ "$DEBUG" == true ]; then
        echo "Debug: PVC_INFO for $PVC_NAME:"
        echo "$PVC_INFO"
    fi

    # Check if the command succeeded
    if [ $? -ne 0 ]; then
        echo "Failed to get PVC info for $PVC_NAME in namespace $PVC_NAMESPACE."
        if [ "$MIGRATE" == true ]; then
            echo "Exiting..."
            exit 1
        else
            return
        fi
    fi

    # Extract the selected node from the PVC annotations
    SELECTED_NODE=$(echo $PVC_INFO | jq -r '.metadata.annotations["volume.kubernetes.io/selected-node"] // empty')

    # Debug logging
    if [ "$DEBUG" == true ]; then
        echo "Debug: SELECTED_NODE from PVC: $SELECTED_NODE"
    fi

    # Check if nodeAffinity or selected-node is missing
    if [ -z "$NODE_NAME" ] || [ -z "$SELECTED_NODE" ]; then
        echo "ERROR: PV $PV_NAME is missing nodeAffinity or volume.kubernetes.io/selected-node!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
        if [ "$MIGRATE" == true ]; then
            echo "Exiting..."
            exit 1
        else
            return
        fi
    fi

    # Find the Xen host corresponding to the selected Kubernetes worker node
    if [ -n "$SELECTED_NODE" ]; then
        XEN_HOST=$(kubectl get node $SELECTED_NODE -o json | jq -r '.metadata.labels["xcp-ng/node"] // empty')
    else
        XEN_HOST=""
    fi

    # Debug logging
    if [ "$DEBUG" == true ]; then
        echo "Debug: XEN_HOST corresponding to selected node: $XEN_HOST"
    fi

    # Output the details only if the PV needs to be moved or if not in quiet mode
    if [ "$XEN_HOST" != "$NODE_NAME" ]; then
        echo "------------------------------------------------------------"
        echo "Selected Node from PVC: $SELECTED_NODE"
        echo "Xen Host corresponding to selected node: $XEN_HOST"
        echo "PVC Name: $PVC_NAME"
        echo "PVC Namespace: $PVC_NAMESPACE"
        echo "ERROR: PV $PV_NAME is on the wrong host!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1"
        echo "To move the PV to the correct host, run the following commands:"
        echo "linstor r c $XEN_HOST $PV_NAME"
        echo "linstor r d $NODE_NAME $PV_NAME"

        if [ "$MIGRATE" == true ]; then
            echo "Migration flag is set. Prompting for confirmation..."
            read -p "Do you want to proceed with moving the PV $PV_NAME? (y/n): " choice
            if [[ "$choice" != "y" ]]; then
                echo "Skipping PV $PV_NAME"
                echo "------------------------------------------------------------"
                return
            fi

            # Run the command to create the resource on the new host
            echo "Running: linstor r c $XEN_HOST $PV_NAME"
            linstor r c $XEN_HOST $PV_NAME
            if [ $? -eq 0 ]; then
                echo "Successfully created resource $PV_NAME on $XEN_HOST"
            else
                echo "Failed to create resource $PV_NAME on $XEN_HOST"
                echo "Exiting..."
                exit 1
            fi
            echo "Waiting for 30 seconds..."
            sleep 30

            # Wait until the resource on the new host is UpToDate
            echo "Waiting for the resource $PV_NAME on $XEN_HOST to be UpToDate..."
            while :; do
                STATUS=$(linstor r l | grep -e "$PV_NAME" | grep -e "$XEN_HOST")
                if echo "$STATUS" | grep -q "UpToDate"; then
                    echo "Resource $PV_NAME on $XEN_HOST is UpToDate."
                    echo "$STATUS"
                    break
                else
                    echo "Current status: $STATUS"
                fi
                sleep 5
            done

            # Run the command to delete the resource from the old host
            echo "Running: linstor r d $NODE_NAME $PV_NAME"
            linstor r d $NODE_NAME $PV_NAME
            if [ $? -eq 0 ]; then
                echo "Successfully deleted resource $PV_NAME from $NODE_NAME"
            else
                echo "Failed to delete resource $PV_NAME from $NODE_NAME"
                echo "Exiting..."
                exit 1
            fi

            # Find the pod using the PVC
            POD_NAME=$(kubectl get pod -n $PVC_NAMESPACE -o json | jq -r ".items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName == \"$PVC_NAME\") | .metadata.name")

            if [ -n "$POD_NAME" ]; then
                echo "Found pod $POD_NAME using PVC $PVC_NAME."

                # Check if the pod is already running with all status good
                POD_STATUS=$(kubectl get pod $POD_NAME -n $PVC_NAMESPACE -o json | jq -r '.status.conditions[] | select(.type=="Ready").status')
                if [ "$POD_STATUS" == "True" ]; then
                    echo "Pod $POD_NAME is already running and in good status. No need to restart."
                else
                    # Delete the pod
                    echo "Deleting pod $POD_NAME..."
                    kubectl delete pod $POD_NAME -n $PVC_NAMESPACE
                    if [ $? -eq 0 ]; then
                        echo "Successfully deleted pod $POD_NAME."
                    else
                        echo "Failed to delete pod $POD_NAME."
                        echo "Exiting..."
                        exit 1
                    fi

                    # Wait for the new pod to come up
                    echo "Waiting for a new pod to start using PVC $PVC_NAME..."
                    while :; do
                        NEW_POD_NAME=$(kubectl get pod -n $PVC_NAMESPACE -o json | jq -r ".items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName == \"$PVC_NAME\") | .metadata.name")
                        if [ -n "$NEW_POD_NAME" ]; then
                            echo "New pod $NEW_POD_NAME is starting."
                            break
                        else
                            echo "No new pod found yet. Waiting..."
                        fi
                        sleep 5
                    done

                    # Wait for the new pod to be on the correct node and fully running
                    echo "Waiting for the new pod $NEW_POD_NAME to start on the correct node and be fully running..."
                    while :; do
                        NEW_NODE=$(kubectl get pod $NEW_POD_NAME -n $PVC_NAMESPACE -o jsonpath='{.spec.nodeName}')
                        POD_STATUS=$(kubectl get pod $NEW_POD_NAME -n $PVC_NAMESPACE -o json | jq -r '.status.conditions[] | select(.type=="Ready").status')
                        if [ "$NEW_NODE" == "$SELECTED_NODE" ] && [ "$POD_STATUS" == "True" ]; then
                            echo "New pod $NEW_POD_NAME is running on the correct node $NEW_NODE and is fully up and running."
                            break
                        else
                            echo "New pod $NEW_POD_NAME is not fully running yet. Current node: $NEW_NODE. Waiting..."
                        fi
                        sleep 5
                    done

                    # If the new pod is not on the correct node, exit with an error
                    if [ "$NEW_NODE" != "$SELECTED_NODE" ]; then
                        echo "ERROR: New pod $NEW_POD_NAME is running on the wrong node $NEW_NODE. Exiting... !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
                        exit 1
                    fi
                fi
            else
                echo "No pod found using PVC $PVC_NAME."
            fi
        fi

    elif [ "$QUIET" != true ]; then
        echo "------------------------------------------------------------"
        echo "Selected Node from PVC: $SELECTED_NODE"
        echo "Xen Host corresponding to selected node: $XEN_HOST"
        echo "PVC Name: $PVC_NAME"
        echo "PVC Namespace: $PVC_NAMESPACE"
        echo "PV $PV_NAME is on the correct host."
    fi

    # Final check to ensure pod is running on the correct node
    POD_NAME=$(kubectl get pod -n $PVC_NAMESPACE -o json | jq -r ".items[] | select(.spec.volumes[]?.persistentVolumeClaim.claimName == \"$PVC_NAME\") | .metadata.name")
    if [ -n "$POD_NAME" ]; then
        POD_NODE=$(kubectl get pod $POD_NAME -n $PVC_NAMESPACE -o jsonpath='{.spec.nodeName}')
        if [ "$POD_NODE" != "$SELECTED_NODE" ]; then
            echo "ERROR: Pod $POD_NAME is running on the wrong node!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!"
            echo "Pod is running on node: $POD_NODE"
            echo "It should be running on node: $SELECTED_NODE"
            echo "PVC Name: $PVC_NAME"
            echo "PVC Namespace: $PVC_NAMESPACE"
            echo "------------------------------------------------------------"
            if [ "$MIGRATE" == true ]; then
                exit 1
            fi
        elif [ "$QUIET" != true ]; then
            echo "Pod $POD_NAME is running on the right node, $POD_NODE."
            echo "------------------------------------------------------------"
        fi
    fi
}

# Initialize optional parameters
QUIET=false
DEBUG=false
MIGRATE=false

# Function to show usage
usage() {
    echo "Usage: $0 <path_to_kubeconfig> [pv_name] [--quiet] [--debug] [--migrate] [-h|--help]"
    echo "Options:"
    echo "  --quiet    : Suppress output for PVs that are on the correct host"
    echo "  --debug    : Enable debug logging"
    echo "  --migrate  : Enable migration of PVs to the correct host with manual approval"
    echo "  -h, --help : Show this help message"
    exit 1
}

# Parse command line arguments
while [[ "$#" -gt 0 ]]; do
    case $1 in
        --quiet) QUIET=true ;;
        --debug) DEBUG=true ;;
        --migrate) MIGRATE=true ;;
        -h|--help) usage ;;
        *) 
            if [ -z "$KUBECONFIG_PATH" ]; then
                KUBECONFIG_PATH=$1
                export KUBECONFIG=$KUBECONFIG_PATH
            elif [ -z "$PV_NAME" ]; then
                PV_NAME=$1
            else
                usage
            fi ;;
    esac
    shift
done

if [ -z "$KUBECONFIG_PATH" ]; then
    usage
fi

# Print the note about storage class
echo "Note: This script processes only PVs with the storage class 'linstor-replica-one-local'."

# Export the kubeconfig path to the environment variable
export KUBECONFIG=$KUBECONFIG_PATH

if [[ -z "$PV_NAME" ]]; then
    # If no PV name is provided, process all PVs
    echo "No PV name provided. Processing all PVs..."
    ALL_PVS=$(kubectl get pv -o json | jq -r '.items[] | select(.spec.storageClassName == "linstor-replica-one-local") | .metadata.name')
    for PV in $ALL_PVS; do
        process_pv $PV $QUIET $DEBUG $MIGRATE
    done
else
    process_pv $PV_NAME $QUIET $DEBUG $MIGRATE
fi

Usage:

jonathon@jonathon-framework:~$ ./pv-magic.sh k8s_configs/new-preprod.yaml --migrate --quiet
Note: This script processes only PVs with the storage class 'linstor-replica-one-local'.
No PV name provided. Processing all PVs...
------------------------------------------------------------
Selected Node from PVC: ovbh-vtest-k8s02-worker01
Xen Host corresponding to selected node: ovbh-pprod-xen10
PVC Name: redis-data-preprod-socket-redis-node-1
PVC Namespace: socket3
ERROR: PV pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 is on the wrong host!
To move the PV to the correct host, run the following commands:
linstor r c ovbh-pprod-xen10 pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348
linstor r d ovbh-pprod-xen13 pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348
Migration flag is set. Prompting for confirmation...
Do you want to proceed with moving the PV pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348? (y/n): y
Running: linstor r c ovbh-pprod-xen10 pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348
INFO:
    Resource-definition property 'DrbdOptions/Resource/quorum' updated from 'off' to 'majority' by auto-quorum
INFO:
    Resource-definition property 'DrbdOptions/Resource/on-no-quorum' updated from 'off' to 'suspend-io' by auto-quorum
SUCCESS:
Description:
    New resource 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' on node 'ovbh-pprod-xen10' registered.
Details:
    Resource 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' on node 'ovbh-pprod-xen10' UUID is: 965817c5-2e8c-43f5-bfa6-8b72f587007a
SUCCESS:
Description:
    Volume with number '0' on resource 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' on node 'ovbh-pprod-xen10' successfully registered
Details:
    Volume UUID is: 55fdf929-3b01-4fc3-9eae-2dd1806f6bab
SUCCESS:
    Added peer(s) 'ovbh-pprod-xen10' to resource 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' on 'ovbh-vtest-k8s02-worker04'
SUCCESS:
    Added peer(s) 'ovbh-pprod-xen10' to resource 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' on 'ovbh-pprod-xen13'
SUCCESS:
    Created resource 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' on 'ovbh-pprod-xen10'
SUCCESS:
Description:
    Resource 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' on 'ovbh-pprod-xen10' ready
Details:
    Node(s): 'ovbh-pprod-xen10', Resource: 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348'
Successfully created resource pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 on ovbh-pprod-xen10
Waiting for the resource pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 on ovbh-pprod-xen10 to be UpToDate...
Current status: | pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348        | ovbh-pprod-xen10                         |      | Unused | Ok    | Inconsistent | 2024-06-26 14:42:09 |
Resource pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 on ovbh-pprod-xen10 is UpToDate.
| pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348        | ovbh-pprod-xen10                         |      | Unused | Ok    | UpToDate | 2024-06-26 14:42:09 |
Running: linstor r d ovbh-pprod-xen13 pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348
SUCCESS:
Description:
    Node: ovbh-pprod-xen13, Resource: pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 preparing for deletion.
Details:
    Node: ovbh-pprod-xen13, Resource: pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 UUID is: 88113cfd-2291-45e7-905c-fe96f6050036
SUCCESS:
    Preparing deletion of resource on 'ovbh-pprod-xen13'
SUCCESS:
    Preparing deletion of resource on 'ovbh-vtest-k8s02-worker04'
SUCCESS:
    Preparing deletion of resource on 'ovbh-pprod-xen10'
INFO:
    Resource-definition property 'DrbdOptions/Resource/quorum' was removed as there are not enough resources for quorum
INFO:
    Resource-definition property 'DrbdOptions/Resource/on-no-quorum' was removed as there are not enough resources for quorum
SUCCESS:
Description:
    Node: ovbh-pprod-xen13, Resource: pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 marked for deletion.
Details:
    Node: ovbh-pprod-xen13, Resource: pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 UUID is: 88113cfd-2291-45e7-905c-fe96f6050036
SUCCESS:
    Notified 'ovbh-vtest-k8s02-worker04' that 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' is being cleaned up on Node(s): [ovbh-pprod-xen13]
SUCCESS:
    Cleaning up 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' on 'ovbh-pprod-xen13'
SUCCESS:
    Notified 'ovbh-pprod-xen10' that 'pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348' is being cleaned up on Node(s): [ovbh-pprod-xen13]
SUCCESS:
Description:
    Node: ovbh-pprod-xen13, Resource: pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 deletion complete.
Details:
    Node: ovbh-pprod-xen13, Resource: pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 UUID was: 88113cfd-2291-45e7-905c-fe96f6050036
Successfully deleted resource pvc-5c855616-da22-48ac-97e8-7ddbd1bdd348 from ovbh-pprod-xen13
Found pod preprod-socket-redis-node-1 using PVC redis-data-preprod-socket-redis-node-1.
Deleting pod preprod-socket-redis-node-1...
pod "preprod-socket-redis-node-1" deleted
Successfully deleted pod preprod-socket-redis-node-1.
Waiting for a new pod to start using PVC redis-data-preprod-socket-redis-node-1...
New pod preprod-socket-redis-node-1 is starting.
Waiting for the new pod preprod-socket-redis-node-1 to start on the correct node...
New pod preprod-socket-redis-node-1 is running on the correct node ovbh-vtest-k8s02-worker01.
------------------------------------------------------------

I think there is a bug in the LINSTOR CSI driver:

time="2024-06-27T08:34:48Z" level=trace msg="try placement on preferred nodes" linstorCSIComponent=client nodes="[k8s-23.test]" scheduler=autoplacetopology segments="map[kubernetes.io/hostname:k8s-23.test linbit.com/hostname:k8s-23.test topology.kubernetes.io/region:ci topology.kubernetes.io/zone:zone-b xcp-ng/node:node2]" volume=pvc-82deef89-7bec-41ff-8af2-bdc3c3f5045b
time="2024-06-27T08:34:48Z" level=debug msg="curl -X 'POST' -d '{\"select_filter\":{\"place_count\":1,\"additional_place_count\":1,\"node_name_list\":[\"k8s-23.test\"]}}\n' -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'User-Agent: linstor-csi/v1.6.0-6b556f4b54ddf410bd29d9d7b237389b2507d6b2' 'http://linstor-controller:3370/v1/resource-definitions/pvc-82deef89-7bec-41ff-8af2-bdc3c3f5045b/autoplace'"
time="2024-06-27T08:34:48Z" level=trace msg="failed to autoplace" error="Message: 'Not enough available nodes'; Details: 'Not enough nodes fulfilling the following auto-place criteria:\n * the current access context has enough privileges to use the node and the storage pool\n * the node is online\n\nAuto-place configuration details:\n  Place Count: 1\n  Don't place with resource (List): [pvc-82deef89-7bec-41ff-8af2-bdc3c3f5045b]\n  Node name: [k8s-23.test]\n  Layer stack: [DRBD, STORAGE]\n\nAuto-placing resource: pvc-82deef89-7bec-41ff-8af2-bdc3c3f5045b'" linstorCSIComponent=client scheduler=autoplacetopology segments="map[kubernetes.io/hostname:k8s-23.test linbit.com/hostname:k8s-23.test topology.kubernetes.io/region:ci topology.kubernetes.io/zone:zone-b xcp-ng/node:node2]" volume=pvc-82deef89-7bec-41ff-8af2-bdc3c3f5045b
time="2024-06-27T08:34:48Z" level=trace msg="exact segments match failed, try with remote access policy" linstorCSIComponent=client scheduler=autoplacetopology segments="map[kubernetes.io/hostname:k8s-23.test linbit.com/hostname:k8s-23.test topology.kubernetes.io/region:ci topology.kubernetes.io/zone:zone-b xcp-ng/node:node2]" volume=pvc-82deef89-7bec-41ff-8af2-bdc3c3f5045b
time="2024-06-27T08:34:48Z" level=trace msg="got accessible segments" accessibleSegments="[map[xcp-ng/node:node2]]" linstorCSIComponent=client scheduler=autoplacetopology segments="map[kubernetes.io/hostname:k8s-23.test linbit.com/hostname:k8s-23.test topology.kubernetes.io/region:ci topology.kubernetes.io/zone:zone-b xcp-ng/node:node2]" volume=pvc-82deef89-7bec-41ff-8af2-bdc3c3f5045b
time="2024-06-27T08:34:48Z" level=debug msg="curl -X 'POST' -d '{\"select_filter\":{\"place_count\":1,\"additional_place_count\":1,\"node_name_list\":[\"k8s-20.test\",\"k8s-21.test\",\"k8s-22.test\",\"k8s-23.test\"]}}\n' -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'User-Agent: linstor-csi/v1.6.0-6b556f4b54ddf410bd29d9d7b237389b2507d6b2' 'http://linstor-controller:3370/v1/resource-definitions/pvc-82deef89-7bec-41ff-8af2-bdc3c3f5045b/autoplace'"

I tried to recreate this scenario. k8s-22.test and k8s-23.test are labelled with xcp-ng/node=node2. While the first attempt, placing it on k8s-23.test is expected to fail, as it is diskless, it should make an attempt to place on k8s-22.test and k8s-23.test in the second round, but instead it seems to select all nodes.

It looks to be related to the new node cache in LINSTOR CSI 1.6.0. You can try downgrading to 1.5.0:

imageConfigOverride:
- base: quay.io/piraeusdatastore
  components:
    linstor-csi:
      tag: v1.5.0
      image: piraeus-csi

quay.io/piraeusdatastore/piraeus-csi:v1.5.0

Unfortunately no :/

0/7 nodes are available: 2 node(s) had volume node affinity conflict, 2 node(s) were unschedulable, 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }. preemption: 0/7 nodes are available: 7 Preemption is not helpful for scheduling..

pv:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: linstor.csi.linbit.com
    volume.kubernetes.io/provisioner-deletion-secret-name: ''
    volume.kubernetes.io/provisioner-deletion-secret-namespace: ''
  creationTimestamp: '2024-06-27T15:37:59Z'
  finalizers:
    - kubernetes.io/pv-protection
  name: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa
spec:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 50Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: redisfailover-persistent-data-rfr-redis-2
    namespace: redis2
    resourceVersion: '774076232'
    uid: d3ae228e-2935-4aca-a915-c93eb0d080fa
  csi:
    driver: linstor.csi.linbit.com
    fsType: ext4
    volumeAttributes:
      linstor.csi.linbit.com/mount-options: ''
      linstor.csi.linbit.com/post-mount-xfs-opts: ''
      linstor.csi.linbit.com/remote-access-policy: |
        - fromSame:
            - xcp-ng/node
      linstor.csi.linbit.com/uses-volume-context: 'true'
      storage.kubernetes.io/csiProvisionerIdentity: 1719502561181-812-linstor.csi.linbit.com
    volumeHandle: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: xcp-ng/node
              operator: In
              values:
                - ovbh-pprod-xen12
  persistentVolumeReclaimPolicy: Delete
  storageClassName: linstor-replica-one-local
  volumeMode: Filesystem
status:
  phase: Bound

pvc:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
    volume.kubernetes.io/selected-node: ovbh-vtest-k8s02-worker01
    volume.kubernetes.io/storage-provisioner: linstor.csi.linbit.com
  creationTimestamp: '2024-06-27T15:37:54Z'
  finalizers:
    - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/component: redis
    app.kubernetes.io/name: redis
    app.kubernetes.io/part-of: redis-failover
  name: redisfailover-persistent-data-rfr-redis-2
  namespace: redis2
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Gi
  storageClassName: linstor-replica-one-local
  volumeMode: Filesystem
  volumeName: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa
status:
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 50Gi
  phase: Bound

Ran script to move it to correct location

jonathon@jonathon-framework:~$ ./pv-magic.sh k8s_configs/new-preprod.yaml pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa      --migrate --quiet
Note: This script processes only PVs with the storage class 'linstor-replica-one-local'.
------------------------------------------------------------
Selected Node from PVC: ovbh-vtest-k8s02-worker01
Xen Host corresponding to selected node: ovbh-pprod-xen10
PVC Name: redisfailover-persistent-data-rfr-redis-2
PVC Namespace: redis2
ERROR: PV pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa is on the wrong host!
To move the PV to the correct host, run the following commands:
linstor r c ovbh-pprod-xen10 pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa
linstor r d ovbh-pprod-xen12 pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa
Migration flag is set. Prompting for confirmation...
Do you want to proceed with moving the PV pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa? (y/n): y
Running: linstor r c ovbh-pprod-xen10 pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa
SUCCESS:
    Successfully set property key(s): StorPoolName
INFO:
    Tie breaker resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' created on DfltDisklessStorPool
INFO:
    Resource-definition property 'DrbdOptions/Resource/quorum' updated from 'off' to 'majority' by auto-quorum
INFO:
    Resource-definition property 'DrbdOptions/Resource/on-no-quorum' updated from 'off' to 'suspend-io' by auto-quorum
SUCCESS:
Description:
    New resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on node 'ovbh-pprod-xen10' registered.
Details:
    Resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on node 'ovbh-pprod-xen10' UUID is: 5fee4bff-95dd-415f-a3ce-ef895349f090
SUCCESS:
Description:
    Volume with number '0' on resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on node 'ovbh-pprod-xen10' successfully registered
Details:
    Volume UUID is: 3c6b2f54-c941-4354-b923-2db85119618f
SUCCESS:
    Added peer(s) 'ovbh-pprod-xen10' to resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen11'
SUCCESS:
    Added peer(s) 'ovbh-pprod-xen10' to resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen12'
SUCCESS:
    Created resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen10'
SUCCESS:
Description:
    Resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen10' ready
Details:
    Node(s): 'ovbh-pprod-xen10', Resource: 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa'
SUCCESS:
    Created resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen11'
SUCCESS:
    Added peer(s) 'ovbh-pprod-xen11' to resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen12'
SUCCESS:
    Added peer(s) 'ovbh-pprod-xen11' to resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen10'
SUCCESS:
Description:
    Resource 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen11' ready
Details:
    Node(s): 'ovbh-pprod-xen10', Resource: 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa'
Successfully created resource pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa on ovbh-pprod-xen10
Waiting for the resource pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa on ovbh-pprod-xen10 to be UpToDate...
Resource pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa on ovbh-pprod-xen10 is UpToDate.
| pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa        | ovbh-pprod-xen10                         |      | Unused | Ok    |   UpToDate | 2024-06-27 08:43:55 |
Running: linstor r d ovbh-pprod-xen12 pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa
INFO:
    Tie breaker marked for deletion
INFO:
    Resource-definition property 'DrbdOptions/Resource/quorum' was removed as there are not enough resources for quorum
INFO:
    Resource-definition property 'DrbdOptions/Resource/on-no-quorum' was removed as there are not enough resources for quorum
SUCCESS:
Description:
    Node: ovbh-pprod-xen12, Resource: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa preparing for deletion.
Details:
    Node: ovbh-pprod-xen12, Resource: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa UUID is: 856083ba-6f83-4661-ae4f-4cb3be9f1a48
SUCCESS:
    Preparing deletion of resource on 'ovbh-pprod-xen11'
SUCCESS:
    Preparing deletion of resource on 'ovbh-pprod-xen12'
SUCCESS:
    Preparing deletion of resource on 'ovbh-pprod-xen10'
INFO:
    Tie breaker marked for deletion
SUCCESS:
Description:
    Node: ovbh-pprod-xen12, Resource: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa marked for deletion.
Details:
    Node: ovbh-pprod-xen12, Resource: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa UUID is: 856083ba-6f83-4661-ae4f-4cb3be9f1a48
SUCCESS:
    Notified 'ovbh-pprod-xen11' that 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' is being cleaned up on Node(s): [ovbh-pprod-xen12]
SUCCESS:
    Notified 'ovbh-pprod-xen10' that 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' is being cleaned up on Node(s): [ovbh-pprod-xen12]
SUCCESS:
    Cleaning up 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen12'
SUCCESS:
Description:
    Node: ovbh-pprod-xen12, Resource: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa deletion complete.
Details:
    Node: ovbh-pprod-xen12, Resource: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa UUID was: 856083ba-6f83-4661-ae4f-4cb3be9f1a48
SUCCESS:
    Cleaning up 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' on 'ovbh-pprod-xen11'
SUCCESS:
    Notified 'ovbh-pprod-xen10' that 'pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa' is being cleaned up on Node(s): [ovbh-pprod-xen11]
SUCCESS:
Description:
    Node: ovbh-pprod-xen11, Resource: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa deletion complete.
Details:
    Node: ovbh-pprod-xen11, Resource: pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa UUID was: ac04fb56-1fb4-444d-82fa-cb16b6941bd4
Successfully deleted resource pvc-d3ae228e-2935-4aca-a915-c93eb0d080fa from ovbh-pprod-xen12
Found pod rfr-redis-2 using PVC redisfailover-persistent-data-rfr-redis-2.
Deleting pod rfr-redis-2...
pod "rfr-redis-2" deleted
Successfully deleted pod rfr-redis-2.
Waiting for a new pod to start using PVC redisfailover-persistent-data-rfr-redis-2...
New pod rfr-redis-2 is starting.
Waiting for the new pod rfr-redis-2 to start on the correct node and be fully running...
New pod rfr-redis-2 is not fully running yet. Current node: . Waiting...
New pod rfr-redis-2 is not fully running yet. Current node: ovbh-vtest-k8s02-worker01. Waiting...
New pod rfr-redis-2 is not fully running yet. Current node: ovbh-vtest-k8s02-worker01. Waiting...
New pod rfr-redis-2 is not fully running yet. Current node: ovbh-vtest-k8s02-worker01. Waiting...
New pod rfr-redis-2 is not fully running yet. Current node: ovbh-vtest-k8s02-worker01. Waiting...
New pod rfr-redis-2 is not fully running yet. Current node: ovbh-vtest-k8s02-worker01. Waiting...
New pod rfr-redis-2 is not fully running yet. Current node: ovbh-vtest-k8s02-worker01. Waiting...
New pod rfr-redis-2 is running on the correct node ovbh-vtest-k8s02-worker01 and is fully up and running.
------------------------------------------------------------
jonathon@jonathon-framework:~$

If you are able to recreate the issue that is great, did not fully read what you said first lol. I may have borked my test, I have been swamped.

I ran my script against our full preprod and it worked great, so anyone can feel free to use it and build on it.

Looking forward to trying out the piraeus operator update that might have a fix for this, but we can get by for now.

Yeah, sorry, seems to have been broken for quite some time (basically with the introduction of caching in the CSI driver, which was in 1.2.3).

You can give the latest CSI version a try, which includes the necessary fixes for caching:

quay.io/piraeusdatastore/piraeus-csi:sha-97757d2

That works 😭
https://github.com/piraeusdatastore/linstor-csi/releases/tag/v1.6.3

Thought I was going crazy there for a bit.

Thanks for getting this resolved so quickly!

Navigation Menu

Repository navigation

Anti-affinity and allowRemoteVolumeAccess broken at some point #680

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Anti-affinity and allowRemoteVolumeAccess broken at some point #680

Description

Activity

WanzenBug commented on Jun 25, 2024

jonathon2nd commented on Jun 25, 2024

jonathon2nd commented on Jun 25, 2024

jonathon2nd commented on Jun 25, 2024

jonathon2nd commented on Jun 25, 2024

jonathon2nd commented on Jun 25, 2024

jonathon2nd commented on Jun 25, 2024

WanzenBug commented on Jun 26, 2024

jonathon2nd commented on Jun 26, 2024

jonathon2nd commented on Jun 26, 2024

jonathon2nd commented on Jun 27, 2024

WanzenBug commented on Jun 27, 2024

WanzenBug commented on Jun 27, 2024

jonathon2nd commented on Jun 27, 2024

jonathon2nd commented on Jun 27, 2024

jonathon2nd commented on Jun 27, 2024

WanzenBug commented on Jun 28, 2024

jonathon2nd commented on Jun 28, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions