Skip to content

CDI importer pod was not created. #3146

Closed

Description

@shusley244

What happened:

After executing the command kubectl create -f dv_fedora.yml as per the instructions on the KubeVirt labs page(https://kubevirt.io/labs/kubernetes/lab2.html), the importer pod was not created. I was unable to find any pod name associated with the import process.

What you expected to happen:

the importer pod was created.

Additional context:

kubectl get cdi cdi -n cdi

NAME   AGE     PHASE
cdi    5h41m   Deployed

kubectl get pvc fedora -oyaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    cdi.kubevirt.io/storage.contentType: kubevirt
    cdi.kubevirt.io/storage.pod.restarts: "0"
    cdi.kubevirt.io/storage.preallocation.requested: "false"
    cdi.kubevirt.io/storage.usePopulator: "true"
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
  creationTimestamp: "2024-03-20T06:14:56Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    alerts.k8s.io/KubePersistentVolumeFillingUp: disabled
    app: containerized-data-importer
    app.kubernetes.io/component: storage
    app.kubernetes.io/managed-by: cdi-controller
  name: fedora
  namespace: default
  ownerReferences:
  - apiVersion: cdi.kubevirt.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: DataVolume
    name: fedora
    uid: cba5ce85-7173-44d2-81f9-ab2372c731df
  resourceVersion: "266638693"
  uid: 60150494-21ed-446b-a540-75ef0240b5a4
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: "5681173672"
  storageClassName: nanhu-sc-db
  volumeMode: Filesystem
  volumeName: pvc-60150494-21ed-446b-a540-75ef0240b5a4
status:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: "5681173672"
  phase: Bound

kubectl get datavolume fedora -oyaml

apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
  annotations:
    cdi.kubevirt.io/storage.usePopulator: "true"
  creationTimestamp: "2024-03-20T06:14:56Z"
  generation: 1
  name: fedora
  namespace: default
  resourceVersion: "266638694"
  uid: cba5ce85-7173-44d2-81f9-ab2372c731df
spec:
  source:
    http:
      url: https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.raw.xz
  storage:
    resources:
      requests:
        storage: 5Gi
status:
  claimName: fedora
  conditions:
  - lastHeartbeatTime: "2024-03-20T06:15:00Z"
    lastTransitionTime: "2024-03-20T06:15:00Z"
    message: PVC fedora Bound
    reason: Bound
    status: "True"
    type: Bound
  - lastHeartbeatTime: "2024-03-20T06:15:00Z"
    lastTransitionTime: "2024-03-20T06:15:00Z"
    status: "False"
    type: Ready
  - lastHeartbeatTime: "2024-03-20T06:15:00Z"
    lastTransitionTime: "2024-03-20T06:15:00Z"
    status: "False"
    type: Running
  phase: PVCBound
  progress: N/A

kubectl logs cdi-deployment-5d4f984b5-kjlj8 -n cdi

{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"events","msg":"PVC fedora Pending","type":"Normal","object":{"kind":"DataVolume","namespace":"default","name":"fedora","uid":"cba5ce85-7173-44d2-81f9-ab2372c731df","apiVersion":"cdi.kubevirt.io/v1beta1","resourceVersion":"266638674"},"reason":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/fedora","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/fedora","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/fedora","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/fedora","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/fedora","isUpload":false,"isCloneTarget":false,"isBound":true,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/fedora","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"events","msg":"PVC fedora Bound","type":"Normal","object":{"kind":"DataVolume","namespace":"default","name":"fedora","uid":"cba5ce85-7173-44d2-81f9-ab2372c731df","apiVersion":"cdi.kubevirt.io/v1beta1","resourceVersion":"266638694"},"reason":"Bound"}

Environment:

  • KubeVirt version (use virtctl version): v1.2.0
  • Kubernetes version (use kubectl version): v1.22.10
  • VM or VMI specifications: N/A
  • Cloud provider or hardware configuration: N/A
  • OS (e.g. from /etc/os-release): Ubuntu 20.04.4 LTS (Focal Fossa)
  • Kernel (e.g. uname -a): Linux onestor 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x8 6_64 GNU/Linux
  • Install tools: N/A
  • Others: N/A

Activity

  1. awels commented on Mar 22, 2024

    @awels
    Member

    Your kubernetes version is very old, I believe populators which CDI uses are beta in 1.24 and enabled by default there. In 1.22 you probably have to enable a feature gate to get it to work.

  2. shusley244 commented on Mar 22, 2024

    @shusley244
    Author

    Thanks for your answer. I will upgrade the kubernetes version and try again.

  3. knkedlaya commented on Mar 26, 2024

    @knkedlaya

    Hi,

    Facing similar issue though my kubernetes(AWS EKS) setup is latest.

    virtctl version
    Client Version: version.Info{GitVersion:"1.2.0", GitCommit:"$Format:%H$", GitTreeState:"", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.22.1", Compiler:"gc", Platform:"darwin/arm64"}
    Server Version: version.Info{GitVersion:"v1.2.0", GitCommit:"f26e45d99ac35743fc33d6a121b629e9a9af6b63", GitTreeState:"clean", BuildDate:"2024-03-05T21:32:21Z", GoVersion:"go1.21.5 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}
    
    kubectl version
    Client Version: v1.29.0-eks-5e0fdde
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.29.1-eks-b9c9ed7
    

    pv.yaml

    kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: models-1-0-0
      labels:
        name: models-1-0-0
    spec:
      capacity:
        storage: 10Gi
      storageClassName: local-storage
      claimRef:
        namespace: default
        name: models-1-0-0-claim
      accessModes:
        - ReadWriteMany
      gcePersistentDisk:
        pdName: models-1-0-0
        fsType: ext4
        readOnly: true
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: models-1-0-0-claim
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 5Gi
      selector:
        matchLabels:
          name: models-1-0-0
    

    Trying to upload image with below command

    virtctl image-upload \
    --pvc-name=models-1-0-0-claim --no-create \
    --image-path=/image/path/image-name-disk1.qcow2 \
    --uploadproxy-url=https://127.0.0.1:18443 --insecure --force-bind
    

    which ends up with

    Flag --pvc-name has been deprecated, specify the name as the second argument instead.
    Using existing PVC default/models-1-0-0-claim
    Waiting for PVC models-1-0-0-claim upload pod to be ready...
    timed out waiting for the condition
    

    Output of

    kubectl logs -f -n cdi cdi-deployment-79b586b894-hxcnh
    ...
    ...
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.RouteReconcile","msg":"No route found, setting to blank","RouteURL":""}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.StorageClassReconcile","msg":"Setting scratch space to default","storageClass.Name":"gp2"}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller","msg":"reconciling CDIConfig","CDIConfig":"/config"}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.IngressReconcile","msg":"No ingress found, setting to blank","IngressURL":""}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.RouteReconcile","msg":"No route found, setting to blank","RouteURL":""}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.StorageClassReconcile","msg":"Setting scratch space to default","storageClass.Name":"gp2"}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/gp2"}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Set metric:gp2 complete:true default:true vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"debug","ts":"2024-03-26T04:38:38Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"info","ts":"2024-03-26T04:38:38Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"info","ts":"2024-03-26T08:32:13Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    
  4. awels commented on Mar 26, 2024

    @awels
    Member

    Hi,

    Facing similar issue though my kubernetes(AWS EKS) setup is latest.

    virtctl version
    Client Version: version.Info{GitVersion:"1.2.0", GitCommit:"$Format:%H$", GitTreeState:"", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.22.1", Compiler:"gc", Platform:"darwin/arm64"}
    Server Version: version.Info{GitVersion:"v1.2.0", GitCommit:"f26e45d99ac35743fc33d6a121b629e9a9af6b63", GitTreeState:"clean", BuildDate:"2024-03-05T21:32:21Z", GoVersion:"go1.21.5 X:nocoverageredesign", Compiler:"gc", Platform:"linux/amd64"}
    
    kubectl version
    Client Version: v1.29.0-eks-5e0fdde
    Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
    Server Version: v1.29.1-eks-b9c9ed7
    

    pv.yaml

    kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: models-1-0-0
      labels:
        name: models-1-0-0
    spec:
      capacity:
        storage: 10Gi
      storageClassName: local-storage
      claimRef:
        namespace: default
        name: models-1-0-0-claim
      accessModes:
        - ReadWriteMany
      gcePersistentDisk:
        pdName: models-1-0-0
        fsType: ext4
        readOnly: true
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: models-1-0-0-claim
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 5Gi
      selector:
        matchLabels:
          name: models-1-0-0
    

    Trying to upload image with below command

    virtctl image-upload \
    --pvc-name=models-1-0-0-claim --no-create \
    --image-path=/image/path/image-name-disk1.qcow2 \
    --uploadproxy-url=https://127.0.0.1:18443 --insecure --force-bind
    

    which ends up with

    Flag --pvc-name has been deprecated, specify the name as the second argument instead.
    Using existing PVC default/models-1-0-0-claim
    Waiting for PVC models-1-0-0-claim upload pod to be ready...
    timed out waiting for the condition
    

    Output of

    kubectl logs -f -n cdi cdi-deployment-79b586b894-hxcnh
    ...
    ...
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.RouteReconcile","msg":"No route found, setting to blank","RouteURL":""}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.StorageClassReconcile","msg":"Setting scratch space to default","storageClass.Name":"gp2"}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller","msg":"reconciling CDIConfig","CDIConfig":"/config"}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.IngressReconcile","msg":"No ingress found, setting to blank","IngressURL":""}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.RouteReconcile","msg":"No route found, setting to blank","RouteURL":""}
    {"level":"info","ts":"2024-03-26T01:40:38Z","logger":"controller.config-controller.CDIconfig.StorageClassReconcile","msg":"Setting scratch space to default","storageClass.Name":"gp2"}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/gp2"}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Set metric:gp2 complete:true default:true vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"info","ts":"2024-03-26T04:11:02Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"debug","ts":"2024-03-26T04:38:38Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"info","ts":"2024-03-26T04:38:38Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"info","ts":"2024-03-26T08:32:13Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T08:32:13Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    

    Looks like you are using the local-storage storage class AND RWX volumes. I am fairly certain local-storage doesn't support RWX, try RWO.

  5. knkedlaya commented on Mar 26, 2024

    @knkedlaya

    Thanks @awels . I changed local-storage to ReadWriteOnce and I see different logs

    This is new pv.yaml after changing to RWO

    kind: PersistentVolume
    apiVersion: v1
    metadata:
      name: models-1-0-0
      labels:
        name: models-1-0-0
    spec:
      capacity:
        storage: 10Gi
      storageClassName: local-storage
      claimRef:
        namespace: default
        name: models-1-0-0-claim
      accessModes:
        - ReadWriteOnce
      gcePersistentDisk:
        pdName: models-1-0-0
        fsType: ext4
        readOnly: true
    ---
    kind: PersistentVolumeClaim
    apiVersion: v1
    metadata:
      name: models-1-0-0-claim
    spec:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 5Gi
      selector:
        matchLabels:
          name: models-1-0-0
    
    
    {"level":"info","ts":"2024-03-26T13:17:39Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
    {"level":"info","ts":"2024-03-26T13:17:39Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
    {"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim","isUpload":true,"isCloneTarget":false,"isBound":true,"podSucceededFromPVC":false,"deletionTimeStamp set?":true}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
    {"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim-scratch","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim-scratch","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":true}
    {"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
    {"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:17:40Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"info","ts":"2024-03-26T13:17:40Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"reconciling StorageProfile","StorageProfile":"/local-storage"}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Updating StorageProfile","StorageProfile":"local-storage","StorageProfile.Name":"local-storage","storageProfile":{"apiVersion":"cdi.kubevirt.io/v1beta1","kind":"StorageProfile","name":"local-storage"}}
    {"level":"info","ts":"2024-03-26T13:17:58Z","logger":"controller.storageprofile-controller","msg":"Set metric:local-storage complete:false default:false vdefault:false rwx:false smartclone:false (deleted 1)"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim","isUpload":false,"isCloneTarget":false,"isBound":true,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:17:58Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"info","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Updating PVC from pod","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Phase is now","pvc.anno.Phase":""}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Updated PVC","PVC":"default/models-1-0-0-claim","pvc.anno.AnnImportPod":"cdi-upload-models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"info","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
    I0326 13:18:15.553328       1 util.go:2109] Applying PVC annotationNamecdi-upload-models-1-0-0-claimsidecar.istio.io/injectfalse
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"upload pod created\n","Namespace":"default","Name":"cdi-upload-models-1-0-0-claim","Image name":"quay.io/kubevirt/cdi-uploadserver:v1.58.3"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim-scratch","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim-scratch","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"upload service created\n","Namespace":"default","Name":"cdi-upload-models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Phase is now","pvc.anno.Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"info","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Phase is now","pvc.anno.Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"info","ts":"2024-03-26T13:18:15Z","logger":"controller.upload-controller","msg":"Calling Upload reconcile PVC","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:15Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim-scratch","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim-scratch","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/models-1-0-0-claim-scratch","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"k8s.io/CloneRequest"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/models-1-0-0-claim-scratch"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/models-1-0-0-claim-scratch","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","annotation":"cdi.kubevirt.io/storage.import.source"}
    {"level":"debug","ts":"2024-03-26T13:18:17Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/models-1-0-0-claim-scratch","Phase":"Pending"}
    
  6. awels commented on Mar 26, 2024

    @awels
    Member

    And the upload pod is still not starting? Do you have any pods in the namespace you are trying to upload to?

  7. changed the title [-]CDI impoter pod was not created.[/-] [+]CDI importer pod was not created.[/+] on Mar 26, 2024
  8. awels commented on Mar 26, 2024

    @awels
    Member

    Also I see no mention of the scratch space pvc. Does that get created? It also seems you are creating the PVC ahead of time, you don't really need to do that, if you specify the datavolume name, CDI should take care of creating all the required resources.

  9. knkedlaya commented on Mar 26, 2024

    @knkedlaya

    You are right, I have a pod running. Never came to my mind that the pod is related to the image I was trying to upload. I will delete this pod, PV, PVC and retry with dv create option while uploading.
    I think upload creates scratch pvc because I havent created it explicitly.

    kubectl get po
    NAME                            READY   STATUS    RESTARTS   AGE
    cdi-upload-models-1-0-0-claim   0/1     Pending   0          25m
    
  10. awels commented on Mar 26, 2024

    @awels
    Member

    it doesn't look running, it looks pending, if it remains pending after retrying, do a kubectl describe pod <name> it should tell you why it is pending.

  11. knkedlaya commented on Mar 26, 2024

    @knkedlaya

    These are the events I am seeing when I describe the pod

    Events:
      Type     Reason            Age                  From               Message
      ----     ------            ----                 ----               -------
      Warning  FailedScheduling  33m                  default-scheduler  0/2 nodes are available: persistentvolumeclaim "models-1-0-0-claim-scratch" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.
      Warning  FailedScheduling  2m58s (x3 over 23m)  default-scheduler  running PreBind plugin "VolumeBinding": binding volumes: context deadline exceeded
    
  12. awels commented on Mar 26, 2024

    @awels
    Member

    Is the scratch PVC created, and is it bound? The events tell me it can't mount the scratch space.

  13. knkedlaya commented on Mar 26, 2024

    @knkedlaya

    yes, scratch pvc is created. When I describe I see following. Hinting at some permission related issue.

    status code: 403, request id: e58dde0e-03e9-4167-9b16-dd18cfe80ca2
      Warning  ProvisioningFailed  18m (x6 over 38m)  ebs.csi.aws.com_ebs-csi-controller-7db7cf79d5-5sq47_d48c1c09-e066-4697-aa55-c1064e3b5068  (combined from similar events): failed to provision volume with StorageClass "gp2": rpc error: code = Internal desc = Could not create volume "pvc-855bd68a-f81c-4e06-9406-e8dfb914459f": could not create volume in EC2: UnauthorizedOperation: You are not authorized to perform this operation. User: arn:aws:sts::<Accound-ID-masked>:assumed-role/eksctl-kubevirt-cluster-1-nodegrou-NodeInstanceRole-2dTUBsU8mLgv/i-023afa1afc95aedde is not authorized to perform: ec2:CreateVolume on resource: arn:aws:ec2:ap-south-1:<Accound-ID-masked>:volume/* because no identity-based policy allows the ec2:CreateVolume action. Encoded authorization failure message: 3PwEe0JdTGHWwrlf1c6ClLMyjwtHtCIpFNlKRq_uinU1OrKB4itwGSUh5eSS786CVBAhBK1tm38KGkKunGL1HM9sA5shKk-LcDm2nKI0xZMUmbuscSdYgwBBvdxpC-xpJpqUdKQ8WUDXCTdKjQGIoRdqmG-TyzBGgiPaNuuMr83dS1td17JU4dbWxBUgJppCL4Iio4J7zLMlutTeXE-H5iKQJK6Rk361E-FXnZBurJhDj5rmYioYFeoeWqRzBAFbJUB2Awd0_-J770LUnnSyARD-Ho56A4KWonOKaXx5KoOVG3gk41tGL1ggFt62V2syzKL204yzlpR4v4vFT5Ulk1PeJXvpNoQIUKzSR6m2IAQ-FtuUEuGPRoGxMsHWlRsteHsdNGmIThrxZ3YCPsXvIElwOv55cUk26z-Rhrr-SOWYKUy3a_7pRRsUXQeYyim3Y0s7rFVHNpzQeyeASR6Gq7-Lg5Rfs45kLHv52KaYbPNSSxm7VUp5aQQUY59BVQdSQxGBGx_dDGbawUe4NT4ZvfYOmASbj6ejJonodDdxyMbLOsRrSHkWH4MOcB7hJbWjW5lWSztKxV-d7kOhiMFP9YWlohagwyciMp6Ffw3cGGJkG9yiWffif81ps_4_CdjittFF4VdgoCdLrkl0wxs5oK9CWhq0p9E8
               status code: 403, request id: 15649ff3-480e-4e28-882a-b84adac025ca
      Normal   ExternalProvisioning  101s (x182 over 46m)  persistentvolume-controller                                                               Waiting for a volume to be created either by the external provisioner 'ebs.csi.aws.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
      Normal   Provisioning          86s (x20 over 46m)    ebs.csi.aws.com_ebs-csi-controller-7db7cf79d5-5sq47_d48c1c09-e066-4697-aa55-c1064e3b5068  External provisioner is provisioning volume for claim "default/models-1-0-0-claim-scratch"
    
  14. awels commented on Mar 26, 2024

    @awels
    Member

    Yeah, seems like it. That is your current problem.

  15. knkedlaya commented on Mar 27, 2024

    @knkedlaya

    Created a new cluster, now the pod is in ContainerCreating state with the following event on the pod. Not sure if some config is gone missing/wrong.

    Events:
      Type     Reason              Age                   From                     Message
      ----     ------              ----                  ----                     -------
      Warning  FailedScheduling    33m                   default-scheduler        0/1 nodes are available: persistentvolumeclaim "sipvc-dv-scratch" not found. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
      Normal   Scheduled           27m                   default-scheduler        Successfully assigned default/cdi-upload-sipvc-dv to ip-192-168-82-77.ap-south-1.compute.internal
      Warning  FailedAttachVolume  3m40s (x10 over 25m)  attachdetach-controller  AttachVolume.Attach failed for volume "sipvc-dv" : timed out waiting for external-attacher of pd.csi.storage.gke.io CSI driver to attach volume projects/UNSPECIFIED/zones/UNSPECIFIED/disks/models-1-0-0
    

    Note that, I had to create scratch pv to bind with scratch pvc. After that, the upload container moved from pending to ContainerCreating state.

  16. awels commented on Mar 27, 2024

    @awels
    Member

    Seems like you still have some storage issues. If the attach is failing it means the volume cannot be mounted on the node the pod is supposed to run on. Looking at the message you have some UNSPECIFIED fields somewhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions