Repository navigation
Closed
Labels
Description
What happened:
After executing the command kubectl create -f dv_fedora.yml as per the instructions on the KubeVirt labs page(https://kubevirt.io/labs/kubernetes/lab2.html), the importer pod was not created. I was unable to find any pod name associated with the import process.
What you expected to happen:
the importer pod was created.
Additional context:
kubectl get cdi cdi -n cdi
NAME AGE PHASE
cdi 5h41m Deployed
kubectl get pvc fedora -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
cdi.kubevirt.io/storage.contentType: kubevirt
cdi.kubevirt.io/storage.pod.restarts: "0"
cdi.kubevirt.io/storage.preallocation.requested: "false"
cdi.kubevirt.io/storage.usePopulator: "true"
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: nfs.csi.k8s.io
creationTimestamp: "2024-03-20T06:14:56Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
alerts.k8s.io/KubePersistentVolumeFillingUp: disabled
app: containerized-data-importer
app.kubernetes.io/component: storage
app.kubernetes.io/managed-by: cdi-controller
name: fedora
namespace: default
ownerReferences:
- apiVersion: cdi.kubevirt.io/v1beta1
blockOwnerDeletion: true
controller: true
kind: DataVolume
name: fedora
uid: cba5ce85-7173-44d2-81f9-ab2372c731df
resourceVersion: "266638693"
uid: 60150494-21ed-446b-a540-75ef0240b5a4
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: "5681173672"
storageClassName: nanhu-sc-db
volumeMode: Filesystem
volumeName: pvc-60150494-21ed-446b-a540-75ef0240b5a4
status:
accessModes:
- ReadWriteMany
capacity:
storage: "5681173672"
phase: Bound
kubectl get datavolume fedora -oyaml
apiVersion: cdi.kubevirt.io/v1beta1
kind: DataVolume
metadata:
annotations:
cdi.kubevirt.io/storage.usePopulator: "true"
creationTimestamp: "2024-03-20T06:14:56Z"
generation: 1
name: fedora
namespace: default
resourceVersion: "266638694"
uid: cba5ce85-7173-44d2-81f9-ab2372c731df
spec:
source:
http:
url: https://download.fedoraproject.org/pub/fedora/linux/releases/37/Cloud/x86_64/images/Fedora-Cloud-Base-37-1.7.x86_64.raw.xz
storage:
resources:
requests:
storage: 5Gi
status:
claimName: fedora
conditions:
- lastHeartbeatTime: "2024-03-20T06:15:00Z"
lastTransitionTime: "2024-03-20T06:15:00Z"
message: PVC fedora Bound
reason: Bound
status: "True"
type: Bound
- lastHeartbeatTime: "2024-03-20T06:15:00Z"
lastTransitionTime: "2024-03-20T06:15:00Z"
status: "False"
type: Ready
- lastHeartbeatTime: "2024-03-20T06:15:00Z"
lastTransitionTime: "2024-03-20T06:15:00Z"
status: "False"
type: Running
phase: PVCBound
progress: N/A
kubectl logs cdi-deployment-5d4f984b5-kjlj8 -n cdi
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"events","msg":"PVC fedora Pending","type":"Normal","object":{"kind":"DataVolume","namespace":"default","name":"fedora","uid":"cba5ce85-7173-44d2-81f9-ab2372c731df","apiVersion":"cdi.kubevirt.io/v1beta1","resourceVersion":"266638674"},"reason":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/fedora","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/fedora","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/fedora","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":false,"has finalizer?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"PVC not bound, skipping pvc","PVC":"default/fedora","Phase":"Pending"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/fedora","isUpload":false,"isCloneTarget":false,"isBound":false,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"reconciling Upload PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"reconciling Import PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.upload-controller","msg":"not doing anything with PVC","PVC":"default/fedora","isUpload":false,"isCloneTarget":false,"isBound":true,"podSucceededFromPVC":false,"deletionTimeStamp set?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"reconciling Clone PVCs","PVC":"default/fedora"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"k8s.io/CloneRequest"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.clone-controller","msg":"Should not reconcile this PVC","PVC":"default/fedora","checkPVC(AnnCloneRequest)":false,"NOT has annotation(AnnCloneOf)":true,"isBound":true,"has finalizer?":false}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.endpoint"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"controller.import-controller","msg":"PVC annotation not found, skipping pvc","PVC":"default/fedora","annotation":"cdi.kubevirt.io/storage.import.source"}
{"level":"debug","ts":"2024-03-20T06:15:00Z","logger":"events","msg":"PVC fedora Bound","type":"Normal","object":{"kind":"DataVolume","namespace":"default","name":"fedora","uid":"cba5ce85-7173-44d2-81f9-ab2372c731df","apiVersion":"cdi.kubevirt.io/v1beta1","resourceVersion":"266638694"},"reason":"Bound"}
Environment:
- KubeVirt version (use
virtctl version): v1.2.0 - Kubernetes version (use
kubectl version): v1.22.10 - VM or VMI specifications: N/A
- Cloud provider or hardware configuration: N/A
- OS (e.g. from /etc/os-release): Ubuntu 20.04.4 LTS (Focal Fossa)
- Kernel (e.g.
uname -a):Linux onestor 5.4.0-131-generic #147-Ubuntu SMP Fri Oct 14 17:07:22 UTC 2022 x86_64 x86_64 x8 6_64 GNU/Linux - Install tools: N/A
- Others: N/A
Activity
awels commentedon Mar 22, 2024
Your kubernetes version is very old, I believe populators which CDI uses are beta in 1.24 and enabled by default there. In 1.22 you probably have to enable a feature gate to get it to work.
shusley244 commentedon Mar 22, 2024
Thanks for your answer. I will upgrade the kubernetes version and try again.
knkedlaya commentedon Mar 26, 2024
Hi,
Facing similar issue though my kubernetes(AWS EKS) setup is latest.
pv.yaml
Trying to upload image with below command
which ends up with
Output of
awels commentedon Mar 26, 2024
Looks like you are using the
local-storagestorage class AND RWX volumes. I am fairly certain local-storage doesn't support RWX, try RWO.knkedlaya commentedon Mar 26, 2024
Thanks @awels . I changed local-storage to ReadWriteOnce and I see different logs
This is new pv.yaml after changing to RWO
awels commentedon Mar 26, 2024
And the upload pod is still not starting? Do you have any pods in the namespace you are trying to upload to?
[-]CDI impoter pod was not created.[/-][+]CDI importer pod was not created.[/+]awels commentedon Mar 26, 2024
Also I see no mention of the scratch space pvc. Does that get created? It also seems you are creating the PVC ahead of time, you don't really need to do that, if you specify the datavolume name, CDI should take care of creating all the required resources.
knkedlaya commentedon Mar 26, 2024
You are right, I have a pod running. Never came to my mind that the pod is related to the image I was trying to upload. I will delete this pod, PV, PVC and retry with dv create option while uploading.
I think upload creates scratch pvc because I havent created it explicitly.
awels commentedon Mar 26, 2024
it doesn't look running, it looks pending, if it remains pending after retrying, do a
kubectl describe pod <name>it should tell you why it is pending.knkedlaya commentedon Mar 26, 2024
These are the events I am seeing when I describe the pod
awels commentedon Mar 26, 2024
Is the scratch PVC created, and is it bound? The events tell me it can't mount the scratch space.
knkedlaya commentedon Mar 26, 2024
yes, scratch pvc is created. When I describe I see following. Hinting at some permission related issue.
awels commentedon Mar 26, 2024
Yeah, seems like it. That is your current problem.
knkedlaya commentedon Mar 27, 2024
Created a new cluster, now the pod is in
ContainerCreatingstate with the following event on the pod. Not sure if some config is gone missing/wrong.Note that, I had to create scratch pv to bind with scratch pvc. After that, the upload container moved from pending to ContainerCreating state.
awels commentedon Mar 27, 2024
Seems like you still have some storage issues. If the attach is failing it means the volume cannot be mounted on the node the pod is supposed to run on. Looking at the message you have some UNSPECIFIED fields somewhere.
feat(kubernetes)!: replace ephemeral emptyDisk with persistent storag…