Repository navigation
Description
What happened
On Kubernetes v1.36 with KubeVirt v1.8.2, every VMI we create ends up in an infinite virt-handler re-enqueue loop trying to update its status. The pod becomes Running, the VMI eventually reports LiveMigratable=True, but live migrations never complete because the status update keeps being rejected by the API server.
The exact error in virt-handler logs:
"component":"virt-handler","controller":"vm","level":"error",
"msg":"Updating the VirtualMachineInstance status failed.",
"name":"kubevirtvmi-1","namespace":"default","pos":"vm.go:1506",
"reason":"VirtualMachineInstance.kubevirt.io \"kubevirtvmi-1\" is invalid: <nil>:
Invalid value: \"\": Checked value must be of type integer with format int32
in status.volumeStatus[1].containerDiskVolume.checksum"
The corresponding virt-controller message is the same shape as the VSOCKCID one already documented in #17688:
"component":"virt-controller","msg":"reenqueuing VirtualMachineInstance default/kubevirtvmi-3",
"pos":"vmi.go:296",
"reason":"patching of vmi conditions and activePods failed:
the server rejected our request due to an error in our request"
In a 4-minute test window we observed 142 status-update rejections across 3 VMIs (64 / 42 / 36), so this is not a flake; once virt-handler enters the loop on a given VMI, it never escapes.
What we expected
VMI status updates from virt-handler to be accepted by the API server, and live migrations to complete normally (as they do on k8s v1.34 and v1.35 with the same KubeVirt version).
Reproducer
This is reproducing 100% on the descheduler v0.36.0 release-prep PR (TestLiveMigrationInBackground) against the matrix entry k8s v1.36.1 + kind v0.35.1 + KubeVirt v1.8.2:
- Failing build: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_descheduler/1874/pull-descheduler-test-e2e-k8s-master-1-36/2056725836169482240
- Passing build on same commit, k8s v1.34: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_descheduler/1874/pull-descheduler-test-e2e-k8s-master-1-34/2056725836387586048
Root cause
This is the same class of bug as #17688 / fixed for VSOCKCID in #17690.
Upstream trigger: kubernetes/kubernetes#136582 ("Bump k8s.io/kube-openapi to latest and enable numeric format validation"), milestoned for k8s v1.36. From its release note:
CRD validation now strictly enforces ranges for numeric formats (int32, int64, float, double) when specified in the schema. Existing objects with out-of-range values are preserved via validation ratcheting.
Before this PR, format: int32 was silently stripped from CRD schemas by StripUnsupportedFormatsPostProcessorForVersion and the underlying validator was not initialized with type/format, so no numeric format validation happened against CRDs. As of k8s v1.36 it is enforced.
KubeVirt mismatch: several *Info status types declare Checksum uint32 but the generated CRD almost certainly publishes format: int32 for them (the same shape that #17690 just fixed for VSOCKCID). The affected fields in staging/src/kubevirt.io/api/core/v1/types.go:
ContainerDiskInfo.Checksum(referenced in the API error)KernelInfo.ChecksumInitrdInfo.Checksum
All three are written by VirtualMachineController.updateChecksumInfo in pkg/virt-handler/vm.go.
One open question
The reported invalid value is "" (empty string), not a large uint32 value out of int32 range. So our case may not be a pure value-range issue like VSOCKCID was; it may be that the serialized status payload includes an empty/null value for the field that the now-strict validator rejects as not-an-integer. The fix shape from #17690 (format: int64 + Minimum: 0 + Maximum: 4294967295) might not be sufficient on its own; the virt-handler write path that produces the empty value may also need to be hardened (either skip writing the field, or write an explicit 0). Maintainers who know the patch-construction path will know better than us.
Suggested fix (same pattern as #17690)
For each of the three fields above:
// +kubebuilder:validation:Format:=int64
// +kubebuilder:validation:Minimum:=0
// +kubebuilder:validation:Maximum:=4294967295
Checksum uint32 `json:"checksum,omitempty"`and regenerate the CRDs. Optionally also audit other uint32 status fields in the API for the same mismatch.
If a release branch fix is feasible (e.g. backport to release-1.8), please consider it; downstream consumers (descheduler, virt-handler-driven test suites) are blocked on k8s v1.36 until then.
Environment
- Kubernetes: v1.36.1 (via kind v0.35.1 node image)
- KubeVirt: v1.8.2
- Reproducer test:
sigs.k8s.io/descheduler/test/e2eTestLiveMigrationInBackground - Tracking on the consumer side: [v0.36.0] release prep: bump k8s/go deps, manifests, docs, and CI matrix kubernetes-sigs/descheduler#1874
Related
- k8s 1.36 sig-compute: VSOCK VMIs stuck in Scheduling + virtiofs token permission denied #17688 (closed, VSOCK leg of the same class of bug)
- vmi: fix VSOCK CID schema mismatch causing VMI requeue loop #17690 (merged, fixed
VSOCKCIDschema for the same reason) - Bump k8s.io/kube-openapi to latest and enable numeric format validation kubernetes/kubernetes#136582 (k8s v1.36 upstream change)
- Standard numeric format validation (int32, int64, float, double) is not enforced in CRDs kubernetes/kubernetes#136091 (upstream tracking issue)
/sig api
/kind bug
/cc @fossedihelm @dhiller
Activity
dosubot commentedon May 19, 2026
Hi @a7i! I'm Dosu and I’m helping the kubevirt team.
This is already fixed on
mainby PR #17469 (merged 2026-04-18), which applied the exact same pattern you suggested —Format:=int64/Minimum:=0/Maximum:=4294967295markers — to all threeChecksumfields (ContainerDiskInfo,KernelInfo,InitrdInfo) [1]. The currenttypes.goonmainhas the correct annotations [2] [3] [4].However, from what I can find, this fix has not been backported to
release-1.8, and there is no v1.8.3 release. The fix will land in v1.9.0 (v1.9.0-alpha.0 was tagged 2026-05-11). So for your k8s v1.36 + KubeVirt v1.8.2 matrix entry, you'd either need to request a backport cherry-pick torelease-1.8, or pin to amain-based build / v1.9 pre-release.Regarding your "open question" about the empty-string
""value — that's worth flagging separately. The schema fix resolves the numeric-format enforcement, but ifvirt-handleris writing an empty/nullchecksumfield into the status patch (rather than omitting it viaomitempty), that could still trip the stricter validator even with the corrected format. Theomitemptytag should handle this for zero-valueuint32(which serializes to nothing), but if there's a code path producing""explicitly, that would need a separate fix in the write path.cc @fossedihelm @dhiller — would a backport of #17469 to
release-1.8be feasible for downstream consumers blocked on k8s 1.36?To reply, just mention @dosu.
Docs are dead. Just use Dosu.
revert: ensureVMIsLiveMigratable retry helper
ci(e2e): bump KubeVirt to v1.9.0-alpha.0 for k8s 1.36 compatibility
a7i commentedon May 19, 2026
Waiting for v1.9 release works for me! For now, I pinned to the RC tag
Moep90 commentedon May 27, 2026
Confirmed on bare-metal Talos Linux + K8s v1.36.0 — VMIs stuck at
ScheduledWe can reproduce this outside
kindon a bare-metal 3-node Talos Linux v1.13.0 cluster with Kubernetes v1.36.0 and KubeVirt v1.8.2.Symptom:
virt-handlerrepeatedly fails to update VMI status. The VMI remains stuck atScheduledand the launcher pod never starts. The virt-handler log shows:Root cause in our case: virt-handler sends
containerDiskVolume.checksum: ""(empty string), but the live CRD schema declareschecksum: type: integer. Kubernetes 1.36's stricter numeric validation (kubernetes/kubernetes#136582) rejects the empty string, whereas earlier Kubernetes versions silently accepted it.Workaround (kubectl-only, no reboot):
kubectl scale deployment virt-operator --replicas=0 -n kubevirtcustomresourcedefinitions virtualmachineinstances.kubevirt.io— remove thechecksumfield fromcontainerDiskVolume,kernelInfo, andinitrdInfoschemaskubectl replace -fthe CRD (do not useapply)After the CRD edit, virt-handler status updates succeed and VMIs progress from
Scheduled→Running.Question for maintainers: Will PR #17469 (merged to
mainin v1.9.0-alpha.0) be backported to the v1.8.x line? Many production deployments are pinned to v1.8 LTS and cannot easily jump to a v1.9 alpha.fossedihelm commentedon May 27, 2026
Hey all. Thank you for raising this. The support matrix shows that k8s 1.36 is not compatible with kv 1.8. The first k8s 1.36 compatible kubevirt will be the release-1.9 https://github.com/kubevirt/sig-release/blob/main/releases/k8s-support-matrix.md.
We can discuss this in the community meeting today, since it involves an api change. WDYT? Thanks
@vladikr FYI
lexfrei commentedon May 29, 2026
Confirming this affects the CNCF project Cozystack as well. Cozystack v1.4.2 ships KubeVirt v1.8.2, and on Kubernetes 1.36 hosts tenant VMs get stuck in
Scheduledwith the exactstatus.volumeStatus[N].containerDiskVolume.checksumint32 rejection described here. Tracked downstream as cozystack/cozystack#2769.This is hitting real users in our community, not just CI — bare-metal Talos + k8s 1.36 among the reported setups. A backport of #17469 to release-1.8 would unblock production deployments pinned to the 1.8 LTS line; strong +1 for that from the Cozystack side.
fossedihelm commentedon Jun 5, 2026
@lexfrei @Moep90 @a7i We are going to backport the PR into 1.8 and cut a z release. However, please note that this is a strict exception. This backport does not mean 1.8 now supports Kubernetes 1.36, nor does it guarantee that future API change backports will be performed. Thank you
#18055
fossedihelm commentedon Jun 16, 2026
@lexfrei @Moep90 @a7i KubeVirt 1.8.4 just released. https://github.com/kubevirt/kubevirt/releases/tag/v1.8.4.
@0xFelix FYI
/close
kubevirt-bot commentedon Jun 16, 2026
@fossedihelm: Closing this issue.
Details
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.