Skip to content

VMI status update loop on k8s 1.36: ContainerDiskInfo/KernelInfo/InitrdInfo Checksum (uint32) hit strict CRD format validation #17858

Closed

Description

@a7i

What happened

On Kubernetes v1.36 with KubeVirt v1.8.2, every VMI we create ends up in an infinite virt-handler re-enqueue loop trying to update its status. The pod becomes Running, the VMI eventually reports LiveMigratable=True, but live migrations never complete because the status update keeps being rejected by the API server.

The exact error in virt-handler logs:

"component":"virt-handler","controller":"vm","level":"error",
"msg":"Updating the VirtualMachineInstance status failed.",
"name":"kubevirtvmi-1","namespace":"default","pos":"vm.go:1506",
"reason":"VirtualMachineInstance.kubevirt.io \"kubevirtvmi-1\" is invalid: <nil>:
  Invalid value: \"\": Checked value must be of type integer with format int32
  in status.volumeStatus[1].containerDiskVolume.checksum"

The corresponding virt-controller message is the same shape as the VSOCKCID one already documented in #17688:

"component":"virt-controller","msg":"reenqueuing VirtualMachineInstance default/kubevirtvmi-3",
"pos":"vmi.go:296",
"reason":"patching of vmi conditions and activePods failed:
  the server rejected our request due to an error in our request"

In a 4-minute test window we observed 142 status-update rejections across 3 VMIs (64 / 42 / 36), so this is not a flake; once virt-handler enters the loop on a given VMI, it never escapes.

What we expected

VMI status updates from virt-handler to be accepted by the API server, and live migrations to complete normally (as they do on k8s v1.34 and v1.35 with the same KubeVirt version).

Reproducer

This is reproducing 100% on the descheduler v0.36.0 release-prep PR (TestLiveMigrationInBackground) against the matrix entry k8s v1.36.1 + kind v0.35.1 + KubeVirt v1.8.2:

Root cause

This is the same class of bug as #17688 / fixed for VSOCKCID in #17690.

Upstream trigger: kubernetes/kubernetes#136582 ("Bump k8s.io/kube-openapi to latest and enable numeric format validation"), milestoned for k8s v1.36. From its release note:

CRD validation now strictly enforces ranges for numeric formats (int32, int64, float, double) when specified in the schema. Existing objects with out-of-range values are preserved via validation ratcheting.

Before this PR, format: int32 was silently stripped from CRD schemas by StripUnsupportedFormatsPostProcessorForVersion and the underlying validator was not initialized with type/format, so no numeric format validation happened against CRDs. As of k8s v1.36 it is enforced.

KubeVirt mismatch: several *Info status types declare Checksum uint32 but the generated CRD almost certainly publishes format: int32 for them (the same shape that #17690 just fixed for VSOCKCID). The affected fields in staging/src/kubevirt.io/api/core/v1/types.go:

All three are written by VirtualMachineController.updateChecksumInfo in pkg/virt-handler/vm.go.

One open question

The reported invalid value is "" (empty string), not a large uint32 value out of int32 range. So our case may not be a pure value-range issue like VSOCKCID was; it may be that the serialized status payload includes an empty/null value for the field that the now-strict validator rejects as not-an-integer. The fix shape from #17690 (format: int64 + Minimum: 0 + Maximum: 4294967295) might not be sufficient on its own; the virt-handler write path that produces the empty value may also need to be hardened (either skip writing the field, or write an explicit 0). Maintainers who know the patch-construction path will know better than us.

Suggested fix (same pattern as #17690)

For each of the three fields above:

// +kubebuilder:validation:Format:=int64
// +kubebuilder:validation:Minimum:=0
// +kubebuilder:validation:Maximum:=4294967295
Checksum uint32 `json:"checksum,omitempty"`

and regenerate the CRDs. Optionally also audit other uint32 status fields in the API for the same mismatch.

If a release branch fix is feasible (e.g. backport to release-1.8), please consider it; downstream consumers (descheduler, virt-handler-driven test suites) are blocked on k8s v1.36 until then.

Environment

Related

/sig api
/kind bug
/cc @fossedihelm @dhiller

Activity

  1. dosubot commented on May 19, 2026

    @dosubot

    Hi @a7i! I'm Dosu and I’m helping the kubevirt team.

    This is already fixed on main by PR #17469 (merged 2026-04-18), which applied the exact same pattern you suggested — Format:=int64 / Minimum:=0 / Maximum:=4294967295 markers — to all three Checksum fields (ContainerDiskInfo, KernelInfo, InitrdInfo) [1]. The current types.go on main has the correct annotations [2] [3] [4].

    However, from what I can find, this fix has not been backported to release-1.8, and there is no v1.8.3 release. The fix will land in v1.9.0 (v1.9.0-alpha.0 was tagged 2026-05-11). So for your k8s v1.36 + KubeVirt v1.8.2 matrix entry, you'd either need to request a backport cherry-pick to release-1.8, or pin to a main-based build / v1.9 pre-release.

    Regarding your "open question" about the empty-string "" value — that's worth flagging separately. The schema fix resolves the numeric-format enforcement, but if virt-handler is writing an empty/null checksum field into the status patch (rather than omitting it via omitempty), that could still trip the stricter validator even with the corrected format. The omitempty tag should handle this for zero-value uint32 (which serializes to nothing), but if there's a code path producing "" explicitly, that would need a separate fix in the write path.

    cc @fossedihelm @dhiller — would a backport of #17469 to release-1.8 be feasible for downstream consumers blocked on k8s 1.36?

    To reply, just mention @dosu.


    Docs are dead. Just use Dosu.

    Leave Feedback Ask Dosu about kubevirt Share Dosu with your team

  2. a7i commented on May 19, 2026

    @a7i
    Author

    Waiting for v1.9 release works for me! For now, I pinned to the RC tag

  3. Moep90 commented on May 27, 2026

    @Moep90

    Confirmed on bare-metal Talos Linux + K8s v1.36.0 — VMIs stuck at Scheduled

    We can reproduce this outside kind on a bare-metal 3-node Talos Linux v1.13.0 cluster with Kubernetes v1.36.0 and KubeVirt v1.8.2.

    Symptom: virt-handler repeatedly fails to update VMI status. The VMI remains stuck at Scheduled and the launcher pod never starts. The virt-handler log shows:

    Invalid value: "": Checked value must be of type integer with format int32 in status.volumeStatus[0].containerDiskVolume.checksum
    

    Root cause in our case: virt-handler sends containerDiskVolume.checksum: "" (empty string), but the live CRD schema declares checksum: type: integer. Kubernetes 1.36's stricter numeric validation (kubernetes/kubernetes#136582) rejects the empty string, whereas earlier Kubernetes versions silently accepted it.

    Workaround (kubectl-only, no reboot):

    1. kubectl scale deployment virt-operator --replicas=0 -n kubevirt
    2. Edit customresourcedefinitions virtualmachineinstances.kubevirt.io — remove the checksum field from containerDiskVolume, kernelInfo, and initrdInfo schemas
    3. kubectl replace -f the CRD (do not use apply)
    4. Restart affected VMIs

    After the CRD edit, virt-handler status updates succeed and VMIs progress from Scheduled → Running.

    Question for maintainers: Will PR #17469 (merged to main in v1.9.0-alpha.0) be backported to the v1.8.x line? Many production deployments are pinned to v1.8 LTS and cannot easily jump to a v1.9 alpha.

  4. fossedihelm commented on May 27, 2026

    @fossedihelm
    Contributor

    Hey all. Thank you for raising this. The support matrix shows that k8s 1.36 is not compatible with kv 1.8. The first k8s 1.36 compatible kubevirt will be the release-1.9 https://github.com/kubevirt/sig-release/blob/main/releases/k8s-support-matrix.md.

    Question for maintainers: Will PR #17469 (merged to main in v1.9.0-alpha.0) be backported to the v1.8.x line? Many production deployments are pinned to v1.8 LTS and cannot easily jump to a v1.9 alpha.

    We can discuss this in the community meeting today, since it involves an api change. WDYT? Thanks
    @vladikr FYI

  5. lexfrei commented on May 29, 2026

    @lexfrei

    Confirming this affects the CNCF project Cozystack as well. Cozystack v1.4.2 ships KubeVirt v1.8.2, and on Kubernetes 1.36 hosts tenant VMs get stuck in Scheduled with the exact status.volumeStatus[N].containerDiskVolume.checksum int32 rejection described here. Tracked downstream as cozystack/cozystack#2769.

    This is hitting real users in our community, not just CI — bare-metal Talos + k8s 1.36 among the reported setups. A backport of #17469 to release-1.8 would unblock production deployments pinned to the 1.8 LTS line; strong +1 for that from the Cozystack side.

  6. fossedihelm commented on Jun 5, 2026

    @fossedihelm
    Contributor

    @lexfrei @Moep90 @a7i We are going to backport the PR into 1.8 and cut a z release. However, please note that this is a strict exception. This backport does not mean 1.8 now supports Kubernetes 1.36, nor does it guarantee that future API change backports will be performed. Thank you
    #18055

  7. fossedihelm commented on Jun 16, 2026

    @fossedihelm
    Contributor
  8. kubevirt-bot commented on Jun 16, 2026

    @kubevirt-bot
    Contributor

    @fossedihelm: Closing this issue.

    Details

    In response to this:

    @lexfrei @Moep90 @a7i KubeVirt 1.8.4 just released. https://github.com/kubevirt/kubevirt/releases/tag/v1.8.4.
    @0xFelix FYI
    /close

    Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions