Repository navigation
fix(platform): accept the container gpu-operator variant in iaas - #4101
Conversation
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (5)
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughThe GPU bundle now accepts the ChangesGPU container variant
Estimated code review effort: 3 (Moderate) | ~20 minutes Merge Risk: ⚪ Minimal · up to This change enables the pod-only container GPU operator variant while preventing KubeVirt GPU wiring from being rendered for it. The intended behavior is covered by validation and rendering tests, with no active merge-blocking risk identified. Sequence Diagram(s)sequenceDiagram
participant Operator
participant PlatformBundle
participant GpuOperator
participant KubeVirt
Operator->>PlatformBundle: Set gpuOperatorVariant: container
PlatformBundle->>GpuOperator: Emit container Package
PlatformBundle->>KubeVirt: Omit HostDevices and GPU device configuration
GpuOperator-->>Operator: Provide pod GPU access through the device plugin
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
The cozystack.gpu-operator PackageSource ships three variants: default, vgpu and container. The iaas bundle's variant guard accepted only the first two, so selecting container failed the render on a value the package ships, values-container.yaml configures deliberately, and the variant bats suite already covers. The exclusion was coherent when it was written and stopped being so immediately. The container variant and the guard landed within minutes of each other from parallel branches, neither an ancestor of the other, variant first. Until the guard, the bundle hardcoded the default variant and container was selected by hand-crafting a Package CR, so leaving it out of a bundle-side list cost nothing. The guard then made the bundle the variant owner and closed that route without noticing it was one; its comment describes itself as typo protection and names passthrough as the example typo. The KubeVirt side is not a bare allowlist edit. The container variant serves GPUs to pods through the device plugin and runs no VM-facing GPU stack: sandboxWorkloads and vfioManager are both off and the host driver stays bound. Both KubeVirt-side pieces the bundle wires are VM-facing — the HostDevices feature gate advertises a capability such a node cannot serve, and permittedHostDevices lists resources no sandbox device plugin registers there. Both are now emitted only for default and vgpu, and container yields the gpu-operator Package with the kubevirt component untouched. Comments and docs that described the KubeVirt mirroring as unconditional on variant, or the container variant as unmerged and unavailable, are corrected to match. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <[email protected]>
1ca93cc to
6e8566a
Compare
IvanHunters
left a comment
There was a problem hiding this comment.
Verdict
LGTM with non-blocking notes
Reviewed at 6e8566a94 against merge-base cd675e113.
Widening the allowlist is right, and skipping the KubeVirt wiring for container is a decision rather than a shortcut. I rendered the iaas bundle on both sides for all three variants: default and vgpu come out byte-identical, so nothing installed today changes shape. container fails on base at iaas.yaml:54 and renders on head. Three notes below, none blocking.
Findings
- [MINOR]
packages/core/platform/templates/bundles/iaas.yaml:73, the accepted-variant set is still hardcoded, and the only thing standing between it and the next drift is a comment - [MINOR]
packages/core/platform/templates/bundles/iaas.yaml:76, gpu.* overrides are dropped silently under container, and nothing tells the operator - [MINOR]
docs/gpu-vgpu.md:221, the docs describe the end state of container but never the transition into it
Claim mismatches
[UNVERIFIABLE] The body says the docs live in cozystack/website#685 as an issue so they cannot land ahead of this change. That is another repository and this run is hermetic, so I did not fetch it. Everything in-repo I re-derived and it holds: the PackageSource ships container, values-container.yaml configures it the way the new comment describes, hack/check-gpu-operator-variants.bats covers it in four cases, and the base guard did reject it.
Caveats
- The live transition is reasoned, not run. A render cannot show the KubeVirt CR converging once
HostDevicesandpermittedHostDevicesare withdrawn, nor what a VMI already holding a host device does in that window. Both fields are chart-owned and get pruned rather than conflicted, and neither is immutable, so I see no server-side-apply hazard. That is an argument though, not evidence. - Non-string values for
gpuOperatorVariant(5,true) slip past thefailand come out as a raw Go template error,incompatible types for comparison: int64 and string. Identical on the merge base, so it predates this PR. Deriving the accepted set as the first finding suggests would not fix it either, thoughhason a non-string at least returns false and reaches the intended message. - Nothing covers
gpuOperatorVariant: containerwithcozystack.gpu-operatorabsent fromenabledPackages, which is newly legal because the guard runs ahead of$gpuEnabled. I rendered it. It is fine. Coverage gap, not a defect. - The
templates/repository.yaml:8render error is alookupguard that fails on any cluster-free render and reproduces on the merge base, andkubeconformskipped all 15 rendered resources becausecozystack.io/v1alpha1 Packagehas no offline schema. Neither says anything about this diff.
Recommended follow-ups
- Derive the accepted variant set from
sources/gpu-operator.yaml, or add a cross-file equality assertion, so a fourth variant cannot ship unreachable the waycontainerdid. - Exercise the transition a static review cannot reach: install at
defaultwith the package enabled, confirm the KubeVirt CR carriesHostDevicesand the passthrough table, flip tocontainer, then watch the CR converge and the HelmRelease swap its artifact ref in place instead of standing up a second release.
| */ -}} | ||
| {{- if not (or (eq $gpuVariant "default") (eq $gpuVariant "vgpu")) -}} | ||
| {{- fail (printf "bundles.iaas.gpuOperatorVariant must be \"default\" or \"vgpu\", got %q" $gpuVariant) -}} | ||
| {{- if not (or (eq $gpuVariant "default") (eq $gpuVariant "vgpu") (eq $gpuVariant "container")) -}} |
There was a problem hiding this comment.
[MINOR] the accepted-variant set is still hardcoded, and the only thing standing between it and the next drift is a comment
This PR exists because the bundle's allowlist and the PackageSource's variant list disagreed. The fix restores agreement and then documents the coupling in prose: "A variant added there needs adding here too". That is the same protection the previous round had, which is none.
sources/gpu-operator.yaml is inside the chart and reachable from templates, so the guard can derive its own accepted set and the class of bug closes permanently rather than for one release. I checked the read works before proposing it (a probe template rendering $.Files.Get "sources/gpu-operator.yaml" printed variants=[default vgpu container]), so the guard can become:
{{- $gpuVariants := list -}}
{{- range (($.Files.Get "sources/gpu-operator.yaml") | fromYaml).spec.variants -}}
{{- $gpuVariants = append $gpuVariants .name -}}
{{- end -}}
{{- if not (has $gpuVariant $gpuVariants) -}}
{{- fail (printf "bundles.iaas.gpuOperatorVariant must be one of %v, got %q" $gpuVariants $gpuVariant) -}}
{{- end -}}
The existing sriov and passthrough rejection cases keep passing against that shape, only the message text changes. If you would rather keep the literal list, the cheap alternative is a helm-unittest or bats case that reads both files and asserts the two sets are equal, which fails the moment a fourth variant lands on one side only.
| {{- fail (printf "bundles.iaas.gpuOperatorVariant must be \"default\", \"vgpu\" or \"container\", got %q" $gpuVariant) -}} | ||
| {{- end -}} | ||
| {{- if $gpuEnabled }} | ||
| {{- if and $gpuEnabled (ne $gpuVariant "container") }} |
There was a problem hiding this comment.
[MINOR] gpu.* overrides are dropped silently under container, and nothing tells the operator
Skipping the KubeVirt wiring for container is right. What the operator experiences is less so: someone who has gpu.permittedHostDevices.pciHostDevices populated (an AMD MI300X, an FPGA, anything they wanted advertised to VMs) and then selects container gets a clean render, a healthy platform, and their entries nowhere. The PR's own second test case pins exactly that silence, using 1002:744C as the fixture. This combination is newly reachable, since base refused container outright, so it is a new silent-ignore path rather than an inherited one.
$ helm template ... --set bundles.iaas.gpuOperatorVariant=container \
--set 'bundles.enabledPackages[0]=cozystack.gpu-operator' | grep -c pciHostDevices
0
I am not asking for a fail: the template already argues, correctly, at lines 99 to 104 that silently dropping mediatedDevices under default beats breaking an operator who keeps one shared gpu block and flips the variant. A fail here would be the harsher, less consistent choice. The chart does have a precedent for the softer signal though. templates/NOTES.txt:3 prints a WARNING: when the deletion-protection guardrail is inactive, and tests/notes_capability_gate_test.yaml holds it in place. A NOTES line along the lines of "gpuOperatorVariant is container; the entries under .gpu are not applied, that variant serves GPUs to pods only" would cost a few lines and close the gap. The same argument applies to gpu.vgpuManager and gpu.vgpuDeviceManager under default, which is pre-existing and out of scope here.
| ## KubeVirt configuration | ||
|
|
||
| When `cozystack.gpu-operator` is in `bundles.enabledPackages` (and not also in `bundles.disabledPackages`), the platform mirrors the chosen GPU variant into the `KubeVirt` CR automatically. There is no manual `kubectl patch` step. | ||
| When `cozystack.gpu-operator` is in `bundles.enabledPackages` (and not also in `bundles.disabledPackages`), the platform mirrors the chosen GPU variant into the `KubeVirt` CR automatically. There is no manual `kubectl patch` step. This applies to the two VM variants only: `container` serves GPUs to pods and never to VMs, so selecting it emits the gpu-operator Package with no host-device wiring on the KubeVirt side. |
There was a problem hiding this comment.
[MINOR] the docs describe the end state of container but never the transition into it
Every changed comment and doc paragraph is accurate about what container renders. None of them says what happens when an existing cluster switches to it. Flipping bundles.iaas.gpuOperatorVariant from default or vgpu to container on a running cluster does not merely stop adding wiring, it withdraws HostDevices from spec.configuration.developerConfiguration.featureGates and removes spec.configuration.permittedHostDevices from the live KubeVirt CR. VMs already consuming a host-device GPU by resourceName stop being admissible on their next start or migration. The operator reading "container serves GPUs to pods and never to VMs" can reasonably read that as a statement about new workloads.
This is a documentation gap, not a code defect: the withdrawal is the intended semantics of the variant. One sentence in this section, and ideally in the release note, saying that selecting container on a cluster that already runs GPU VMs takes the gate and the permit table away from those VMs, is enough. Note that default and vgpu do not have this property between themselves, since both keep the gate and only the device table differs, so container is the first variant whose selection removes a KubeVirt feature gate.
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does ## What this PR does Two stale references in `docs/gpu-vgpu.md`. The `Last verified` line says "this PR's `vgpu` variant". The variant ships, so there is no PR left for a reader to resolve that against. The link to `packages/system/gpu-operator/examples/README.md` calls it the flow "that predates the `container` variant". That README now opens with the container variant and sends non-Talos hosts to it. The link is there because it is the Talos path. The rest is gone. #4101 rewrote the same three passages in main, and got right the one thing this PR had wrong: `bundles.iaas.gpuOperatorVariant` takes `container` now, so the `disabledPackages` plus hand-written `Package` route described here is not the way in. The rebase takes main's text for all three and keeps only the two references #4101 did not touch. ### Screenshots Docs only, no UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Checked cozystack/website: neither line exists in `content/en/docs/next/virtualization/gpu.md` or `vgpu.md`. The `v1.2` snapshot carries its own PR-relative `Last verified` line, pointing at #2323, but that is a released version's record and out of scope here. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Documentation** - Updated GPU/vGPU verification details to reference the released vGPU variant. - Documented the available container variant, including deployment guidance and its scope compared with passthrough. - Updated known limitations to reflect that the container variant is shipped and does not configure host-device wiring through KubeVirt. - Clarified when KubeVirt GPU device wiring is applied based on GPU configuration settings. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
The container GPU variant can now be set up through the bundle, and the docs say so. cozystack/cozystack#4101 made the `iaas` bundle accept `gpuOperatorVariant: container`, so a hand-written Package CR is no longer the only way to get it. The same change stopped rendering the KubeVirt `HostDevices` gate and `permittedHostDevices` table for `container`. Only `default` and `vgpu` get them. The container workloads page now shows the bundle values and keeps the Package CR for overrides the bundle does not expose. The passthrough and vGPU pages say which variants get the KubeVirt wiring. Only `next/` is changed. The fix is on cozystack `main` and not in any v1.6.x tag, so the `v1.6/` pages still describe the released behaviour. Closes #685
…or variant in iaas (#5073) ## What this PR does Backport of #4101 to release-1.6. - Conflict only in comments of `packages/core/platform/values.yaml`, release-1.6's KubeVirt settings are kept as they are. ### Screenshots Not a UI change. ### Downstream repositories The website v1.6 page `operations/gpu-container-workloads.md` says the iaas bundle accepts only `default` or `vgpu`, it needs an update once this ships, I will open a website issue for it. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(platform): accept `container` as a value for `bundles.iaas.gpuOperatorVariant`, so the gpu-operator variant the platform already ships can be selected from platform values instead of a hand-crafted Package CR. The KubeVirt `HostDevices` feature gate and `permittedHostDevices` table are now emitted for the `default` and `vgpu` variants only; `container` serves GPUs to pods and gets neither. ```
What this PR does
bundles.iaas.gpuOperatorVariant: containerfailed the whole platform render. Thecozystack.gpu-operatorPackageSource ships that variant,values-container.yamlconfigures it deliberately, andhack/check-gpu-operator-variants.batscovers it, but theiaasbundle guard only accepteddefaultandvgpu.The exclusion made sense when it was written and stopped making sense a few minutes later. The variant and the guard came from parallel branches, neither an ancestor of the other, variant merged first. Before the guard the bundle hardcoded
default, socontainerwas selected by hand-writing a Package CR, and leaving it out of a bundle-side list cost nothing. The guard made the bundle the variant owner and closed that route without noticing it was one. Its comment calls itself typo protection and namespassthroughas the example typo.Allowing the value is not the whole fix. The same file puts
extraFeatureGates: [HostDevices]on the KubeVirt CR whenever gpu-operator is enabled, and builds apermittedHostDevicestable under it. Both are VM-facing. The container variant keeps the host driver bound and pinsvfioManageroff, so no GPU reaches a VM: the gate would advertise something the node cannot serve, and the table would name resources no sandbox device plugin registers there. Both now render fordefaultandvgpuonly, andcontainergets the gpu-operator Package with the kubevirt component untouched.The helm-unittest case that pinned the rejection is gone. Two replace it: the Package renders at the container variant with no component overrides, and the KubeVirt wiring stays absent even with
gpu.permittedHostDevicesandgpu.mediatedDevicesConfigurationset. Comments and docs that called the mirroring unconditional on variant, or the container variant unmerged, are fixed in the same pass.No issue for this one, so the history is in the body.
Screenshots
No UI change.
Downstream repositories
Walked the trigger map against the diff. The only file it touches that appears there is
packages/core/platform/values.yaml, and the website hit is not the hand-written key table (that page has no GPU section): it is the documented workaround this change makes obsolete.content/en/docs/next/operations/gpu-container-workloads.mdcurrently tells readers the bundle rejectscontainerand routes them to a hand-crafted Package CR, andvirtualization/gpu.md/vgpu.mddescribe the KubeVirt mirroring as unconditional on variant.Filed as an issue rather than a PR because the docs must not land ahead of this change, and the
v1.6copies describe released behaviour and stay correct as they are.Nothing else matches: no package added, renamed or removed, no
values.schema.jsonor version enum touched, noApplicationDefinitionsemantics, no installer values key, nohack/layout change, no platform variant or bundle recomposed.gpuOperatorVariantappears in none ofccp,ansible-cozystackorterraform-provider-cozystack.Release note
Summary by CodeRabbit
New Features
containerGPU operator variant for workloads that do not require VM passthrough.Documentation