Skip to content

Commit cd66a2b

Browse files
committed
fix(vm-disk): fail on storageClass/source edit of existing disk
Editing a VMDisk's storageClass or source after its DataVolume already exists was silently ignored: dv.yaml reused the existing DataVolume.spec verbatim (DataVolume.spec is immutable in CDI), so the edit rendered as a no-op while the HelmRelease reported success and the CR diverged from the cluster. When a DataVolume already exists, fail with a clear message if the requested storageClass or source differs from what the disk was created with. Both compare the operator's input recorded in an annotation at creation (vm-disk.cozystack.io/storage-class and .../source), written only at creation and preserved verbatim afterwards — never overwritten with an unapplied edit. Comparing recorded input rather than the rendered spec matters because the image-clone PVC name carries a chart-version prefix (vm-image- was renamed to vm-default-images-, commit bec35e3) that drifts across upgrades without any edit, and because clearing storageClass to "" (cluster default) must still be caught. A disk that predates the annotation falls back: storageClass to the bound PVC's class (then the DV spec), compared only when the request is non-empty so a default-class disk is not misread as a change; source to the stored spec.source for every type except image, whose prefix drift makes it the one documented exemption. An unrecognized source type now fails explicitly instead of rendering an empty source. Storage growth is unaffected and still handled by pvc-resize-hook.yaml. Mark source as immutable in values.yaml so the generated schema carries the same x-kubernetes-validations rule already present on storageClass, and document the vm-disk source contract in docs/storage-immutability.md and the chart README. Aggregated-apiserver CEL enforcement is tracked separately in #2657; the chart-level fail is the substantive guard today. Add helm-unittest coverage for the immutability guard (rejection incl. clearing storageClass to empty, no-op reuse with annotations preserved, legacy image-disk drift, legacy non-image fallback, annotation non-poisoning, PVC-backed and default-class storageClass on adopted disks, unknown source) and initial-creation source rendering. Fixes #2985 Signed-off-by: Alexey Artamonov <[email protected]>
1 parent eef95f4 commit cd66a2b

9 files changed

Lines changed: 626 additions & 26 deletions

File tree

‎api/apps/v1alpha1/vmdisk/types.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/storage-immutability.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,12 @@ The following charts annotate at least one `storageClass` field as immutable:
2222

2323
The `kubernetes-nodes` chart annotates both its per-pool `storageClass` (the worker node system-disk class, defaulted `replicated`) and its `cluster` field immutable. `storageClass` binds the same PVC-template contract as every other chart in the table, so the plain `self == oldSelf` rule applies for the reason in "Why" above. `cluster` is immutable for a different reason: it wires the pool's CAPI objects to one parent cluster `kubernetes-<cluster>`, and repointing a live pool at another cluster would orphan its running worker VMs rather than migrate them.
2424

25+
## `vm-disk` also freezes `source`
26+
27+
The `vm-disk` chart annotates `source` immutable in addition to `storageClass`, for a mechanism distinct from the PVC-template reasoning above. A `vm-disk` renders a single `DataVolume`, and `DataVolume.spec` is immutable in CDI: once the DataVolume exists, editing `spec.source` (or `spec.storage.storageClassName`) has no effect. The chart therefore looks the existing DataVolume up and reuses its stored `spec`, so an edit to `.Values.source` would otherwise render as a silent no-op while the HelmRelease still reported success — the CR and the cluster diverge with no diagnostic (cozystack/cozystack#2985).
28+
29+
Because the aggregated apiserver does not yet evaluate the CEL rule (see the table above, cozystack/cozystack#2657), `vm-disk` adds a second, chart-level guard that is the substantive enforcement today: `templates/dv.yaml` fails the render when the requested `storageClass` or `source` differs from the existing disk. The `source` comparison keys off the operator's input recorded in a `vm-disk.cozystack.io/source` annotation at creation, not the rendered `spec.source` — the image-clone PVC target name carries a chart-version prefix (`vm-image-` was renamed to `vm-default-images-`) that drifts across upgrades without any operator edit, and comparing the rendered form would falsely reject long-lived image disks. A disk created before that annotation existed is instead compared against its stored `spec.source` directly for every source type except image clones: `http`, `upload`, `disk` and `blank` all reproduce the stored spec exactly, so an edit to one of them is still caught, while the image case is skipped rather than fired on drift. `storageClass` is compared against the bound PVC's class (falling back to the DataVolume spec) and is not subject to that exemption. To actually change `storageClass` or `source`, delete and recreate the disk; size growth is the one mutable dimension and is handled separately by `templates/pvc-resize-hook.yaml`.
30+
2531
If you add a new stateful chart that exposes `storageClass`, annotate it the same way:
2632

2733
```yaml

‎packages/apps/vm-disk/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
A Virtual Machine Disk
44

5-
> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it.
5+
> `storageClass` and `source` are annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. `DataVolume.spec` is immutable in CDI, so the chart also fails the release on an attempted `storageClass`/`source` edit of an existing disk rather than silently reusing the old spec; delete and recreate the disk to change either field.
66
77
## Parameters
88

‎packages/apps/vm-disk/templates/dv.yaml‎

Lines changed: 98 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,118 @@
11
{{- $existingDV := lookup "cdi.kubevirt.io/v1beta1" "DataVolume" .Release.Namespace .Release.Name }}
2+
{{- $existingPVC := lookup "v1" "PersistentVolumeClaim" .Release.Namespace .Release.Name }}
3+
{{- $sourceAnnotation := "vm-disk.cozystack.io/source" }}
4+
{{- $storageClassAnnotation := "vm-disk.cozystack.io/storage-class" }}
5+
{{- $requestedSource := toJson (.Values.source | default dict) }}
6+
{{- $requestedSC := (.Values.storageClass | default "") | toString }}
7+
{{- /* Build the desired source spec from .Values so it can be both rendered and compared. */}}
8+
{{- $desiredSource := dict }}
9+
{{- if .Values.source }}
10+
{{- if gt (len .Values.source) 1 }}
11+
{{- fail "Exactly one type of source is expected!" }}
12+
{{- end }}
13+
{{- if hasKey .Values.source "image" }}
14+
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-default-images-%s" (required "A valid .Values.source.image.name entry required!" .Values.source.image.name)) "namespace" "cozy-public") }}
15+
{{- else if hasKey .Values.source "http" }}
16+
{{- $desiredSource = dict "http" (dict "url" (required "A valid .Values.source.http.url entry required!" .Values.source.http.url)) }}
17+
{{- else if hasKey .Values.source "upload" }}
18+
{{- $desiredSource = dict "upload" dict }}
19+
{{- else if hasKey .Values.source "disk" }}
20+
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-disk-%s" (required "A valid .Values.source.disk.name entry required!" .Values.source.disk.name)) "namespace" .Release.Namespace) }}
21+
{{- else }}
22+
{{- fail (printf "Unknown source type: %s. Expected one of: image, http, upload, disk." (keys .Values.source | join ", ")) }}
23+
{{- end }}
24+
{{- else }}
25+
{{- $desiredSource = dict "blank" dict }}
26+
{{- end }}
27+
{{- /*
28+
DataVolume.spec is immutable in CDI, so editing storageClass or source on an
29+
existing disk cannot take effect. Reusing the stored spec silently would let
30+
the CR diverge from the cluster while the release reports success (#2985);
31+
fail loudly instead.
32+
33+
storageClass: like source, the operator input is recorded in an annotation at
34+
creation and compared back on later renders, so clearing a class to "" (which
35+
binds the PVC to the cluster default) is still caught. A disk that predates the
36+
annotation falls back to the effective class read from the bound PVC
37+
(authoritative, and present even for a DataVolume created via spec.pvc or
38+
without an explicit storageClassName), then the DataVolume spec; that fallback
39+
compares only when the requested value is non-empty, because a legacy disk
40+
created with "" binds its PVC to a concrete default class that must not be read
41+
as a change. When neither records a class the check is skipped rather than
42+
misreading it as "<nil>".
43+
44+
source: the operator's input is recorded in an annotation at creation and
45+
compared back on later renders. Comparing the rendered spec.source instead
46+
would be wrong for image clones — the PVC name carries a chart-version prefix
47+
(vm-image- -> vm-default-images-, commit bec35e3a) that drifts across upgrades
48+
with no operator edit. A disk that predates the annotation is matched against
49+
its stored spec.source directly for every source type except image (all of
50+
which reproduce the stored spec exactly); the image case is the one
51+
documented exemption. The annotation is written only at creation and preserved
52+
verbatim afterwards, never overwritten with an unapplied edit.
53+
54+
Size growth is intentionally excluded and handled by pvc-resize-hook.yaml.
55+
*/}}
56+
{{- if $existingDV }}
57+
{{- $recordedSC := (index ($existingDV.metadata.annotations | default dict) $storageClassAnnotation) | default "" }}
58+
{{- if $recordedSC }}
59+
{{- if ne $recordedSC $requestedSC }}
60+
{{- fail (printf "VMDisk storageClass is immutable after creation (current: %q, requested: %q). Delete and recreate the disk to change it." $recordedSC $requestedSC) }}
61+
{{- end }}
62+
{{- else }}
63+
{{- $existingSC := "" }}
64+
{{- if $existingPVC }}
65+
{{- $existingSC = ($existingPVC.spec.storageClassName | default "" | toString) }}
66+
{{- end }}
67+
{{- if not $existingSC }}
68+
{{- $existingSC = ((($existingDV.spec.storage | default dict).storageClassName) | default "" | toString) }}
69+
{{- end }}
70+
{{- if and $requestedSC $existingSC (ne $existingSC $requestedSC) }}
71+
{{- fail (printf "VMDisk storageClass is immutable after creation (current: %q, requested: %q). Delete and recreate the disk to change it." $existingSC $requestedSC) }}
72+
{{- end }}
73+
{{- end }}
74+
{{- $recordedSource := (index ($existingDV.metadata.annotations | default dict) $sourceAnnotation) | default "" }}
75+
{{- if $recordedSource }}
76+
{{- if ne $recordedSource $requestedSource }}
77+
{{- fail (printf "VMDisk source is immutable after creation (current: %s, requested: %s). Delete and recreate the disk to change it." $recordedSource $requestedSource) }}
78+
{{- end }}
79+
{{- else if not (hasKey .Values.source "image") }}
80+
{{- if ne (toYaml $existingDV.spec.source) (toYaml $desiredSource) }}
81+
{{- fail (printf "VMDisk source is immutable after creation (current: %s, requested: %s). Delete and recreate the disk to change it." (toJson $existingDV.spec.source) $requestedSource) }}
82+
{{- end }}
83+
{{- end }}
84+
{{- end }}
285
apiVersion: cdi.kubevirt.io/v1beta1
386
kind: DataVolume
487
metadata:
588
annotations:
689
{{- if hasKey .Values.source "upload" }}
790
cdi.kubevirt.io/storage.bind.immediate.requested: ""
891
{{- end }}
92+
{{- if $existingDV }}
93+
{{- with (index ($existingDV.metadata.annotations | default dict) $sourceAnnotation) }}
94+
{{ $sourceAnnotation }}: {{ . | quote }}
95+
{{- end }}
96+
{{- with (index ($existingDV.metadata.annotations | default dict) $storageClassAnnotation) }}
97+
{{ $storageClassAnnotation }}: {{ . | quote }}
98+
{{- end }}
99+
{{- else }}
100+
{{ $sourceAnnotation }}: {{ $requestedSource | quote }}
101+
{{- with $requestedSC }}
102+
{{ $storageClassAnnotation }}: {{ . | quote }}
103+
{{- end }}
104+
{{- end }}
9105
vm-disk.cozystack.io/optical: "{{ .Values.optical }}"
10106
labels:
11107
app.kubernetes.io/instance: {{ .Release.Name }}
12108
name: {{ .Release.Name }}
13109
spec:
14110
{{- if $existingDV }}
15111
{{- toYaml $existingDV.spec | nindent 2 }}
16-
{{- else }}
17-
contentType: kubevirt
18-
{{- if .Values.source }}
19-
{{- if gt (len .Values.source) 1 }}
20-
{{- fail "Exactly one type of source is expected!" }}
21-
{{- end }}
22-
source:
23-
{{- if hasKey .Values.source "image" }}
24-
pvc:
25-
name: vm-default-images-{{ required "A valid .Values.source.image.name entry required!" .Values.source.image.name }}
26-
namespace: cozy-public
27-
{{- else if hasKey .Values.source "http" }}
28-
http:
29-
url: {{ required "A valid .Values.source.http.url entry required!" .Values.source.http.url }}
30-
{{- else if hasKey .Values.source "upload" }}
31-
upload: {}
32-
{{- else if hasKey .Values.source "disk" }}
33-
pvc:
34-
name: vm-disk-{{ required "A valid .Values.source.disk.name entry required!" .Values.source.disk.name }}
35-
namespace: {{ $.Release.Namespace }}
36-
{{- end }}
37112
{{- else }}
113+
contentType: kubevirt
38114
source:
39-
blank: {}
40-
{{- end }}
115+
{{- toYaml $desiredSource | nindent 4 }}
41116
storage:
42117
resources:
43118
requests:

0 commit comments

Comments
 (0)