Skip to content

Commit b2e00f6

Browse files
committed
fix(vm-disk): fail on storageClass/source edit of existing disk
Editing a VMDisk's storageClass or source after its DataVolume already exists was silently ignored: dv.yaml reused the existing DataVolume.spec verbatim (DataVolume.spec is immutable in CDI), so the edit rendered as a no-op while the HelmRelease reported success and the CR diverged from the cluster. When a DataVolume already exists, fail with a clear message if the requested storageClass or source differs from what the disk was created with. The source check compares the operator's input, recorded in a vm-disk.cozystack.io/source annotation at creation, rather than the rendered spec.source: the image-clone PVC name carries a chart-version prefix (vm-image- was renamed to vm-default-images-) that drifts across upgrades without any edit, so comparing the rendered form would falsely reject long-lived image disks. A disk created before the annotation existed carries no record and its source check is skipped rather than fired on drift. storageClass is read defensively so an adopted DataVolume without an explicit storageClassName is not misread as "<nil>". An unrecognized source type now fails with an explicit message instead of rendering an empty source. Storage growth is unaffected and still handled by pvc-resize-hook.yaml. Mark source as immutable in values.yaml so the generated schema carries the same x-kubernetes-validations rule already present on storageClass, and document the vm-disk source contract in docs/storage-immutability.md and the chart README. Aggregated-apiserver CEL enforcement is tracked separately in #2657; the chart-level fail is the substantive guard today. Add helm-unittest coverage for the immutability guard (rejection, no-op reuse, legacy image-disk drift, adopted DataVolume, unknown source) and initial-creation source rendering. Fixes #2985 Signed-off-by: Alexey Artamonov <[email protected]>
1 parent eef95f4 commit b2e00f6

9 files changed

Lines changed: 364 additions & 26 deletions

File tree

‎api/apps/v1alpha1/vmdisk/types.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/storage-immutability.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,12 @@ The following charts annotate at least one `storageClass` field as immutable:
2222

2323
The `kubernetes-nodes` chart annotates both its per-pool `storageClass` (the worker node system-disk class, defaulted `replicated`) and its `cluster` field immutable. `storageClass` binds the same PVC-template contract as every other chart in the table, so the plain `self == oldSelf` rule applies for the reason in "Why" above. `cluster` is immutable for a different reason: it wires the pool's CAPI objects to one parent cluster `kubernetes-<cluster>`, and repointing a live pool at another cluster would orphan its running worker VMs rather than migrate them.
2424

25+
## `vm-disk` also freezes `source`
26+
27+
The `vm-disk` chart annotates `source` immutable in addition to `storageClass`, for a mechanism distinct from the PVC-template reasoning above. A `vm-disk` renders a single `DataVolume`, and `DataVolume.spec` is immutable in CDI: once the DataVolume exists, editing `spec.source` (or `spec.storage.storageClassName`) has no effect. The chart therefore looks the existing DataVolume up and reuses its stored `spec`, so an edit to `.Values.source` would otherwise render as a silent no-op while the HelmRelease still reported success — the CR and the cluster diverge with no diagnostic (cozystack/cozystack#2985).
28+
29+
Because the aggregated apiserver does not yet evaluate the CEL rule (see the table above, cozystack/cozystack#2657), `vm-disk` adds a second, chart-level guard that is the substantive enforcement today: `templates/dv.yaml` fails the render when the requested `storageClass` or `source` differs from the existing disk. The `source` comparison keys off the operator's input recorded in a `vm-disk.cozystack.io/source` annotation at creation, not the rendered `spec.source` — the image-clone PVC target name carries a chart-version prefix (`vm-image-` was renamed to `vm-default-images-`) that drifts across upgrades without any operator edit, and comparing the rendered form would falsely reject long-lived image disks. A disk created before that annotation existed carries no record and its `source` check is skipped rather than fired on drift. To actually change `storageClass` or `source`, delete and recreate the disk; size growth is the one mutable dimension and is handled separately by `templates/pvc-resize-hook.yaml`.
30+
2531
If you add a new stateful chart that exposes `storageClass`, annotate it the same way:
2632

2733
```yaml

‎packages/apps/vm-disk/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
A Virtual Machine Disk
44

5-
> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it.
5+
> `storageClass` and `source` are annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. `DataVolume.spec` is immutable in CDI, so the chart also fails the release on an attempted `storageClass`/`source` edit of an existing disk rather than silently reusing the old spec; delete and recreate the disk to change either field.
66
77
## Parameters
88

‎packages/apps/vm-disk/templates/dv.yaml‎

Lines changed: 49 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,69 @@
11
{{- $existingDV := lookup "cdi.kubevirt.io/v1beta1" "DataVolume" .Release.Namespace .Release.Name }}
2+
{{- $sourceAnnotation := "vm-disk.cozystack.io/source" }}
3+
{{- $requestedSource := toJson (.Values.source | default dict) }}
4+
{{- /* Build the desired source spec from .Values so it can be both rendered and compared. */}}
5+
{{- $desiredSource := dict }}
6+
{{- if .Values.source }}
7+
{{- if gt (len .Values.source) 1 }}
8+
{{- fail "Exactly one type of source is expected!" }}
9+
{{- end }}
10+
{{- if hasKey .Values.source "image" }}
11+
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-default-images-%s" (required "A valid .Values.source.image.name entry required!" .Values.source.image.name)) "namespace" "cozy-public") }}
12+
{{- else if hasKey .Values.source "http" }}
13+
{{- $desiredSource = dict "http" (dict "url" (required "A valid .Values.source.http.url entry required!" .Values.source.http.url)) }}
14+
{{- else if hasKey .Values.source "upload" }}
15+
{{- $desiredSource = dict "upload" dict }}
16+
{{- else if hasKey .Values.source "disk" }}
17+
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-disk-%s" (required "A valid .Values.source.disk.name entry required!" .Values.source.disk.name)) "namespace" .Release.Namespace) }}
18+
{{- else }}
19+
{{- fail (printf "Unknown source type: %s. Expected one of: image, http, upload, disk." (keys .Values.source | join ", ")) }}
20+
{{- end }}
21+
{{- else }}
22+
{{- $desiredSource = dict "blank" dict }}
23+
{{- end }}
24+
{{- /*
25+
DataVolume.spec is immutable in CDI, so editing storageClass or source on an
26+
existing disk cannot take effect. Reusing the stored spec silently would let
27+
the CR diverge from the cluster while the release reports success (#2985);
28+
fail loudly instead. Both checks compare the operator's *input*, not the
29+
rendered DataVolume.spec: the image-clone PVC name carries a chart-version
30+
prefix (vm-image- -> vm-default-images-, commit bec35e3a) that drifts across
31+
upgrades without any operator edit, so comparing the rendered spec.source
32+
would falsely reject long-lived image disks. The requested source is recorded
33+
in an annotation at creation and compared back on later renders; a disk
34+
created before that annotation existed carries no record, so its source check
35+
is skipped rather than fired on drift. Size growth is intentionally excluded
36+
and handled by pvc-resize-hook.yaml.
37+
*/}}
38+
{{- if $existingDV }}
39+
{{- $existingSC := (($existingDV.spec.storage | default dict).storageClassName) | default "" | toString }}
40+
{{- if and .Values.storageClass $existingSC (ne $existingSC (.Values.storageClass | toString)) }}
41+
{{- fail (printf "VMDisk storageClass is immutable after creation (current: %q, requested: %q). Delete and recreate the disk to change it." $existingSC .Values.storageClass) }}
42+
{{- end }}
43+
{{- $recordedSource := (index ($existingDV.metadata.annotations | default dict) $sourceAnnotation) | default "" }}
44+
{{- if and $recordedSource (ne $recordedSource $requestedSource) }}
45+
{{- fail (printf "VMDisk source is immutable after creation (current: %s, requested: %s). Delete and recreate the disk to change it." $recordedSource $requestedSource) }}
46+
{{- end }}
47+
{{- end }}
248
apiVersion: cdi.kubevirt.io/v1beta1
349
kind: DataVolume
450
metadata:
551
annotations:
652
{{- if hasKey .Values.source "upload" }}
753
cdi.kubevirt.io/storage.bind.immediate.requested: ""
854
{{- end }}
55+
{{ $sourceAnnotation }}: {{ $requestedSource | quote }}
956
vm-disk.cozystack.io/optical: "{{ .Values.optical }}"
1057
labels:
1158
app.kubernetes.io/instance: {{ .Release.Name }}
1259
name: {{ .Release.Name }}
1360
spec:
1461
{{- if $existingDV }}
1562
{{- toYaml $existingDV.spec | nindent 2 }}
16-
{{- else }}
17-
contentType: kubevirt
18-
{{- if .Values.source }}
19-
{{- if gt (len .Values.source) 1 }}
20-
{{- fail "Exactly one type of source is expected!" }}
21-
{{- end }}
22-
source:
23-
{{- if hasKey .Values.source "image" }}
24-
pvc:
25-
name: vm-default-images-{{ required "A valid .Values.source.image.name entry required!" .Values.source.image.name }}
26-
namespace: cozy-public
27-
{{- else if hasKey .Values.source "http" }}
28-
http:
29-
url: {{ required "A valid .Values.source.http.url entry required!" .Values.source.http.url }}
30-
{{- else if hasKey .Values.source "upload" }}
31-
upload: {}
32-
{{- else if hasKey .Values.source "disk" }}
33-
pvc:
34-
name: vm-disk-{{ required "A valid .Values.source.disk.name entry required!" .Values.source.disk.name }}
35-
namespace: {{ $.Release.Namespace }}
36-
{{- end }}
3763
{{- else }}
64+
contentType: kubevirt
3865
source:
39-
blank: {}
40-
{{- end }}
66+
{{- toYaml $desiredSource | nindent 4 }}
4167
storage:
4268
resources:
4369
requests:
Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
suite: vm-disk DataVolume immutable fields (storageClass/source)
2+
3+
# Regression for #2985: editing storageClass/source on an existing VMDisk used
4+
# to be silently ignored — dv.yaml reused the existing DataVolume.spec verbatim
5+
# and the edit rendered as a no-op while the HelmRelease reported success. The
6+
# template now fails loudly when the operator's requested storageClass/source
7+
# differs from what the disk was created with. The source check compares the
8+
# operator's input against the source recorded in an annotation at creation
9+
# (NOT the rendered spec.source, whose image-clone PVC name carries a
10+
# chart-version prefix that drifts without any edit — commit bec35e3a renamed
11+
# vm-image- to vm-default-images-). Existing DataVolumes are mocked via the
12+
# kubernetes provider so the lookup in dv.yaml resolves without a live cluster.
13+
release:
14+
name: test-vm-disk
15+
namespace: tenant-test
16+
templates:
17+
- templates/dv.yaml
18+
tests:
19+
- it: rejects a storageClass change on an existing DataVolume
20+
set:
21+
storageClass: other
22+
source:
23+
http:
24+
url: https://example.com/original.qcow2
25+
kubernetesProvider:
26+
scheme:
27+
"cdi.kubevirt.io/v1beta1/DataVolume":
28+
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
29+
namespaced: true
30+
objects:
31+
- kind: DataVolume
32+
apiVersion: cdi.kubevirt.io/v1beta1
33+
metadata:
34+
name: test-vm-disk
35+
namespace: tenant-test
36+
annotations:
37+
vm-disk.cozystack.io/source: '{"http":{"url":"https://example.com/original.qcow2"}}'
38+
spec:
39+
source: {http: {url: https://example.com/original.qcow2}}
40+
storage: {resources: {requests: {storage: 5Gi}}, storageClassName: replicated}
41+
asserts:
42+
- failedTemplate:
43+
errorPattern: 'VMDisk storageClass is immutable after creation'
44+
45+
- it: rejects a source change on an existing DataVolume
46+
set:
47+
storageClass: replicated
48+
source:
49+
http:
50+
url: https://example.com/changed.qcow2
51+
kubernetesProvider:
52+
scheme:
53+
"cdi.kubevirt.io/v1beta1/DataVolume":
54+
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
55+
namespaced: true
56+
objects:
57+
- kind: DataVolume
58+
apiVersion: cdi.kubevirt.io/v1beta1
59+
metadata:
60+
name: test-vm-disk
61+
namespace: tenant-test
62+
annotations:
63+
vm-disk.cozystack.io/source: '{"http":{"url":"https://example.com/original.qcow2"}}'
64+
spec:
65+
source: {http: {url: https://example.com/original.qcow2}}
66+
storage: {resources: {requests: {storage: 5Gi}}, storageClassName: replicated}
67+
asserts:
68+
- failedTemplate:
69+
errorPattern: 'VMDisk source is immutable after creation'
70+
71+
- it: reuses the existing spec when storageClass and source are unchanged
72+
set:
73+
storageClass: replicated
74+
source:
75+
http:
76+
url: https://example.com/original.qcow2
77+
kubernetesProvider:
78+
scheme:
79+
"cdi.kubevirt.io/v1beta1/DataVolume":
80+
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
81+
namespaced: true
82+
objects:
83+
- kind: DataVolume
84+
apiVersion: cdi.kubevirt.io/v1beta1
85+
metadata:
86+
name: test-vm-disk
87+
namespace: tenant-test
88+
annotations:
89+
vm-disk.cozystack.io/source: '{"http":{"url":"https://example.com/original.qcow2"}}'
90+
spec:
91+
source: {http: {url: https://example.com/original.qcow2}}
92+
storage: {resources: {requests: {storage: 5Gi}}, storageClassName: replicated}
93+
asserts:
94+
- notFailedTemplate: {}
95+
- equal:
96+
path: spec.storage.storageClassName
97+
value: replicated
98+
- equal:
99+
path: spec.source.http.url
100+
value: https://example.com/original.qcow2
101+
102+
- it: allows a storage grow while immutable fields are unchanged
103+
set:
104+
storageClass: replicated
105+
storage: 10Gi
106+
source:
107+
http:
108+
url: https://example.com/original.qcow2
109+
kubernetesProvider:
110+
scheme:
111+
"cdi.kubevirt.io/v1beta1/DataVolume":
112+
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
113+
namespaced: true
114+
objects:
115+
- kind: DataVolume
116+
apiVersion: cdi.kubevirt.io/v1beta1
117+
metadata:
118+
name: test-vm-disk
119+
namespace: tenant-test
120+
annotations:
121+
vm-disk.cozystack.io/source: '{"http":{"url":"https://example.com/original.qcow2"}}'
122+
spec:
123+
source: {http: {url: https://example.com/original.qcow2}}
124+
storage: {resources: {requests: {storage: 5Gi}}, storageClassName: replicated}
125+
asserts:
126+
- notFailedTemplate: {}
127+
128+
- it: treats a matching image clone as a no-op (rendered PVC name round-trips)
129+
set:
130+
storageClass: replicated
131+
source:
132+
image:
133+
name: ubuntu
134+
kubernetesProvider:
135+
scheme:
136+
"cdi.kubevirt.io/v1beta1/DataVolume":
137+
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
138+
namespaced: true
139+
objects:
140+
- kind: DataVolume
141+
apiVersion: cdi.kubevirt.io/v1beta1
142+
metadata:
143+
name: test-vm-disk
144+
namespace: tenant-test
145+
annotations:
146+
vm-disk.cozystack.io/source: '{"image":{"name":"ubuntu"}}'
147+
spec:
148+
source: {pvc: {name: vm-default-images-ubuntu, namespace: cozy-public}}
149+
storage: {resources: {requests: {storage: 5Gi}}, storageClassName: replicated}
150+
asserts:
151+
- notFailedTemplate: {}
152+
- equal:
153+
path: spec.source.pvc.name
154+
value: vm-default-images-ubuntu
155+
156+
- it: does not falsely reject a legacy image disk whose PVC name predates the rename
157+
# CRITICAL regression (#3969 review): a disk cloned before commit bec35e3a
158+
# carries spec.source.pvc.name=vm-image-<name> forever, and has no source
159+
# annotation. Comparing the rendered vm-default-images-<name> against it must
160+
# NOT fire — the operator changed nothing.
161+
set:
162+
storageClass: replicated
163+
source:
164+
image:
165+
name: ubuntu
166+
kubernetesProvider:
167+
scheme:
168+
"cdi.kubevirt.io/v1beta1/DataVolume":
169+
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
170+
namespaced: true
171+
objects:
172+
- kind: DataVolume
173+
apiVersion: cdi.kubevirt.io/v1beta1
174+
metadata:
175+
name: test-vm-disk
176+
namespace: tenant-test
177+
spec:
178+
source: {pvc: {name: vm-image-ubuntu, namespace: cozy-public}}
179+
storage: {resources: {requests: {storage: 5Gi}}, storageClassName: replicated}
180+
asserts:
181+
- notFailedTemplate: {}
182+
- equal:
183+
path: spec.source.pvc.name
184+
value: vm-image-ubuntu
185+
186+
- it: does not nil-deref or falsely reject a DataVolume adopted without storageClassName
187+
# MINOR (#3969 review): an out-of-band/adopted DataVolume may omit
188+
# spec.storage.storageClassName (or use spec.pvc). The read must not render
189+
# "<nil>" and fire against the default, and must not panic.
190+
set:
191+
storageClass: other
192+
source:
193+
http:
194+
url: https://example.com/original.qcow2
195+
kubernetesProvider:
196+
scheme:
197+
"cdi.kubevirt.io/v1beta1/DataVolume":
198+
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
199+
namespaced: true
200+
objects:
201+
- kind: DataVolume
202+
apiVersion: cdi.kubevirt.io/v1beta1
203+
metadata:
204+
name: test-vm-disk
205+
namespace: tenant-test
206+
spec:
207+
source: {http: {url: https://example.com/original.qcow2}}
208+
storage: {resources: {requests: {storage: 5Gi}}}
209+
asserts:
210+
- notFailedTemplate: {}

0 commit comments

Comments
 (0)