Skip to content

Commit a616850

Browse files
committed
fix(vm-disk): fail on storageClass/source edit of existing disk
Editing a VMDisk's storageClass or source after its DataVolume already exists was silently ignored: dv.yaml reused the existing DataVolume.spec verbatim (DataVolume.spec is immutable in CDI), so the edit rendered as a no-op while the HelmRelease reported success and the CR diverged from the cluster. When a DataVolume already exists, fail with a clear message if the requested storageClass or source differs from what the disk was created with. Both compare the operator's input recorded in an annotation at creation (vm-disk.cozystack.io/storage-class and .../source), written only at creation and preserved verbatim afterwards — never overwritten with an unapplied edit. Recording the input rather than reading the live object matters on both axes: the image-clone PVC name carries a chart-version prefix (vm-image- was renamed to vm-default-images-, commit bec35e3) that drifts across upgrades with no edit, and the live PVC storageClassName of an adopted or foreign DataVolume may legitimately differ from the CR default without being a change. The annotation key presence (not its value) marks a recorded class, so a disk created with the cluster default "" is recorded and still caught if later changed to a concrete class. Disks with no annotation (created earlier, or out-of-band) are handled conservatively so a working release never deadlocks: source is compared against the stored spec.source, with the exemption keyed on the stored source being an image clone (a pvc in cozy-public), not on the requested type, so a change away from a non-image source is still caught; storageClass has no safe live fallback, so an un-annotated disk is not guarded here (apiserver CEL, #2657, is the eventual guard). An unrecognized source type now fails explicitly. Storage growth is unaffected and still handled by pvc-resize-hook.yaml. Mark source as immutable in values.yaml so the generated schema carries the same x-kubernetes-validations rule already present on storageClass, and document the contract, the exemption, and the source-annotation schema coupling in docs/storage-immutability.md and the chart README. Add helm-unittest coverage for the immutability guard (rejection incl. clearing storageClass to empty and changing a non-image disk to image, no-op reuse with annotations preserved, legacy image-disk drift, annotation non-poisoning, un-annotated storageClass skip, adopted default-class disk, unknown source) and initial-creation source rendering. Fixes #2985 Signed-off-by: Alexey Artamonov <[email protected]>
1 parent eef95f4 commit a616850

9 files changed

Lines changed: 689 additions & 26 deletions

File tree

‎api/apps/v1alpha1/vmdisk/types.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/storage-immutability.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,16 @@ The following charts annotate at least one `storageClass` field as immutable:
2222

2323
The `kubernetes-nodes` chart annotates both its per-pool `storageClass` (the worker node system-disk class, defaulted `replicated`) and its `cluster` field immutable. `storageClass` binds the same PVC-template contract as every other chart in the table, so the plain `self == oldSelf` rule applies for the reason in "Why" above. `cluster` is immutable for a different reason: it wires the pool's CAPI objects to one parent cluster `kubernetes-<cluster>`, and repointing a live pool at another cluster would orphan its running worker VMs rather than migrate them.
2424

25+
## `vm-disk` also freezes `source`
26+
27+
The `vm-disk` chart annotates `source` immutable in addition to `storageClass`, for a mechanism distinct from the PVC-template reasoning above. A `vm-disk` renders a single `DataVolume`, and `DataVolume.spec` is immutable in CDI: once the DataVolume exists, editing `spec.source` (or `spec.storage.storageClassName`) has no effect. The chart therefore looks the existing DataVolume up and reuses its stored `spec`, so an edit to `.Values.source` would otherwise render as a silent no-op while the HelmRelease still reported success — the CR and the cluster diverge with no diagnostic (cozystack/cozystack#2985).
28+
29+
Because the aggregated apiserver does not yet evaluate the CEL rule (see the table above, cozystack/cozystack#2657), `vm-disk` adds a second, chart-level guard that is the substantive enforcement today: `templates/dv.yaml` fails the render when the requested `storageClass` or `source` differs from what the disk was created with. Both checks compare the operator's input recorded in an annotation at creation — `vm-disk.cozystack.io/storage-class` and `vm-disk.cozystack.io/source` — written only at creation and preserved verbatim afterward, never overwritten with an unapplied edit. The storage-class annotation is written even for the cluster default (`""`), and its key presence — not its value — marks a recorded class, so a disk created with `""` is still caught if later changed to a concrete class. Recording the input rather than reading the live object matters on both axes: the image-clone PVC name carries a chart-version prefix (`vm-image-` was renamed to `vm-default-images-`) that drifts across upgrades with no operator edit, and the live PVC's `storageClassName` on an adopted or foreign DataVolume may legitimately differ from the CR's default without being a change.
30+
31+
Disks created before these annotations existed (or out-of-band) are handled conservatively so a working release never deadlocks. For `source`, such a disk is compared against its stored `spec.source` directly, with the exemption keyed on the **stored** source being an image clone (a `pvc` in namespace `cozy-public`) — not on the requested type — so a change away from a non-image source is still caught, while a stored image clone is left alone because its rendered PVC name would drift. For `storageClass` there is no safe live fallback (the CR default need not reflect an adopted disk's real class), so an un-annotated disk is simply not guarded here; apiserver CEL (cozystack/cozystack#2657) is the eventual guard for those. To actually change `storageClass` or `source`, delete and recreate the disk; size growth is the one mutable dimension and is handled separately by `templates/pvc-resize-hook.yaml`.
32+
33+
The `source` annotation is a byte-for-byte `toJson` of `.Values.source`, so it is coupled to the exact serialization of the source schema. Adding a subfield with a schema default (which the apiserver would stamp onto every CR) would make the recorded and requested JSON diverge on an untouched disk; such a change needs an annotation-migration story rather than a bare schema bump.
34+
2535
If you add a new stateful chart that exposes `storageClass`, annotate it the same way:
2636

2737
```yaml

‎packages/apps/vm-disk/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
A Virtual Machine Disk
44

5-
> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it.
5+
> `storageClass` and `source` are annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. `DataVolume.spec` is immutable in CDI, so the chart also fails the release on an attempted `storageClass`/`source` edit of an existing disk rather than silently reusing the old spec; delete and recreate the disk to change either field.
66
77
## Parameters
88

‎packages/apps/vm-disk/templates/dv.yaml‎

Lines changed: 93 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,113 @@
11
{{- $existingDV := lookup "cdi.kubevirt.io/v1beta1" "DataVolume" .Release.Namespace .Release.Name }}
2+
{{- $sourceAnnotation := "vm-disk.cozystack.io/source" }}
3+
{{- $storageClassAnnotation := "vm-disk.cozystack.io/storage-class" }}
4+
{{- $requestedSource := toJson (.Values.source | default dict) }}
5+
{{- $requestedSC := (.Values.storageClass | default "") | toString }}
6+
{{- /* Build the desired source spec from .Values so it can be both rendered and compared. */}}
7+
{{- $desiredSource := dict }}
8+
{{- if .Values.source }}
9+
{{- if gt (len .Values.source) 1 }}
10+
{{- fail "Exactly one type of source is expected!" }}
11+
{{- end }}
12+
{{- if hasKey .Values.source "image" }}
13+
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-default-images-%s" (required "A valid .Values.source.image.name entry required!" .Values.source.image.name)) "namespace" "cozy-public") }}
14+
{{- else if hasKey .Values.source "http" }}
15+
{{- $desiredSource = dict "http" (dict "url" (required "A valid .Values.source.http.url entry required!" .Values.source.http.url)) }}
16+
{{- else if hasKey .Values.source "upload" }}
17+
{{- $desiredSource = dict "upload" dict }}
18+
{{- else if hasKey .Values.source "disk" }}
19+
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-disk-%s" (required "A valid .Values.source.disk.name entry required!" .Values.source.disk.name)) "namespace" .Release.Namespace) }}
20+
{{- else }}
21+
{{- fail (printf "Unknown source type: %s. Expected one of: image, http, upload, disk." (keys .Values.source | join ", ")) }}
22+
{{- end }}
23+
{{- else }}
24+
{{- $desiredSource = dict "blank" dict }}
25+
{{- end }}
26+
{{- /*
27+
DataVolume.spec is immutable in CDI, so editing storageClass or source on an
28+
existing disk cannot take effect. Reusing the stored spec silently would let
29+
the CR diverge from the cluster while the release reports success (#2985);
30+
fail loudly instead.
31+
32+
storageClass: the operator input is recorded in an annotation at creation and
33+
compared only against that. Reading the live PVC/DV class instead would break
34+
adopted or foreign DataVolumes — a disk on one class with the CR left at the
35+
default (replicated) was Ready before and must not deadlock the render. The
36+
annotation KEY presence (not its value) marks a recorded class, so a disk
37+
created with the cluster default ("") is recorded as "" and still guarded
38+
against a later change to a concrete class; only a disk with no annotation at
39+
all (created before it existed, or out-of-band) is left unguarded here, with
40+
apiserver CEL (#2657) as the eventual guard for those.
41+
42+
source: the operator input is recorded in an annotation at creation and
43+
compared back on later renders. Comparing the rendered spec.source instead
44+
would be wrong for image clones — the PVC name carries a chart-version prefix
45+
(vm-image- -> vm-default-images-, commit bec35e3a) that drifts across upgrades
46+
with no operator edit. A disk that predates the annotation is matched against
47+
its stored spec.source directly; the exemption is keyed on the STORED source
48+
being an image clone (a pvc in namespace cozy-public), not on the requested
49+
type, so a change away from a non-image source is still caught. The annotation
50+
is written only at creation and preserved verbatim afterward, never
51+
overwritten with an unapplied edit.
52+
53+
A byte-for-byte toJson compare couples the source check to the exact schema
54+
serialization; a future Source subfield with a kubebuilder default would need
55+
an annotation migration (see docs/storage-immutability.md).
56+
57+
Size growth is intentionally excluded and handled by pvc-resize-hook.yaml.
58+
*/}}
59+
{{- if $existingDV }}
60+
{{- $existingAnnotations := $existingDV.metadata.annotations | default dict }}
61+
{{- if hasKey $existingAnnotations $storageClassAnnotation }}
62+
{{- $recordedSC := (index $existingAnnotations $storageClassAnnotation) | toString }}
63+
{{- if ne $recordedSC $requestedSC }}
64+
{{- fail (printf "VMDisk storageClass is immutable after creation (current: %q, requested: %q). Delete and recreate the disk to change it." $recordedSC $requestedSC) }}
65+
{{- end }}
66+
{{- end }}
67+
{{- $recordedSource := (index $existingAnnotations $sourceAnnotation) | default "" }}
68+
{{- if $recordedSource }}
69+
{{- if ne $recordedSource $requestedSource }}
70+
{{- fail (printf "VMDisk source is immutable after creation (current: %s, requested: %s). Delete and recreate the disk to change it." $recordedSource $requestedSource) }}
71+
{{- end }}
72+
{{- else }}
73+
{{- $storedSource := $existingDV.spec.source | default dict }}
74+
{{- $storedIsImageClone := and $storedSource.pvc (eq (($storedSource.pvc).namespace | toString) "cozy-public") }}
75+
{{- if not $storedIsImageClone }}
76+
{{- if ne (toYaml $storedSource) (toYaml $desiredSource) }}
77+
{{- fail (printf "VMDisk source is immutable after creation (current: %s, requested: %s). Delete and recreate the disk to change it." (toJson $storedSource) $requestedSource) }}
78+
{{- end }}
79+
{{- end }}
80+
{{- end }}
81+
{{- end }}
282
apiVersion: cdi.kubevirt.io/v1beta1
383
kind: DataVolume
484
metadata:
585
annotations:
686
{{- if hasKey .Values.source "upload" }}
787
cdi.kubevirt.io/storage.bind.immediate.requested: ""
888
{{- end }}
89+
{{- if $existingDV }}
90+
{{- with (index ($existingDV.metadata.annotations | default dict) $sourceAnnotation) }}
91+
{{ $sourceAnnotation }}: {{ . | quote }}
92+
{{- end }}
93+
{{- if hasKey ($existingDV.metadata.annotations | default dict) $storageClassAnnotation }}
94+
{{ $storageClassAnnotation }}: {{ index ($existingDV.metadata.annotations | default dict) $storageClassAnnotation | quote }}
95+
{{- end }}
96+
{{- else }}
97+
{{ $sourceAnnotation }}: {{ $requestedSource | quote }}
98+
{{ $storageClassAnnotation }}: {{ $requestedSC | quote }}
99+
{{- end }}
9100
vm-disk.cozystack.io/optical: "{{ .Values.optical }}"
10101
labels:
11102
app.kubernetes.io/instance: {{ .Release.Name }}
12103
name: {{ .Release.Name }}
13104
spec:
14105
{{- if $existingDV }}
15106
{{- toYaml $existingDV.spec | nindent 2 }}
16-
{{- else }}
17-
contentType: kubevirt
18-
{{- if .Values.source }}
19-
{{- if gt (len .Values.source) 1 }}
20-
{{- fail "Exactly one type of source is expected!" }}
21-
{{- end }}
22-
source:
23-
{{- if hasKey .Values.source "image" }}
24-
pvc:
25-
name: vm-default-images-{{ required "A valid .Values.source.image.name entry required!" .Values.source.image.name }}
26-
namespace: cozy-public
27-
{{- else if hasKey .Values.source "http" }}
28-
http:
29-
url: {{ required "A valid .Values.source.http.url entry required!" .Values.source.http.url }}
30-
{{- else if hasKey .Values.source "upload" }}
31-
upload: {}
32-
{{- else if hasKey .Values.source "disk" }}
33-
pvc:
34-
name: vm-disk-{{ required "A valid .Values.source.disk.name entry required!" .Values.source.disk.name }}
35-
namespace: {{ $.Release.Namespace }}
36-
{{- end }}
37107
{{- else }}
108+
contentType: kubevirt
38109
source:
39-
blank: {}
40-
{{- end }}
110+
{{- toYaml $desiredSource | nindent 4 }}
41111
storage:
42112
resources:
43113
requests:

0 commit comments

Comments
 (0)