Skip to content

Commit 06624ba

Browse files
committed
fix(vm-disk): fail on storageClass/source edit of existing disk
Editing a VMDisk's storageClass or source after its DataVolume already exists was silently ignored: dv.yaml reused the existing DataVolume.spec verbatim (DataVolume.spec is immutable in CDI), so the edit rendered as a no-op while the HelmRelease reported success and the CR diverged from the cluster. When a DataVolume already exists, fail with a clear message if the requested storageClass or source differs from what the disk was created with. Both compare the operator's input recorded in an annotation at creation (vm-disk.cozystack.io/storage-class and .../source), written only at creation and preserved verbatim afterwards — never overwritten with an unapplied edit. Recording the input rather than reading the live object matters on both axes: the image-clone PVC name carries a chart-version prefix (vm-image- was renamed to vm-default-images-, commit bec35e3) that drifts across upgrades with no edit, and the live PVC storageClassName of an adopted or foreign DataVolume may legitimately differ from the CR default without being a change. The annotation key presence (not its value) marks a recorded class, so a disk created with the cluster default "" is recorded and still caught if later changed to a concrete class. Disks with no annotation (created earlier, or out-of-band) are handled conservatively so a working release never deadlocks: source is compared against the stored spec.source only when that stored source is a shape the chart itself produces, so an adopted DataVolume using spec.sourceRef or a non-chart source (registry, snapshot, http+secretRef, ...) is left alone rather than deadlocked. The image-clone exemption fires only when the stored source is a cozy-public pvc AND the request is itself an image, so a change away from an image source to another type is still caught. storageClass has no safe live fallback, so an un-annotated disk is not guarded here (apiserver CEL, #2657, is the eventual guard). An unrecognized source type now fails explicitly. Storage growth is unaffected and still handled by pvc-resize-hook.yaml. Mark source as immutable in values.yaml so the generated schema carries the same x-kubernetes-validations rule already present on storageClass, and document the contract, the exemption, and the source-annotation schema coupling in docs/storage-immutability.md and the chart README. Add helm-unittest coverage for the immutability guard (rejection incl. clearing storageClass to empty and changing a non-image disk to image, no-op reuse with annotations preserved, legacy image-disk drift, annotation non-poisoning, un-annotated storageClass skip, adopted default-class disk, unknown source) and initial-creation source rendering. Fixes #2985 Signed-off-by: Alexey Artamonov <[email protected]>
1 parent eef95f4 commit 06624ba

9 files changed

Lines changed: 1151 additions & 27 deletions

File tree

‎api/apps/v1alpha1/vmdisk/types.go‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/storage-immutability.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,18 @@ The following charts annotate at least one `storageClass` field as immutable:
2222

2323
The `kubernetes-nodes` chart annotates both its per-pool `storageClass` (the worker node system-disk class, defaulted `replicated`) and its `cluster` field immutable. `storageClass` binds the same PVC-template contract as every other chart in the table, so the plain `self == oldSelf` rule applies for the reason in "Why" above. `cluster` is immutable for a different reason: it wires the pool's CAPI objects to one parent cluster `kubernetes-<cluster>`, and repointing a live pool at another cluster would orphan its running worker VMs rather than migrate them.
2424

25+
## `vm-disk` also freezes `source`
26+
27+
The `vm-disk` chart annotates `source` immutable in addition to `storageClass`, for a mechanism distinct from the PVC-template reasoning above. A `vm-disk` renders a single `DataVolume`, and `DataVolume.spec` is immutable in CDI: once the DataVolume exists, editing `spec.source` (or `spec.storage.storageClassName`) has no effect. The chart therefore looks the existing DataVolume up and reuses its stored `spec`, so an edit to `.Values.source` would otherwise render as a silent no-op while the HelmRelease still reported success — the CR and the cluster diverge with no diagnostic (cozystack/cozystack#2985).
28+
29+
Because the aggregated apiserver does not yet evaluate the CEL rule (see the table above, cozystack/cozystack#2657), `vm-disk` adds a second, chart-level guard that is the substantive enforcement today: `templates/dv.yaml` fails the render when the requested `storageClass` or `source` differs from what the disk was created with. Both checks compare the operator's input recorded in an annotation at creation — `vm-disk.cozystack.io/storage-class` and `vm-disk.cozystack.io/source` — written only at creation and preserved verbatim afterward, never overwritten with an unapplied edit. The storage-class annotation is written even for the cluster default (`""`), and its key presence — not its value — marks a recorded class, so a disk created with `""` is still caught if later changed to a concrete class. Recording the input rather than reading the live object matters on both axes: the image-clone PVC name carries a chart-version prefix (`vm-image-` was renamed to `vm-default-images-`) that drifts across upgrades with no operator edit, and the live PVC's `storageClassName` on an adopted or foreign DataVolume may legitimately differ from the CR's default without being a change.
30+
31+
Disks created before these annotations existed (or out-of-band) are handled conservatively so a working release never deadlocks. For `source`, such a disk is compared against its stored `spec.source` directly, but only when that stored source is a shape the chart itself produces (`blank`, `http` with just a `url`, `upload`, or a `pvc` with just `name`/`namespace`). An adopted DataVolume using `spec.sourceRef`, or a source type the chart cannot render (`registry`, `snapshot`, `imageio`, `http` with a `secretRef`, a foreign-namespace `pvc`), has no comparable provenance and is left alone rather than failed on a mismatch. The image-clone exemption fires only when the stored source is a `cozy-public` `pvc` **and** the request is itself an image — the one case where the `vm-image-` → `vm-default-images-` prefix drift is indistinguishable from a real image-name change; a change away from an image source to another type is still caught. For `storageClass` there is no safe live fallback (the CR default need not reflect an adopted disk's real class), so an un-annotated disk is simply not guarded here; apiserver CEL (cozystack/cozystack#2657) is the eventual guard for those. To actually change `storageClass` or `source`, delete and recreate the disk; size growth is the one mutable dimension and is handled separately by `templates/pvc-resize-hook.yaml`.
32+
33+
The `storageClass` annotation records the effective value applied at creation, including when it came from the chart's `values.yaml` default (`replicated`) rather than an explicit operator value. Applications created through the aggregated apiserver have the default stamped into the CR, so it stays stable; but a direct HelmRelease consumer that omits `storageClass` records the chart default of the day, so changing that default in a future chart version would make every such untouched release start failing the render until its CR pins the old value or the disk is recreated. Treat the `storageClass` default as a frozen contract for the same reason the `source` serialization above is.
34+
35+
The `source` annotation is a byte-for-byte `toJson` of `.Values.source`, so it is coupled to the exact serialization of the source schema. Adding a subfield with a schema default (which the apiserver would stamp onto every CR) would make the recorded and requested JSON diverge on an untouched disk; such a change needs an annotation-migration story rather than a bare schema bump.
36+
2537
If you add a new stateful chart that exposes `storageClass`, annotate it the same way:
2638

2739
```yaml

‎packages/apps/vm-disk/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22

33
A Virtual Machine Disk
44

5-
> `storageClass` is annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it.
5+
> `storageClass` and `source` are annotated as immutable in the chart schema — see [`docs/storage-immutability.md`](../../../docs/storage-immutability.md) for the contract and which consumers enforce it. `DataVolume.spec` is immutable in CDI, so the chart also fails the release on an attempted `storageClass`/`source` edit of an existing disk rather than silently reusing the old spec; delete and recreate the disk to change either field.
66
77
## Parameters
88

‎packages/apps/vm-disk/templates/dv.yaml‎

Lines changed: 101 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,43 +1,120 @@
11
{{- $existingDV := lookup "cdi.kubevirt.io/v1beta1" "DataVolume" .Release.Namespace .Release.Name }}
2+
{{- $sourceAnnotation := "vm-disk.cozystack.io/source" }}
3+
{{- $storageClassAnnotation := "vm-disk.cozystack.io/storage-class" }}
4+
{{- $requestedSource := toJson (.Values.source | default dict) }}
5+
{{- $requestedSC := (.Values.storageClass | default "") | toString }}
6+
{{- /* Build the desired source spec from .Values so it can be both rendered and compared. */}}
7+
{{- $desiredSource := dict }}
8+
{{- if .Values.source }}
9+
{{- if gt (len .Values.source) 1 }}
10+
{{- fail "Exactly one type of source is expected!" }}
11+
{{- end }}
12+
{{- if hasKey .Values.source "image" }}
13+
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-default-images-%s" (required "A valid .Values.source.image.name entry required!" .Values.source.image.name)) "namespace" "cozy-public") }}
14+
{{- else if hasKey .Values.source "http" }}
15+
{{- $desiredSource = dict "http" (dict "url" (required "A valid .Values.source.http.url entry required!" .Values.source.http.url)) }}
16+
{{- else if hasKey .Values.source "upload" }}
17+
{{- $desiredSource = dict "upload" dict }}
18+
{{- else if hasKey .Values.source "disk" }}
19+
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-disk-%s" (required "A valid .Values.source.disk.name entry required!" .Values.source.disk.name)) "namespace" .Release.Namespace) }}
20+
{{- else }}
21+
{{- fail (printf "Unknown source type: %s. Expected one of: image, http, upload, disk." (keys .Values.source | join ", ")) }}
22+
{{- end }}
23+
{{- else }}
24+
{{- $desiredSource = dict "blank" dict }}
25+
{{- end }}
26+
{{- /*
27+
DataVolume.spec is immutable in CDI, so editing storageClass/source on an
28+
existing disk is a silent no-op that diverges the CR from the cluster (#2985);
29+
fail instead. Both are compared against the operator input recorded in an
30+
annotation at creation (legacy disks fall back to stored spec.source). Full
31+
contract + caveats: docs/storage-immutability.md. Size growth: pvc-resize-hook.
32+
*/}}
33+
{{- if $existingDV }}
34+
{{- $existingAnnotations := $existingDV.metadata.annotations | default dict }}
35+
{{- if hasKey $existingAnnotations $storageClassAnnotation }}
36+
{{- $recordedSC := (index $existingAnnotations $storageClassAnnotation) | toString }}
37+
{{- if ne $recordedSC $requestedSC }}
38+
{{- fail (printf "VMDisk storageClass is immutable after creation (current: %q, requested: %q). Delete and recreate the disk to change it." $recordedSC $requestedSC) }}
39+
{{- end }}
40+
{{- end }}
41+
{{- $recordedSource := (index $existingAnnotations $sourceAnnotation) | default "" }}
42+
{{- if $recordedSource }}
43+
{{- if ne $recordedSource $requestedSource }}
44+
{{- fail (printf "VMDisk source is immutable after creation (current: %s, requested: %s). Delete and recreate the disk to change it." $recordedSource $requestedSource) }}
45+
{{- end }}
46+
{{- else }}
47+
{{- $storedSource := $existingDV.spec.source | default dict }}
48+
{{- /* Legacy disk (no annotation): compare only chart-producible stored
49+
shapes; anything else (sourceRef, non-map, foreign source) is left
50+
alone, never crashed or deadlocked. Vars are declared before the map
51+
guards so they stay in scope below. See docs/storage-immutability.md. */}}
52+
{{- $storedType := "" }}
53+
{{- $chartShaped := false }}
54+
{{- $storedIsImageClone := false }}
55+
{{- if kindIs "map" $storedSource }}
56+
{{- if eq (len (keys $storedSource)) 1 }}{{- $storedType = first (keys $storedSource) }}{{- end }}
57+
{{- $storedBody := index $storedSource $storedType }}
58+
{{- if kindIs "map" $storedBody }}
59+
{{- if or (eq $storedType "blank") (eq $storedType "upload") }}
60+
{{- $chartShaped = true }}
61+
{{- else if eq $storedType "http" }}
62+
{{- if eq (keys $storedBody | sortAlpha | join ",") "url" }}{{- $chartShaped = true }}{{- end }}
63+
{{- else if eq $storedType "pvc" }}
64+
{{- /* The chart only ever clones a pvc from cozy-public (image) or the
65+
release namespace (disk); a pvc in any other namespace is a
66+
foreign source with no chart-producible provenance. */}}
67+
{{- if eq (keys $storedBody | sortAlpha | join ",") "name,namespace" }}
68+
{{- if or (eq ($storedBody.namespace | toString) "cozy-public") (eq ($storedBody.namespace | toString) .Release.Namespace) }}
69+
{{- $chartShaped = true }}
70+
{{- end }}
71+
{{- if eq ($storedBody.namespace | toString) "cozy-public" }}{{- $storedIsImageClone = true }}{{- end }}
72+
{{- end }}
73+
{{- end }}
74+
{{- end }}
75+
{{- end }}
76+
{{- /* The image-clone exemption applies only when the request is ALSO an
77+
image: the vm-image- -> vm-default-images- prefix drift is
78+
indistinguishable from a real image-name change only then. A change
79+
away from an image source to another type is a different shape and is
80+
still caught. */}}
81+
{{- $exemptImage := and $storedIsImageClone (hasKey (.Values.source | default dict) "image") }}
82+
{{- if and $chartShaped (not $exemptImage) }}
83+
{{- if ne (toYaml $storedSource) (toYaml $desiredSource) }}
84+
{{- fail (printf "VMDisk source is immutable after creation (current: %s, requested: %s). Delete and recreate the disk to change it." (toJson $storedSource) $requestedSource) }}
85+
{{- end }}
86+
{{- end }}
87+
{{- end }}
88+
{{- end }}
289
apiVersion: cdi.kubevirt.io/v1beta1
390
kind: DataVolume
491
metadata:
592
annotations:
6-
{{- if hasKey .Values.source "upload" }}
93+
{{- if hasKey (.Values.source | default dict) "upload" }}
794
cdi.kubevirt.io/storage.bind.immediate.requested: ""
895
{{- end }}
96+
{{- if $existingDV }}
97+
{{- if hasKey ($existingDV.metadata.annotations | default dict) $sourceAnnotation }}
98+
{{ $sourceAnnotation }}: {{ index ($existingDV.metadata.annotations | default dict) $sourceAnnotation | quote }}
99+
{{- end }}
100+
{{- if hasKey ($existingDV.metadata.annotations | default dict) $storageClassAnnotation }}
101+
{{ $storageClassAnnotation }}: {{ index ($existingDV.metadata.annotations | default dict) $storageClassAnnotation | quote }}
102+
{{- end }}
103+
{{- else }}
104+
{{ $sourceAnnotation }}: {{ $requestedSource | quote }}
105+
{{ $storageClassAnnotation }}: {{ $requestedSC | quote }}
106+
{{- end }}
9107
vm-disk.cozystack.io/optical: "{{ .Values.optical }}"
10108
labels:
11109
app.kubernetes.io/instance: {{ .Release.Name }}
12110
name: {{ .Release.Name }}
13111
spec:
14112
{{- if $existingDV }}
15113
{{- toYaml $existingDV.spec | nindent 2 }}
16-
{{- else }}
17-
contentType: kubevirt
18-
{{- if .Values.source }}
19-
{{- if gt (len .Values.source) 1 }}
20-
{{- fail "Exactly one type of source is expected!" }}
21-
{{- end }}
22-
source:
23-
{{- if hasKey .Values.source "image" }}
24-
pvc:
25-
name: vm-default-images-{{ required "A valid .Values.source.image.name entry required!" .Values.source.image.name }}
26-
namespace: cozy-public
27-
{{- else if hasKey .Values.source "http" }}
28-
http:
29-
url: {{ required "A valid .Values.source.http.url entry required!" .Values.source.http.url }}
30-
{{- else if hasKey .Values.source "upload" }}
31-
upload: {}
32-
{{- else if hasKey .Values.source "disk" }}
33-
pvc:
34-
name: vm-disk-{{ required "A valid .Values.source.disk.name entry required!" .Values.source.disk.name }}
35-
namespace: {{ $.Release.Namespace }}
36-
{{- end }}
37114
{{- else }}
115+
contentType: kubevirt
38116
source:
39-
blank: {}
40-
{{- end }}
117+
{{- toYaml $desiredSource | nindent 4 }}
41118
storage:
42119
resources:
43120
requests:

0 commit comments

Comments
 (0)