Skip to content

Commit 339be8d

Browse files
authored
fix(vm-default-images,kubernetes-worker-image,vm-disk): wait for golden images to import and refuse cloning a failed one (#4392)
## What this PR does Both golden image catalogs, `vm-default-images` and `kubernetes-worker-image`, went Ready before any image had imported. Each one renders a DataVolume per image with immediate bind, so every import starts at install, but neither PackageSource had `healthCheckExprs`. The HelmRelease went Ready as soon as helm applied the DataVolumes, and an image that failed or stalled stayed green. This PR makes both releases wait for the imports. Each component's `install:` block now has a DataVolume health check. `current` holds only on `Succeeded`, and `failed` holds on `Failed`. Any other phase, `Unknown` included, matches neither, and flux keeps waiting. A DataVolume with no `status` key makes the CEL evaluation error out. Flux reports that as Unknown and keeps waiting too. The other settled phases are safe to treat as not ready for these charts. `PendingPopulation` and `WaitForFirstConsumer` are where a volume waits for a consumer, and immediate bind removes that wait. `UploadReady` and `Paused` belong to upload and multi-stage sources, and both charts render only `source.http`. There is no per-component timeout, so I did not set one. The window is the operator-wide `--helmrelease-install-timeout`, 10m by default. The operator sets `RetryOnFailure` on both install and upgrade, so a download that outlasts the window is retried as an upgrade until the import finishes. Nothing uninstalls the DataVolumes in between, and the import keeps running. One image that never imports keeps its whole catalog not Ready until the image imports or its entry is removed. There is no time limit, and I kept it that way on purpose. A group/kind health check cannot tell one image from another, so it cannot be made per image. A per-component timeout would be an API change for one opt-in package. Both charts' `values.yaml` document this. Nothing on `main` lists either package in `dependsOn`, and the images that did import stay usable in the meantime. Tenant worker pools look up each golden directly and never wait on the catalog's status. The price is a helm-controller slot. The operator labels the generated HelmRelease with `cozystack.io/package`, plus `cozystack.io/privileged` when privileged. The flux-shard-operator acts only on HelmReleases that already carry `sharding.fluxcd.io/key`. So both catalogs go to the system helm-controller, which picks up releases without that key and runs with `--concurrent=5`. A DataVolume in `Failed` ends each attempt at once and costs next to nothing. An import that stalls without reaching `Failed` is the expensive case, for example an importer that keeps retrying an unreachable URL. Each attempt waits out the full window, and the next one starts 30s later. So one of those five slots stays busy about 95% of the time for as long as the image is stuck. #4145 describes the same cost on the shard controllers. The status change alone does not protect a vm-disk, because vm-disk clones from a golden without checking it. A golden still importing is fine, since CDI holds the clone until the source is populated. A golden whose import has Failed is never populated, so the clone would wait forever. vm-disk now refuses to render a new disk on a Failed golden and names the image. This matches the check the worker pools already have. A disk that already exists is exempt, so a golden that fails later does not break that disk's upgrades. The check fires only when CDI sets Failed, and at CDI v1.66.1 an import reaches Failed only on a checksum mismatch. The goldens carry no checksum, so a broken URL keeps the importer retrying and the new disk still waits, with only the catalog's not-Ready status pointing at the image. #4564 tracks that gap. A golden that does not exist at all is a separate gap, tracked in #4548. Tests pin the expressions for both catalogs. A helm-unittest case checks them on the rendered PackageSource. A Go test builds the HelmRelease spec from the shipped file and checks that they arrive with the poller wait strategy. The repo has no harness that evaluates flux CEL, so the phase behaviour above is argued from flux's evaluator and not exercised by a test. The vm-disk check has helm-unittest cases for a Failed, a Succeeded, and a still-importing golden, for a render with no cluster, and for an existing disk. Closes #4169 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(vm-default-images, kubernetes-worker-image): the golden image catalogs now report Ready only after their images have finished importing, and a failed or stalled import keeps the catalog not Ready. A vm-disk created from a golden image whose DataVolume is in phase Failed is now refused with an error naming the image, instead of waiting forever on a clone that cannot complete. ```
2 parents 9c9a8c2 + 5d2c82c commit 339be8d

8 files changed

Lines changed: 223 additions & 0 deletions

File tree

‎internal/operator/package_reconciler_test.go‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@ import (
2020
"encoding/json"
2121
"os"
2222
"path/filepath"
23+
"reflect"
2324
"testing"
2425
"time"
2526

@@ -281,3 +282,51 @@ func TestPackageSourceCRDHasUpgradeCRDsEnum(t *testing.T) {
281282
}
282283
}
283284
}
285+
286+
// TestGoldenImageHelmReleasesWaitForImport builds the HelmRelease spec from
287+
// each shipped golden image catalog. Its DataVolumes start importing at
288+
// install, and without the gate the release is Ready before any image exists.
289+
func TestGoldenImageHelmReleasesWaitForImport(t *testing.T) {
290+
want := []kustomize.CustomHealthCheck{{
291+
APIVersion: "cdi.kubevirt.io/v1beta1",
292+
Kind: "DataVolume",
293+
HealthCheckExpressions: kustomize.HealthCheckExpressions{
294+
Failed: "has(status.phase) && status.phase == 'Failed'",
295+
Current: "has(status.phase) && status.phase == 'Succeeded'",
296+
},
297+
}}
298+
for _, component := range []string{"vm-default-images", "kubernetes-worker-image"} {
299+
t.Run(component, func(t *testing.T) {
300+
path := filepath.Join("..", "..", "packages", "core", "platform", "sources", component+".yaml")
301+
data, err := os.ReadFile(path)
302+
if err != nil {
303+
t.Fatalf("read %s: %v", path, err)
304+
}
305+
var ps cozyv1alpha1.PackageSource
306+
if err := yaml.Unmarshal(data, &ps); err != nil {
307+
t.Fatalf("unmarshal %s: %v", path, err)
308+
}
309+
310+
var install *cozyv1alpha1.ComponentInstall
311+
for _, v := range ps.Spec.Variants {
312+
for _, c := range v.Components {
313+
if c.Name == component {
314+
install = c.Install
315+
}
316+
}
317+
}
318+
if install == nil {
319+
t.Fatalf("%s: component %s has no install block", path, component)
320+
}
321+
322+
spec := (&PackageReconciler{}).buildHelmReleaseSpec(install, "x")
323+
324+
if !reflect.DeepEqual(spec.HealthCheckExprs, want) {
325+
t.Errorf("HealthCheckExprs = %+v, want %+v", spec.HealthCheckExprs, want)
326+
}
327+
if spec.WaitStrategy == nil || spec.WaitStrategy.Name != helmv2.WaitStrategyPoller {
328+
t.Errorf("WaitStrategy = %+v, want poller: the expressions are not evaluated under any other strategy", spec.WaitStrategy)
329+
}
330+
})
331+
}
332+
}

‎packages/apps/vm-disk/templates/dv.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,17 @@
1111
{{- end }}
1212
{{- if hasKey .Values.source "image" }}
1313
{{- $desiredSource = dict "pvc" (dict "name" (printf "vm-default-images-%s" (required "A valid .Values.source.image.name entry required!" .Values.source.image.name)) "namespace" "cozy-public") }}
14+
{{- /* Refuse only a golden whose import Failed: CDI never populates it, so the
15+
clone would wait forever. A golden still importing is left alone because
16+
CDI holds the clone until the source is populated. An existing disk is
17+
exempt: its DataVolume spec is immutable, so failing the render cannot
18+
rescue it and would only block this release's upgrades. */}}
19+
{{- if not $existingDV }}
20+
{{- $golden := lookup "cdi.kubevirt.io/v1beta1" "DataVolume" "cozy-public" $desiredSource.pvc.name }}
21+
{{- if eq (dig "status" "phase" "" $golden | toString) "Failed" }}
22+
{{- fail (printf "source.image %q: golden DataVolume %q in cozy-public failed to import (phase Failed), so a clone of it would never complete. Ask the platform administrator to check the importer pod in cozy-public, then delete that DataVolume to retry the import." .Values.source.image.name $desiredSource.pvc.name) }}
23+
{{- end }}
24+
{{- end }}
1425
{{- else if hasKey .Values.source "http" }}
1526
{{- $desiredSource = dict "http" (dict "url" (required "A valid .Values.source.http.url entry required!" .Values.source.http.url)) }}
1627
{{- else if hasKey .Values.source "upload" }}
Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
suite: vm-disk refuses to clone a golden image whose import failed
2+
# A golden whose import Failed is never populated, and CDI holds a clone of it
3+
# forever, so a new disk on it would sit pending with nothing pointing at the
4+
# image. Only Failed is refused: a golden still importing is safe to clone
5+
# because CDI waits for it, and helm template has no cluster to look it up in.
6+
release:
7+
name: test-vm-disk
8+
namespace: tenant-test
9+
templates:
10+
- templates/dv.yaml
11+
tests:
12+
- it: refuses a new disk on a golden whose import Failed
13+
set:
14+
source: {image: {name: ubuntu}}
15+
kubernetesProvider:
16+
scheme: &scheme
17+
"cdi.kubevirt.io/v1beta1/DataVolume":
18+
gvr: {group: cdi.kubevirt.io, version: v1beta1, resource: datavolumes}
19+
namespaced: true
20+
objects:
21+
- kind: DataVolume
22+
apiVersion: cdi.kubevirt.io/v1beta1
23+
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
24+
status: {phase: Failed}
25+
asserts:
26+
- failedTemplate:
27+
errorPattern: 'golden DataVolume "vm-default-images-ubuntu" in cozy-public failed to import \(phase Failed\)'
28+
29+
- it: clones a golden whose import Succeeded
30+
set:
31+
source: {image: {name: ubuntu}}
32+
kubernetesProvider:
33+
scheme: *scheme
34+
objects:
35+
- kind: DataVolume
36+
apiVersion: cdi.kubevirt.io/v1beta1
37+
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
38+
status: {phase: Succeeded}
39+
asserts:
40+
- equal:
41+
path: spec.source.pvc
42+
value: {name: vm-default-images-ubuntu, namespace: cozy-public}
43+
44+
- it: clones a golden that is still importing
45+
set:
46+
source: {image: {name: ubuntu}}
47+
kubernetesProvider:
48+
scheme: *scheme
49+
objects:
50+
- kind: DataVolume
51+
apiVersion: cdi.kubevirt.io/v1beta1
52+
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
53+
status: {phase: ImportInProgress}
54+
asserts:
55+
- equal:
56+
path: spec.source.pvc
57+
value: {name: vm-default-images-ubuntu, namespace: cozy-public}
58+
59+
- it: renders without a cluster to look the golden up in
60+
set:
61+
source: {image: {name: ubuntu}}
62+
asserts:
63+
- equal:
64+
path: spec.source.pvc
65+
value: {name: vm-default-images-ubuntu, namespace: cozy-public}
66+
67+
- it: leaves an existing disk alone when its golden later Failed
68+
set:
69+
source: {image: {name: ubuntu}}
70+
kubernetesProvider:
71+
scheme: *scheme
72+
objects:
73+
- kind: DataVolume
74+
apiVersion: cdi.kubevirt.io/v1beta1
75+
metadata: {name: vm-default-images-ubuntu, namespace: cozy-public}
76+
status: {phase: Failed}
77+
- kind: DataVolume
78+
apiVersion: cdi.kubevirt.io/v1beta1
79+
metadata:
80+
name: test-vm-disk
81+
namespace: tenant-test
82+
annotations:
83+
vm-disk.cozystack.io/source: '{"image":{"name":"ubuntu"}}'
84+
vm-disk.cozystack.io/storage-class: replicated
85+
spec:
86+
source: {pvc: {name: vm-default-images-ubuntu, namespace: cozy-public}}
87+
storage: {resources: {requests: {storage: 5Gi}}, storageClassName: replicated}
88+
asserts:
89+
- notFailedTemplate: {}

‎packages/core/platform/sources/kubernetes-worker-image.yaml‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,13 @@ spec:
2020
install:
2121
namespace: cozy-system
2222
releaseName: kubernetes-worker-image
23+
# Ready only once every golden has imported. Worker pools read each
24+
# golden's phase directly and do not wait on this status, so the
25+
# gate delays no pool; it only stops the catalog reporting green
26+
# over an import that never finished. See vm-default-images.yaml
27+
# for how the expressions treat each phase.
28+
healthCheckExprs:
29+
- apiVersion: cdi.kubevirt.io/v1beta1
30+
kind: DataVolume
31+
failed: "has(status.phase) && status.phase == 'Failed'"
32+
current: "has(status.phase) && status.phase == 'Succeeded'"

‎packages/core/platform/sources/vm-default-images.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,3 +20,14 @@ spec:
2020
install:
2121
namespace: cozy-system
2222
releaseName: vm-default-images
23+
# Ready only once every image has imported, so the status says
24+
# whether the collection is actually there. Any phase other than
25+
# Succeeded or Failed matches neither expression and flux keeps
26+
# waiting. There is no per-component timeout: a download that
27+
# outlasts the operator-wide install window is retried as an
28+
# upgrade until it finishes, and nothing is uninstalled in between.
29+
healthCheckExprs:
30+
- apiVersion: cdi.kubevirt.io/v1beta1
31+
kind: DataVolume
32+
failed: "has(status.phase) && status.phase == 'Failed'"
33+
current: "has(status.phase) && status.phase == 'Succeeded'"
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
suite: golden image catalogs wait for their images to import
2+
# Every image is a DataVolume with immediate bind, so the import starts at
3+
# install. Without healthCheckExprs the HelmRelease goes Ready as soon as helm
4+
# has applied the DataVolumes, and an image that never imports stays green.
5+
# current must hold only on Succeeded: a looser expression lets an unfinished
6+
# import pass.
7+
templates:
8+
- templates/sources.yaml
9+
release:
10+
name: cozystack
11+
namespace: cozy-system
12+
tests:
13+
- it: gates the vm-default-images component on DataVolume Succeeded
14+
documentSelector:
15+
path: metadata.name
16+
value: cozystack.vm-default-images
17+
asserts:
18+
- equal:
19+
path: spec.variants[0].components[0].name
20+
value: vm-default-images
21+
- equal:
22+
path: spec.variants[0].components[0].install.healthCheckExprs
23+
value: &exprs
24+
- apiVersion: cdi.kubevirt.io/v1beta1
25+
kind: DataVolume
26+
failed: "has(status.phase) && status.phase == 'Failed'"
27+
current: "has(status.phase) && status.phase == 'Succeeded'"
28+
29+
- it: gates the kubernetes-worker-image component on DataVolume Succeeded
30+
documentSelector:
31+
path: metadata.name
32+
value: cozystack.kubernetes-worker-image
33+
asserts:
34+
- equal:
35+
path: spec.variants[0].components[0].name
36+
value: kubernetes-worker-image
37+
- equal:
38+
path: spec.variants[0].components[0].install.healthCheckExprs
39+
value: *exprs

‎packages/system/kubernetes-worker-image/values.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,13 @@
1515
## `cozystack.kubernetes-worker-image` to `bundles.enabledPackages` in the Cozystack
1616
## config; without it no golden is created and every worker imports over HTTP.
1717
##
18+
## The package reports Ready only once every golden in images[] has imported. One
19+
## golden that cannot be fetched keeps the whole package not Ready, retrying with
20+
## no time limit, until it imports or its entry is removed. A golden that reached
21+
## phase Failed does not import again by itself: delete it and the next retry
22+
## recreates it. Pools do not wait on that status: they read each golden
23+
## directly, as described below.
24+
##
1825
## A tenant node group opts into a clone explicitly by setting `osImage.builtin`
1926
## (optionally overriding schematicID/version) on its own `KubernetesNodes` resource
2027
## -- one per pool since the Phase 2 split, NOT `nodeGroups` on the parent `Kubernetes`

‎packages/system/vm-default-images/values.yaml‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,13 @@
11
##
22
## @section Global Golden Image Collection
33
##
4+
## The package reports Ready only once every image below has imported. One image
5+
## that cannot be fetched, such as a removed upstream URL, keeps the whole package
6+
## not Ready, retrying with no time limit, until that image imports or its entry
7+
## is removed from the list. An import that reached phase Failed does not start
8+
## again by itself: delete its DataVolume and the next retry recreates it. The
9+
## images that did import stay usable meanwhile.
10+
##
411

512
## @param {string} storageClass - Default StorageClass for all images. If empty, uses the cluster default StorageClass.
613
## NOTE: The default set of images requires approximately 320Gi of storage (16 images × 20Gi each).

0 commit comments

Comments
 (0)