Skip to content

Commit 15035a9

Browse files
authored
fix(vm-instance): disable helm readiness wait on the VMInstance release (#3735)
## What this PR does A VMInstance is served as a Flux HelmRelease that renders a KubeVirt VirtualMachine. Since v1.5.0 (helm-controller v1.4.3 to v1.5.0) that release is health-assessed by the kstatus poller. vm-instance sets no `healthCheckExprs`, so the VirtualMachine goes through kstatus's own rules in `fluxcd/cli-utils` (`pkg/kstatus/status`): `checkGeneration` returns InProgress for any object whose `status.observedGeneration` differs from `metadata.generation`, and past that check a `Ready=False` condition is InProgress too, so a halted VM is never Current even with its generation in sync. KubeVirt keeps a VirtualMachine's `observedGeneration` behind its `generation` whenever a `spec.template` change is pending a restart, the VM is stopped, or a live-updatable change never converges (kubevirt/kubevirt#18700). So a healthy, Running VM is read as permanently InProgress, the upgrade waits its full timeout, and fails with `timeout waiting for: [VirtualMachine/... status: 'InProgress']`. On releases still carrying the pre-1.5.0 `remediation.retries: -1` + rollback spec, the timeout then reverts the previous manifest, dropping just-applied changes such as opened ports. This sets `release.cozystack.io/helm-install-disable-wait: "true"` on the vm-instance ApplicationDefinition, so cozystack-api generates the HelmRelease with Install/Upgrade DisableWait. The release settles once the VirtualMachine is applied, and VM health is read off the VirtualMachine itself (`status.printableStatus` / `Ready`) and off the WorkloadMonitor the chart already renders, which is what `WorkloadsReady` on the Application reports. This restores the pre-1.5.0 behaviour, under which the VM was never a helm readiness gate. The same mechanism is already used by `kubernetes-rd` and `kubernetes-nodes-rd`. On its own that annotation does not reach a running cluster, so two gaps in cozystack-controller are closed with it. The first is the rollout. `applicationdefinition-controller` hashed each definition's name and spec only, so a change confined to a `release.cozystack.io/*` annotation left the hash unmoved and cozystack-api kept serving the configuration it read at its last start-up, which is the only place those annotations are ever read. The hash now covers them too. Only that prefix, because kubectl and Helm rewrite annotations of their own on every apply and hashing those would roll the api Deployment on every reconcile. The second is the releases that already exist. `applicationdefinition-helm-reconciler` brings chartRef, valuesFrom and labels on existing HelmReleases in line with their definition; it now does the same for the readiness wait. Without it the annotation reaches new instances only, which on an affected cluster is the wrong half: #3734 reports 134 of 136 releases already looping upgrade and timeout, and re-applying every VMInstance by hand was the only way out. The setting is driven in both directions, since cozystack-api rebuilds the whole spec on every write and emits `DisableWait=false` once the annotation is gone. An absent install or upgrade block already behaves as `DisableWait=false` and is materialised only when the definition asks to disable the wait, so a release with nothing to change is not rewritten. Two alternatives were considered. A per-kind `healthCheckExprs` keyed on `status.printableStatus` cannot work: setting it moves the VirtualMachine from kstatus to the CEL status evaluator (`fluxcd/pkg/runtime/cel`), which runs the same `observedGeneration` check ahead of the expressions, and any `healthCheckExprs` added to this kind later is a no-op while the wait is disabled. The typed `spec.release.waitStrategy: legacy` is the narrower option, since the legacy checker never waits on a custom resource, but it still gates the release on the chart's Service, and with `external: true` that is a LoadBalancer the legacy checker holds not ready until it has ingress. So an external VM would still wait on IP allocation, which `WorkloadsReady` already watches. Disable-wait avoids that and is the mechanism this repository already uses for the same class of problem. Tests: a helm-unittest suite (`tests/readiness_gate_test.yaml`) guards that the annotation stays on the rendered ApplicationDefinition, and Go unit tests cover both cozystack-controller paths: the hash moves on a release annotation and stands still on an unrelated one, and the wait lands on an existing release, is withdrawn when the annotation goes, does not materialise empty install/upgrade blocks, does not rewrite a release already in sync, leaves a release alone when the annotation does not parse, and does not reach a release of another kind. Note on existing clusters: no manual step. The definition changes on upgrade, the config-hash rolls cozystack-api, and the helm reconciler brings existing VMInstance releases in line. Releases that predate v1.5.0 also carry a `remediation.retries: -1` + rollback spec, which is left alone: with the wait disabled the upgrade no longer times out, so that remediation no longer fires, and any write through the apps API drops it. Closes #3734. ### Downstream repositories - [x] No downstream repository is affected by this change <!-- Walked the trigger map against the diff. The change is a readiness annotation on an ApplicationDefinition plus two cozystack-controller paths. It does not touch openAPISchema, kind, plural, release.prefix, any app package, any values.schema.json or any installer/platform values key. The website's application-definitions.md describes the config-hash as a checksum "over the sorted definitions", which stays accurate now that the hash also covers the release.cozystack.io/* annotations, and no page documents those annotations today. --> ### Release note ```release-note fix(vm-instance): VMInstance HelmReleases no longer hang InProgress or roll back healthy VMs; the release no longer waits on VirtualMachine readiness, which helm-controller's kstatus poller cannot assess correctly for KubeVirt VMs. Existing VMInstance releases are brought in line automatically on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * VM installations no longer wait indefinitely when Kubernetes reports the VM’s observed generation behind its current generation. * Changes to release settings are now detected and applied to generated Helm releases, including install and upgrade wait behavior. * **Tests** * Added coverage for VM readiness and release-setting reconciliation. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 95ece88 + 70ea242 commit 15035a9

9 files changed

Lines changed: 431 additions & 2 deletions

‎internal/controller/applicationdefinition_controller.go‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,12 @@ import (
66
"encoding/hex"
77
"encoding/json"
88
"slices"
9+
"strings"
910
"sync"
1011
"time"
1112

1213
cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1"
14+
"github.com/cozystack/cozystack/pkg/config"
1315

1416
appsv1 "k8s.io/api/apps/v1"
1517
corev1 "k8s.io/api/core/v1"
@@ -67,6 +69,33 @@ func (r *ApplicationDefinitionReconciler) SetupWithManager(mgr ctrl.Manager) err
6769
type appDefHashView struct {
6870
Name string `json:"name"`
6971
Spec cozyv1alpha1.ApplicationDefinitionSpec `json:"spec"`
72+
// Annotations holds the release.cozystack.io/* subset of the definition's
73+
// metadata annotations. They belong in the hash because cozystack-api reads
74+
// them once, at start-up (buildResourceFromCRD in pkg/cmd/server/start.go),
75+
// and this rollout is the only thing that delivers a change in one: a
76+
// definition whose whole change is an annotation would otherwise leave the
77+
// running api serving the previous behaviour until its pod restarted for an
78+
// unrelated reason. Only that prefix is hashed, because kubectl and Helm
79+
// rewrite annotations of their own on every apply and hashing those would
80+
// roll the api Deployment on every reconcile.
81+
Annotations map[string]string `json:"annotations,omitempty"`
82+
}
83+
84+
// releaseAnnotations returns the release.cozystack.io/* annotations of a
85+
// definition, or nil when it carries none, so a definition without them hashes
86+
// exactly as it did before this field existed.
87+
func releaseAnnotations(annotations map[string]string) map[string]string {
88+
var selected map[string]string
89+
for k, v := range annotations {
90+
if !strings.HasPrefix(k, config.ReleaseAnnotationPrefix) {
91+
continue
92+
}
93+
if selected == nil {
94+
selected = make(map[string]string)
95+
}
96+
selected[k] = v
97+
}
98+
return selected
7099
}
71100

72101
func (r *ApplicationDefinitionReconciler) computeConfigHash(ctx context.Context) (string, error) {
@@ -80,8 +109,9 @@ func (r *ApplicationDefinitionReconciler) computeConfigHash(ctx context.Context)
80109
views := make([]appDefHashView, 0, len(list.Items))
81110
for i := range list.Items {
82111
views = append(views, appDefHashView{
83-
Name: list.Items[i].Name,
84-
Spec: list.Items[i].Spec,
112+
Name: list.Items[i].Name,
113+
Spec: list.Items[i].Spec,
114+
Annotations: releaseAnnotations(list.Items[i].Annotations),
85115
})
86116
}
87117
b, err := json.Marshal(views)

‎internal/controller/applicationdefinition_controller_test.go‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import (
66
"time"
77

88
cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1"
9+
"github.com/cozystack/cozystack/pkg/config"
910
appsv1 "k8s.io/api/apps/v1"
1011
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
1112
"k8s.io/apimachinery/pkg/runtime"
@@ -229,3 +230,65 @@ func TestApplicationDefinition_HashUnchangedSkipsPatch(t *testing.T) {
229230
t.Fatalf("expected lastHandled advanced after no-op reconcile")
230231
}
231232
}
233+
234+
// hashWithAnnotations computes the config hash for a single ApplicationDefinition
235+
// carrying the given metadata annotations.
236+
func hashWithAnnotations(t *testing.T, annotations map[string]string) string {
237+
t.Helper()
238+
239+
scheme := newAppDefScheme(t)
240+
appDef := &cozyv1alpha1.ApplicationDefinition{
241+
ObjectMeta: metav1.ObjectMeta{Name: "vm-instance", Annotations: annotations},
242+
}
243+
r := &ApplicationDefinitionReconciler{
244+
Client: fake.NewClientBuilder().WithScheme(scheme).WithObjects(appDef).Build(),
245+
Scheme: scheme,
246+
}
247+
248+
hash, err := r.computeConfigHash(context.TODO())
249+
if err != nil {
250+
t.Fatalf("computeConfigHash: %v", err)
251+
}
252+
return hash
253+
}
254+
255+
// TestApplicationDefinition_HashCoversReleaseAnnotations pins that a change
256+
// confined to a release.cozystack.io/* annotation moves the hash. cozystack-api
257+
// reads those annotations once, at start-up (pkg/cmd/server/start.go), so the
258+
// config-hash rollout is the only thing that makes such a change take effect.
259+
// A hash built from spec alone leaves the annotation inert on an existing
260+
// cluster until the api pod happens to restart for an unrelated reason, which
261+
// is indistinguishable from the change never having been delivered.
262+
func TestApplicationDefinition_HashCoversReleaseAnnotations(t *testing.T) {
263+
bare := hashWithAnnotations(t, nil)
264+
annotated := hashWithAnnotations(t, map[string]string{
265+
config.HelmInstallDisableWaitAnnotation: "true",
266+
})
267+
268+
if bare == annotated {
269+
t.Fatalf("hash ignores %s: %q for both", config.HelmInstallDisableWaitAnnotation, bare)
270+
}
271+
272+
flipped := hashWithAnnotations(t, map[string]string{
273+
config.HelmInstallDisableWaitAnnotation: "false",
274+
})
275+
if flipped == annotated {
276+
t.Fatalf("hash ignores the annotation's value: %q for both true and false", annotated)
277+
}
278+
}
279+
280+
// TestApplicationDefinition_HashIgnoresUnrelatedAnnotations pins the other half:
281+
// only the release.cozystack.io/ prefix counts. kubectl and Flux rewrite their
282+
// own annotations on every apply, and hashing those would roll out cozystack-api
283+
// on every reconcile of every ApplicationDefinition.
284+
func TestApplicationDefinition_HashIgnoresUnrelatedAnnotations(t *testing.T) {
285+
bare := hashWithAnnotations(t, nil)
286+
noisy := hashWithAnnotations(t, map[string]string{
287+
"kubectl.kubernetes.io/last-applied-configuration": `{"metadata":{"name":"vm-instance"}}`,
288+
"meta.helm.sh/release-name": "vm-instance-rd",
289+
})
290+
291+
if bare != noisy {
292+
t.Fatalf("unrelated annotations moved the hash: %q vs %q", bare, noisy)
293+
}
294+
}

‎internal/controller/applicationdefinition_helmreconciler.go‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import (
55
"fmt"
66

77
cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1"
8+
"github.com/cozystack/cozystack/pkg/config"
89
helmv2 "github.com/fluxcd/helm-controller/api/v2"
910

1011
"k8s.io/apimachinery/pkg/runtime"
@@ -163,6 +164,44 @@ func (r *ApplicationDefinitionHelmReconciler) updateHelmReleaseChart(ctx context
163164
updated = true
164165
}
165166

167+
// Check and update the readiness wait. cozystack-api reads
168+
// release.cozystack.io/helm-install-disable-wait off the definition when it
169+
// builds a release, and only there, so a release created before the
170+
// annotation appeared keeps waiting on readiness until its Application is
171+
// written through the apps API again. Where the wait is what broke, that is
172+
// every existing instance of the kind at once, each looping upgrade and
173+
// timeout until an operator rewrites it by hand. The definition is the
174+
// source of truth in both directions, since the API rebuilds the whole spec
175+
// on every write and emits DisableWait=false once the annotation is gone.
176+
disableWait, err := config.ParseHelmInstallDisableWaitAnnotation(
177+
appDef.Annotations[config.HelmInstallDisableWaitAnnotation],
178+
)
179+
if err != nil {
180+
// cozystack-api refuses to start on this same value, so the cluster
181+
// already has a loud signal; leave the field as it is rather than guess.
182+
logger.Error(err, "Skipping HelmRelease wait update: invalid annotation",
183+
"appDef", appDef.Name, "annotation", config.HelmInstallDisableWaitAnnotation)
184+
} else {
185+
// An absent install/upgrade block already behaves as DisableWait=false,
186+
// so it is only materialised when the definition asks to disable the wait.
187+
if disableWait && hrCopy.Spec.Install == nil {
188+
hrCopy.Spec.Install = &helmv2.Install{}
189+
}
190+
if hrCopy.Spec.Install != nil && hrCopy.Spec.Install.DisableWait != disableWait {
191+
logger.V(4).Info("Updating HelmRelease install wait", "name", hr.Name, "namespace", hr.Namespace, "disableWait", disableWait)
192+
hrCopy.Spec.Install.DisableWait = disableWait
193+
updated = true
194+
}
195+
if disableWait && hrCopy.Spec.Upgrade == nil {
196+
hrCopy.Spec.Upgrade = &helmv2.Upgrade{}
197+
}
198+
if hrCopy.Spec.Upgrade != nil && hrCopy.Spec.Upgrade.DisableWait != disableWait {
199+
logger.V(4).Info("Updating HelmRelease upgrade wait", "name", hr.Name, "namespace", hr.Namespace, "disableWait", disableWait)
200+
hrCopy.Spec.Upgrade.DisableWait = disableWait
201+
updated = true
202+
}
203+
}
204+
166205
// Check and update labels from ApplicationDefinition
167206
if len(appDef.Spec.Release.Labels) > 0 {
168207
if hrCopy.Labels == nil {

0 commit comments

Comments
 (0)