Skip to content

Commit bf4dfdd

Browse files
fix(backport): batch for release-1.6, 2026-10-23 (#5089)
## What this PR does Backports to release-1.6, picked from main with `cherry-pick -x` in merge order. Each commit keeps the original message and its `(cherry picked from commit …)` line. - [#3581](#3581) fix(kubernetes): retry the tenant CSI install instead of uninstalling it - [#3621](#3621) fix(harbor): retry the nested system install instead of uninstalling it - [#3630](#3630) fix(tenant): retry failed etcd and seaweedfs installs in place - [#3735](#3735) fix(vm-instance): disable helm readiness wait on the VMInstance release - [#4799](#4799) fix: typo in monitoring-agents values.yaml ### Conflicts resolved by hand - [#3735](#3735): only `pkg/config/config_test.go` conflicted. release-1.6 has no `HelmServerSideApplyAnnotation`, so that test entry is left out, and two comments in `pkg/config/config.go` and `internal/controller/applicationdefinition_controller.go` are adjusted to what exists on this line. The rest is the original as is. Checked on this branch before opening: `helm template` of `monitoring-agents` and `vm-instance-rd`, `go build`, `go vet`, `go test` and the commit trailer check pass. Not run locally: `helm unittest` (plugin not installed) and `helm template` of `harbor`, `kubernetes` and `tenant`, which need values to render at base and head alike. CI runs the unit suites. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): the tenant CSI HelmRelease now retries a failed install instead of being uninstalled between attempts, so a failed install no longer takes the tenant's CSI node plugin and its propagated StorageClasses away while it retries. The same strategy applies to the upgrade action, which it has to: a retried install runs as an upgrade. One consequence is operator-visible: a failed CSI upgrade in a tenant cluster now retries the new manifests every 30s instead of rolling back to the previous release. fix(harbor): retry a failed Harbor install instead of uninstalling the release, so a slow first install no longer deletes the Redis volume and starts over fix(tenant): retry a failed etcd or seaweedfs install instead of uninstalling the release, so a slow first install no longer runs the cleanup hooks that delete the etcd data volumes, the SeaweedFS volumes and the filer metadata database fix(vm-instance): VMInstance HelmReleases no longer hang InProgress or roll back healthy VMs; the release no longer waits on VirtualMachine readiness, which helm-controller's kstatus poller cannot assess correctly for KubeVirt VMs. Existing VMInstance releases are brought in line automatically on upgrade. fix: typo in monitoring-agents values.yaml ```
2 parents d7c280c + cf81d29 commit bf4dfdd

17 files changed

Lines changed: 776 additions & 3 deletions

File tree

‎internal/controller/applicationdefinition_controller.go‎

Lines changed: 32 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -6,10 +6,12 @@ import (
66
"encoding/hex"
77
"encoding/json"
88
"slices"
9+
"strings"
910
"sync"
1011
"time"
1112

1213
cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1"
14+
"github.com/cozystack/cozystack/pkg/config"
1315

1416
appsv1 "k8s.io/api/apps/v1"
1517
corev1 "k8s.io/api/core/v1"
@@ -67,6 +69,33 @@ func (r *ApplicationDefinitionReconciler) SetupWithManager(mgr ctrl.Manager) err
6769
type appDefHashView struct {
6870
Name string `json:"name"`
6971
Spec cozyv1alpha1.ApplicationDefinitionSpec `json:"spec"`
72+
// Annotations holds the release.cozystack.io/* subset of the definition's
73+
// metadata annotations. They belong in the hash because cozystack-api reads
74+
// them once, at start-up (CozyServerOptions.Complete in pkg/cmd/server/start.go),
75+
// and this rollout is the only thing that delivers a change in one: a
76+
// definition whose whole change is an annotation would otherwise leave the
77+
// running api serving the previous behaviour until its pod restarted for an
78+
// unrelated reason. Only that prefix is hashed, because kubectl and Helm
79+
// rewrite annotations of their own on every apply and hashing those would
80+
// roll the api Deployment on every reconcile.
81+
Annotations map[string]string `json:"annotations,omitempty"`
82+
}
83+
84+
// releaseAnnotations returns the release.cozystack.io/* annotations of a
85+
// definition, or nil when it carries none, so a definition without them hashes
86+
// exactly as it did before this field existed.
87+
func releaseAnnotations(annotations map[string]string) map[string]string {
88+
var selected map[string]string
89+
for k, v := range annotations {
90+
if !strings.HasPrefix(k, config.ReleaseAnnotationPrefix) {
91+
continue
92+
}
93+
if selected == nil {
94+
selected = make(map[string]string)
95+
}
96+
selected[k] = v
97+
}
98+
return selected
7099
}
71100

72101
func (r *ApplicationDefinitionReconciler) computeConfigHash(ctx context.Context) (string, error) {
@@ -80,8 +109,9 @@ func (r *ApplicationDefinitionReconciler) computeConfigHash(ctx context.Context)
80109
views := make([]appDefHashView, 0, len(list.Items))
81110
for i := range list.Items {
82111
views = append(views, appDefHashView{
83-
Name: list.Items[i].Name,
84-
Spec: list.Items[i].Spec,
112+
Name: list.Items[i].Name,
113+
Spec: list.Items[i].Spec,
114+
Annotations: releaseAnnotations(list.Items[i].Annotations),
85115
})
86116
}
87117
b, err := json.Marshal(views)

‎internal/controller/applicationdefinition_controller_test.go‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ import (
66
"time"
77

88
cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1"
9+
"github.com/cozystack/cozystack/pkg/config"
910
appsv1 "k8s.io/api/apps/v1"
1011
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
1112
"k8s.io/apimachinery/pkg/runtime"
@@ -229,3 +230,65 @@ func TestApplicationDefinition_HashUnchangedSkipsPatch(t *testing.T) {
229230
t.Fatalf("expected lastHandled advanced after no-op reconcile")
230231
}
231232
}
233+
234+
// hashWithAnnotations computes the config hash for a single ApplicationDefinition
235+
// carrying the given metadata annotations.
236+
func hashWithAnnotations(t *testing.T, annotations map[string]string) string {
237+
t.Helper()
238+
239+
scheme := newAppDefScheme(t)
240+
appDef := &cozyv1alpha1.ApplicationDefinition{
241+
ObjectMeta: metav1.ObjectMeta{Name: "vm-instance", Annotations: annotations},
242+
}
243+
r := &ApplicationDefinitionReconciler{
244+
Client: fake.NewClientBuilder().WithScheme(scheme).WithObjects(appDef).Build(),
245+
Scheme: scheme,
246+
}
247+
248+
hash, err := r.computeConfigHash(context.TODO())
249+
if err != nil {
250+
t.Fatalf("computeConfigHash: %v", err)
251+
}
252+
return hash
253+
}
254+
255+
// TestApplicationDefinition_HashCoversReleaseAnnotations pins that a change
256+
// confined to a release.cozystack.io/* annotation moves the hash. cozystack-api
257+
// reads those annotations once, at start-up (pkg/cmd/server/start.go), so the
258+
// config-hash rollout is the only thing that makes such a change take effect.
259+
// A hash built from spec alone leaves the annotation inert on an existing
260+
// cluster until the api pod happens to restart for an unrelated reason, which
261+
// is indistinguishable from the change never having been delivered.
262+
func TestApplicationDefinition_HashCoversReleaseAnnotations(t *testing.T) {
263+
bare := hashWithAnnotations(t, nil)
264+
annotated := hashWithAnnotations(t, map[string]string{
265+
config.HelmInstallDisableWaitAnnotation: "true",
266+
})
267+
268+
if bare == annotated {
269+
t.Fatalf("hash ignores %s: %q for both", config.HelmInstallDisableWaitAnnotation, bare)
270+
}
271+
272+
flipped := hashWithAnnotations(t, map[string]string{
273+
config.HelmInstallDisableWaitAnnotation: "false",
274+
})
275+
if flipped == annotated {
276+
t.Fatalf("hash ignores the annotation's value: %q for both true and false", annotated)
277+
}
278+
}
279+
280+
// TestApplicationDefinition_HashIgnoresUnrelatedAnnotations pins the other half:
281+
// only the release.cozystack.io/ prefix counts. kubectl and Flux rewrite their
282+
// own annotations on every apply, and hashing those would roll out cozystack-api
283+
// on every reconcile of every ApplicationDefinition.
284+
func TestApplicationDefinition_HashIgnoresUnrelatedAnnotations(t *testing.T) {
285+
bare := hashWithAnnotations(t, nil)
286+
noisy := hashWithAnnotations(t, map[string]string{
287+
"kubectl.kubernetes.io/last-applied-configuration": `{"metadata":{"name":"vm-instance"}}`,
288+
"meta.helm.sh/release-name": "vm-instance-rd",
289+
})
290+
291+
if bare != noisy {
292+
t.Fatalf("unrelated annotations moved the hash: %q vs %q", bare, noisy)
293+
}
294+
}

‎internal/controller/applicationdefinition_helmreconciler.go‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ import (
55
"fmt"
66

77
cozyv1alpha1 "github.com/cozystack/cozystack/api/v1alpha1"
8+
"github.com/cozystack/cozystack/pkg/config"
89
helmv2 "github.com/fluxcd/helm-controller/api/v2"
910

1011
"k8s.io/apimachinery/pkg/runtime"
@@ -163,6 +164,44 @@ func (r *ApplicationDefinitionHelmReconciler) updateHelmReleaseChart(ctx context
163164
updated = true
164165
}
165166

167+
// Check and update the readiness wait. cozystack-api reads
168+
// release.cozystack.io/helm-install-disable-wait off the definition when it
169+
// builds a release, and only there, so a release created before the
170+
// annotation appeared keeps waiting on readiness until its Application is
171+
// written through the apps API again. Where the wait is what broke, that is
172+
// every existing instance of the kind at once, each looping upgrade and
173+
// timeout until an operator rewrites it by hand. The definition is the
174+
// source of truth in both directions, since the API rebuilds the whole spec
175+
// on every write and emits DisableWait=false once the annotation is gone.
176+
disableWait, err := config.ParseHelmInstallDisableWaitAnnotation(
177+
appDef.Annotations[config.HelmInstallDisableWaitAnnotation],
178+
)
179+
if err != nil {
180+
// cozystack-api refuses to start on this same value, so the cluster
181+
// already has a loud signal; leave the field as it is rather than guess.
182+
logger.Error(err, "Skipping HelmRelease wait update: invalid annotation",
183+
"appDef", appDef.Name, "annotation", config.HelmInstallDisableWaitAnnotation)
184+
} else {
185+
// An absent install/upgrade block already behaves as DisableWait=false,
186+
// so it is only materialised when the definition asks to disable the wait.
187+
if disableWait && hrCopy.Spec.Install == nil {
188+
hrCopy.Spec.Install = &helmv2.Install{}
189+
}
190+
if hrCopy.Spec.Install != nil && hrCopy.Spec.Install.DisableWait != disableWait {
191+
logger.V(4).Info("Updating HelmRelease install wait", "name", hr.Name, "namespace", hr.Namespace, "disableWait", disableWait)
192+
hrCopy.Spec.Install.DisableWait = disableWait
193+
updated = true
194+
}
195+
if disableWait && hrCopy.Spec.Upgrade == nil {
196+
hrCopy.Spec.Upgrade = &helmv2.Upgrade{}
197+
}
198+
if hrCopy.Spec.Upgrade != nil && hrCopy.Spec.Upgrade.DisableWait != disableWait {
199+
logger.V(4).Info("Updating HelmRelease upgrade wait", "name", hr.Name, "namespace", hr.Namespace, "disableWait", disableWait)
200+
hrCopy.Spec.Upgrade.DisableWait = disableWait
201+
updated = true
202+
}
203+
}
204+
166205
// Check and update labels from ApplicationDefinition
167206
if len(appDef.Spec.Release.Labels) > 0 {
168207
if hrCopy.Labels == nil {

0 commit comments

Comments
 (0)