fix(backport): batch for release-1.6, 2026-10-23 - #5089

…se (#3735)

## What this PR does

A VMInstance is served as a Flux HelmRelease that renders a KubeVirt
VirtualMachine. Since v1.5.0 (helm-controller v1.4.3 to v1.5.0) that
release is health-assessed by the kstatus poller. vm-instance sets no
`healthCheckExprs`, so the VirtualMachine goes through kstatus's own
rules in `fluxcd/cli-utils` (`pkg/kstatus/status`): `checkGeneration`
returns InProgress for any object whose `status.observedGeneration`
differs from `metadata.generation`, and past that check a `Ready=False`
condition is InProgress too, so a halted VM is never Current even with
its generation in sync. KubeVirt keeps a VirtualMachine's
`observedGeneration` behind its `generation` whenever a `spec.template`
change is pending a restart, the VM is stopped, or a live-updatable
change never converges (kubevirt/kubevirt#18700). So a healthy, Running
VM is read as permanently InProgress, the upgrade waits its full
timeout, and fails with `timeout waiting for: [VirtualMachine/...
status: 'InProgress']`. On releases still carrying the pre-1.5.0
`remediation.retries: -1` + rollback spec, the timeout then reverts the
previous manifest, dropping just-applied changes such as opened ports.

This sets `release.cozystack.io/helm-install-disable-wait: "true"` on
the vm-instance ApplicationDefinition, so cozystack-api generates the
HelmRelease with Install/Upgrade DisableWait. The release settles once
the VirtualMachine is applied, and VM health is read off the
VirtualMachine itself (`status.printableStatus` / `Ready`) and off the
WorkloadMonitor the chart already renders, which is what
`WorkloadsReady` on the Application reports. This restores the pre-1.5.0
behaviour, under which the VM was never a helm readiness gate. The same
mechanism is already used by `kubernetes-rd` and `kubernetes-nodes-rd`.

On its own that annotation does not reach a running cluster, so two gaps
in cozystack-controller are closed with it.

The first is the rollout. `applicationdefinition-controller` hashed each
definition's name and spec only, so a change confined to a
`release.cozystack.io/*` annotation left the hash unmoved and
cozystack-api kept serving the configuration it read at its last
start-up, which is the only place those annotations are ever read. The
hash now covers them too. Only that prefix, because kubectl and Helm
rewrite annotations of their own on every apply and hashing those would
roll the api Deployment on every reconcile.

The second is the releases that already exist.
`applicationdefinition-helm-reconciler` brings chartRef, valuesFrom and
labels on existing HelmReleases in line with their definition; it now
does the same for the readiness wait. Without it the annotation reaches
new instances only, which on an affected cluster is the wrong half:
#3734 reports 134 of 136 releases already looping upgrade and timeout,
and re-applying every VMInstance by hand was the only way out. The
setting is driven in both directions, since cozystack-api rebuilds the
whole spec on every write and emits `DisableWait=false` once the
annotation is gone. An absent install or upgrade block already behaves
as `DisableWait=false` and is materialised only when the definition asks
to disable the wait, so a release with nothing to change is not
rewritten.

Two alternatives were considered. A per-kind `healthCheckExprs` keyed on
`status.printableStatus` cannot work: setting it moves the
VirtualMachine from kstatus to the CEL status evaluator
(`fluxcd/pkg/runtime/cel`), which runs the same `observedGeneration`
check ahead of the expressions, and any `healthCheckExprs` added to this
kind later is a no-op while the wait is disabled. The typed
`spec.release.waitStrategy: legacy` is the narrower option, since the
legacy checker never waits on a custom resource, but it still gates the
release on the chart's Service, and with `external: true` that is a
LoadBalancer the legacy checker holds not ready until it has ingress. So
an external VM would still wait on IP allocation, which `WorkloadsReady`
already watches. Disable-wait avoids that and is the mechanism this
repository already uses for the same class of problem.

Tests: a helm-unittest suite (`tests/readiness_gate_test.yaml`) guards
that the annotation stays on the rendered ApplicationDefinition, and Go
unit tests cover both cozystack-controller paths: the hash moves on a
release annotation and stands still on an unrelated one, and the wait
lands on an existing release, is withdrawn when the annotation goes,
does not materialise empty install/upgrade blocks, does not rewrite a
release already in sync, leaves a release alone when the annotation does
not parse, and does not reach a release of another kind.

Note on existing clusters: no manual step. The definition changes on
upgrade, the config-hash rolls cozystack-api, and the helm reconciler
brings existing VMInstance releases in line. Releases that predate
v1.5.0 also carry a `remediation.retries: -1` + rollback spec, which is
left alone: with the wait disabled the upgrade no longer times out, so
that remediation no longer fires, and any write through the apps API
drops it.

Closes #3734.

### Downstream repositories

- [x] No downstream repository is affected by this change

<!-- Walked the trigger map against the diff. The change is a readiness
annotation on an ApplicationDefinition plus two cozystack-controller
paths. It does not touch openAPISchema, kind, plural, release.prefix,
any app package, any values.schema.json or any installer/platform values
key. The website's application-definitions.md describes the config-hash
as a checksum "over the sorted definitions", which stays accurate now
that the hash also covers the release.cozystack.io/* annotations, and no
page documents those annotations today. -->

### Release note

```release-note
fix(vm-instance): VMInstance HelmReleases no longer hang InProgress or roll back healthy VMs; the release no longer waits on VirtualMachine readiness, which helm-controller's kstatus poller cannot assess correctly for KubeVirt VMs. Existing VMInstance releases are brought in line automatically on upgrade.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* VM installations no longer wait indefinitely when Kubernetes reports
the VM’s observed generation behind its current generation.
* Changes to release settings are now detected and applied to generated
Helm releases, including install and upgrade wait behavior.
* **Tests**
  * Added coverage for VM readiness and release-setting reconciliation.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Backport adaptation: release-1.6 has no server-side-apply release
annotation, so TestReleaseAnnotationsShareThePrefix lists the three
annotations this line has, main's TestParseHelmServerSideApplyAnnotation
is not brought in, and the comments count three constants and name
CozyServerOptions.Complete, where this line reads the annotations.

(cherry picked from commit 15035a9)
Signed-off-by: Myasnikov Daniil <[email protected]>
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <[email protected]>

…it (#3621)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

Harbor's app chart renders a nested HelmRelease that owns the whole
workload: the Deployments, the inline RedisFailover and its volume. It
had no install strategy, so it got Flux's default, and install
remediation is an uninstall in every case Flux offers. A first install
that misses its budget gets torn down, and the next one provisions the
redis volume and runs the master election again from scratch.

That happens on a loaded runner. harbor-core waits on redis, redis waits
on its volume and the master election, and the nginx Deployment hits the
default `progressDeadlineSeconds` of 600s. Flux reads that as a stalled
resource and fails the install early. The e2e wait budget is 10m, the
same as the deadline, so after a teardown the second attempt has less
time than a first install needs.

`RetryOnFailure` keeps the manifests applied and retries as an upgrade.
Nothing is deleted, so the volume, the elected master and harbor-core's
startup all survive, and the retry costs 30s instead of a full install.
A Deployment clears `ProgressDeadlineExceeded` by itself once its pods
go ready, so the retry lands as soon as the stack converges.

This drops the teardown, not the deadline. The suite applies the release
in one step and waits for it in later ones, so part of the budget is
still left when the fail-early hits. That leftover is useless to a fresh
install and usable to a retry that resumes. Whether the deadline or the
budget also needs moving is a separate question, and an early deadline
only becomes safe once a failed install stops being torn down. In
production the win holds without any of that: a slow first install no
longer deletes the Redis volume and the database.

Both actions carry the strategy. The controller runs the retry of a
failed install as an upgrade, so with install alone the retry's own
failure falls through to upgrade remediation. The remediation blocks
stay: an absent release consults install, an out-of-sync one consults
upgrade, both by retry count rather than by strategy.

It goes on the nested release, not the parent. The parent is built from
the Application by the apiserver, which already stamps this strategy on
both actions. The nested one owns the workload and the volume a
remediation deletes.

Same shape as the tenant CNI and CSI releases in #3552 and #3581, and
the unit test follows theirs. Relates to #3445.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(harbor): retry a failed Harbor install instead of uninstalling the release, so a slow first install no longer deletes the Redis volume and starts over
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Harbor installations and upgrades now automatically retry every 30
seconds when failures occur.
* Existing unlimited remediation retries remain available for both
operations.

* **Tests**
* Added automated coverage to verify Harbor retry behavior and
remediation settings.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

(cherry picked from commit 79ceaf1)
Signed-off-by: Myasnikov Daniil <[email protected]>

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does

The tenant chart renders the etcd and seaweedfs releases with no install
strategy, so they get Flux's default, and install remediation is an
uninstall in every case Flux offers. Neither release sets
`spec.uninstall.disableHooks`, which defaults to false, so that
uninstall runs the app chart's `post-delete` cleanup Job. Nothing
unusual is configured anywhere along that chain: fail early, uninstall,
hooks.

Those Jobs delete data on purpose. The etcd one removes the
`data-etcd-<ordinal>` PVCs by name, because the etcd-operator does not
reclaim them when the EtcdCluster goes. The seaweedfs one removes the
volume PVCs by label and explicitly reclaims the keep-protected CNPG
Cluster holding the filer metadata, because nothing else would. Both are
right for a real uninstall, and neither hook is at fault here.
Remediation is what calls them, on a release that is only starting
slowly.

So an install that misses its budget does not restart. It destroys the
datastore the tenant control planes keep their state in, or the object
store's data together with the metadata database that indexes it, and
the next attempt begins from nothing.

That budget is not a comfortable margin. The seaweedfs release has
`timeout: 10m`, and `hack/e2e-install-cozystack.bats` already records
that on a loaded runner the tenant stack only starts creating pods
around 9 to 10 minutes in. The window CI observes and the window that
triggers this are the same window.

`RetryOnFailure` keeps the manifests applied and retries the failed
release as an upgrade. No uninstall happens, so neither cleanup Job runs
and the install already under way carries on instead of starting over.

Both actions carry the strategy. The controller runs the retry of a
failed install as an upgrade, so with install alone the retry's own
failure falls through to upgrade remediation. The remediation blocks
stay: an absent release consults install, an out-of-sync one consults
upgrade, both by retry count rather than by strategy.

One trade is that a real upgrade failure retries instead of rolling
back. That costs no stable safety net. With `retries: -1` and the
default strategy, upgrade remediation was already rolling back and
retrying without end, so the release never rested in the rolled-back
state either.

The other side of the trade is that the uninstall was also the only
automatic clean slate. On a tenant holding no data yet, an install that
wedges on a bad resource state now retries against that state until an
operator removes the release. For a release that owns data that is the
right exchange, because lost data does not come back and a wedged
install does.

This covers two of the tenant's data-owning releases. The monitoring
release carries the same default and a cleanup Job that deletes its PVCs
as well, and is left for its own change, as is the rest of the catalog,
whose nested releases want the question asked one at a time. Same shape
as #3552 and #3581. Relates to #3555 and #3576.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(tenant): retry a failed etcd or seaweedfs install instead of uninstalling the release, so a slow first install no longer runs the cleanup hooks that delete the etcd data volumes, the SeaweedFS volumes and the filer metadata database
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Reliability Improvements**
* Improved installation and upgrade recovery for etcd and SeaweedFS by
automatically retrying failed operations every 30 seconds.
* Preserved ongoing remediation attempts to help services recover
without uninstalling or rolling back after failures.

* **Tests**
* Added coverage to verify retry behavior and remediation settings for
both services.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

(cherry picked from commit 36f83fa)
Signed-off-by: Myasnikov Daniil <[email protected]>