[Backport release-1.6] fix(monitoring-agents): let the release uninstall when the VictoriaMetrics operator is gone first - #5158

…trics operator is gone first (#5152)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Disabling the Monitoring Agents addon of a tenant Kubernetes cluster
leaves `<cluster>-monitoring-agents` (release `cozy-monitoring-agents`,
installed into the tenant cluster) in `UninstallFailed` for good. Seen
in production on 1.6.5 and still reproducible from `main`.

`monitoring-agents`, `victoria-metrics-operator` and
`vertical-pod-autoscaler-crds` in
`packages/apps/kubernetes/templates/helmreleases/` are all gated on
`addons.monitoringAgents.enabled`, so they disappear together and Flux
uninstalls them in parallel: `dependsOn` is not honoured on deletion.
Two things then break the uninstall of `monitoring-agents`:

1. The VictoriaMetrics operator is often gone first. The operator's
`apps.victoriametrics.com/finalizer` stays on the `VMAgent`, nobody
removes it, and the deletion hangs. The manual workaround was to strip
the finalizer, after which GC removes the VMAgent's Deployment, Service,
Secrets and ServiceAccount through their owner references.
2. The VPA CRD was deleted with its own release while the stored
manifest still contains `VerticalPodAutoscaler vpa-vmagent`, so Helm
cannot build the manifest to uninstall. That root cause is fixed by
#3586 (`helm.sh/resource-policy: keep` on the tenant addon CRDs), which
this PR does not redo.

This PR:

- adds a `pre-delete` hook (`templates/vm-cleanup-hook.yaml`) that
deletes every VictoriaMetrics object the release owns, waits up to 60s
for a live operator to finalize them properly, and clears their
finalizers by hand when it has not. Objects are selected through Helm's
`meta.helm.sh/release-name` / `release-namespace` annotations rather
than by name, so the 34 `VMRule`s rendered from `alerts/` and the
`VMServiceScrape` in `cozy-victoria-metrics-operator` are covered along
with the VMAgent. The kinds handled are every VictoriaMetrics kind the
chart renders: `VMAgent`, `VMRule`, `VMServiceScrape`, `VMPodScrape`,
`VMNodeScrape`. Operator v0.68.4 only finalizes the VMAgent; the other
kinds are covered in case another operator version finalizes them, and
for them the patch is a no-op. A kind whose CRD is already gone is
skipped, and the script always exits 0, since a failing pre-delete hook
would itself block the uninstall. The Job is bounded by
`activeDeadlineSeconds: 180` and `backoffLimit: 0`; its kubectl calls
carry no `--request-timeout`, which would make kubectl skip the
in-cluster credentials (`hack/check-hook-request-timeout.bats`) and
reuses the digest-pinned `docker.io/clastix/kubectl:v1.32` image the
etcd-operator hook uses, via a new `kubectlImage` value. The shape
follows
`packages/system/forklift/templates/forklift-cr-cleanup-hook.yaml`.
- guards `templates/vpa.yaml` on `.Capabilities.APIVersions.Has
"autoscaling.k8s.io/v1/VerticalPodAutoscaler"`, so a cluster that does
not serve the CRD never records a VPA in the release manifest. Cheap
defence in depth next to #3586.

Points for review:

- The hook's RBAC is a ClusterRole/ClusterRoleBinding, limited to `get,
list, delete, patch` on the five `operator.victoriametrics.com`
resources, rather than Roles. The objects span two namespaces, and a
Role in `cozy-victoria-metrics-operator` could not be created if that
namespace were already gone or terminating, which would fail the hook
and the uninstall with it. All hook objects carry
`hook-succeeded,hook-failed,before-hook-creation` except the Job, which
keeps a failed pod for its logs (`ttlSecondsAfterFinished: 600`), as in
the forklift hook.
- When the operator is already gone, clearing the finalizer skips
whatever the operator does on finalize, notably the cluster-scoped RBAC
it creates for the VMAgent, which has no owner reference. That residue
is the same the manual workaround left, and it is no worse than today's
stuck release; removing it would mean hard-coding the operator's naming
scheme for those objects, which I preferred not to do.
- When the operator is alive, the hook only brings forward deletions
Helm was about to make, so the operator's normal finalization still
runs.

Tests: `tests/uninstall_test.yaml` asserts that the hook is a
`pre-delete` Job in the release namespace with a bounded runtime, that
its RBAC is hook-scoped and grants exactly the verbs needed on every
VictoriaMetrics kind the chart renders, that the binding targets the
hook's ServiceAccount, that the script selects by release ownership,
deletes, clears finalizers and exits 0, and that the VPA renders only
when the capability is present (using `capabilities.apiVersions`). With
the hook removed and `vpa.yaml` reverted, 6 of the 7 tests fail; with
the fix, `helm unittest packages/system/monitoring-agents` passes 26/26.

`hack/monitoring-agents-vm-cleanup-hook.bats` runs the rendered script
against a fake `kubectl` that keeps the objects in a state file, with
fake `date` and `sleep`. It checks that only objects annotated for this
release are deleted and patched, that with the operator gone the hook
polls through the grace period and then clears the finalizers, that a
live operator's objects are never patched, that a missing CRD still
exits 0 and cleans the other kinds, and that a release with nothing left
exits 0 without deleting anything. Removing the ownership condition,
making the patch a no-op, or inverting the polling condition each turns
this file red. The hook has not yet been run on a live tenant cluster.

Backport: labelled `kind/backport` (release-1.7). The release-1.6
backport is done by hand in #5158. The commit cherry-picks onto
`release-1.6` with a single trivial conflict in `values.yaml` (context
only: 1.6 lacks the `prometheus-node-exporter` resources block the new
`kubectlImage` block sits after on `main`). Resolved locally, the
chart's tests pass on 1.6 (9/9); 1.6's chart renders the same
VictoriaMetrics kinds. Note that `release-1.6`'s monitoring-agents
`Makefile` has no `test:` target, so CI does not run this chart's unit
tests on that branch.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff (a system chart's templates,
`values.yaml` and tests): no package added or renamed, no platform
values, no app schema, no CRD, no tooling or convention change. I also
searched cozystack/website for a documented VMAgent finalizer workaround
that this would make obsolete and found none.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(monitoring-agents): disabling the Monitoring Agents addon of a tenant Kubernetes cluster no longer leaves the monitoring-agents release stuck in UninstallFailed when the VictoriaMetrics operator is removed first; a pre-delete hook now releases the VictoriaMetrics objects' finalizers, and the VPA is only rendered when its CRD is served.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* VictoriaMetrics resources belonging to a release are cleaned up during
uninstall, including resources that remain because of finalizers.
Cleanup is limited to resources owned by that release.
* Vertical Pod Autoscaler resources are rendered only when the cluster
supports the required API.
* **Configuration**
* Added an image setting for the uninstall cleanup job, including a
pinned image version and pull policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->