fix(kubernetes): keep tenant addon CRDs when a release is uninstalled - #3586

## What this PR does

`make update` in `packages/system/etcd-operator-crds` vendors three CRDs
and stamps `helm.sh/resource-policy: keep` onto each with an awk step
anchored on the `controller-gen.kubebuilder.io/version:` line. That line
is upstream's to emit. If a future controller-gen stops writing it, the
awk matches nothing, writes the CRD out unstamped and exits zero, so the
regeneration reports success and the CRDs land without the annotation
that keeps a chart uninstall from deleting them, and with them every
EtcdCluster, EtcdMember and EtcdSnapshot in the cluster.

The step now counts what it produced and requires exactly one annotation
per CRD. Zero means the anchor is gone; more than one means it matched
somewhere unintended. Every CRD is staged in a temporary directory and
`templates/` is replaced only after all of them have been counted, so a
failure on any one leaves the vendored files exactly as they were rather
than wiping the directory and refilling part of the set.

I checked it by substituting the upstream response, so what ran was the
recipe that ships rather than a copy of its logic: a CRD with no
controller-gen line fails with `stamped 0 keep annotations, expected 1`,
a CRD carrying that line twice fails with 2, a failure on the second of
the three leaves all three committed files untouched, and the real
upstream still vendors byte-identical output, with `make update` leaving
`git status` showing nothing but this Makefile.

On `main` today this is the only thing guarding those three CRDs. A
helm-unittest suite that fails when one of them renders without the
annotation is in #3590, which is open and not merged. Once that lands
the two catch the same loss at different moments: the suite on a test
run, this step in front of whoever ran the regeneration.

The same stamping shape exists in three more packages, but only on the
branch of #3586, which adds it to `gateway-api-crds`, `vsnap-crd` and
`vertical-pod-autoscaler-crds`. There is nothing to guard there on
`main` yet, so they are deliberately out of this change and need the
same step once #3586 merges.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

I walked the trigger map in `docs/agents/contributing.md` against the
file list of this diff, which is one package Makefile and nothing else.
`hack/package.mk` is untouched, so the `ccp` and `external-apps-example`
triggers that key on it do not fire, and the website's "developer
tooling (the package Makefiles)" line is the only near miss:
`content/en/docs/next/development.md` describes `make update` in general
terms and does not go stale from a package validating its own output.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved CRD update validation to ensure generated definitions include
the required preservation annotation.
* Prevented incomplete or invalid updates from replacing existing CRD
templates.
  * Added automatic cleanup of temporary update files.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

…trics operator is gone first (#5152)

<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

Disabling the Monitoring Agents addon of a tenant Kubernetes cluster
leaves `<cluster>-monitoring-agents` (release `cozy-monitoring-agents`,
installed into the tenant cluster) in `UninstallFailed` for good. Seen
in production on 1.6.5 and still reproducible from `main`.

`monitoring-agents`, `victoria-metrics-operator` and
`vertical-pod-autoscaler-crds` in
`packages/apps/kubernetes/templates/helmreleases/` are all gated on
`addons.monitoringAgents.enabled`, so they disappear together and Flux
uninstalls them in parallel: `dependsOn` is not honoured on deletion.
Two things then break the uninstall of `monitoring-agents`:

1. The VictoriaMetrics operator is often gone first. The operator's
`apps.victoriametrics.com/finalizer` stays on the `VMAgent`, nobody
removes it, and the deletion hangs. The manual workaround was to strip
the finalizer, after which GC removes the VMAgent's Deployment, Service,
Secrets and ServiceAccount through their owner references.
2. The VPA CRD was deleted with its own release while the stored
manifest still contains `VerticalPodAutoscaler vpa-vmagent`, so Helm
cannot build the manifest to uninstall. That root cause is fixed by
#3586 (`helm.sh/resource-policy: keep` on the tenant addon CRDs), which
this PR does not redo.

This PR:

- adds a `pre-delete` hook (`templates/vm-cleanup-hook.yaml`) that
deletes every VictoriaMetrics object the release owns, waits up to 60s
for a live operator to finalize them properly, and clears their
finalizers by hand when it has not. Objects are selected through Helm's
`meta.helm.sh/release-name` / `release-namespace` annotations rather
than by name, so the 34 `VMRule`s rendered from `alerts/` and the
`VMServiceScrape` in `cozy-victoria-metrics-operator` are covered along
with the VMAgent. The kinds handled are every VictoriaMetrics kind the
chart renders: `VMAgent`, `VMRule`, `VMServiceScrape`, `VMPodScrape`,
`VMNodeScrape`. Operator v0.68.4 only finalizes the VMAgent; the other
kinds are covered in case another operator version finalizes them, and
for them the patch is a no-op. A kind whose CRD is already gone is
skipped, and the script always exits 0, since a failing pre-delete hook
would itself block the uninstall. The Job is bounded by
`activeDeadlineSeconds: 180` and `backoffLimit: 0`; its kubectl calls
carry no `--request-timeout`, which would make kubectl skip the
in-cluster credentials (`hack/check-hook-request-timeout.bats`) and
reuses the digest-pinned `docker.io/clastix/kubectl:v1.32` image the
etcd-operator hook uses, via a new `kubectlImage` value. The shape
follows
`packages/system/forklift/templates/forklift-cr-cleanup-hook.yaml`.
- guards `templates/vpa.yaml` on `.Capabilities.APIVersions.Has
"autoscaling.k8s.io/v1/VerticalPodAutoscaler"`, so a cluster that does
not serve the CRD never records a VPA in the release manifest. Cheap
defence in depth next to #3586.

Points for review:

- The hook's RBAC is a ClusterRole/ClusterRoleBinding, limited to `get,
list, delete, patch` on the five `operator.victoriametrics.com`
resources, rather than Roles. The objects span two namespaces, and a
Role in `cozy-victoria-metrics-operator` could not be created if that
namespace were already gone or terminating, which would fail the hook
and the uninstall with it. All hook objects carry
`hook-succeeded,hook-failed,before-hook-creation` except the Job, which
keeps a failed pod for its logs (`ttlSecondsAfterFinished: 600`), as in
the forklift hook.
- When the operator is already gone, clearing the finalizer skips
whatever the operator does on finalize, notably the cluster-scoped RBAC
it creates for the VMAgent, which has no owner reference. That residue
is the same the manual workaround left, and it is no worse than today's
stuck release; removing it would mean hard-coding the operator's naming
scheme for those objects, which I preferred not to do.
- When the operator is alive, the hook only brings forward deletions
Helm was about to make, so the operator's normal finalization still
runs.

Tests: `tests/uninstall_test.yaml` asserts that the hook is a
`pre-delete` Job in the release namespace with a bounded runtime, that
its RBAC is hook-scoped and grants exactly the verbs needed on every
VictoriaMetrics kind the chart renders, that the binding targets the
hook's ServiceAccount, that the script selects by release ownership,
deletes, clears finalizers and exits 0, and that the VPA renders only
when the capability is present (using `capabilities.apiVersions`). With
the hook removed and `vpa.yaml` reverted, 6 of the 7 tests fail; with
the fix, `helm unittest packages/system/monitoring-agents` passes 26/26.

`hack/monitoring-agents-vm-cleanup-hook.bats` runs the rendered script
against a fake `kubectl` that keeps the objects in a state file, with
fake `date` and `sleep`. It checks that only objects annotated for this
release are deleted and patched, that with the operator gone the hook
polls through the grace period and then clears the finalizers, that a
live operator's objects are never patched, that a missing CRD still
exits 0 and cleans the other kinds, and that a release with nothing left
exits 0 without deleting anything. Removing the ownership condition,
making the patch a no-op, or inverting the polling condition each turns
this file red. The hook has not yet been run on a live tenant cluster.

Backport: labelled `kind/backport` (release-1.7). The release-1.6
backport is done by hand in #5158. The commit cherry-picks onto
`release-1.6` with a single trivial conflict in `values.yaml` (context
only: 1.6 lacks the `prometheus-node-exporter` resources block the new
`kubectlImage` block sits after on `main`). Resolved locally, the
chart's tests pass on 1.6 (9/9); 1.6's chart renders the same
VictoriaMetrics kinds. Note that `release-1.6`'s monitoring-agents
`Makefile` has no `test:` target, so CI does not run this chart's unit
tests on that branch.

### Screenshots

Not a UI change.

### Downstream repositories

I walked the trigger map against the diff (a system chart's templates,
`values.yaml` and tests): no package added or renamed, no platform
values, no app schema, no CRD, no tooling or convention change. I also
searched cozystack/website for a documented VMAgent finalizer workaround
that this would make obsolete and found none.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(monitoring-agents): disabling the Monitoring Agents addon of a tenant Kubernetes cluster no longer leaves the monitoring-agents release stuck in UninstallFailed when the VictoriaMetrics operator is removed first; a pre-delete hook now releases the VictoriaMetrics objects' finalizers, and the VPA is only rendered when its CRD is served.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* VictoriaMetrics resources belonging to a release are cleaned up during
uninstall, including resources that remain because of finalizers.
Cleanup is limited to resources owned by that release.
* Vertical Pod Autoscaler resources are rendered only when the cluster
supports the required API.
* **Configuration**
* Added an image setting for the uninstall cleanup job, including a
pinned image version and pull policy.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->