Changing a platform value upgrades every app and re-runs their Jobs although the render is unchanged #5092

Describe the bug

Changing a platform-level value that most applications never read, such as publishing.apiServerEndpoint, upgrades every application HelmRelease in every tenant, and each of those upgrades starts Jobs again even though the chart renders byte-for-byte the same manifest. On a production cluster with ~1,580 HelmReleases this created about 80 Jobs within a minute and saturated pod creation for a while.

Two mechanisms, neither of which is a content hash picking up unrelated values:

1. _cluster is copied into each namespace's cozystack-values Secret and fed to every HelmRelease through valuesFrom. Flux upgrades a release whenever its values digest changes, regardless of whether the render changes. So any _cluster key change upgrades every app, including apps that never read that key (here api-server-endpoint is read only by packages/extra/info, and expose-services by none of the apps below).
2. On that no-op upgrade, Jobs run again:
   • post-install,post-upgrade hooks run on every upgrade by Helm's definition: postgres <release>-init-job, kubernetes <release>-oidc-bootstrap, the monitoring oidc-users hook.
   • The main-phase Jobs kubernetes <release>-bootstrap-token-tenant-<hash> and kubernetes-nodes (formerly kubernetes) <release>-talos-reconcile-<group>-<hash> keep the same content-hash name, but ttlSecondsAfterFinished: 600 deleted them minutes after their previous run. Helm finds a resource of the new manifest missing from the cluster and creates it again. The template comment anticipates this ("only an actual upgrade does"), but a platform-level value change makes it an upgrade of every cluster at once.

Environment

• Cozystack v1.6.5 (the templates involved are unchanged in this respect on main at 55aad6f)
• Provider: on-prem

To Reproduce

1. Run a platform with a few Postgres and Kubernetes applications.
2. Change publishing.apiServerEndpoint (or any other _cluster key) in the platform Package.
3. Watch kubectl get jobs -A --sort-by=.metadata.creationTimestamp.

Expected behaviour

A platform value that an application does not use does not run that application's Jobs again, and ideally does not upgrade its release at all.

Actual behaviour

Measured on the cluster above after changing publishing.apiServerEndpoint (https://api.<domain>:443 → https://api.<other-domain>:6443; the same change also dropped api from expose-services). Jobs created within the following minute:

For the 36 releases behind those Jobs, I compared the manifest stored in the last two Helm release Secrets (the revision before the change and the one it produced). 31 are byte-identical. The other 5 are Postgres releases whose only difference is the order of the database names in the init script, a separate rendering bug fixed in #5091. Example, one kubernetes release, revisions 49 → 50:

_cluster.api-server-endpoint: https://api.<domain>:443  ->  https://api.<other-domain>:6443
_cluster.expose-services:     api,dashboard,...         ->  dashboard,...
manifest diff:                0 lines
Jobs in manifest (both):      <release>-bootstrap-token-tenant-53dbf6, <release>-talos-reconcile-md0-f308bf
talos-reconcile Job creationTimestamp: the minute of the change

The same holds with helm template on v1.6.5, rendering packages/apps/kubernetes and packages/apps/postgres with two _cluster blocks that differ only in api-server-endpoint: the only differences in the output are the random values helm template generates for <release>-talos-secrets (no lookup offline), and the Job names are the same:

a: kubernetes-x-bootstrap-token-tenant-b741a0 kubernetes-x-talos-reconcile-md0-8d5cb1 kubernetes-x-pre-cleanup kubernetes-x-oidc-bootstrap
b: kubernetes-x-bootstrap-token-tenant-b741a0 kubernetes-x-talos-reconcile-md0-8d5cb1 kubernetes-x-pre-cleanup kubernetes-x-oidc-bootstrap
postgres (a, b): postgres-x-init-job, no diff

Additional context

Options, roughly from the root cause outward. They are design choices for maintainers, which is why this is an issue and not a PR:

• Do not upgrade an app for a _cluster key it does not read. For instance, let an ApplicationDefinition declare the _cluster keys its chart consumes and project only those into the values it receives, or split the rarely read keys (api-server-endpoint, expose-services, branding) out of the block every app gets. This removes the upgrade storm itself, Jobs or not.
• Make the main-phase Jobs survive a successful run. With no ttlSecondsAfterFinished on success, a no-op upgrade finds the Complete Job in place and does nothing. The TTL exists so that a Failed Job is retried on the next upgrade (fix(kubernetes): run the Talos reconcile Job in the main install phase #3145, fix(kubernetes): break bootstrap-token Job out of Helm hook to unblock fresh installs #3887); keeping that needs something other than a TTL that applies to both outcomes, e.g. a podFailurePolicy with a large backoffLimit, or a periodic cleanup of Failed Jobs only.
• Hooks. fix(postgres): run init-job in main phase and wait for the superuser secret #3923 moves the postgres init Job to the main phase with a content-hash name; it would then inherit the TTL behaviour above, so the two decisions interact. oidc-bootstrap and the monitoring oidc-users hook would need the same treatment to stop running on every upgrade.

Related: #2497 (umbrella on bootstrap workarounds), #4342 (another source of upgrades the user never asked for).

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs