generated HelmReleases retry a failed install forever, with no cap, backoff, or notion of a terminal failure #4155

Every HelmRelease the platform generates retries a failed install or upgrade forever, and nothing in the policy can tell a release that is slow from one that will never succeed.

Both generators, pkg/registry/apps/application/rest.go and internal/operator/package_reconciler.go, set Install.Strategy and Upgrade.Strategy to RetryOnFailure with an explicit RetryInterval, defaulting to 30s from pkg/cmd/server/start.go:90. Neither sets Remediation, so there is no retry count and no exhaustion. A release whose failure is permanent re-enters its wait 30 seconds after each timeout, for as long as the object exists.

For a transient cause that is exactly right, and it is why the setting exists. The comment at rest.go:1580 records the reasoning: RetryOnFailure with an explicit interval was chosen instead of Remediation{Retries: -1} precisely to decouple retry timing from Spec.Interval, because the previous coupling made failed installs retry only every five minutes and blew past E2E budgets. Any fix here has to keep that property, so reaching back for a remediation-based cap would undo it.

What is missing is the other half. A release that cannot succeed, for instance one whose readiness gate waits on an object the storage layer will never be able to create, is indistinguishable from one that is merely slow, and the policy treats both the same forever. The cost is not only the retry itself: each attempt re-enters a wait that occupies one of the shard's helm-controller slots for its whole window, which is what turns a permanent failure into near-continuous occupancy rather than a one-off. That side is #4145, and this is the other lever on the same symptom; either one alone leaves the other standing.

Worth deciding together rather than separately, because the options trade against each other. A retry cap bounds the damage but makes the platform stop converging on a cause that later clears itself. Backoff keeps convergence and reduces occupancy without a cliff. Detecting terminal failure is the most precise and the most work, and helm-controller already has the notion, since it stalls a release with a terminal error for an uncompilable CEL expression; the distinction exists in the controller and simply is not reachable from a wait timeout.

Nothing here measures how often a permanently failing release actually occurs in a running cluster, and that number is what should decide how much of this is worth building.