Skip to content

[Backport release-1.6] fix(kubernetes): bound the remaining pre-delete hook deletions - #5060

Merged
myasnikovdaniil merged 1 commit into
release-1.6from
backport-3604-to-release-1.6
Oct 22, 2026
Merged

myasnikovdaniil merged 1 commit into
release-1.6from
backport-3604-to-release-1.6

Conversation

@github-actions

Copy link
Copy Markdown

Description

Backport of #3604 to release-1.6.

Three of the six steps in the pre-delete hook call kubectl delete
with neither --wait=false nor --timeout. kubectl defaults --wait to
true and reads --timeout=0 as "wait forever", which it implements as
a week, so the KamajiControlPlane delete in step 3, the DataVolume
delete in step 5 and the LoadBalancer Service delete in step 6 each
park the hook for 168 hours on a single stuck finalizer. A DataVolume
held by a CDI importer or a PVC finalizer is the reachable case.

None of the three waits now. Nothing downstream reads the result of
steps 5 and 6, and what matters after step 3 is the
TenantControlPlane, which step 4 deletes and waits for directly;
blocking on the KamajiControlPlane first only adds a second place for
a CAPI finalizer to stall. An expired wait and --wait=false leave the
same cluster state either way, with the same objects finalizing in
the background once the hook has moved on.

Not waiting also keeps these steps under `set -e` with no guard. A
guard cannot tell an expired wait from a rejected request, so it
would report a failed delete as cleanup that happened; with no
timeout to expire, every non-zero exit left is a real failure.

Drop the watch verb from the three resources those steps delete, and
pin the whole rule set. kubectl opens a watch only on the wait path,
so the grant was unexercised for the three; for helmreleases and
tenantcontrolplanes its absence would not error but hang, which is
the failure mode hardest to read from the outside.

Set backoffLimit to 0, against a default of 6 that restartPolicy
Never turns into seven pods in series. Zero rather than one because
both remaining waits are guarded, so an attempt can spend its whole
budget without failing and a retry would start from step 1 with the
clock already there; on a nodeless tenant it would also re-pay step
4, since attempt one leaves the TenantControlPlane still terminating.
The failure a retry would cover is an eviction, which is not bounded
in time and is worst exactly when it arrives late. The price is that
such an eviction fails the whole attempt and flux repeats it on its
own interval.

No activeDeadlineSeconds: the waits and helm-controller's uninstall
budget leave no gap a deadline could sit in, once a Docker Hub image
pull is counted. That needs the budget raised first, through
spec.uninstall.timeout on the generated HelmRelease, and is left out
rather than set to a number that would cut runs which were going to
finish.

These bounds cover the waiting, not the request: --request-timeout is
still 0 on every call in the hook, so an unresponsive apiserver hangs
it as before. That is a different failure from the stuck finalizer,
and it wants one answer across every call rather than a flag on the
deletes.

Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>
(cherry picked from commit b60c08f)
@coderabbitai

coderabbitai Bot commented Oct 22, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@myasnikovdaniil
myasnikovdaniil merged commit 24e23cf into release-1.6 Oct 22, 2026
12 checks passed
@myasnikovdaniil
myasnikovdaniil deleted the backport-3604-to-release-1.6 branch October 22, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants