Repository navigation
Commit 0827bcb
committed
fix(apps/kubernetes): report the pre-delete cleanup outcome honestly
The pre-delete cleanup Job ended with an unconditional "Cleanup
completed successfully". Several of its steps swallow a failure behind
`|| echo "WARNING: ..."` so that `set -e` cannot end the run, so that
line was printed in exactly the situation the warnings exist to report:
child HelmReleases still holding finalizers, or a Kamaji datastore
Secret still pinning its namespace in Terminating.
Record what a step leaves behind and close with either the success line
or the list of steps that left something. Residue is the thing worth
recording, not every fallback: a bounded delete that times out has a
remedy right behind it -- Step 2 falls through to force-clearing
finalizers, Step 4 falls through to Step 4b -- and whether anything
survived is decided by that remedy, which reports its own outcome. A
tenant with no working nodes takes that route as a matter of course,
which is why the wait is bounded in the first place, and calling such a
run incomplete would send an operator hunting for objects that are not
there.
A read that fails looks exactly like one that found nothing: an empty
list and a failed list both give a loop nothing to iterate, and an
absent object and an unreachable apiserver both make a probe non-zero.
Every read the summary rests on now keeps its status. Step 2's
force-clear re-list reports when it cannot list what it was about to
clear, because everything it could not return is still holding its
finalizer. Step 4b's two probes became three-way -- present, absent or
unknown -- so a Secret whose existence could not be established is no
longer announced as absent. Step 1's list is the one read still
swallowing its status, and that is a decision rather than an omission:
Step 2 deletes by label rather than from that list, so a suspend that
never happened leaves nothing of its own behind and reporting it would
name residue that does not exist.
Making the TenantControlPlane probe three-way also repairs the gate
above it, and that is intended rather than incidental. The gate exists
so the datastore Secret's finalizer is not stripped while Kamaji may
still own it, and a two-way probe defeated it in the one direction that
does damage: any error reads as "confirmed absent" and the strip goes
ahead. Its likeliest cause, a throttled or briefly unreachable
apiserver, is also likeliest exactly when a broken tenant is being torn
down.
Step 4b declining to strip a finalizer while the TenantControlPlane is
still alive is the one residue that no failing command marks: every
call in that branch succeeds and the namespace stays in Terminating
regardless. It now reports, and names the TenantControlPlane as the
thing holding the namespace up.
The exit code is deliberately unchanged and still zero. These are
backstops, not the main path: they run after the normal deletion has
already timed out, and they exist to push teardown further along, so
failing there would abort the Helm uninstall at the very moment the
backstop was written for, leaving the release in storage and the same
teardown retried forever. That makes the closing message the only
outcome report an operator gets, which is why it has to be accurate.
The steps that are not backstopped still end the run under `set -e` and
never reach that message.
Rework the runtime warning and the RBAC comment that pointed at the
tracking issue for the Kamaji datastore-secret leak. That issue is
closed, and closed by this very step rather than by a fix in Kamaji, so
a reader who followed the pointer found a resolved report and no reason
for the code to exist. The step comment now says the ordering it covers
is still reachable, and the warning names the finalizer and the object
to clear by hand.
Cover the behaviour with a bats file that renders the Job, extracts the
script and runs it against a stub kubectl, since the helm-unittest cases
match the script's source text and cannot tell whether the closing
message is reachable.
Assisted-By: Claude <[email protected]>
Signed-off-by: Aleksei Sviridkin <[email protected]>1 parent b4e2031 commit 0827bcb
2 files changed
Lines changed: 542 additions & 15 deletions
0 commit comments