Skip to content

Commit 1f92bcd

Browse files
fix(fluxcd): close the flux-plunger review follow-ups for the hook, drain and Warnings (#4914)
## What this PR does Follow-up to the flux-plunger uninstall fence: the points the last review round raised and left for after the merge, plus four found by a later independent review. None changes the fence's design; each closes a gap in what it reports, how long it waits, or what its documentation says. **Pre-delete hook.** - A release being deleted that only the fence still holds is no longer unfenced in the same step that lifts its suspension. helm-controller runs the uninstall of a release being deleted once it is no longer suspended, but removing the fence at once deleted the object before helm-controller could reconcile it. The hook now keeps that fence while helm-controller handles the generation the lifted suspension wrote, until shortly before its retry window closes, and names a release it still had to let go of as before. A release whose state cannot be read keeps its fence for the wait the same way, rather than being let go of on one failed read. - A release being deleted while still suspended, by anyone but flux-plunger, was reported as cleanly released, although helm-controller skips the uninstall of a suspended release. It is now named as deleted without an uninstall and counted with the others. - Each release gets its own share of the retry window, capped at the window's end. One release that could never be read used to take the whole window and leave every release listed after it a single attempt. The share is a cap, not a floor of attempts: a floor would let enough failing releases run the Job past its deadline, which kills it before its report is written. - The hook stops, waits for and lists again only for the flux-plunger of its own release. Another one (a second installation under another release name, changed labels, a binary outside the cluster) is not stopped by the hook, keeps the fences it sets and gives them up itself; the README now says so instead of the hook re-listing unconditionally, which would not close that case and would fail the uninstall for as long as a second installation is recovering anything. - helm-controller's finalizer, which the hook reads to tell whether a release being deleted is still held, is tied to the constant the controller uses, as the fence already was. Two comments are corrected: the retry window counts from the script's start, not the Job's, and kubectl scale's internal conflict retry never runs here. **Shutdown drain.** - A drain deadline of zero or less is refused at startup, and the chart's lowest accepted grace period is held to the same constants as the drain subtraction, so raising the manager's stop bound cannot make the chart render a deadline that is already spent. The timings the README and the values comment state are held to those constants too. - Fences left behind are reported as an error naming each release, not an Info line; a drain that never got a listing answered says so instead of reporting a count of -1; a flux-plunger that never led no longer claims it released every fence. - The drain's Events fold repeats into one counted Event per release and reason, as the manager's recorder does, instead of one object per pass. The fold is a patch, the verb the chart grants on events, and every write is bounded by the drain's deadline. It does not add a per-release Event for fences left at the end: by then the drain's deadline is spent, and writing one would reach past it. - Tests now hold the outcome line, the retry of a release whose fence cannot be given up yet, and the pause between passes, none of which any suite noticed when removed. **Controller.** The `SuspensionNotOwned` and `TerminatingWhileSuspended` Warnings were keyed on the processed-version annotation, which stays on every release flux-plunger ever recovered, so a later restore, migration or operator suspension of a healthy release drew a Warning telling the reader to resume it, and a migration's deliberate suspend-then-delete was called an accident. Both now key on the state a leftover actually freezes the release in, `Ready=False` with `has no deployed releases`, since helm-controller rewrites no condition of a suspended release; the README's candidate query follows. Two comments that claimed more than holds are qualified. **Chart and README.** A `shutdownGracePeriodSeconds` of a million or more now gets the range message rather than being refused as a fraction. The README gives the recovery steps in the order the controller runs them (the annotation is written before the Secret is deleted), and names the window in which a backup restore started during a recovery is unsuspended under it, since the etcd, CNPG and Velero restore steps write `spec.suspend` only when it differs. ### Testing Each change was written test first: a test that builds the binary and runs it with a deadline of zero, helm-unittest for the chart, unit tests with a captured logger and the fake client for the drain's outcome, Events and passes (the Event fold under a client that refuses the update verb, as the chart's role does), table tests for both Warnings, and the bats suite for the hook, whose fake kubectl now models helm-controller handling a release some reads after its suspension is lifted and a state read that fails. Every fix was then reverted on its own, and the suite went red each time. The envtest suites pass unchanged. The README's candidate query was run against HelmReleases read from a live cluster, with a stuck, a healthy, an owned and an unsuspended variant, and lists only the stuck one. ### Screenshots Not applicable: no UI change. ### Downstream repositories The trigger map was walked against the diff: `internal/controller/fluxplunger`, `cmd/flux-plunger`, the `packages/system/flux-plunger` chart (templates, tests, README) and its bats test under `hack/`. The website's FluxPlunger section in `guides/concepts.md` draws the recovery with the Secret deleted before the annotation is written, which is the order before the fence change; it needs the same correction as the chart README. That change is held as a draft until v1.7.0 is released and the website's `next/` is re-initialized: the fence change is not part of release-1.7, so the corrected order would be wrong for v1.7 docs, which are copied from `next/`. No other repository is reached by this diff. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#752 (draft, merge after v1.7.0) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(fluxcd): flux-plunger's pre-delete hook now lets helm-controller run the uninstall of a release being deleted before it removes the release's fence, gives each release its own share of its retries, and reports releases deleted under another manager's suspension; the shutdown drain reports the fences it leaves as errors and refuses a deadline that leaves no time ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Shutdown drain timeouts must be positive, and invalid values now stop startup with a clear error. * Uninstall cleanup waits for Helm to handle releases before removing their fences, while bounding retries and reporting releases that remain incomplete. * Recovery and suspension warnings now reflect the release’s actual failure state, and recovery can retry stale storage cleanup against the same revision. * **Documentation** * Clarified shutdown grace-period requirements, uninstall behavior, and guidance for backup restores and existing suspensions. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents d1d4e92 + 7011510 commit 1f92bcd

12 files changed

Lines changed: 926 additions & 125 deletions

File tree

‎cmd/flux-plunger/main.go‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -82,6 +82,11 @@ func main() {
8282

8383
ctrl.SetLogger(zap.New(zap.UseFlagOptions(&opts)))
8484

85+
if err := fluxplunger.ValidateShutdownDrainTimeout(shutdownDrainTimeout); err != nil {
86+
setupLog.Error(err, "invalid flag")
87+
os.Exit(1)
88+
}
89+
8590
// if the enable-http2 flag is false (the default), http/2 should be disabled
8691
// due to its vulnerabilities. More specifically, disabling http/2 will
8792
// prevent from being vulnerable to the HTTP/2 Stream Cancellation and

‎hack/flux-plunger-fence-hook_test.bats‎

Lines changed: 175 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,9 @@ fences_left() {
8989
# * <ns>@<name>.rv resourceVersion, bumped by every write;
9090
# * <ns>@<name>.suspended spec.suspend is true;
9191
# * <ns>@<name>.deleting the release is being deleted;
92+
# * <ns>@<name>.hcdelay helm-controller handles the release, once it is no
93+
# longer suspended, after this many reads of its state:
94+
# the uninstall runs and the generation is observed;
9295
# * <ns>@<name>.owned flux-plunger's Apply entry owns spec.suspend,
9396
# holding what a correct ownership template prints;
9497
# * <ns>@<name>.coowned another manager also owns spec.suspend at true;
@@ -115,7 +118,8 @@ fences_left() {
115118
# for a reason other than pods (wait), the listing
116119
# (list), a listing after the first (relist), the
117120
# pod listing (pods), a release's ownership read
118-
# (<ns>@<name>.ownedread), unsuspend
121+
# (<ns>@<name>.ownedread), a release's state read
122+
# (<ns>@<name>.stateread), unsuspend
119123
# (<ns>@<name>.apply), unsuspend racing another
120124
# write (<ns>@<name>.applyconflict) or finalizer patch
121125
# (<ns>@<name>.patch);
@@ -132,7 +136,10 @@ HELMFIN=finalizers.fluxcd.io
132136
# The one predicate that tells a release helm-controller has not yet handled in
133137
# its unsuspended form: being deleted, no longer held by its finalizer, and its
134138
# latest generation not yet observed.
135-
UNHANDLED='{{if .metadata.deletionTimestamp}}{{$held := false}}{{range .metadata.finalizers}}{{if eq . "finalizers.fluxcd.io"}}{{$held = true}}{{end}}{{end}}{{if and (not $held) (ne (printf "%v" .status.observedGeneration) (printf "%v" .metadata.generation))}}'
139+
# A release being deleted under a suspension, which helm-controller skips the
140+
# uninstall of, is told apart first.
141+
SUSPENDED_DELETING='{{if .metadata.deletionTimestamp}}{{if .spec.suspend}}suspended{{else}}'
142+
UNHANDLED='{{$held := false}}{{range .metadata.finalizers}}{{if eq . "finalizers.fluxcd.io"}}{{$held = true}}{{end}}{{end}}{{if and (not $held) (ne (printf "%v" .status.observedGeneration) (printf "%v" .metadata.generation))}}'
136143
# In the hook's pod kubectl finds the API server only through the in-cluster
137144
# config, which client-go stops falling back to once a client override such as
138145
# --request-timeout is set: the call then goes to localhost:8080 and fails.
@@ -263,8 +270,16 @@ if [ "$verb" = get ]; then
263270
case "$tmpl" in *'ne .manager "flux-plunger"'*) [ -e "$rel.coowned" ] && echo other ;; esac
264271
exit 0 ;;
265272
*deletionTimestamp*)
266-
case "$tmpl" in *"$UNHANDLED"*) ;; *) exit 0 ;; esac
273+
if take_fail "$rel.stateread"; then echo "fake: the state read timed out" >&2; exit 1; fi
267274
[ -e "$rel.deleting" ] || exit 0
275+
case "$tmpl" in *"$SUSPENDED_DELETING"*) if [ -e "$rel.suspended" ]; then printf 'suspended'; exit 0; fi ;; esac
276+
case "$tmpl" in *"$UNHANDLED"*) ;; *) exit 0 ;; esac
277+
# helm-controller handles an unsuspended release being deleted after a
278+
# number of reads: it runs the uninstall and records the generation.
279+
if [ ! -e "$rel.suspended" ] && [ -s "$rel.hcdelay" ]; then
280+
n=$(cat "$rel.hcdelay")
281+
if [ "$n" -le 0 ]; then cp "$rel.gen" "$rel.obs"; else echo $((n - 1)) > "$rel.hcdelay"; fi
282+
fi
268283
grep -q -x "$HELMFIN" "$rel.fin" && exit 0
269284
[ "$(cat "$rel.gen")" = "$(cat "$rel.obs")" ] && exit 0
270285
printf 'unhandled'
@@ -447,6 +462,7 @@ run_hook() {
447462
# Being deleted, helm-controller has handled its current generation.
448463
add_release "$STATE" tenant-c app-c "$FENCE"
449464
touch "$STATE/[email protected]"
465+
rm "$STATE/[email protected]"
450466
# Its uninstall completes mid-sweep, and the release is gone when read.
451467
add_release "$STATE" tenant-d app-d "$FENCE"
452468
touch "$STATE/[email protected]"
@@ -460,6 +476,162 @@ run_hook() {
460476
grep -q -x "released $FENCE from tenant-c/app-c" "$tmp/out"
461477
grep -q -x "released $FENCE from tenant-d/app-d" "$tmp/out"
462478
grep -q -x 'released 4 of 4 fenced HelmReleases, 1 without an uninstall' "$tmp/out"
479+
[ "$(cat "$CLOCK")" -lt "$DEADLINE" ]
480+
rm -rf "$tmp"
481+
}
482+
483+
@test "a release being deleted under another manager's suspension is named as deleted without an uninstall" {
484+
tmp=$(mktemp -d)
485+
prepare "$tmp"
486+
# helm-controller skips the uninstall of a suspended release; the hook leaves a
487+
# suspension another manager also holds, so this release goes without one.
488+
add_release "$STATE" tenant-x app-x "$FENCE"
489+
touch "$STATE/[email protected]"
490+
echo yes > "$STATE/[email protected]"
491+
touch "$STATE/[email protected]"
492+
493+
run_hook "$tmp"
494+
495+
[ "$status" -eq 0 ]
496+
[ "$(fences_left "$STATE")" -eq 0 ]
497+
grep -q -x "released $FENCE from tenant-x/app-x without an uninstall: it is still suspended, not by flux-plunger" "$tmp/out"
498+
grep -q -x 'released 1 of 1 fenced HelmReleases, 1 without an uninstall' "$tmp/out"
499+
rm -rf "$tmp"
500+
}
501+
502+
@test "a release that can never be read does not leave the ones listed after it a single attempt" {
503+
tmp=$(mktemp -d)
504+
prepare "$tmp"
505+
# Listed first, never readable: it must not spend the whole retry window.
506+
add_release "$STATE" tenant-a app-a "$FENCE"
507+
echo 9999 > "$STATE/[email protected]"
508+
# Listed after it, and its fence comes off only on the third try.
509+
add_release "$STATE" tenant-z app-z "$FENCE"
510+
echo 2 > "$STATE/[email protected]"
511+
512+
run_hook "$tmp"
513+
514+
[ "$status" -ne 0 ]
515+
grep -q '^NOT released tenant-a/app-a: it could not be read' "$tmp/out"
516+
grep -q -x "released $FENCE from tenant-z/app-z" "$tmp/out"
517+
[ "$(cat "$CLOCK")" -lt "$DEADLINE" ]
518+
rm -rf "$tmp"
519+
}
520+
521+
@test "releases swept after the retry window closed get one attempt each, so the report is still written" {
522+
tmp=$(mktemp -d)
523+
prepare "$tmp"
524+
# A slow scale-down and the stuck pod's wait take the window to its end; a
525+
# share of it per release must not reach past the window for these.
526+
echo $((WINDOW - 10)) > "$STATE/scale.slow"
527+
touch "$STATE/pods.stuck"
528+
for n in a b c; do
529+
add_release "$STATE" "tenant-$n" "app-$n" "$FENCE"
530+
echo 9999 > "$STATE/tenant-$n@app-$n.ownedread.fail"
531+
done
532+
533+
run_hook "$tmp"
534+
535+
[ "$status" -ne 0 ]
536+
grep -q -x 'released 0 of 3 fenced HelmReleases, 0 without an uninstall' "$tmp/out"
537+
[ "$(cat "$CLOCK")" -lt "$DEADLINE" ]
538+
rm -rf "$tmp"
539+
}
540+
541+
@test "the listing after the sweep gets the rest of the window, not the last release's share" {
542+
tmp=$(mktemp -d)
543+
prepare "$tmp"
544+
# The pod stays, so the fences are listed again; that listing fails for longer
545+
# than one release's share before it answers.
546+
echo 1 > "$STATE/wait.fail"
547+
touch "$STATE/pods.stuck"
548+
add_release "$STATE" tenant-a app-a "$FENCE"
549+
echo 20 > "$STATE/relist.fail"
550+
551+
run_hook "$tmp"
552+
553+
[ "$status" -eq 0 ]
554+
[ "$(fences_left "$STATE")" -eq 0 ]
555+
grep -q -x 'released 1 of 1 fenced HelmReleases, 0 without an uninstall' "$tmp/out"
556+
rm -rf "$tmp"
557+
}
558+
559+
@test "the wait for helm-controller ends early enough that the last fences still get their retries" {
560+
tmp=$(mktemp -d)
561+
prepare "$tmp"
562+
# helm-controller never handles this one, and the first removals of its fence
563+
# fail: the wait must leave the removal time to be retried.
564+
add_release "$STATE" tenant-a app-a "$FENCE"
565+
touch "$STATE/[email protected]"
566+
echo yes > "$STATE/[email protected]"
567+
echo 2 > "$STATE/[email protected]"
568+
569+
run_hook "$tmp"
570+
571+
[ "$status" -eq 0 ]
572+
[ "$(fences_left "$STATE")" -eq 0 ]
573+
grep -q -x "released $FENCE from tenant-a/app-a without an uninstall: helm-controller had not handled it" "$tmp/out"
574+
[ "$(cat "$CLOCK")" -lt "$DEADLINE" ]
575+
rm -rf "$tmp"
576+
}
577+
578+
@test "a state read that fails once does not end the wait for helm-controller" {
579+
tmp=$(mktemp -d)
580+
prepare "$tmp"
581+
add_release "$STATE" tenant-a app-a "$FENCE"
582+
touch "$STATE/[email protected]"
583+
echo yes > "$STATE/[email protected]"
584+
echo 3 > "$STATE/[email protected]"
585+
echo 1 > "$STATE/[email protected]"
586+
587+
run_hook "$tmp"
588+
589+
[ "$status" -eq 0 ]
590+
[ "$(fences_left "$STATE")" -eq 0 ]
591+
[ "$(cat "$STATE/[email protected]")" = "$(cat "$STATE/[email protected]")" ]
592+
grep -q -x "released $FENCE from tenant-a/app-a" "$tmp/out"
593+
grep -q -x 'released 1 of 1 fenced HelmReleases, 0 without an uninstall' "$tmp/out"
594+
rm -rf "$tmp"
595+
}
596+
597+
@test "a release whose state can never be read is still released inside the deadline, and says so" {
598+
tmp=$(mktemp -d)
599+
prepare "$tmp"
600+
add_release "$STATE" tenant-a app-a "$FENCE"
601+
touch "$STATE/[email protected]"
602+
echo yes > "$STATE/[email protected]"
603+
echo 9999 > "$STATE/[email protected]"
604+
605+
run_hook "$tmp"
606+
607+
[ "$status" -eq 0 ]
608+
[ "$(fences_left "$STATE")" -eq 0 ]
609+
grep -q -x "released $FENCE from tenant-a/app-a; whether helm-controller had handled it could not be read" "$tmp/out"
610+
# It kept the fence for the whole wait, as one helm-controller had not handled.
611+
[ "$(cat "$CLOCK")" -ge $((WINDOW - 30)) ]
612+
[ "$(cat "$CLOCK")" -lt "$DEADLINE" ]
613+
rm -rf "$tmp"
614+
}
615+
616+
@test "a release helm-controller handles while the hook waits keeps its fence until the uninstall has run" {
617+
tmp=$(mktemp -d)
618+
prepare "$tmp"
619+
# helm-controller dropped its finalizer on the suspended delete. Once the hook
620+
# lifts flux-plunger's suspension, helm-controller runs the uninstall a few
621+
# seconds later; removing the fence before that deletes the release without it.
622+
add_release "$STATE" tenant-a app-a "$FENCE"
623+
touch "$STATE/[email protected]"
624+
echo yes > "$STATE/[email protected]"
625+
echo 3 > "$STATE/[email protected]"
626+
627+
run_hook "$tmp"
628+
629+
[ "$status" -eq 0 ]
630+
[ "$(fences_left "$STATE")" -eq 0 ]
631+
[ "$(cat "$STATE/[email protected]")" = "$(cat "$STATE/[email protected]")" ]
632+
grep -q -x "released $FENCE from tenant-a/app-a" "$tmp/out"
633+
grep -q -x 'released 1 of 1 fenced HelmReleases, 0 without an uninstall' "$tmp/out"
634+
[ "$(cat "$CLOCK")" -lt "$DEADLINE" ]
463635
rm -rf "$tmp"
464636
}
465637

0 commit comments

Comments
 (0)