Skip to content

Commit 8a6fe02

Browse files
authored
fix(bucket): stop the ACME cleanup hook from talking to localhost, and gate the class (cozystack#4289)
## What this PR does Fixes cozystack#4288, and closes the class behind it and cozystack#4276. `packages/apps/bucket/templates/hooks/cleanup-acme.yaml` passed `--request-timeout=30s` on all four of its kubectl calls. A non-zero value there makes kubectl skip its in-cluster credentials: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so every call went to kubectl's built-in default cluster and was refused. This package failed more quietly than kafka. Each call is guarded by `|| echo "WARNING: ..."` and the script sets no `-e`, so the hook exited 0 and printed `ACME cleanup complete.`, and `hook-succeeded` then deleted the Job holding those warnings. The ACME solver objects it exists to reap were left behind, with the only record removed moments after it was written. Its own suite matched the rendered command including the flag, so the broken form was pinned in place. The gate is the point of the change rather than a bonus. Two packages carried this, neither was visible in a rendered manifest, and both passed their unit suites. Nothing stops the next hook from reaching for the same flag, because it reads as ordinary defensive practice. `hack/check-hook-request-timeout.bats` fails any non-vendored template whose kubectl call carries a non-zero `--request-timeout`. The coarse part is stated in the file rather than left to be discovered: a template that names `KUBECONFIG` is skipped, because an explicit kubeconfig is not served by the in-cluster branch and the flag is harmless there. That is a per-file judgement, not per-command. It is what lets `apps/kubernetes` keep the flag on two calls that pass their own kubeconfig and stay green, and it is the first thing to tighten if a file ever mixes both styles. Checked in both directions rather than only on the clean tree: the gate passes as committed, fails if the flag is put back on any bucket call, fails if it is put back on the kafka call, and leaves the two `apps/kubernetes` files alone. It also carries a vacuity guard, because a collapsed glob would otherwise report success for having examined nothing. Bounding the work is still available where it belongs: `--timeout` for a watch, `activeDeadlineSeconds` for the Job, which this hook already had at 120s. **Depends on cozystack#4280** and is branched off it, so the gate is green here. cozystack#4280 removes the same flag from the kafka hook; merged before it, this check would be red on `main`. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff. A hook's kubectl flags and a new `hack/*.bats` check; no values key, schema, API, default or naming contract moves, and nothing under `hack/` is moved or renamed, nor does any make target change what it does. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(bucket): the ACME cleanup hook now reaches the apiserver instead of reporting success while doing nothing ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved ACME cleanup reliability by preventing cleanup commands from targeting an unavailable local Kubernetes endpoint. * Improved Kafka uninstall cleanup reliability by ensuring the pre-delete hook retains access to required credentials throughout execution. * Preserved existing execution time limits and warning-based failure handling for cleanup operations. * **Tests** * Added coverage to prevent regressions in Kubernetes cleanup command configuration and hook resource lifecycle behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents b323d5d + 3615958 commit 8a6fe02

3 files changed

Lines changed: 76 additions & 7 deletions

File tree

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/usr/bin/env bats
2+
# Repo-wide invariant: a kubectl call that relies on in-cluster credentials must
3+
# not carry a non-zero --request-timeout.
4+
#
5+
# client-go loads in-cluster credentials only while the assembled config still
6+
# equals the default. --request-timeout lands in that config, so any non-zero
7+
# value diverts the call to kubectl's built-in default cluster, localhost:8080,
8+
# where it is refused. Nothing about this is visible in a rendered manifest: the
9+
# flag reads as ordinary defensive practice, the chart renders, and the failure
10+
# appears only when the container runs against a real apiserver. It cost two
11+
# packages before it was found (#4276, #4288), and in one of them the hook
12+
# reported success while doing nothing.
13+
#
14+
# Scope and its limit. A file that names KUBECONFIG is skipped: it hands kubectl
15+
# an explicit kubeconfig, so the in-cluster branch is not what serves it and the
16+
# flag is harmless there. That is a per-file judgement, not per-command, which
17+
# is the coarse part of this check and is why it is written down rather than
18+
# inferred. Vendored charts under packages/*/*/charts are upstream and out of
19+
# scope.
20+
#
21+
# Bounding the call is still available: --timeout bounds a watch, and
22+
# activeDeadlineSeconds bounds the whole Job, which is what
23+
# packages/apps/tenant/templates/cleanup-job.yaml uses and documents.
24+
25+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
26+
27+
@test "no in-cluster kubectl call carries a non-zero --request-timeout" {
28+
cd "$REPO_ROOT"
29+
30+
# Files in scope: non-vendored templates that invoke kubectl.
31+
candidates="$(grep -rl 'kubectl' packages/*/*/templates/ 2>/dev/null || true)"
32+
[ -n "$candidates" ] || { echo "FAIL: found no templates invoking kubectl; this check is broken, not clean"; false; }
33+
34+
# Guard against a vacuous pass: the tree ships many such templates, so a
35+
# collapsed glob has to fail here rather than report success.
36+
count="$(printf '%s\n' "$candidates" | grep -c . || true)"
37+
[ "$count" -ge 10 ] || { echo "FAIL: only $count template(s) invoke kubectl; expected the tree's usual population"; false; }
38+
39+
offenders=""
40+
for f in $candidates; do
41+
# An explicit kubeconfig means the in-cluster branch is not in play here.
42+
if grep -q 'KUBECONFIG' "$f"; then
43+
continue
44+
fi
45+
hits="$(grep -nE -- '--request-timeout=[^0[:space:]]' "$f" || true)"
46+
if [ -n "$hits" ]; then
47+
offenders="${offenders}${f}
48+
${hits}
49+
"
50+
fi
51+
done
52+
53+
if [ -n "$offenders" ]; then
54+
echo "FAIL: non-zero --request-timeout on a call that depends on in-cluster credentials."
55+
echo "It diverts kubectl to localhost:8080 and the call never reaches the cluster."
56+
echo "Bound the work with --timeout or activeDeadlineSeconds instead."
57+
printf '%s\n' "$offenders"
58+
false
59+
fi
60+
}

‎packages/apps/bucket/templates/hooks/cleanup-acme.yaml‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -59,11 +59,20 @@ spec:
5959
# one reliable per-release identifier for ACME solver objects (they
6060
# carry no app/release label of their own).
6161
HOST="{{ .Release.Name }}.{{ .Values._namespace.host }}"
62+
# None of the calls below carries --request-timeout. A non-zero
63+
# value makes kubectl skip its in-cluster credentials: client-go
64+
# consults them only while the assembled config still equals the
65+
# default, and the flag lands in that config. Every call then goes
66+
# to localhost:8080 and is refused, which here is worse than
67+
# failing -- each one is guarded by `|| echo WARNING`, so the hook
68+
# exits 0, prints "ACME cleanup complete.", and hook-succeeded
69+
# deletes the Job carrying those warnings (#4288). The whole
70+
# cleanup stays bounded by activeDeadlineSeconds above.
6271
# Delete the per-bucket Certificate/Order/Challenge. cert-manager
6372
# then cascade-removes the cm-acme-http-solver-* solver objects it
6473
# owns. The release's TLS Certificate is named after the secret it
6574
# populates ({{ .Release.Name }}-ui-tls).
66-
kubectl delete certificate {{ .Release.Name }}-ui-tls -n {{ .Release.Namespace }} --ignore-not-found --wait=false --request-timeout=30s \
75+
kubectl delete certificate {{ .Release.Name }}-ui-tls -n {{ .Release.Namespace }} --ignore-not-found --wait=false \
6776
|| echo "WARNING: cert-manager cleanup failed for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
6877
# Orders are owned by the Certificate and are removed by the cascade
6978
# above; Challenges carry no labels and are reaped by the
@@ -81,21 +90,21 @@ spec:
8190
#
8291
# Delete Challenges for our host (cascades the solver
8392
# Ingress/Service/Pod they own).
84-
challenges="$(kubectl get challenge -n {{ .Release.Namespace }} --request-timeout=30s -o jsonpath="{range .items[?(@.spec.dnsName=='$HOST')]}{.metadata.name} {end}")" \
93+
challenges="$(kubectl get challenge -n {{ .Release.Namespace }} -o jsonpath="{range .items[?(@.spec.dnsName=='$HOST')]}{.metadata.name} {end}")" \
8594
|| { echo "WARNING: failed to list challenges for {{ .Release.Name }}; orphaned ACME resources may remain" >&2; challenges=""; }
8695
for ch in $challenges; do
87-
kubectl delete challenge -n {{ .Release.Namespace }} "$ch" --ignore-not-found --wait=false --request-timeout=30s \
96+
kubectl delete challenge -n {{ .Release.Namespace }} "$ch" --ignore-not-found --wait=false \
8897
|| echo "WARNING: failed to delete challenge $ch for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
8998
done
9099
# Delete any orphaned solver Ingress for our host (no live owner
91100
# left to cascade it). Its solver Service/Pod are owned by a
92101
# Challenge and are cascade-deleted with it; truly owner-less
93102
# solver Service/Pod with no matching Challenge are not expected
94103
# and are not swept namespace-wide.
95-
ingresses="$(kubectl get ingress -n {{ .Release.Namespace }} -l acme.cert-manager.io/http01-solver=true --request-timeout=30s -o jsonpath="{range .items[?(@.spec.rules[0].host=='$HOST')]}{.metadata.name} {end}")" \
104+
ingresses="$(kubectl get ingress -n {{ .Release.Namespace }} -l acme.cert-manager.io/http01-solver=true -o jsonpath="{range .items[?(@.spec.rules[0].host=='$HOST')]}{.metadata.name} {end}")" \
96105
|| { echo "WARNING: failed to list solver ingresses for {{ .Release.Name }}; orphaned ACME resources may remain" >&2; ingresses=""; }
97106
for ing in $ingresses; do
98-
kubectl delete ingress -n {{ .Release.Namespace }} "$ing" --ignore-not-found --wait=false --request-timeout=30s \
107+
kubectl delete ingress -n {{ .Release.Namespace }} "$ing" --ignore-not-found --wait=false \
99108
|| echo "WARNING: failed to delete solver ingress $ing for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
100109
done
101110
echo "ACME cleanup complete."

‎packages/apps/bucket/tests/cleanup_acme_test.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -84,14 +84,14 @@ tests:
8484
# Host-scoped challenge sweep filtering on .spec.dnsName == our host
8585
- matchRegex:
8686
path: spec.template.spec.containers[0].command[2]
87-
pattern: "kubectl get challenge -n tenant-root --request-timeout=30s -o jsonpath=.*@.spec.dnsName=='\\$HOST'"
87+
pattern: "kubectl get challenge -n tenant-root -o jsonpath=.*@.spec.dnsName=='\\$HOST'"
8888
- matchRegex:
8989
path: spec.template.spec.containers[0].command[2]
9090
pattern: 'kubectl delete challenge -n tenant-root "\$ch" --ignore-not-found'
9191
# Host-scoped orphaned solver Ingress sweep filtering on rules[0].host
9292
- matchRegex:
9393
path: spec.template.spec.containers[0].command[2]
94-
pattern: "kubectl get ingress -n tenant-root -l acme.cert-manager.io/http01-solver=true --request-timeout=30s -o jsonpath=.*@.spec.rules\\[0\\].host=='\\$HOST'"
94+
pattern: "kubectl get ingress -n tenant-root -l acme.cert-manager.io/http01-solver=true -o jsonpath=.*@.spec.rules\\[0\\].host=='\\$HOST'"
9595
- matchRegex:
9696
path: spec.template.spec.containers[0].command[2]
9797
pattern: 'kubectl delete ingress -n tenant-root "\$ing" --ignore-not-found'

0 commit comments

Comments
 (0)