Skip to content

Commit 3615958

Browse files
committed
fix(bucket): stop the ACME cleanup hook from talking to localhost, and gate the class
The hook's four kubectl calls carried --request-timeout=30s. A non-zero value there makes kubectl skip its in-cluster credentials: client-go consults them only while the assembled config still equals the default, and the flag lands in that config, so every call went to kubectl's built-in default cluster and was refused. This package failed more quietly than kafka did. Each call is guarded by `|| echo WARNING` and the script sets no -e, so the hook exited 0 and printed "ACME cleanup complete."; hook-succeeded then deleted the Job that held those warnings. The ACME solver objects it exists to reap were left behind, and the only record went with the Job. Its own suite matched the rendered command including the flag, so the broken form was pinned. The gate is the point of this change. Two packages had it, neither was visible in a rendered manifest, and nothing stops the next hook from reaching for the same flag, since it reads as ordinary defensive practice. hack/check-hook-request-timeout.bats fails any non-vendored template whose kubectl call carries a non-zero --request-timeout, skipping files that name KUBECONFIG because an explicit kubeconfig is not served by the in-cluster branch. That per-file judgement is the coarse part and is written down in the file rather than left to be inferred: apps/kubernetes keeps the flag on two calls that pass their own kubeconfig, and stays green. Bounding the work is still available where it belongs: --timeout for a watch, activeDeadlineSeconds for the Job, which is what this hook already had at 120s. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
1 parent e63b859 commit 3615958

3 files changed

Lines changed: 76 additions & 7 deletions

File tree

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,60 @@
1+
#!/usr/bin/env bats
2+
# Repo-wide invariant: a kubectl call that relies on in-cluster credentials must
3+
# not carry a non-zero --request-timeout.
4+
#
5+
# client-go loads in-cluster credentials only while the assembled config still
6+
# equals the default. --request-timeout lands in that config, so any non-zero
7+
# value diverts the call to kubectl's built-in default cluster, localhost:8080,
8+
# where it is refused. Nothing about this is visible in a rendered manifest: the
9+
# flag reads as ordinary defensive practice, the chart renders, and the failure
10+
# appears only when the container runs against a real apiserver. It cost two
11+
# packages before it was found (#4276, #4288), and in one of them the hook
12+
# reported success while doing nothing.
13+
#
14+
# Scope and its limit. A file that names KUBECONFIG is skipped: it hands kubectl
15+
# an explicit kubeconfig, so the in-cluster branch is not what serves it and the
16+
# flag is harmless there. That is a per-file judgement, not per-command, which
17+
# is the coarse part of this check and is why it is written down rather than
18+
# inferred. Vendored charts under packages/*/*/charts are upstream and out of
19+
# scope.
20+
#
21+
# Bounding the call is still available: --timeout bounds a watch, and
22+
# activeDeadlineSeconds bounds the whole Job, which is what
23+
# packages/apps/tenant/templates/cleanup-job.yaml uses and documents.
24+
25+
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
26+
27+
@test "no in-cluster kubectl call carries a non-zero --request-timeout" {
28+
cd "$REPO_ROOT"
29+
30+
# Files in scope: non-vendored templates that invoke kubectl.
31+
candidates="$(grep -rl 'kubectl' packages/*/*/templates/ 2>/dev/null || true)"
32+
[ -n "$candidates" ] || { echo "FAIL: found no templates invoking kubectl; this check is broken, not clean"; false; }
33+
34+
# Guard against a vacuous pass: the tree ships many such templates, so a
35+
# collapsed glob has to fail here rather than report success.
36+
count="$(printf '%s\n' "$candidates" | grep -c . || true)"
37+
[ "$count" -ge 10 ] || { echo "FAIL: only $count template(s) invoke kubectl; expected the tree's usual population"; false; }
38+
39+
offenders=""
40+
for f in $candidates; do
41+
# An explicit kubeconfig means the in-cluster branch is not in play here.
42+
if grep -q 'KUBECONFIG' "$f"; then
43+
continue
44+
fi
45+
hits="$(grep -nE -- '--request-timeout=[^0[:space:]]' "$f" || true)"
46+
if [ -n "$hits" ]; then
47+
offenders="${offenders}${f}
48+
${hits}
49+
"
50+
fi
51+
done
52+
53+
if [ -n "$offenders" ]; then
54+
echo "FAIL: non-zero --request-timeout on a call that depends on in-cluster credentials."
55+
echo "It diverts kubectl to localhost:8080 and the call never reaches the cluster."
56+
echo "Bound the work with --timeout or activeDeadlineSeconds instead."
57+
printf '%s\n' "$offenders"
58+
false
59+
fi
60+
}

‎packages/apps/bucket/templates/hooks/cleanup-acme.yaml‎

Lines changed: 14 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -59,11 +59,20 @@ spec:
5959
# one reliable per-release identifier for ACME solver objects (they
6060
# carry no app/release label of their own).
6161
HOST="{{ .Release.Name }}.{{ .Values._namespace.host }}"
62+
# None of the calls below carries --request-timeout. A non-zero
63+
# value makes kubectl skip its in-cluster credentials: client-go
64+
# consults them only while the assembled config still equals the
65+
# default, and the flag lands in that config. Every call then goes
66+
# to localhost:8080 and is refused, which here is worse than
67+
# failing -- each one is guarded by `|| echo WARNING`, so the hook
68+
# exits 0, prints "ACME cleanup complete.", and hook-succeeded
69+
# deletes the Job carrying those warnings (#4288). The whole
70+
# cleanup stays bounded by activeDeadlineSeconds above.
6271
# Delete the per-bucket Certificate/Order/Challenge. cert-manager
6372
# then cascade-removes the cm-acme-http-solver-* solver objects it
6473
# owns. The release's TLS Certificate is named after the secret it
6574
# populates ({{ .Release.Name }}-ui-tls).
66-
kubectl delete certificate {{ .Release.Name }}-ui-tls -n {{ .Release.Namespace }} --ignore-not-found --wait=false --request-timeout=30s \
75+
kubectl delete certificate {{ .Release.Name }}-ui-tls -n {{ .Release.Namespace }} --ignore-not-found --wait=false \
6776
|| echo "WARNING: cert-manager cleanup failed for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
6877
# Orders are owned by the Certificate and are removed by the cascade
6978
# above; Challenges carry no labels and are reaped by the
@@ -81,21 +90,21 @@ spec:
8190
#
8291
# Delete Challenges for our host (cascades the solver
8392
# Ingress/Service/Pod they own).
84-
challenges="$(kubectl get challenge -n {{ .Release.Namespace }} --request-timeout=30s -o jsonpath="{range .items[?(@.spec.dnsName=='$HOST')]}{.metadata.name} {end}")" \
93+
challenges="$(kubectl get challenge -n {{ .Release.Namespace }} -o jsonpath="{range .items[?(@.spec.dnsName=='$HOST')]}{.metadata.name} {end}")" \
8594
|| { echo "WARNING: failed to list challenges for {{ .Release.Name }}; orphaned ACME resources may remain" >&2; challenges=""; }
8695
for ch in $challenges; do
87-
kubectl delete challenge -n {{ .Release.Namespace }} "$ch" --ignore-not-found --wait=false --request-timeout=30s \
96+
kubectl delete challenge -n {{ .Release.Namespace }} "$ch" --ignore-not-found --wait=false \
8897
|| echo "WARNING: failed to delete challenge $ch for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
8998
done
9099
# Delete any orphaned solver Ingress for our host (no live owner
91100
# left to cascade it). Its solver Service/Pod are owned by a
92101
# Challenge and are cascade-deleted with it; truly owner-less
93102
# solver Service/Pod with no matching Challenge are not expected
94103
# and are not swept namespace-wide.
95-
ingresses="$(kubectl get ingress -n {{ .Release.Namespace }} -l acme.cert-manager.io/http01-solver=true --request-timeout=30s -o jsonpath="{range .items[?(@.spec.rules[0].host=='$HOST')]}{.metadata.name} {end}")" \
104+
ingresses="$(kubectl get ingress -n {{ .Release.Namespace }} -l acme.cert-manager.io/http01-solver=true -o jsonpath="{range .items[?(@.spec.rules[0].host=='$HOST')]}{.metadata.name} {end}")" \
96105
|| { echo "WARNING: failed to list solver ingresses for {{ .Release.Name }}; orphaned ACME resources may remain" >&2; ingresses=""; }
97106
for ing in $ingresses; do
98-
kubectl delete ingress -n {{ .Release.Namespace }} "$ing" --ignore-not-found --wait=false --request-timeout=30s \
107+
kubectl delete ingress -n {{ .Release.Namespace }} "$ing" --ignore-not-found --wait=false \
99108
|| echo "WARNING: failed to delete solver ingress $ing for {{ .Release.Name }}; orphaned ACME resources may remain" >&2
100109
done
101110
echo "ACME cleanup complete."

‎packages/apps/bucket/tests/cleanup_acme_test.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -84,14 +84,14 @@ tests:
8484
# Host-scoped challenge sweep filtering on .spec.dnsName == our host
8585
- matchRegex:
8686
path: spec.template.spec.containers[0].command[2]
87-
pattern: "kubectl get challenge -n tenant-root --request-timeout=30s -o jsonpath=.*@.spec.dnsName=='\\$HOST'"
87+
pattern: "kubectl get challenge -n tenant-root -o jsonpath=.*@.spec.dnsName=='\\$HOST'"
8888
- matchRegex:
8989
path: spec.template.spec.containers[0].command[2]
9090
pattern: 'kubectl delete challenge -n tenant-root "\$ch" --ignore-not-found'
9191
# Host-scoped orphaned solver Ingress sweep filtering on rules[0].host
9292
- matchRegex:
9393
path: spec.template.spec.containers[0].command[2]
94-
pattern: "kubectl get ingress -n tenant-root -l acme.cert-manager.io/http01-solver=true --request-timeout=30s -o jsonpath=.*@.spec.rules\\[0\\].host=='\\$HOST'"
94+
pattern: "kubectl get ingress -n tenant-root -l acme.cert-manager.io/http01-solver=true -o jsonpath=.*@.spec.rules\\[0\\].host=='\\$HOST'"
9595
- matchRegex:
9696
path: spec.template.spec.containers[0].command[2]
9797
pattern: 'kubectl delete ingress -n tenant-root "\$ing" --ignore-not-found'

0 commit comments

Comments
 (0)