Skip to content

Commit e02adad

Browse files
fix(seaweedfs): read the chart name first-match, fail the fake on unmodelled calls
Review findings from #3436, plus what an adversarial re-check of the first attempt at them turned up. The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals "config" (the release`s values) after "chart", so a values subtree that spells chart.metadata.name shadowed the real chart name: the release then read as non-SeaweedFS, the tenant silently vanished from the report, and the script exited 0 -- the precise false clean this PR exists to prevent, reachable without any corruption at all. It now takes the FIRST match. The path stays adjacent on purpose: a looser "any name after metadata" matches chart.templates[].name, which Helm emits immediately after metadata on every healthy release, so the review`s suggested relaxation would have returned a template path for every tenant. Over-strictness fails loudly and recoverably; over-looseness deletes data quietly. Newlines are now folded before matching, so a pretty-printed payload parses at all rather than aborting, and whitespace around the punctuation is tolerated. first_deployed got the same treatment -- it had the identical greedy, single- line-only shape, and a spaced payload silently dropped the PV-vintage row from the report. The FATAL message names the path it read and the shape it expected, so a future Helm format change is diagnosable instead of looking like real corruption. Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy) each byte-compared against the same golden as the compact payload, so the shape must make no difference to the report. Mutation-checked -- restoring last-match extraction fails the decoy test on its own, restoring the single-line matcher fails the pretty test. The test fake answered any unmodelled kubectl invocation with exit 0 and empty stdout, the fail-open shape this script was rewritten to reject, and enough to let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97 naming the invocation. Verified inert first by instrumenting the fake: 88 invocations across the suite, none unmodelled. Also added the missing test for the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it. The runbook now tells the operator to read the exit code: a non-zero exit means the table is incomplete and no step may be taken on it. That contract lived only in the script header, while the runbook is what the operator follows. 29/29 tests green; sh -n clean. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 3dc6cbd commit e02adad

3 files changed

Lines changed: 144 additions & 15 deletions

File tree

‎docs/operations/seaweedfs-431-rename-recovery.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,8 @@ hack/seaweedfs-naming-audit.sh # whole cluster
5555
hack/seaweedfs-naming-audit.sh tenant-foo # or named namespaces
5656
```
5757

58+
**Read the exit code, not just the table.** The audit fails closed: any error it cannot interpret — a kubectl call that fails, a Helm release payload it cannot decode — makes it print `FATAL` and exit non-zero, and the table it printed up to that point is incomplete. A non-zero exit means you do not yet know the state of the fleet, so none of the steps below may be taken on the strength of it. Only a zero exit means the table is the whole answer; an empty table with exit 0 is a genuinely clean fleet.
59+
5860
It mutates nothing. Earlier revisions of this runbook inlined the classification as a shell snippet here; it is a tested script now (`hack/seaweedfs-naming-audit.bats`), because it is what the chart's refusal hands you to and acting on it deletes PVCs. Two inline versions shipped wrong — one whose selector matched both generations at once and so inverted its own primary rule, one that could not see a long instance name at all — so it is not a snippet any more.
5961

6062
It reports one class per SeaweedFS instance, matching exactly what the chart's guard decides:
@@ -297,7 +299,7 @@ Do **not** start by deleting PVCs. If the duplicate ever served, they may hold o
297299

298300
Never delete `data1-seaweedfs-volume-*` here — those are the live data PVCs of the authoritative set. (For an `S-damaged` tenant it is the other way round; that is Step 2a's job, and it has its own precondition check.)
299301

300-
5. **Re-run the audit.** The tenant must now read `L` (or `S`, if you are on the Step 2 path). Only then upgrade.
302+
5. **Re-run the audit.** The tenant must now read `L` (or `S`, if you are on the Step 2 path), and the audit must exit zero — a `FATAL` here means the re-check never completed, not that the tenant is fine. Only then upgrade.
301303

302304
```sh
303305
hack/seaweedfs-naming-audit.sh "$ns"

‎hack/seaweedfs-naming-audit.bats‎

Lines changed: 105 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -172,13 +172,34 @@ EOF
172172
# chart.metadata.name (as real Helm payloads carry it). Defaults to a cozy-seaweedfs
173173
# release so the audit confirms the tenant; pass another chart name for a non-
174174
# SeaweedFS release, or the literal EMPTY for a chartless {} payload.
175+
#
176+
# Three payload SHAPES beyond the default compact one, because the chart-name
177+
# extraction is now fatal on a miss and every shape below is something a
178+
# re-serializer or a user's values could produce:
179+
# SPACED whitespace around the JSON punctuation (json.dumps' default)
180+
# PRETTY indented and MULTI-LINE (jq . / yq -o=json), which a line-based
181+
# matcher cannot read at all unless newlines are folded first
182+
# DECOY a real cozy-seaweedfs chart PLUS a values subtree that spells
183+
# chart.metadata.name with a different value. Helm marshals "config"
184+
# (the values) AFTER "chart", so a last-match extraction returns the
185+
# decoy and silently declares the tenant non-SeaweedFS.
175186
_release_blob() {
176187
_rb_name=${1:-cozy-seaweedfs}
177-
if [ "$_rb_name" = EMPTY ]; then
178-
_rb_json='{}'
179-
else
180-
_rb_json='{"name":"seaweedfs-system","chart":{"metadata":{"name":"'"$_rb_name"'"}}}'
181-
fi
188+
case "$_rb_name" in
189+
EMPTY) _rb_json='{}' ;;
190+
SPACED) _rb_json='{"name": "seaweedfs-system", "chart": {"metadata": {"name": "cozy-seaweedfs", "version": "1.0.0"}}}' ;;
191+
PRETTY) _rb_json='{
192+
"name": "seaweedfs-system",
193+
"chart": {
194+
"metadata": {
195+
"name": "cozy-seaweedfs",
196+
"version": "1.0.0"
197+
}
198+
}
199+
}' ;;
200+
DECOY) _rb_json='{"name":"seaweedfs-system","chart":{"metadata":{"name":"cozy-seaweedfs"}},"config":{"chart":{"metadata":{"name":"decoy-not-seaweedfs"}}}}' ;;
201+
*) _rb_json='{"name":"seaweedfs-system","chart":{"metadata":{"name":"'"$_rb_name"'"}}}' ;;
202+
esac
182203
printf '%s' "$_rb_json" | gzip | base64 | base64 | tr -d '\n'
183204
}
184205

@@ -212,6 +233,10 @@ _write_fake_kubectl() {
212233
cat <<'FAKE'
213234
verb=${1:-}; res=${2:-}; args="$*"
214235
fail() { echo "fake kubectl: $1 (real error)" >&2; exit 1; }
236+
# Anything this fake does not model must NOT look like a successful empty
237+
# answer: that is the fail-open shape the audited script exists to reject, and
238+
# it would let a new query added to the script pass these goldens unnoticed.
239+
unmodelled() { echo "fake kubectl: unmodelled invocation: $args" >&2; exit 97; }
215240
if [ "$verb $res" = "get ns" ]; then
216241
[ "$FAIL" = ns ] && fail "get ns"
217242
printf 'namespace/tenant-test\n'; exit 0
@@ -247,7 +272,7 @@ if [ "$verb $res" = "get pvc" ]; then
247272
*data1-seaweedfs-volume-1*) [ "$FAIL" = pvcget ] && fail "pvc GET"; printf 'pv-legacy2\n'; exit 0 ;;
248273
*data1-seaweedfs-volume-0*) [ "$FAIL" = pvcget ] && fail "pvc GET"; printf 'pv-legacy\n'; exit 0 ;;
249274
esac
250-
exit 0
275+
unmodelled
251276
fi
252277
if [ "$verb $res" = "get sts" ]; then
253278
[ "$FAIL" = sts ] && fail "sts LIST"
@@ -260,9 +285,9 @@ if [ "$verb $res" = "get pv" ]; then
260285
*pv-legacy*) [ "$ABSENT_PV" = pv-legacy ] && exit 0; printf '2020-01-01T00:00:00Z\n'; exit 0 ;;
261286
*pv-renamed*) [ "$ABSENT_PV" = pv-renamed ] && exit 0; printf '2020-06-01T00:00:00Z\n'; exit 0 ;;
262287
esac
263-
exit 0
288+
unmodelled
264289
fi
265-
exit 0
290+
unmodelled
266291
FAKE
267292
} > "$1/kubectl"
268293
chmod +x "$1/kubectl"
@@ -396,7 +421,10 @@ _expected_mixed_overlap() {
396421
echo "rc=$rc"; echo "$out"
397422
[ "$rc" -ne 0 ]
398423
printf '%s\n' "$out" | grep -q 'FATAL'
399-
printf '%s\n' "$out" | grep -q 'no chart name'
424+
printf '%s\n' "$out" | grep -q 'could not read the chart name'
425+
# The diagnostic must name the path it looked at, so a future Helm payload
426+
# format change is diagnosable as such instead of reading as real corruption.
427+
printf '%s\n' "$out" | grep -q '\.chart\.metadata\.name'
400428
}
401429

402430
@test "a present release secret for a non-SeaweedFS chart is a silent legitimate skip" {
@@ -507,3 +535,71 @@ _expected_mixed_overlap() {
507535
diff "$d/want" "$d/got"
508536
rm -rf "$d"
509537
}
538+
539+
@test "a whitespace-spaced Helm payload still classifies the tenant" {
540+
# The chart-name read is FATAL on a miss, so any payload shape a re-serializer
541+
# can produce must parse. Byte-compare against the same golden as the compact
542+
# payload: the shape must make no difference to the report at all.
543+
d=$(mktemp -d)
544+
_write_fake_kubectl "$d" "" "" "$(_release_blob SPACED)" "persistentvolumeclaim/data1-seaweedfs-volume-0"
545+
rc=0
546+
out=$(PATH="$d:$PATH" SEAWEEDFS_AUDIT_LIB=0 sh "$PWD/hack/seaweedfs-naming-audit.sh" tenant-test 2>&1) || rc=$?
547+
printf '%s\n' "$out" > "$d/got"
548+
_expected_L > "$d/want"
549+
echo "rc=$rc"; echo "--- got ---"; cat "$d/got"
550+
[ "$rc" -eq 0 ]
551+
diff "$d/want" "$d/got"
552+
rm -rf "$d"
553+
}
554+
555+
@test "a pretty-printed multi-line Helm payload still classifies the tenant" {
556+
# Line-based sed/grep cannot see across newlines at all, so this shape is the
557+
# one that fails hardest without the newline fold -- and jq/yq re-serialization
558+
# is exactly how a payload would arrive indented.
559+
d=$(mktemp -d)
560+
_write_fake_kubectl "$d" "" "" "$(_release_blob PRETTY)" "persistentvolumeclaim/data1-seaweedfs-volume-0"
561+
rc=0
562+
out=$(PATH="$d:$PATH" SEAWEEDFS_AUDIT_LIB=0 sh "$PWD/hack/seaweedfs-naming-audit.sh" tenant-test 2>&1) || rc=$?
563+
printf '%s\n' "$out" > "$d/got"
564+
_expected_L > "$d/want"
565+
echo "rc=$rc"; echo "--- got ---"; cat "$d/got"
566+
[ "$rc" -eq 0 ]
567+
diff "$d/want" "$d/got"
568+
rm -rf "$d"
569+
}
570+
571+
@test "a values subtree spelling chart.metadata.name cannot shadow the real chart" {
572+
# Helm marshals "config" (the user's values) AFTER "chart", so a LAST-match
573+
# extraction returns the decoy, the release reads as non-SeaweedFS, and the
574+
# tenant vanishes from the report with exit 0 -- a false clean, the failure this
575+
# whole script exists to prevent. First-match extraction is what stops it.
576+
d=$(mktemp -d)
577+
_write_fake_kubectl "$d" "" "" "$(_release_blob DECOY)" "persistentvolumeclaim/data1-seaweedfs-volume-0"
578+
rc=0
579+
out=$(PATH="$d:$PATH" SEAWEEDFS_AUDIT_LIB=0 sh "$PWD/hack/seaweedfs-naming-audit.sh" tenant-test 2>&1) || rc=$?
580+
printf '%s\n' "$out" > "$d/got"
581+
_expected_L > "$d/want"
582+
echo "rc=$rc"; echo "--- got ---"; cat "$d/got"
583+
[ "$rc" -eq 0 ]
584+
# The decoy name must never reach the report.
585+
[ "$(printf '%s\n' "$out" | grep -c 'decoy-not-seaweedfs')" -eq 0 ]
586+
diff "$d/want" "$d/got"
587+
rm -rf "$d"
588+
}
589+
590+
@test "fails closed when the by-name PVC GET hits a real error" {
591+
# pv_epoch resolves each claim's bound PV by name before reading the PV's age.
592+
# A real error there (RBAC/timeout) must abort: silently dropping the claim
593+
# shrinks the observed vintage range, which is what produces a wrong "candidate
594+
# duplicate" verdict. The PV half of this pair was already covered; this is the
595+
# PVC half, whose fake FAIL mode existed with no test behind it.
596+
d=$(mktemp -d)
597+
_write_fake_kubectl "$d" pvcget "" "$(_release_blob)" "$(_pvcs_mixed)"
598+
rc=0
599+
out=$(PATH="$d:$PATH" SEAWEEDFS_AUDIT_LIB=0 sh "$PWD/hack/seaweedfs-naming-audit.sh" tenant-test 2>&1) || rc=$?
600+
rm -rf "$d"
601+
echo "rc=$rc"; echo "$out"
602+
[ "$rc" -ne 0 ]
603+
printf '%s\n' "$out" | grep -q 'FATAL'
604+
printf '%s\n' "$out" | grep -q "get PVC '"
605+
}

‎hack/seaweedfs-naming-audit.sh‎

Lines changed: 36 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -191,11 +191,33 @@ system_releases() {
191191
# the very false-clean this guard exists to stop -- so it is FATAL. A present
192192
# but different chart name is a real, non-SeaweedFS release and stays a
193193
# legitimate skip, filtered exactly as before by the cozy-seaweedfs test.
194-
# ("chart" is the release struct's only such key -- vendored subcharts nest
195-
# under "dependencies" -- so this match is unambiguous.)
194+
#
195+
# Two properties make this match trustworthy, and both are load-bearing now
196+
# that a miss is FATAL:
197+
#
198+
# * FIRST match, not last. Helm's Release marshals "chart" before
199+
# "config" (the user's values), so a values subtree that happens to
200+
# spell chart.metadata.name sits LATER in the payload -- and a greedy
201+
# `sed 's/.*"chart"...'` would return that decoy instead of the real
202+
# chart. A wrong name reads downstream as "not SeaweedFS" and drops a
203+
# real release from the report: the exact silent false-clean this guard
204+
# exists to stop. `grep -o | head -1` takes the leftmost match.
205+
# * The chart -> metadata -> name key path stays ADJACENT. A looser "any
206+
# 'name' after 'metadata'" matches chart.templates[].name, which Helm
207+
# serializes immediately after metadata on EVERY healthy release, so it
208+
# would return a template path for every tenant.
209+
#
210+
# Newlines are folded first so a pretty-printed payload parses at all (sed
211+
# and grep are line-based), and whitespace around the punctuation is
212+
# tolerated. Failing loudly on a payload this cannot read is the safe
213+
# direction -- over-strictness stops the runbook, over-looseness lets it
214+
# delete data -- so the message says what shape was expected.
196215
if [ -n "$_sr_json" ]; then
197-
_sr_chart=$(printf '%s' "$_sr_json" | sed -n 's/.*"chart":{"metadata":{"name":"\([^"]*\)".*/\1/p' | head -1)
198-
[ -n "$_sr_chart" ] || { audit_fatal "secret 'sh.helm.release.v1.$rel.v$rev' in namespace '$1' decoded to a payload with no chart name (corrupt Helm release)."; return 1; }
216+
_sr_chart=$(printf '%s' "$_sr_json" | tr '\n' ' ' \
217+
| grep -o '"chart"[[:space:]]*:[[:space:]]*{[[:space:]]*"metadata"[[:space:]]*:[[:space:]]*{[[:space:]]*"name"[[:space:]]*:[[:space:]]*"[^"]*"' \
218+
| head -1 \
219+
| sed -n 's/.*"\([^"]*\)"$/\1/p')
220+
[ -n "$_sr_chart" ] || { audit_fatal "could not read the chart name of secret 'sh.helm.release.v1.$rel.v$rev' in namespace '$1' at .chart.metadata.name (expected 'name' as metadata's first key; a corrupt Helm release, or a payload format this parser does not handle)."; return 1; }
199221
if [ "$_sr_chart" = cozy-seaweedfs ]; then printf '%s\n' "$rel"; fi
200222
fi
201223
break
@@ -209,7 +231,16 @@ first_deployed() {
209231
_fd_revs=$(revisions "$1" "$2") || return $?
210232
for rev in $_fd_revs; do
211233
_fd_json=$(release_json "$1" "$2" "$rev") || return $?
212-
ts=$(printf '%s' "$_fd_json" | sed -n 's/.*"first_deployed":"\([^"]*\)".*/\1/p' | head -1)
234+
# Same extraction discipline as the chart name above: fold newlines so a
235+
# pretty-printed payload parses at all, tolerate whitespace around the
236+
# punctuation, and take the FIRST match (.info precedes .config, so a values
237+
# subtree cannot shadow the real timestamp). A miss here is not fatal -- the
238+
# caller degrades to the documented safe fallback -- but a WRONG timestamp
239+
# would silently change a vintage verdict, which is worse than none.
240+
ts=$(printf '%s' "$_fd_json" | tr '\n' ' ' \
241+
| grep -o '"first_deployed"[[:space:]]*:[[:space:]]*"[^"]*"' \
242+
| head -1 \
243+
| sed -n 's/.*"\([^"]*\)"$/\1/p')
213244
if [ -n "$ts" ]; then date -u -d "$(printf '%s' "$ts" | cut -c1-19)" +%s 2>/dev/null; return; fi
214245
done
215246
}

0 commit comments

Comments
 (0)