Skip to content

Commit 2c45377

Browse files
fix(seaweedfs): make naming audit fail closed on kubectl and payload errors (#3436)
## What this PR does Fixes #3431. `hack/seaweedfs-naming-audit.sh` was fail-open: any kubectl failure produced an empty table indistinguishable from an honestly clean fleet — a failed namespace LIST meant zero namespaces walked, failed secret/PVC/STS LISTs meant zero findings, all silenced by `2>/dev/null` with no error handling anywhere in the script. The runbook uses this output as the gate before PVC deletion, as post-deletion verification, and as an upgrade precondition, so a transient API error could green-light destroying data. Every kubectl call now routes through a `run_kubectl` helper: a non-zero exit prints a FATAL line naming the failed query and the code propagates up the whole chain (enumerations restructured to capture-then-check, since an `exit` inside `$(...)` dies with the subshell). By-name GETs separate existence from extraction: a `--ignore-not-found -o name` probe keeps legitimate absence (Helm-pruned revision, unbound PVC) a clean skip, while everything else fails loudly — including a release Secret that exists but has no decodable payload or decodes without a chart name, which previously dropped the whole tenant from the audit. Incomplete PV-age evidence now marks the generation incomplete and falls to the safe "direction cannot be established" branch, so a failed GET can no longer flip an OVERLAP verdict into a wrong Step-3 deletion candidate. A successful audit prints the same bytes as before. Tests: 14 new cozytest cases — failure injection for every LIST and by-name GET site, corrupt-payload variants (missing `.data.release`, undecodable base64/gzip, chartless `{}`), and full-output golden diffs proving the success paths stay byte-identical to the pre-change script. 25/25 green via `hack/cozytest.sh`. ### Release note ```release-note fix(seaweedfs): the naming-audit script fails loudly when a kubectl query or a Helm release payload read fails, instead of printing an empty table indistinguishable from a clean fleet ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Audits now fail clearly (non-zero exit + fatal messaging) when Kubernetes queries or Helm release data is unavailable or corrupt, instead of producing incomplete results. * Missing release revisions are skipped safely; missing/unreadable PV age evidence no longer leads to incorrect deletion candidates, and MIXED now reports conservative “direction cannot be established” when evidence is incomplete. * Non-SeaweedFS releases remain silently ignored. * **Tests** * Added end-to-end acceptance coverage for fail-closed behavior and byte-for-byte stable output across key naming scenarios. * **Documentation** * Updated the runbook to require relying on the audit script exit code before proceeding, with a dedicated MIXED re-check step. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 4922276 + e02adad commit 2c45377

3 files changed

Lines changed: 670 additions & 34 deletions

File tree

‎docs/operations/seaweedfs-431-rename-recovery.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,8 @@ hack/seaweedfs-naming-audit.sh # whole cluster
5555
hack/seaweedfs-naming-audit.sh tenant-foo # or named namespaces
5656
```
5757

58+
**Read the exit code, not just the table.** The audit fails closed: any error it cannot interpret — a kubectl call that fails, a Helm release payload it cannot decode — makes it print `FATAL` and exit non-zero, and the table it printed up to that point is incomplete. A non-zero exit means you do not yet know the state of the fleet, so none of the steps below may be taken on the strength of it. Only a zero exit means the table is the whole answer; an empty table with exit 0 is a genuinely clean fleet.
59+
5860
It mutates nothing. Earlier revisions of this runbook inlined the classification as a shell snippet here; it is a tested script now (`hack/seaweedfs-naming-audit.bats`), because it is what the chart's refusal hands you to and acting on it deletes PVCs. Two inline versions shipped wrong — one whose selector matched both generations at once and so inverted its own primary rule, one that could not see a long instance name at all — so it is not a snippet any more.
5961

6062
It reports one class per SeaweedFS instance, matching exactly what the chart's guard decides:
@@ -297,7 +299,7 @@ Do **not** start by deleting PVCs. If the duplicate ever served, they may hold o
297299

298300
Never delete `data1-seaweedfs-volume-*` here — those are the live data PVCs of the authoritative set. (For an `S-damaged` tenant it is the other way round; that is Step 2a's job, and it has its own precondition check.)
299301

300-
5. **Re-run the audit.** The tenant must now read `L` (or `S`, if you are on the Step 2 path). Only then upgrade.
302+
5. **Re-run the audit.** The tenant must now read `L` (or `S`, if you are on the Step 2 path), and the audit must exit zero — a `FATAL` here means the re-check never completed, not that the tenant is fine. Only then upgrade.
301303

302304
```sh
303305
hack/seaweedfs-naming-audit.sh "$ns"

0 commit comments

Comments
 (0)