Skip to content

Commit ac3e2d3

Browse files
committed
docs(platform): document the upgrade preflight gate
A default-on upgrade blocker shipped with no operator-facing documentation. The values file was the only description of it, and the block message named preflight.override without saying where that key lives, which is not guessable and is silently ignored when nested wrong. The page covers when the gate runs, what each check asserts, where to read a verdict that outlives the hook Job, the full Package CR path for every hatch, and how to add a check. The runner's own block message now carries the same path and points at the status ConfigMap. Signed-off-by: IvanHunters <[email protected]>
1 parent 2795a80 commit ac3e2d3

3 files changed

Lines changed: 91 additions & 5 deletions

File tree

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Upgrade Preflight
2+
3+
Cozystack runs a health gate before a platform upgrade touches anything. It executes as a `pre-upgrade` Helm hook at weight 0, ahead of the schema migrations at weight 1, and refuses to let the upgrade start when the cluster is not in a state where it can safely absorb one. The point is to surface a broken node or a degraded storage layer up front, with a message naming what is wrong, instead of halfway through a HelmRelease rollout.
4+
5+
The gate is enabled by default.
6+
7+
## When it runs
8+
9+
The hook renders only when the `cozystack-version` ConfigMap in the release namespace reports a version **below** `migrations.targetVersion`, which is the same condition the migration hook uses. Three consequences worth knowing:
10+
11+
- A fresh install is never gated. There is no prior state to protect and nothing to query.
12+
- A release that ships no new migration is not gated either, because the version does not advance.
13+
- The checks do not re-run on every Flux reconcile of a healthy cluster, which keeps a live `linstor` exec off the steady-state path.
14+
15+
## What it checks
16+
17+
| Check id | What it asserts | Blocking by default |
18+
| --- | --- | --- |
19+
| `nodes-ready` | Every Kubernetes node carries `Ready=True`. A node with no `Ready` condition at all counts as not ready. | yes |
20+
| `linstor` | Every LINSTOR satellite is `ONLINE` and no DRBD volume sits in `Inconsistent`, `Failed`, `DUnknown` or `Outdated`. Skipped as not-applicable when LINSTOR is not installed. | no, advisory |
21+
22+
Cordoned nodes are exempt from `nodes-ready` whatever their readiness, and are reported as a warning instead. A cordon is the operator declaring a node out of service, and the normal maintenance sequence (cordon, drain, power off) leaves the node at `Ready=Unknown` with its workloads already moved. The trade-off is deliberate: a node that failed and was then cordoned is waved through on that same signal.
23+
24+
Every check fails closed. If a check cannot determine health, because the API server refused the call, timed out, or returned output it could not parse, that counts as a failure rather than a pass. A gate that ran no checks at all also fails.
25+
26+
The `linstor` check is advisory until its detection of faulty states has been exercised against a genuinely degraded cluster: it runs and reports, but a problem it finds warns rather than blocks.
27+
28+
## Reading the verdict
29+
30+
Read the status ConfigMap, not the Job:
31+
32+
```bash
33+
kubectl --namespace cozy-system get configmap cozystack-preflight-status --output yaml
34+
```
35+
36+
It carries `verdict` (`running`, `passed` or `blocked`), the check ids by outcome (`failedChecks`, `advisoryChecks`, `okChecks`, `naChecks`, `skippedChecks`), `completedAt`, and the full run log.
37+
38+
The hook Job is not a durable record. A failed upgrade is retried automatically by the platform HelmRelease (`Upgrade.Strategy` is `RetryOnFailure`, with a retry interval that defaults to 30 seconds), and Helm deletes the previous hook Job at the top of every attempt, so the Job and its log survive roughly one retry interval before an identical run replaces them. The HelmRelease itself reports only a generic `pre-upgrade hooks failed`, which names neither the check nor the node. The ConfigMap is written by every run, including one that starts and is then killed, and no hook lifecycle deletes it.
39+
40+
While the gate is blocking, the upgrade stays pending and the checks keep re-running at the retry interval. Fixing the reported problem is enough; no manual retry is needed.
41+
42+
## Overriding the gate
43+
44+
These keys live under `spec.components.platform.values.preflight` on the `cozystack.cozystack-platform` Package CR. Nothing else reads them, and a misspelled path is silently ignored, so check the nesting.
45+
46+
```yaml
47+
apiVersion: cozystack.io/v1alpha1
48+
kind: Package
49+
metadata:
50+
name: cozystack.cozystack-platform
51+
spec:
52+
components:
53+
platform:
54+
values:
55+
preflight:
56+
# Proceed past every failing check. The failures are logged for the
57+
# audit trail and downgraded to warnings.
58+
override: false
59+
# Skip named checks entirely, instead of overriding all of them.
60+
skipChecks: []
61+
# Named checks still run and report, but do not block.
62+
advisoryChecks: [linstor]
63+
# Do not render the gate at all.
64+
enabled: true
65+
```
66+
67+
Prefer the narrowest hatch that clears your situation: fix what the check reports, then `skipChecks` for a single check, then `override` for all of them, then `enabled: false`. Both booleans accept `true`, `1`, `yes` and `on` for yes and `false`, `0`, `no` and `off` for no, in any of the shapes the Package CR's untyped JSON can deliver, so a quoted `"false"` behaves the same as a bare one. Anything unrecognised leaves the gate up.
68+
69+
## Adding a check
70+
71+
Drop an executable script into `packages/core/platform/images/migrations/preflight/checks/`, named `<order>-<id>.sh`. The runner discovers it, derives the check id from the filename, and interprets its exit code: `0` is OK, `2` blocks the upgrade, `3` is not-applicable, and any other non-zero exit is treated as a failure. Source `../lib/preflight-lib.sh` and use `kubectl_t` for API calls so the call is bounded and cannot hang the hook.
72+
73+
Anything a check needs from RBAC must be added to `packages/core/platform/templates/preflight-hook.yaml`. The gate holds read-only cluster-wide access plus two namespaced grants, `pods/exec` in `cozy-linstor` and write access to its own status ConfigMap, and it should stay that way.

‎packages/core/platform/images/migrations/preflight/run-preflight.sh‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -152,11 +152,18 @@ run_checks() {
152152
echo "Preflight FAILED for:${fail_checks}"
153153
echo "The upgrade is BLOCKED because the cluster is not in a safe state."
154154
echo ""
155-
echo "Fix the issues reported above and retry the upgrade."
156-
echo "If this degraded state is expected and you accept the risk, re-run with"
157-
echo "either:"
158-
echo " preflight.override: true # proceed past all failing checks"
159-
echo " preflight.skipChecks: [<check-id>] # skip specific checks only"
155+
echo "Fix the issues reported above and retry the upgrade. The upgrade is"
156+
echo "retried automatically, so there is nothing to re-run by hand."
157+
echo ""
158+
echo "This verdict is also in configmap/${STATUS_CM:-cozystack-preflight-status}"
159+
echo "in namespace ${STATUS_NS:-<release namespace>}, which outlives this Job."
160+
echo ""
161+
echo "If this degraded state is expected and you accept the risk, set one of"
162+
echo "these under spec.components.platform.values.preflight on the"
163+
echo "cozystack.cozystack-platform Package CR:"
164+
echo " override: true # proceed past all failing checks"
165+
echo " skipChecks: [<check-id>] # skip specific checks only"
166+
echo "See docs/operations/upgrade-preflight.md."
160167
return 1
161168
fi
162169

‎packages/core/platform/templates/preflight-hook.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,12 @@ spec:
7070
# check bounds its own kubectl calls with a `timeout` wrapper, so a healthy
7171
# run finishes far under this; the deadline only fires on a pathological hang
7272
# the per-call timeouts somehow did not bound.
73+
#
74+
# The budget covers pulling the image too, and this Job is the first thing to
75+
# pull it on an upgrade. That is deliberate rather than a hazard: the
76+
# migration hook pulls the same image seconds later, under the release's own
77+
# 10m upgrade timeout, so a cluster that cannot fetch it inside 5 minutes was
78+
# not going to finish the upgrade either.
7379
activeDeadlineSeconds: 300
7480
template:
7581
metadata:

0 commit comments

Comments
 (0)