Repository navigation
docs: reorganize the upgrade runbook for better UX - #1624
Conversation
Raji (Raji14)
left a comment
Raji (Raji14)
left a comment
There was a problem hiding this comment.
A few suggestions before merge:
- Warning 1 under "Three things that break an upgrade":
The heading is "Three things that break an upgrade", but Warning 1 is phrased as the correct action: "Flip the node's version label before deleting its old worker pods."
Consider rephrasing it to state the anti-pattern (like warnings 2 & 3 do):Do not delete old worker pods before flipping the node's version label. Otherwise the old pool reschedules replacements onto the same node next to the new atelet...
- Pre-flight check for multi-node clusters:
In the previous version, line 199 noted:# At least two nodes; on a single node the roll is a full stop.This was omitted from the new checklist. We should add a checklist item confirming the cluster has at least 2 worker nodes, since a rolling zero-downtime upgrade cannot work on a single node. - Rollback list completeness (Steps 3 & 4):
In## Something's wrong, or I need to undo, the list jumps from "Past step 5" to "Past step 2". For an operator who reached Step 3 (deployed new atelet) or Step 4 (cloned workerpool) but hasn't flipped nodes yet, it would be helpful to include:- Past step 4: delete the cloned WorkerPool (kubectl -n $NS delete workerpool $NEW_WORKERPOOL)- Past step 3: delete the new DaemonSet (kubectl delete daemonset -n ate-system -l app=atelet,ate.dev/substrate-version=$NEW_VERSION)
- "Check your progress" note phrasing:
"Steps 1, 2 and 6 leave no mark you can read back from the cluster."
Those steps do update cluster objects (e.g.ate-controller/apideployments, CRDs); it's just that the 3 quick-check commands above don't query them. Clarifying: "The three quick-check commands above do not report whether steps 1, 2, or 6 have run..." would avoid confusion. - Step 2 readiness check:
In Step 2: "wait for READY to equal DESIRED again on every serving pool... before step 5."
Because Step 4 clones the serving pool, we should probably advise waiting before Step 4 so operators don't clone a pool whose Deployment is actively rolling. - Doc consistency (60s vs 30m):
Good catch updating the suspend window from 60s to 30 minutes to matchworkloadGracePeriod. As a follow-up (or in this PR),tools/setup-gcp/README.md#2-create-clusterstill mentions "about 60 seconds" and should be updated as well.
|
Thanks, pushed second commit. On point 2: |
|
Two open questions for you: |
Talked offline and add both points. |
|
LGTM, thanks! |
|
ci is broken, #1643 should fix it |
Phrase the first warning as the mistake, like the other two. Add rollback entries for the new DaemonSet and the cloned pools, so an operator who stopped before flipping any node finds their step in the list. Say what the progress commands do not show instead of claiming that steps leave no trace. Wait for a controller-triggered roll to settle before cloning pools. Note that a single-node cluster makes the per-node step a full stop. The setup-gcp README's eviction warning still said 60 seconds; it is 30 minutes.
The intro said at most one node's worth of capacity is out of service during the roll, which is only true when there is more than one node. Say so, and point at the single-node note in the per-node step.
The port-forward, CA bundle and token only need the running cluster, so they can be prepared before the roll starts. Doing it there lets the checklist confirm the installed ate-api-server serves DrainWorker before anything changes, instead of finding out in the per-node step.
3bebc7a to
9de0879
Compare
7d2abc3
into
agent-substrate:main
Part of #1197