Repository navigation
atecontroller: set the worker Deployment rollout strategy - #1783
Scott Funkenhauser (sfunkenhauser) merged 1 commit into
Conversation
| const ( | ||
| workerRolloutMaxSurge = 0 | ||
| workerRolloutMaxUnavailable = "10%" | ||
| workerRolloutProgressDeadlineSeconds = int32(3600) |
There was a problem hiding this comment.
Should we make the rollout deadline slightly longer than the termination grace period? If they are exactly the same we could potentially still hit failed rollout errors.
There was a problem hiding this comment.
updated to 4800
| // - progressDeadlineSeconds above that 30min window, so a batch waiting out the | ||
| // drain does not have kubectl rollout status report a failed rollout. | ||
| const ( | ||
| workerRolloutMaxSurge = 0 |
There was a problem hiding this comment.
As it stands, you're guaranteed to have 90% capacity during upgrades, correct?
What if we had 2 configurations? If your workerpool is large enough, we use the default settings to potentially give you better availability (surge first), but if your workerpool is small enough that the 25% unavailable (<4 replicas) could cause it to get stuck we use the more conservative rollout strategy?
There was a problem hiding this comment.
Yes, 90% is guaranteed floor.
nit: k8s is not surge-first, its default is removing and adding at the same time.
I don't buy 2 configuration too much, it's a little bit burden for documentation and tests and could have edge cases when replica=4. Also the current "remove" is graceful, controller mark the first batch worker pods as TERMINATING but every actor can still run for 30mins, there are three cases for the new pods:
- if there are still spaces, or the pod has no resource request: it can lands immediately, no availability lost.
- if there are no space, but this cluster has autoscaler or similar: new pods land on new nodes, no availability lost.
- if there are no space and the nodes count(pool) is fixed: the 10% needs to exit so new pods can land, this is the 90% floor. notice the default k8s rollout will also stuck the new pods since there are no spaces from k8s scheduler perspective
There was a problem hiding this comment.
I'm fine with this for now, but my understanding was that with maxSurge: 0 the deployment would never have any additional pods created during upgrade. So 1) and 2) won't happen, there will always be at least 1 pod that's unavailable (and more than likely closer to 10%).
There was a problem hiding this comment.
with maxSurge: 0 the deployment would never have any additional pods created during upgrade
I think maxSurge=0 is to make active pods number capped at total, but a TERMINATING pod is not counted as active pods. So the moment the 10% get marked as TERMINATING, the new 10% can come up if there's space (which is 1) and 2)). Physically that is 90 old + 10 draining + 10 new.
12ccc35 to
24768cd
Compare
The Kubernetes default strategy (
maxSurge=25%, maxUnavailable=25%) can stall small pools when a workerpool edit and get k8s deployment rolling:below 4 replic, as
maxUnavailablerounds to 0, the surge pod needs to land first, and if existing pods already take all spaces of their node, the new one never lands and the rollout is stuck.Use
maxSurge0 withmaxUnavailable10%, which Kubernetes will evaluate as removing remove least one pod first so it will never block.Also modify
progressDeadlineSecondsto be 3600s so a batch waiting out the drain window (currently 30min) is not reported as a failed rollout.