Repository navigation
fix: do not hold an allocation exclusion a drain cannot use - #1586
IvanHunters wants to merge 3 commits into
Conversation
When drainDataNodes is on and a shard on the candidate node can be placed nowhere else, the node never empties. PreparePodForDelete excluded it from allocation before it could know that, and RemoveExcludeNodeHost only runs after a successful DeletePod, so cluster.routing.allocation.exclude._name kept a data node out of allocation for as long as the restart stayed stuck. The restart itself was already stuck before the yellow health gate landed; what changed is that the drain path is now reached on a permanently yellow cluster, which is exactly the case where the drain cannot finish. Nothing clears the exclusion there: CleanStaleExclusionList only treats a pod as stale once it has restarted, and this one never does. Ask the allocator first, and only while nothing is moving, so a drain in progress pays nothing. If any started shard on the node reports can_move_to_other_node: no, skip the exclusion and withdraw one left by an earlier pass. A drain that can still make progress behaves as it did. The two-data-node path is untouched, since it never waits for a full drain. Resolves opensearch-project#1585 Signed-off-by: IvanHunters <[email protected]>
…nish (#4323) ## What this PR does The previous patch let a permanently yellow cluster past the restart health gate. With three or more nodes `packages/apps/opensearch` renders `drainDataNodes: true`, and the restart then reaches `PreparePodForDelete`, which excludes the candidate node from allocation before it can know whether the drain will finish. On the cluster that made the gate wait it never finishes: a shard on that node has nowhere else to go, which is the same condition holding the cluster yellow. The exclusion is only withdrawn after a pod is really deleted (`rollingRestart.go`, `upgrade.go`), so a data node stays out of allocation for as long as the restart is stuck, shedding its shards onto the rest of the cluster and buying nothing. Before the yellow gate this path was unreachable with draining on, so nothing was written at all. `patches/skip-unusable-drain-exclusion.diff` asks the allocator before excluding anything, and only while nothing is moving, so a drain making progress costs nothing extra. A started shard answering `can_move_to_other_node: no` means the drain cannot finish: no exclusion is written, and one left by an earlier pass is withdrawn. A drain that can still progress behaves exactly as before, and the two-data-node path is untouched because it never waits for a full drain. Checking after the exclusion instead would flap: the next reconcile ten seconds later would apply it again. The v2.8.0 base has no allocation-explain call of its own, so the patch adds the client method and a response type carrying only the field the decision reads. The second commit makes the shipped tests real. The image build ran `go build` alone, so a test file that stopped compiling, or stopped asserting, still produced an image. `go test` over the two package trees the patches touch now runs in the builder stage; both run without a cluster. Upstream as opensearch-project/opensearch-k8s-operator#1585, fixed there by opensearch-project/opensearch-k8s-operator#1586. Both patches go away with the first operator release carrying that change. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff: two files under `packages/system/opensearch-operator/images/opensearch-operator/`. No package README or values schema, no API or CRD, no node prerequisites in `hack/`, no vendored chart, no make target. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): a rolling restart or upgrade that cannot drain its next node no longer leaves that node excluded from shard allocation for as long as the restart is stuck ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved node draining during OpenSearch maintenance and restarts. * The operator now checks whether shards can move before excluding a node from allocation. * If a shard cannot relocate, the existing exclusion is withdrawn and draining pauses, helping prevent nodes from remaining unnecessarily unavailable. * Existing behavior is preserved for two-node clusters and while shard relocation is already active. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
Thanks for digging into this one, and for the very precise write-up in #1585 — the bug report is correct and the chain you traced is exactly right. I walked it against
So the node stays in One framing note before the review: this isn't strictly a #1569 regression. A green cluster holding an immovable shard (an index pinned with The check doesn't fire on the pass that matters
moveDecision = localShardsBalancer.decideMove(shard);
if (moveDecision.isDecisionTaken() && moveDecision.canRemain()) {
MoveDecision rebalanceDecision = localShardsBalancer.decideRebalance(shard);
moveDecision = rebalanceDecision.withRemainDecision(moveDecision.getCanRemainDecision());
}With a rebalance decision in hand, Which gives:
Probed on this branch with the explain responder modelling that contract (shards and health otherwise identical to the repro in #1585): Two transient The suite here is green, but Other risks worth weighing
SuggestionAsking the allocator up front can't work: before the exclusion there is no answer to read, and after it there is already a write to undo. Bounding the wait avoids both. Exclude as today, and if the node still holds shards after N consecutive passes with If you'd rather keep the explain check, it needs to run after the exclusion is applied and key on Happy to look again once it's reworked — the underlying report is a good catch and worth fixing properly. |
The allocation-explain check in the previous commit cannot work. OpenSearch only emits can_move_to_other_node once something already stops the shard from remaining where it is (MoveDecision.toXContent, reached from BalancedShardsAllocator.decideShardAllocation, which swaps in a rebalance decision whenever the shard can remain). Before the exclusion there is nothing to read, so the check passed and the node was excluded exactly as before; after it, the field says no and the exclusion was withdrawn, and the pass after that applied it again. A permanent exclusion became a flapping one, which is worse. The test that let this through returned the move fields unconditionally, a shape the API cannot produce. Bound the wait instead. The exclusion is applied as it always was. Passes where the allocator is idle accumulate, movement restarts the clock, and once the quiet has run past the threshold the exclusion is withdrawn once, a Warning is emitted naming a shard still on the node, and the candidate is left alone until shards move again or the node turns out to be empty. No allocation-explain calls at all, one write and one unwrite, and it covers a green cluster holding an immovable shard the same way, which the explain check never addressed. The withdrawal defers to the scaler while it owns the exclude list, for the same reason CleanStaleExclusionList does. The drain state lives under its own Drain component status, because the restarter and the upgrader both rewrite their conditions from scratch every pass. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
AppendExcludeNodeHost wrote cluster settings on every call, including when the node it is asked to exclude is already excluded. The restart and upgrade paths call it once per pass for the whole of a drain, so a drain that takes ten minutes republishes cluster state sixty times to no effect. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
|
You were right on every point. The explain check is gone and the fix is now a bounded wait. Three things need you: approving the workflow runs, a re-review of 1) Approve the queued workflow runsSix of them sit at 2) Re-review
|
synhershko
left a comment
There was a problem hiding this comment.
The PR description still describes the allocation-explain check, which is gone. It names drain_progress_test.go and says the two-data-node path is untouched, but recordUnfinishedDrain now runs for dataCount == 2 too. Rewrite it for the bounded wait, 150 words or fewer.
1038b8e: take it.
The stale Waiting for node to drain condition with draining off (secondary point in #1585) is tracked in #1619.
| if !notEmpty { | ||
| return false, nil, nil | ||
| } | ||
| return true, current, nil |
There was a problem hiding this comment.
Once stalled, the restart only resumes on unrelated shard activity, and fixing the cause usually produces none. Example: lower number_of_replicas, or drop the require._name filter in the green case. Without the exclusion nothing moves, the cluster goes green and idle, and this returns true on every pass with no end. Before this PR the held exclusion let the drain finish on its own once fixed, and the Warning event still says the restart waits until the shards can move. My earlier "and stop" was wrong: re-apply the exclusion once per drainStallWarningAfter while stalled, so a fixed cluster resumes for one write and one unwrite per window.
| return ctrl.Result{Requeue: true, RequeueAfter: 10 * time.Second}, nil | ||
| } | ||
|
|
||
| if err := r.client.DeletePod(&corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: c.podName, Namespace: c.podNS}}); err != nil { |
There was a problem hiding this comment.
The drain status entry is never cleared after deletePod. If it still says drainStalled, the next restart of this pod stands still forever on a healthy cluster and the node is never excluded.
| return drainStallOutcome{conditions: drainConditions(podName, now, false)} | ||
| } | ||
|
|
||
| startedAt, ok := drainStartedAt(current) |
There was a problem hiding this comment.
this can come from an earlier restart of the same pod.
| } | ||
|
|
||
| if activity { | ||
| return drainStallOutcome{conditions: drainConditions(podName, now, false)} |
There was a problem hiding this comment.
when shards are moving, drainStarted is set to now on every pass, so the status is rewritten every 10s during every normal drain.
Description
With
drainDataNodes: true,PreparePodForDeleteexcludes the candidate node from allocation before it can know whether the drain will finish. When a shard on that node can be placed nowhere else, it never does: the function keeps returning false, andRemoveExcludeNodeHostruns only after a successfulDeletePod, socluster.routing.allocation.exclude._namekeeps a data node out of allocation for as long as the restart is stuck.CleanStaleExclusionListdoes not reach this one, becauseisPodStaletreats a pod as stale only once it has already restarted, and this pod never gets that far.The restart standing still is not new. What changed with the settled-yellow health gate is that the drain path is now reached on a permanently yellow cluster, which is exactly the case where the drain cannot complete, so the exclusion is now written where previously nothing was written at all.
This asks the allocator before excluding anything, and only while nothing is moving, so a drain that is making progress costs nothing extra. If any started shard on the node answers
can_move_to_other_node: no, the exclusion is skipped, and one left behind by an earlier pass is withdrawn. A drain that can still progress behaves exactly as before. The two-data-node path is untouched, since it never waits for a full drain, only for system-index primaries.Cost is one
_cluster/allocation/explainper started shard on the candidate node, on passes where the allocator is idle, and the loop stops at the first shard that cannot move.Issues Resolved
Resolves #1585
Check List
make lint)Tests added in
opensearch-gateway/services/drain_progress_test.go:drainCanProceed: a movable shard, a shard with nowhere to go, one immovable shard among movable ones, an empty node, shards belonging to other nodes, and a relocating shard already on its way out.PreparePodForDelete: an exclusion already in place is withdrawn, nothing is written when none was there, a drain that can still progress still excludes and waits, and the allocation explanation is never requested while shards are moving.Each new branch was checked by mutation: dropping the withdrawal, dropping the shard-activity guard, applying the check at two data nodes, and never blocking the drain each turn a test red.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.