Skip to content

fix: do not hold an allocation exclusion a drain cannot use - #1586

Open
IvanHunters wants to merge 3 commits into
opensearch-project:mainfrom
IvanHunters:fix/release-unproductive-drain-exclusion
Open

IvanHunters wants to merge 3 commits into
opensearch-project:mainfrom
IvanHunters:fix/release-unproductive-drain-exclusion

Conversation

@IvanHunters

Copy link
Copy Markdown

Description

With drainDataNodes: true, PreparePodForDelete excludes the candidate node from allocation before it can know whether the drain will finish. When a shard on that node can be placed nowhere else, it never does: the function keeps returning false, and RemoveExcludeNodeHost runs only after a successful DeletePod, so cluster.routing.allocation.exclude._name keeps a data node out of allocation for as long as the restart is stuck. CleanStaleExclusionList does not reach this one, because isPodStale treats a pod as stale only once it has already restarted, and this pod never gets that far.

The restart standing still is not new. What changed with the settled-yellow health gate is that the drain path is now reached on a permanently yellow cluster, which is exactly the case where the drain cannot complete, so the exclusion is now written where previously nothing was written at all.

This asks the allocator before excluding anything, and only while nothing is moving, so a drain that is making progress costs nothing extra. If any started shard on the node answers can_move_to_other_node: no, the exclusion is skipped, and one left behind by an earlier pass is withdrawn. A drain that can still progress behaves exactly as before. The two-data-node path is untouched, since it never waits for a full drain, only for system-index primaries.

Cost is one _cluster/allocation/explain per started shard on the candidate node, on passes where the allocator is idle, and the loop stops at the first shard that cannot move.

Issues Resolved

Resolves #1585

Check List

  • Commits are signed per the DCO using --signoff
  • Unittest added for the new/changed functionality and all unit tests are successful
  • Customer-visible features documented (behaviour fix, no new configuration surface)
  • No linter warnings (make lint)

Tests added in opensearch-gateway/services/drain_progress_test.go:

  • drainCanProceed: a movable shard, a shard with nowhere to go, one immovable shard among movable ones, an empty node, shards belonging to other nodes, and a relocating shard already on its way out.
  • PreparePodForDelete: an exclusion already in place is withdrawn, nothing is written when none was there, a drain that can still progress still excludes and waits, and the allocation explanation is never requested while shards are moving.

Each new branch was checked by mutation: dropping the withdrawal, dropping the shard-activity guard, applying the check at two data nodes, and never blocking the drain each turn a test red.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

When drainDataNodes is on and a shard on the candidate node can be placed
nowhere else, the node never empties. PreparePodForDelete excluded it from
allocation before it could know that, and RemoveExcludeNodeHost only runs
after a successful DeletePod, so cluster.routing.allocation.exclude._name
kept a data node out of allocation for as long as the restart stayed stuck.

The restart itself was already stuck before the yellow health gate landed;
what changed is that the drain path is now reached on a permanently yellow
cluster, which is exactly the case where the drain cannot finish. Nothing
clears the exclusion there: CleanStaleExclusionList only treats a pod as
stale once it has restarted, and this one never does.

Ask the allocator first, and only while nothing is moving, so a drain in
progress pays nothing. If any started shard on the node reports
can_move_to_other_node: no, skip the exclusion and withdraw one left by an
earlier pass. A drain that can still make progress behaves as it did. The
two-data-node path is untouched, since it never waits for a full drain.

Resolves opensearch-project#1585

Signed-off-by: IvanHunters <[email protected]>
IvanHunters added a commit to cozystack/cozystack that referenced this pull request Sep 18, 2026
…nish (#4323)

## What this PR does

The previous patch let a permanently yellow cluster past the restart
health gate. With three or more nodes `packages/apps/opensearch` renders
`drainDataNodes: true`, and the restart then reaches
`PreparePodForDelete`, which excludes the candidate node from allocation
before it can know whether the drain will finish.

On the cluster that made the gate wait it never finishes: a shard on
that node has nowhere else to go, which is the same condition holding
the cluster yellow. The exclusion is only withdrawn after a pod is
really deleted (`rollingRestart.go`, `upgrade.go`), so a data node stays
out of allocation for as long as the restart is stuck, shedding its
shards onto the rest of the cluster and buying nothing. Before the
yellow gate this path was unreachable with draining on, so nothing was
written at all.

`patches/skip-unusable-drain-exclusion.diff` asks the allocator before
excluding anything, and only while nothing is moving, so a drain making
progress costs nothing extra. A started shard answering
`can_move_to_other_node: no` means the drain cannot finish: no exclusion
is written, and one left by an earlier pass is withdrawn. A drain that
can still progress behaves exactly as before, and the two-data-node path
is untouched because it never waits for a full drain.

Checking after the exclusion instead would flap: the next reconcile ten
seconds later would apply it again.

The v2.8.0 base has no allocation-explain call of its own, so the patch
adds the client method and a response type carrying only the field the
decision reads.

The second commit makes the shipped tests real. The image build ran `go
build` alone, so a test file that stopped compiling, or stopped
asserting, still produced an image. `go test` over the two package trees
the patches touch now runs in the builder stage; both run without a
cluster.

Upstream as opensearch-project/opensearch-k8s-operator#1585, fixed there
by opensearch-project/opensearch-k8s-operator#1586. Both patches go away
with the first operator release carrying that change.

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: two files under
`packages/system/opensearch-operator/images/opensearch-operator/`. No
package README or values schema, no API or CRD, no node prerequisites in
`hack/`, no vendored chart, no make target.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(opensearch-operator): a rolling restart or upgrade that cannot drain its next node no longer leaves that node excluded from shard allocation for as long as the restart is stuck
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
  * Improved node draining during OpenSearch maintenance and restarts.
* The operator now checks whether shards can move before excluding a
node from allocation.
* If a shard cannot relocate, the existing exclusion is withdrawn and
draining pauses, helping prevent nodes from remaining unnecessarily
unavailable.
* Existing behavior is preserved for two-node clusters and while shard
relocation is already active.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@synhershko

Copy link
Copy Markdown
Collaborator

Thanks for digging into this one, and for the very precise write-up in #1585 — the bug report is correct and the chain you traced is exactly right. I walked it against main (1bd5b8a):

  • the yellow branch in CheckClusterStatusForRestart (os_data_service.go:236-263) sits above if drainNodes { return false } (:266), so a settled yellow cluster returns ready=true with draining on;
  • PreparePodForDelete then calls AppendExcludeNodeHost as its first action, before anything is known about whether the drain can finish;
  • CheckPodSafeToDelete only forgives replicas stuck on the node_version decider, so a shard immovable for any other reason (allocation filter, awareness, watermark) keeps it false;
  • RemoveExcludeNodeHost runs only after a successful DeletePod (rollingRestart.go:394, upgrade.go:567), and isPodStale keeps the entry on purpose because the pod never restarted.

So the node stays in cluster.routing.allocation.exclude._name indefinitely. That part is not in dispute.

One framing note before the review: this isn't strictly a #1569 regression. A green cluster holding an immovable shard (an index pinned with index.routing.allocation.require._name and number_of_replicas: 0, say) leaked the same exclusion before #1569 too, since green short-circuits to ready=true at :232. #1569 widened the exposure rather than creating it, and a fix is worth judging against both shapes.

The check doesn't fire on the pass that matters

can_move_to_other_node is not emitted when the shard can remain where it is. From BalancedShardsAllocator.decideShardAllocation:

moveDecision = localShardsBalancer.decideMove(shard);
if (moveDecision.isDecisionTaken() && moveDecision.canRemain()) {
    MoveDecision rebalanceDecision = localShardsBalancer.decideRebalance(shard);
    moveDecision = rebalanceDecision.withRemainDecision(moveDecision.getCanRemainDecision());
}

With a rebalance decision in hand, MoveDecision.toXContent prints can_rebalance_cluster / can_rebalance_to_other_node and not can_move_to_other_node — that field only appears on the cannotRemain path, i.e. only once something already makes can_remain_on_current_node: "no". In this flow that something is our own exclusion.

Which gives:

  • pass 1, no exclusion yet: explain answers can_remain: yes and omits can_move_to_other_node, so explain.CanMoveToOtherNode == "", drainCanProceed returns true, and AppendExcludeNodeHost runs exactly as it does on main;
  • pass 2, exclusion in place: can_remain: no, can_move: no, so the exclusion is withdrawn;
  • pass 3: can_remain is yes again, so it is re-applied.

Probed on this branch with the explain responder modelling that contract (shards and health otherwise identical to the repro in #1585):

pass 1: ready=false excludedAfter="n0" puts=1
pass 2: ready=false excludedAfter=""   puts=2
pass 3: ready=false excludedAfter="n0" puts=3
pass 4: ready=false excludedAfter=""   puts=4

Two transient _cluster/settings writes per reconcile pair, indefinitely. The permanent exclusion becomes a flapping one, which is arguably harder on an already-unhealthy cluster: every toggle republishes cluster state, and shards that can move get shed off the node and are free to drift back each time the exclusion lifts.

The suite here is green, but explainBody() returns can_remain_on_current_node:"no" together with can_move_to_other_node unconditionally, independent of the exclude list. That shape can't come out of OpenSearch, and it's what lets TestPreparePodForDeleteReleasesAnUnproductiveExclusion's "no exclusion is applied in the first place" case pass. A mock whose explain response depends on whether the node is currently excluded would turn that test red.

Other risks worth weighing

  • An explain burst before every drain, for every user. The guard runs whenever hasShardActivity is false, which is exactly the state at the start of every drain, healthy cluster or not. Pre-exclusion no shard short-circuits the loop (they all answer ""), so it is one serialized _cluster/allocation/explain — a master-level API — per started shard on the candidate node, ahead of every single pod restart. On a node holding a few hundred shards that is a few hundred sequential round trips per reconcile. The "stops at the first shard that cannot move" cost in the description only holds under the mocked response.
  • No scaler guard on the withdrawal. CleanStaleExclusionList deliberately refuses to touch the exclude list while a Scaler component is Excluded/Drained (util/util.go:519-522). The new RemoveExcludeNodeHost call has no equivalent, so if a restart candidate ever coincides with a node the smart scaler is draining, it cancels that drain.
  • Throttled is not immovable. MoveDecision.forceMove() is canRemain == false && decision == YES, so a THROTTLED move serializes as can_move_to_other_node: "no". The hasShardActivity gate shields most of this, but health is sampled before a potentially long explain loop, so the two can disagree by the time the verdict is used.
  • New failure mode. A health blip now fails PreparePodForDelete outright; it had no health dependency before.
  • Smaller things: strconv.Atoi failing continues and silently counts the shard as movable, where DetectShardStuckVersionMismatch errors out; strings.EqualFold where the neighbouring code uses == "no"; and a second GetHealth immediately after CheckClusterStatusForRestart already fetched one.
  • The give-up path is a log line only. The secondary point in [BUG] With drainDataNodes on, a permanently yellow cluster leaves the candidate node in allocation.exclude._name forever #1585 — upgrade.go:544 reporting Waiting for node to drain when draining is not what's blocking — is still open, so an operator sees an upgrade that never moves with no condition saying why.

Suggestion

Asking the allocator up front can't work: before the exclusion there is no answer to read, and after it there is already a write to undo. Bounding the wait avoids both. Exclude as today, and if the node still holds shards after N consecutive passes with hasShardActivity == false, withdraw the exclusion once, set a component condition naming the shard that cannot move, and stop. No explain calls, no flapping, one write and one unwrite, and it covers the green-cluster shape identically.

If you'd rather keep the explain check, it needs to run after the exclusion is applied and key on can_remain_on_current_node == "no" && can_move_to_other_node == "no" — with tests whose mock derives the explain response from the current exclude list.

Happy to look again once it's reworked — the underlying report is a good catch and worth fixing properly.

The allocation-explain check in the previous commit cannot work. OpenSearch
only emits can_move_to_other_node once something already stops the shard from
remaining where it is (MoveDecision.toXContent, reached from
BalancedShardsAllocator.decideShardAllocation, which swaps in a rebalance
decision whenever the shard can remain). Before the exclusion there is nothing
to read, so the check passed and the node was excluded exactly as before; after
it, the field says no and the exclusion was withdrawn, and the pass after that
applied it again. A permanent exclusion became a flapping one, which is worse.
The test that let this through returned the move fields unconditionally, a shape
the API cannot produce.

Bound the wait instead. The exclusion is applied as it always was. Passes where
the allocator is idle accumulate, movement restarts the clock, and once the
quiet has run past the threshold the exclusion is withdrawn once, a Warning is
emitted naming a shard still on the node, and the candidate is left alone until
shards move again or the node turns out to be empty. No allocation-explain calls
at all, one write and one unwrite, and it covers a green cluster holding an
immovable shard the same way, which the explain check never addressed.

The withdrawal defers to the scaler while it owns the exclude list, for the same
reason CleanStaleExclusionList does. The drain state lives under its own Drain
component status, because the restarter and the upgrader both rewrite their
conditions from scratch every pass.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
AppendExcludeNodeHost wrote cluster settings on every call, including when the
node it is asked to exclude is already excluded. The restart and upgrade paths
call it once per pass for the whole of a drain, so a drain that takes ten
minutes republishes cluster state sixty times to no effect.

Assisted-by: LLM
Signed-off-by: IvanHunters <[email protected]>
@IvanHunters

Copy link
Copy Markdown
Author

You were right on every point. The explain check is gone and the fix is now a bounded wait. Three things need you: approving the workflow runs, a re-review of a6096f6, and a yes or no on 1038b8e, which is deliberately separate.

1) Approve the queued workflow runs

Six of them sit at action_required on 1038b8e, so nothing ran on the rework. The one I would most like to see is data-integrity-upgrade: on the first push it failed by timing out after thirty minutes waiting for the upgrade to complete, which is exactly the area this change touches, so I am treating it as a likely regression of that first attempt rather than a flake until it reports on this commit.

2) Re-review a6096f6, starting with TestStalledDrainSettlesAfterOneExclusionAndOneWithdrawal

That test is the whole claim in one place: ten passes against a node holding a shard whose replica can never be assigned, with a live exclude list in the mock, asserting one write, one unwrite, and the node not left excluded.

What the commit does. The exclusion is applied exactly as before. Passes where the allocator is idle accumulate, any movement restarts the clock, and once the quiet runs past drainStallWarningAfter the exclusion is withdrawn once, a Warning names a shard still on the node, and the candidate is left alone until shards move again or the node turns out to be empty. No allocation-explain calls anywhere, so the burst you flagged goes with them, and a green cluster holding an immovable shard is covered the same way, which the explain check never was.

Your other points, in the same commit:

  • The withdrawal defers to the scaler while a Scaler component is Excluded/Drained, for the reason CleanStaleExclusionList does. Pinned by mutation: remove the guard and a test goes red.
  • The state lives under its own Drain component status, because the restarter and the upgrader both rewrite their conditions from scratch every pass.
  • THROTTLED, the swallowed strconv.Atoi and strings.EqualFold went out with the explain code.
  • The give-up path is no longer a log line alone: the upgrader reports that the drain is not progressing instead of Waiting for node to drain.
  • One GetHealth, and only on the pass where the pod could not be deleted, not ahead of every restart.

Every new decision term is pinned by its own mutation: movement restarting the clock, the release firing once, the emptied node, candidate identity, the scaler guard.

3) Decide whether to take 1038b8e or drop it

It makes AppendExcludeNodeHost skip the write when the node is already in the exclude list. Today it writes cluster settings on every call, and the restart path calls it once per pass for the whole of a drain, so a ten-minute drain republishes cluster state sixty times to no effect. It is unrelated to the bug, so it is its own commit and easy to drop.

4) For the record, on the first attempt

The flapping reproduces here exactly as you described it:

pass 1: ready=false excludedAfter="n0" puts=1
pass 2: ready=false excludedAfter=""   puts=2
pass 3: ready=false excludedAfter="n0" puts=3
pass 4: ready=false excludedAfter=""   puts=4

I read the contract this time instead of assuming it. BalancedShardsAllocator.decideShardAllocation swaps in a rebalance decision whenever the shard can remain (2.11.1, lines 253-256), and MoveDecision.toXContent emits can_move_to_other_node only when clusterRebalanceDecision is nil (lines 328-343). So there was nothing to read before the exclusion, and the guard passed for the wrong reason. My test hid it because explainBody() returned the move fields unconditionally, which is a response the API cannot produce. That is the actual defect: I guessed a third-party contract and then wrote a fixture that agreed with the guess.

Your framing note is right too, and #1585 is wrong on it. A green cluster holding an immovable shard leaks the same exclusion with #1569 nowhere near it, since green short-circuits at :232. I will correct the issue text.

@synhershko synhershko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR description still describes the allocation-explain check, which is gone. It names drain_progress_test.go and says the two-data-node path is untouched, but recordUnfinishedDrain now runs for dataCount == 2 too. Rewrite it for the bounded wait, 150 words or fewer.

1038b8e: take it.

The stale Waiting for node to drain condition with draining off (secondary point in #1585) is tracked in #1619.

if !notEmpty {
return false, nil, nil
}
return true, current, nil

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Once stalled, the restart only resumes on unrelated shard activity, and fixing the cause usually produces none. Example: lower number_of_replicas, or drop the require._name filter in the green case. Without the exclusion nothing moves, the cluster goes green and idle, and this returns true on every pass with no end. Before this PR the held exclusion let the drain finish on its own once fixed, and the Warning event still says the restart waits until the shards can move. My earlier "and stop" was wrong: re-apply the exclusion once per drainStallWarningAfter while stalled, so a fixed cluster resumes for one write and one unwrite per window.

@synhershko synhershko added the bug Something isn't working label Oct 1, 2026
return ctrl.Result{Requeue: true, RequeueAfter: 10 * time.Second}, nil
}

if err := r.client.DeletePod(&corev1.Pod{ObjectMeta: metav1.ObjectMeta{Name: c.podName, Namespace: c.podNS}}); err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The drain status entry is never cleared after deletePod. If it still says drainStalled, the next restart of this pod stands still forever on a healthy cluster and the node is never excluded.

return drainStallOutcome{conditions: drainConditions(podName, now, false)}
}

startedAt, ok := drainStartedAt(current)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this can come from an earlier restart of the same pod.

}

if activity {
return drainStallOutcome{conditions: drainConditions(podName, now, false)}

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when shards are moving, drainStarted is set to now on every pass, so the status is rewritten every 10s during every normal drain.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

Status: 👀 In Review

Development

Successfully merging this pull request may close these issues.

[BUG] With drainDataNodes on, a permanently yellow cluster leaves the candidate node in allocation.exclude._name forever

3 participants