Skip to content

fix: use voting config exclusions when removing master-eligible nodes - #1483

Merged
synhershko merged 3 commits into
opensearch-project:mainfrom
data-ops-pulse:fix/voting-config-exclusions-master-scale-down
Sep 24, 2026
Merged

synhershko merged 3 commits into
opensearch-project:mainfrom
data-ops-pulse:fix/voting-config-exclusions-master-scale-down

Conversation

@josedev-union

@josedev-union josedev-union commented Aug 3, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Removing a master-eligible node never touched the voting configuration: the scaler treated master pools like data pools (allocation exclusion, wait for shards, delete the pod), so scaling a 3-master pool to 1 removed two voters back to back and could leave the cluster without quorum. The same gap existed when a whole master pool was removed from spec.nodePools and when the bootstrap pod was deleted after initialization.

Master-eligible nodes now follow the documented sequence, one node at a time:

  1. POST /_cluster/voting_config_exclusions?node_names=<pod>&timeout=10s, which blocks until the node has left the voting configuration. The POST is skipped for a node that is not a cluster member, since OpenSearch would otherwise record an _absent_ tombstone that only counts toward cluster.max_voting_config_exclusions.
  2. Shrink the StatefulSet (or delete the pod). The exclusion is re-applied right before the shrink because the list is cluster-wide and can be cleared underneath a pool.
  3. Once the node has left the cluster (_cat/nodes), DELETE /_cluster/voting_config_exclusions?wait_for_removal=true. The scaler keeps the Drained status until this succeeds. The waiting clear is never issued while the node is still a member: it would sit on OpenSearch's 30s wait_for_removal deadline, which is also the operator's client deadline, and fail.

This applies whether or not SmartScaler is enabled. With SmartScaler off only the voting step is performed; the shard drain stays opt-in. The same steps run in removeStatefulSet for a removed master pool and in the bootstrap teardown.

Other behaviour:

  • A master scale-down reverted while its target is still excluded clears the list with wait_for_removal=false, the one case where a non-waiting clear is correct, since the node stays.
  • ScalerReconciler sweeps the exclusions list on every reconcile of an initialized cluster. Exclusions whose nodes have all left (a clear that failed, an operator restart mid-removal, the bootstrap pod) are cleared with wait. A live pool member that nothing is removing but is still excluded is cleared without wait with a Warning event, because such a node is silently out of the voting configuration and a later restart of another master could lose quorum. While any excluded node is still leaving, the sweep does nothing.
  • IsMasterStatefulSet falls back to the container's node.roles env when the opensearch.role label is missing or overridden by a user label.
  • removeBootstrapPod failures return RequeueAfter without an error so the reconciler chain keeps running.

The DELETE endpoint only consumes wait_for_removal; passing timeout on it is rejected with a 400, so the timeout is only sent on the POST.

Docs: docs/userguide/main.md (SmartScaler section) describes the master-removal sequence and the cluster-wide clear; docs/userguide/webhooks.md notes the existing guard against clusters without a master-eligible pool.

Issues Resolved

Closes #1448

Check List

  • Commits are signed per the DCO using --signoff
  • Unittest added for the new/changed functionality and all unit tests are successful
  • Customer-visible features documented
  • No linter warnings (make lint)

If CRDs are changed:

  • CRD YAMLs updated (make manifests) and also copied into the helm chart
  • Changes to CRDs documented

Please refer to the PR guidelines before submitting this pull request.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@josedev-union
josedev-union marked this pull request as ready for review August 4, 2026 12:34
@josedev-union
josedev-union marked this pull request as draft August 4, 2026 12:54
@josedev-union
josedev-union force-pushed the fix/voting-config-exclusions-master-scale-down branch from e0c0689 to a572a44 Compare August 6, 2026 11:33
@josedev-union
josedev-union marked this pull request as ready for review August 6, 2026 11:35
@josedev-union
josedev-union marked this pull request as draft August 6, 2026 11:40
@josedev-union
josedev-union force-pushed the fix/voting-config-exclusions-master-scale-down branch from a572a44 to ba18ecf Compare August 20, 2026 16:59
@josedev-union
josedev-union marked this pull request as ready for review August 20, 2026 17:00
@josedev-union
josedev-union force-pushed the fix/voting-config-exclusions-master-scale-down branch from ba18ecf to baa5e79 Compare August 21, 2026 06:56
@josedev-union
josedev-union force-pushed the fix/voting-config-exclusions-master-scale-down branch 2 times, most recently from 7c9209c to 7ad0950 Compare September 1, 2026 13:00

@synhershko synhershko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on - the approach is the right one and it addresses all three parts of #1448 (master scale-down, removing a whole master pool, and the bootstrap pod), plus the webhook guard. A few things need to change before this can go in though.

Blocking

Nil-pointer panic when smartScaler: false and the pool is master-eligible (the default config). In decreaseOneNode, clusterClient is only created inside if smartDecrease { ... }, but the new code then calls services.RemoveExcludeNodeHost(clusterClient, ...) and services.ClearVotingConfigExclusions(clusterClient, ...) whenever isMaster is true. With SmartScaler off, reconcileNodePool routes master pools through exclude → drain → decreaseOneNode(..., smartDecrease=false, isMaster=true), and that dereferences a nil client.

I reproduced it on this branch with a small Ginkgo test that calls decreaseOneNode(status, sts, "masters", false, true) with a mocked k8s client:

runtime error: invalid memory address or nil pointer dereference

Because the StatefulSet has already been shrunk and the scaler status removed by that point, nothing retries: the allocation exclusion and the voting-config exclusion for the removed node are never cleared. Voting exclusions then accumulate on every master scale-down until the default cluster.max_voting_config_exclusions (10) is hit, after which every further master removal fails.

Fix: create the client when smartDecrease || isMaster.

Should fix

  1. The "retry without wait" fallback is unsafe. On every clear path (decreaseOneNode, both branches of removeStatefulSet, removeBootstrapPod), if DELETE _cluster/voting_config_exclusions?wait_for_removal=true fails, the code immediately retries with wait_for_removal=false. That clears the exclusion while the node may still be alive, which lets OpenSearch put it back into the voting configuration right before the pod is killed - i.e. the original bug, re-introduced on a timeout. Lingering exclusions are harmless; please just return an error / requeue and retry the waiting DELETE later. Never clear without wait.

  2. Client and server timeouts collide. The operator's HTTP transport has ResponseHeaderTimeout: 30s, and OpenSearch's default server-side timeout for both the POST and the DELETE is also 30s. The client will usually give up first, which makes the fallback in (1) more likely to trigger. Pass an explicit shorter timeout= query parameter (e.g. 10s) on both requests and rely on requeue.

  3. removeBootstrapPod errors now short-circuit the whole reconciler chain. If the OpenSearch client can't be built or the POST fails, ClusterReconciler.Reconcile returns an error and opensearchController stops before running scaler/upgrade/restart/etc. Bootstrap deletion could not fail this way before. Either log and continue, or explain why blocking here is acceptable.

  4. No tests for the new behaviour. The added tests cover CountMasterEligibleReplicas, IsMasterStatefulSet, the URL builder and the webhook, but neither the master scale-down path in the scaler nor removeBootstrapPod is exercised - which is how the panic above got through. scaler_test.go already has httpmock helpers (registerClusterSettingsResponders etc.); one test per path that asserts the voting-config POST/DELETE calls happen in the right order would be enough.

Minor

  • if smartDecrease || isMaster { directly after if !smartDecrease && !isMaster { return ... } is always true.
  • In decreaseOneNode, returning requeue=true when the clear fails does nothing useful: the scaler status has already been removed, so the next reconcile sees desireReplicaDiff == 0.
  • The webhook now rejects any cluster with no master/cluster_manager pool, on create as well as update. That's reasonable, but the chart enables the webhook with failurePolicy: Fail by default, so it's a user-visible change and should be mentioned in the changelog and the PR description.
  • The PR description is still the empty template; please fill in the description and tick the checklist.
  • The CVE Scan failure is unrelated - the branch is behind main and is missing the CVE fix that landed after this was opened. A rebase should clear it.

Once the nil-client panic and the no-wait fallback are addressed I'm happy to take another look.

@josedev-union
josedev-union force-pushed the fix/voting-config-exclusions-master-scale-down branch from 7ad0950 to ad3a7a1 Compare September 10, 2026 09:20
@synhershko
synhershko force-pushed the fix/voting-config-exclusions-master-scale-down branch from f05ce72 to 09868b6 Compare September 11, 2026 07:06
@synhershko

Copy link
Copy Markdown
Collaborator

Thanks @josedev-union, f05ce72 addresses everything from my review: the nil client is fixed, the no-wait fallback is gone, both requests pass timeout=10s, bootstrap removal no longer fails the reconciler chain, and the new tests cover the scale-down, StatefulSet removal and bootstrap paths.

I pushed two things on top:

  • Rebased onto main (it conflicted). main now has 6793450, which rejects clusters without a cluster-manager pool that has replicas >= 1 (version-mapped). I dropped this PR's validateMasterEligibleNodes and CountMasterEligibleReplicas in favor of it and updated the webhooks doc note to match.
  • 09868b6 fixes a regression from the new desireReplicaDiff == 0 && Drained branch. If a master scale-down is reverted after the voting exclusion but before the StatefulSet shrinks, finishDecreaseCleanup's waiting DELETE can never succeed, because the node is not leaving. The scaler retried forever, the node stayed excluded, and every later master removal's waiting clear would hang the same way. Now that cleanup only runs when the target is past the StatefulSet. Otherwise cancelDecrease removes the allocation exclusion and clears voting exclusions with wait_for_removal=false, which is the one place no-wait is correct because the node stays. That clear is cluster-wide, and the scaler reconciles every pool on each pass, so decreaseOneNode now re-POSTs the target's exclusion right before shrinking. Both branches have tests.

Unit tests and the controllers envtest suite pass locally. One item from the review is still open: the PR description is the empty template. Could you fill it in?

@josedev-union
josedev-union force-pushed the fix/voting-config-exclusions-master-scale-down branch from 09868b6 to e29d3ca Compare September 11, 2026 10:57
IvanHunters added a commit to cozystack/cozystack that referenced this pull request Sep 14, 2026
…er bootstrap (#4231)

## What this PR does

Builds the OpenSearch operator in-tree, the way kamaji and
redis-operator already are, and carries one source patch.

The operator deletes its bootstrap pod the moment `status.initialized`
flips, without looking at the voting configuration. With `replicas: 1`
there are exactly two cluster-manager-eligible nodes, OpenSearch keeps
the voting configuration at an odd size of one, and the voter it keeps
is the bootstrap node that formed the cluster. Deleting that pod takes
away the only vote there was, and the survivor can never hold an
election again, because changing the voting configuration itself needs a
quorum. Full write-up in #4230.

The patch excludes the bootstrap node from the voting configuration
before the pod goes, and confirms the exclusion committed before
deleting anything:

1. Pod already gone: nothing to do.
2. Build a cluster client. Cannot: requeue, keep the pod.
3. `POST
/_cluster/voting_config_exclusions?node_names=<release>-bootstrap-0`.
Fails: requeue, keep the pod.
4. Read the voting configuration back. Still a voter: requeue, keep the
pod.
5. Delete the pod.
6. `DELETE /_cluster/voting_config_exclusions`. Fails: log and carry on,
the pod is already gone and the list is only capped, not load-bearing.

Step 4 is the one place this differs from the upstream attempt, which
relies on the POST being blocking. I did not verify that property, and
polling the committed configuration is correct whether or not it holds.

Every failure to reach the cluster keeps the bootstrap pod and requeues.
That is a deliberate trade: a bootstrap pod outliving its purpose costs
one pod, a cluster that cannot elect a cluster-manager needs a human.

### How the values were arrived at

Not from the documentation. Reproduced outside Kubernetes first, two
containers with the env the operator sets,
`cluster.initial_master_nodes` pointing at the first:

```text
last_committed_config: ["D6Fijf-SRN-FwIXVGNehPw"]        # the bootstrapper, alone
POST .../voting_config_exclusions?node_names=osA  ->  200
last_committed_config: ["oy8DPOV-SKKildaIZpgoPg"]        # moved to the survivor
```

Without the exclusion, removing the first container leaves the second
printing `cluster-manager not discovered or elected yet, an election
requires a node with id [...]` forever, which is the same sentence the
e2e run produced.

### Why carry it here rather than wait

Upstream knows:
[opensearch-k8s-operator#1448](opensearch-project/opensearch-k8s-operator#1448)
is open with `release-blocker`, and its third point is this exact case.
Three PRs address it and none looks close:
[#1483](opensearch-project/opensearch-k8s-operator#1483)
has changes-requested, a merge conflict and a red lint;
[#1150](opensearch-project/opensearch-k8s-operator#1150)
and
[#1430](opensearch-project/opensearch-k8s-operator#1430)
have not moved since July. When a release carries the fix, this patch
and this image build come back out.

The alternative considered and rejected was refusing `replicas: 1` in
`values.schema.json`. It is a breaking change for existing releases and
removes a configuration rather than fixing it.

### Before this can merge

`values.yaml` deliberately carries no `manager.image` yet. The image
does not exist, and committing a first-party reference without a digest
is exactly what `docs/agents/image-refs.md` forbids. Once the first
build here publishes it, I will add `repository` plus `tag@sha256:...`
in a second commit and take the PR out of draft.

### Note on CI cost

Adding the build line to the root `Makefile` puts it in
`full_rebuild_pattern`, so every push to this branch rebuilds all 28
units rather than one. Verified with `hack/build-matrix.sh`.
Unavoidable: the line is what puts the image in the matrix at all.

### What was checked locally

- The image builds and the patch applies cleanly; `/manager --help` runs
from the built image.
- `go build ./...`, `go vet` and the operator's own `pkg/reconcilers`
tests pass with the patch applied.
- `gofmt` clean, `helm unittest` for the package green, `pre-commit`
green on the changed files.
- `hack/build-matrix.sh` selects the package for a package-only change.

Not checked locally, and the reason this needs the e2e lane: that the
fix actually keeps a real cluster electable. The `opensearch` chainsaw
suite is the test for it.

### Screenshots

Not a UI change.

### Downstream repositories


- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note


```release-note
fix(opensearch-operator): exclude the bootstrap node from the voting configuration before removing it, so a single-replica OpenSearch cluster stays able to elect a cluster-manager
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added support for building and publishing the OpenSearch operator
image as part of the standard build workflow.
- Operator image versions now stay aligned automatically with the
packaged source version.
- Added a packaged operator image configuration for consistent
deployment.
- Improved cluster initialization by safely excluding the bootstrap node
from voting before removal and clearing the exclusion afterward.
- Added retry handling when cluster state is unavailable or voting
changes are not yet committed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@synhershko

Copy link
Copy Markdown
Collaborator

Took another pass over this at 97aecca. The direction is right — POST the exclusion (which blocks until the voting config has actually dropped the node), stop the node, then clear — and that matches the documented procedure. Node names are the right key too, since we set cluster.initial_master_nodes to the bootstrap pod name and never set node.name, so OpenSearch node names equal pod names (the existing AllMastersJoinedCluster/ExpectedMasterNodeNames code already leans on that). go build, go vet and the unit suites are clean, envtest suites pass with assets set.

One blocker, though, and it comes straight out of the timeout item from my last review.

Blocker: every clear call returns 400

ClearVotingConfigExclusions (opensearch-gateway/services/os_client.go:403) sends:

DELETE /_cluster/voting_config_exclusions?wait_for_removal=<bool>&timeout=10s

RestClearVotingConfigExclusionsAction.prepareRequest consumes only wait_for_removal:

ClearVotingConfigExclusionsRequest req = new ClearVotingConfigExclusionsRequest();
if (request.hasParam("wait_for_removal")) {
    req.setWaitForRemoval(request.paramAsBoolean("wait_for_removal", true));
}

ClearVotingConfigExclusionsRequest's timeout is not settable over REST at all, and BaseRestHandler.handleRequest rejects any parameter the handler didn't consume. Same on 2.x and main, and the handler is inherited from ES 7.10 so 1.x behaves the same. RestAddVotingConfigExclusionAction does consume timeout — the constant is shared between the two calls, which is how this slipped through.

Reproduced against a real opensearchproject/opensearch:2.19.0 running this branch's gateway code unmodified:

ADD err: <nil>
CLEAR(wait=true)  err: voting config exclusions failed: [400 Bad Request]
  illegal_argument_exception: request [/_cluster/voting_config_exclusions] contains unrecognized parameter: [timeout]
CLEAR(wait=false) err: ... same 400 ...

The same DELETE without &timeout=10s returns 200.

So the PR can add voting exclusions and never remove any of them. What that does at each call site:

  • scaler.go:403 (finishDecreaseCleanup) returns (true, clearErr) and deliberately retains the Scaler status. Trace a 3→1 master scale-down: the last decreaseOneNode shrinks the STS, the clear 400s, status stays Drained with the target past the end of the STS. Next pass, desireReplicaDiff == 0 and that same status routes straight back into finishDecreaseCleanup → 400 again. Permanent error loop. And because Reconcile folds poolErr into results.Err (scaler.go:95-105) and opensearchController.go returns on any non-terminal reconciler error, the dashboards, upgrade, rolling-restart and snapshot-repository reconcilers stop running for that cluster altogether. Any master-pool scale-down deadlocks the operator for that CR.
  • scaler.go:438 (cancelDecrease) — same 400, same loop on a reverted scale-down.
  • scaler.go:700 / :721 (removeStatefulSet) — error and requeue, but the STS is already gone, so the exclusion is simply never cleared.
  • cluster.go:848 (removeBootstrapPod) — logged only, so no loop, but the bootstrap node stays excluded forever.

They then accumulate against cluster.max_voting_config_exclusions (default 10); past the cap AddVotingConfigExclusion starts returning 400 too and every master removal fails at excludeNode (scaler.go:465). A master scaled down and back up reuses its node id from the retained PVC, so a stale exclusion permanently bars it from voting.

The fix is to drop &timeout= from the DELETE only, keeping it on the POST, plus the four test assertions in scaler_test.go / cluster_test.go that currently hard-code wait_for_removal=true&timeout=10s.

Which puts the timeout collision back

My earlier point about the client/server deadlines can't be solved with a query parameter on this endpoint. Once timeout comes off, the DELETE's server-side wait falls back to 30s, which is exactly the transport's ResponseHeaderTimeout: 30 * time.Second (os_client.go:105). And the waiting DELETE is issued immediately after the STS shrink (scaler.go:403, :700, :721), so the node is normally still terminating — this call will routinely sit for ~30s and then fail on whichever deadline wins. Either confirm the node is actually gone before issuing it (there's already a _cat/nodes helper in util) or give this one request its own longer context deadline.

Also worth fixing before merge

  • The POST succeeds for a node that isn't there. POST ...?node_names=does-not-exist&timeout=10s returns 200 on 2.19.0 and writes {"node_id":"_absent_","node_name":"does-not-exist"} into cluster state — resolveVotingConfigExclusions only errors for the deprecated nodeDescriptions form. So AddVotingConfigExclusion returning nil doesn't mean the target left the voting config; it may mean it was never in it. The tombstone counts toward the cap and only a successful clear removes it, which compounds the above. Reachable at scaler.go:297 and :646.
  • Silent behaviour change for smartScaler: false, and the two paths disagree. reconcileNodePool now routes master pools through excludeNode → drainNode → decreaseOneNode regardless of SmartScaler (scaler.go:203-215), and drainNode (scaler.go:517) has no SmartScaler guard — it reactivates shard allocation and then blocks on HasShardsOnNode. For the common single master,data pool with SmartScaler explicitly off, a scale-down that used to just delete the pod now waits on a full shard drain that may never finish. Meanwhile removeStatefulSet takes the graceful path for masters (scaler.go:628-630) but does guard the drain wait with if r.instance.Spec.ConfMgmt.SmartScaler (:665-684). Only the voting exclusion should bypass the opt-out; the shard drain should stay opt-in in both paths. Nothing covers this in the tests.
  • ReactivateShardAllocation now fires on a SmartScaler-off master-pool removal (scaler.go:659), PUT-ing allocation.enable=all, which can clobber the primaries guard an in-flight upgrade or rolling restart set.
  • scaler.go:161: Drained && (target == "" || target past-the-end) sends a reverted scale-down with empty status conditions into finishDecreaseCleanup, i.e. a waiting DELETE for a node that isn't leaving — the hang e29d3ca removed, reintroduced for legacy CRs whose Conditions were never populated.
  • helpers.IsMasterStatefulSet (pkg/helpers/helpers.go:900-903) keys on the opensearch.role label, but user-supplied nodePool.labels are merged after the role label in pkg/builders/cluster.go:363-372, so a user label can silently disable the guard; STSes predating the label get no exclusion either. Using the label is unavoidable here (the NodePool is gone from the CR by the time removeStatefulSet runs), but a fallback or at least a warning when a removed pool's role is unknown would be safer.
  • cancelDecrease clearing all exclusions cluster-wide is inherent to the API and acceptable given the re-POST in decreaseOneNode, but it belongs in the docs, not only in a code comment.
  • The Makefile:326 change (always go install mockery rather than test -s ||) is unrelated and slows every make mockery/make generate. Separate PR, please.

Tests

The set of paths covered is the right one — master scale-down with SmartScaler off, failed clear retaining status, reverted scale-down, pending-cleanup resume, whole-STS removal, bootstrap removal — and asserting call ordering and that no wait_for_removal=false leaks into the removal paths is exactly right. But it's all httpmock, which answers 200 to any query string, and the assertions pin the exact query including timeout=10s. So they lock the defect in and will fail when it's fixed. Nothing exercises the _absent_ no-op POST, the max_voting_config_exclusions rejection, or a master pool with SmartScaler off going through drainNode. For a change that rests entirely on an API contract, one functional test against a real multi-master cluster in opensearch-operator/functionaltests would be worth it — it's the only thing that would have caught this.

Status of my earlier review

Fixed: the nil-client panic with smartScaler: false (scaler.go:282), the unsafe retry-without-wait fallback, the removeBootstrapPod short-circuit (RequeueAfter with a nil error is right — only Requeue: true short-circuits the chain), the always-true condition, and the useless requeue=true on a failed clear. Tests added. The webhook guard landing via 6793450 on main is fine and the new docs/userguide/webhooks.md paragraph matches what the webhook actually does — but there's still no CHANGELOG entry, and the PR description is the unmodified template with an unticked checklist. Please fill that in.

Since I authored the merge commit at the head and two of the commits here, this should get a second approver before it goes in.

@josedev-union
josedev-union force-pushed the fix/voting-config-exclusions-master-scale-down branch 2 times, most recently from 7e5867f to 0e64afc Compare September 22, 2026 16:09

@josedev-union josedev-union left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@josedev-union
josedev-union force-pushed the fix/voting-config-exclusions-master-scale-down branch from 0e64afc to c985a16 Compare September 24, 2026 08:05
@synhershko
synhershko force-pushed the fix/voting-config-exclusions-master-scale-down branch from 38f4ebe to 6fd50b1 Compare September 24, 2026 12:41
@synhershko synhershko changed the title Fix/voting config exclusions master scale down fix: use voting config exclusions when removing master-eligible nodes Sep 24, 2026
@synhershko
synhershko force-pushed the fix/voting-config-exclusions-master-scale-down branch from 6fd50b1 to c2d83a2 Compare September 24, 2026 12:48
Removing a master-eligible node never touched the voting configuration, so
scaling a master pool down, removing a master pool from spec.nodePools or
deleting the bootstrap pod could permanently remove voters and lose quorum.

Master-eligible nodes are now removed one at a time following the documented
sequence: POST /_cluster/voting_config_exclusions for the node (blocks until it
has left the voting configuration), shrink the StatefulSet or delete the pod,
and once the node has left the cluster clear the exclusions with
wait_for_removal=true. The scaler keeps its Drained status until the clear
succeeds. The waiting clear is never issued while the node is still a member,
since it would sit on OpenSearch's 30s wait_for_removal deadline, which is also
the operator's client deadline. This applies regardless of SmartScaler; with
SmartScaler off only the voting step runs and the shard drain stays opt-in.

A reverted master scale-down clears the list without waiting, the one case
where that is correct because the node stays. The scaler also sweeps the
exclusions list on every reconcile of an initialized cluster: exclusions whose
nodes have all left are cleared with wait, and a live pool member nothing is
removing that is still excluded is cleared without wait with a Warning event.

The POST is skipped for a node that is not a cluster member, since OpenSearch
would otherwise record an _absent_ tombstone that only counts toward
cluster.max_voting_config_exclusions. The DELETE endpoint only consumes
wait_for_removal; it rejects a timeout parameter with a 400, so the timeout is
only sent on the POST. IsMasterStatefulSet falls back to the container's
node.roles env when the opensearch.role label is missing or overridden.

Closes opensearch-project#1448

Signed-off-by: josedev-union <[email protected]>
Signed-off-by: Itamar Syn-Hershko <[email protected]>
@josedev-union
josedev-union force-pushed the fix/voting-config-exclusions-master-scale-down branch from c2d83a2 to d33303d Compare September 24, 2026 13:46

@josedev-union josedev-union left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@synhershko
synhershko merged commit 4debbb8 into opensearch-project:main Sep 24, 2026
18 checks passed
@synhershko
synhershko deleted the fix/voting-config-exclusions-master-scale-down branch September 24, 2026 18:32
monotek pushed a commit to monotek/opensearch-k8s-operator that referenced this pull request Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

[BUG] No voting config exclusions when removing master-eligible nodes — master scale-down and bootstrap removal can lose quorum permanently

2 participants