Repository navigation
Description
What is the bug?
The operator never calls the voting-config-exclusions API — grep -rni "voting" --include="*.go" over the repository returns zero hits. Every code path that removes a master-eligible node relies solely on shard-allocation exclusion, which affects shard placement only and does nothing for cluster-manager voting membership. Consequences:
1. Scaling a master pool down can permanently lose quorum. The scaler treats master pools identically to data pools (pkg/reconcilers/scaler.go, reconcileNodePool): exclude via shard allocation, check HasShardsOnNode — trivially empty for a dedicated master — mark Drained, delete the pod. Scaling replicas: 3 → 1 therefore removes two masters back-to-back with no health or quorum check in between. OpenSearch's auto-reconfiguration will not shrink the voting configuration below 3 nodes, so after the second removal only 1 of 3 voters is alive → no election possible → the whole cluster becomes unavailable, with risk of stale cluster state on recovery. Per OpenSearch/Elasticsearch operational guidance, each master-eligible node must be added to POST /_cluster/voting_config_exclusions and its removal confirmed before the next one is taken down.
2. Nothing prevents scaling a master pool to 0 or deleting the entire master node pool from spec.nodePools (removeStatefulSet, scaler.go:433-498, has no master-role guard either).
3. Bootstrap pod removal has the same gap for small master pools. Status.Initialized flips the moment all master-pool pods are Ready (controllers/opensearchController.go:255-265), and the bootstrap pod is then deleted (pkg/reconcilers/cluster.go:128-129). During initialization the cluster has N+1 master-eligible nodes (bootstrap + pool). With a 1-master pool that is 2 master-eligible nodes; OpenSearch keeps an odd voting configuration, and the elected master (the bootstrap node, which formed the cluster) is typically the voter it keeps. Deleting the bootstrap pod at that moment can remove the sole voter — the remaining single master cannot elect itself and the cluster is stuck (unauthenticated 503s) until manual intervention. 3+ master pools are safe; 1-master (and even-count) pools are exposed.
Contrast: the rolling-restart reconciler does do quorum math before restarting a master (pkg/reconcilers/rollingRestart.go:274-302) — but restarts return the same node; scale-down and bootstrap removal permanently remove a voter, which is exactly the case that needs voting-config exclusions.
How can one reproduce the bug?
- Deploy a cluster with a dedicated master pool,
replicas: 3, plus a data pool. - Edit the CR: master pool
replicas: 1. - Watch the scaler delete
masters-2thenmasters-1within one or two reconcile cycles; the cluster loses quorum and all API calls fail.
For (3): deploy a cluster with a single master+data node pool of 1 replica and observe initialization races where the cluster never becomes available after the bootstrap pod is removed.
What is the expected behavior?
Removing a master-eligible node should follow the documented sequence: add the node to voting-config exclusions → wait until it has left the voting configuration → stop the node → clear the exclusions list (DELETE /_cluster/voting_config_exclusions?wait_for_removal=true). Scaling masters below 1 (arguably below 3) should be rejected by the admission webhook. The bootstrap pod should be excluded from voting before deletion.
Suggested fix
- Add
AddVotingConfigExclusion(nodeName)/ClearVotingConfigExclusions()to the gateway (opensearch-gateway/services), call them fromdrainNode/decreaseOneNode/removeStatefulSetwhen the pool has thecluster_manager/masterrole, and from the bootstrap teardown incontrollers/opensearchController.go:255-265/pkg/reconcilers/cluster.go:117-132. - Add a webhook validation rejecting master-pool scale-down to 0 and (optionally, behind a flag) any scale-down that would go below 3 masters.
- One removal at a time for masters, verified against
GET /_cluster/statevoting config, mirroring what the scaler already does per-pool for data nodes.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fields✅ Done
Activity
fix: use voting config exclusions when removing master nodes
fix: use voting config exclusions when removing master nodes
fix: use voting config exclusions when removing master nodes
fix: use voting config exclusions when removing master nodes
fix: use voting config exclusions when removing master nodes
fix: use voting config exclusions when removing master nodes
fix: use voting config exclusions when removing master nodes
fix: use voting config exclusions when removing master nodes
fix(opensearch-operator): keep a single-replica cluster electable aft…
fix(opensearch-operator): keep a single-replica cluster electable aft…
fix: use voting config exclusions when removing master nodes
fix: use voting config exclusions when removing master nodes
fix(opensearch-operator): keep a single-replica cluster electable aft…
fix: use voting config exclusions when removing master nodes
fix: use voting config exclusions when removing master-eligible nodes
fix: use voting config exclusions when removing master-eligible nodes
fix: use voting config exclusions when removing master-eligible nodes
fix(opensearch-operator): keep a single-replica cluster electable aft…