Skip to content

[BUG] No voting config exclusions when removing master-eligible nodes — master scale-down and bootstrap removal can lose quorum permanently #1448

Closed

Description

@synhershko

What is the bug?

The operator never calls the voting-config-exclusions API — grep -rni "voting" --include="*.go" over the repository returns zero hits. Every code path that removes a master-eligible node relies solely on shard-allocation exclusion, which affects shard placement only and does nothing for cluster-manager voting membership. Consequences:

1. Scaling a master pool down can permanently lose quorum. The scaler treats master pools identically to data pools (pkg/reconcilers/scaler.go, reconcileNodePool): exclude via shard allocation, check HasShardsOnNode — trivially empty for a dedicated master — mark Drained, delete the pod. Scaling replicas: 3 → 1 therefore removes two masters back-to-back with no health or quorum check in between. OpenSearch's auto-reconfiguration will not shrink the voting configuration below 3 nodes, so after the second removal only 1 of 3 voters is alive → no election possible → the whole cluster becomes unavailable, with risk of stale cluster state on recovery. Per OpenSearch/Elasticsearch operational guidance, each master-eligible node must be added to POST /_cluster/voting_config_exclusions and its removal confirmed before the next one is taken down.

2. Nothing prevents scaling a master pool to 0 or deleting the entire master node pool from spec.nodePools (removeStatefulSet, scaler.go:433-498, has no master-role guard either).

3. Bootstrap pod removal has the same gap for small master pools. Status.Initialized flips the moment all master-pool pods are Ready (controllers/opensearchController.go:255-265), and the bootstrap pod is then deleted (pkg/reconcilers/cluster.go:128-129). During initialization the cluster has N+1 master-eligible nodes (bootstrap + pool). With a 1-master pool that is 2 master-eligible nodes; OpenSearch keeps an odd voting configuration, and the elected master (the bootstrap node, which formed the cluster) is typically the voter it keeps. Deleting the bootstrap pod at that moment can remove the sole voter — the remaining single master cannot elect itself and the cluster is stuck (unauthenticated 503s) until manual intervention. 3+ master pools are safe; 1-master (and even-count) pools are exposed.

Contrast: the rolling-restart reconciler does do quorum math before restarting a master (pkg/reconcilers/rollingRestart.go:274-302) — but restarts return the same node; scale-down and bootstrap removal permanently remove a voter, which is exactly the case that needs voting-config exclusions.

How can one reproduce the bug?

  1. Deploy a cluster with a dedicated master pool, replicas: 3, plus a data pool.
  2. Edit the CR: master pool replicas: 1.
  3. Watch the scaler delete masters-2 then masters-1 within one or two reconcile cycles; the cluster loses quorum and all API calls fail.

For (3): deploy a cluster with a single master+data node pool of 1 replica and observe initialization races where the cluster never becomes available after the bootstrap pod is removed.

What is the expected behavior?

Removing a master-eligible node should follow the documented sequence: add the node to voting-config exclusions → wait until it has left the voting configuration → stop the node → clear the exclusions list (DELETE /_cluster/voting_config_exclusions?wait_for_removal=true). Scaling masters below 1 (arguably below 3) should be rejected by the admission webhook. The bootstrap pod should be excluded from voting before deletion.

Suggested fix

  • Add AddVotingConfigExclusion(nodeName) / ClearVotingConfigExclusions() to the gateway (opensearch-gateway/services), call them from drainNode/decreaseOneNode/removeStatefulSet when the pool has the cluster_manager/master role, and from the bootstrap teardown in controllers/opensearchController.go:255-265 / pkg/reconcilers/cluster.go:117-132.
  • Add a webhook validation rejecting master-pool scale-down to 0 and (optionally, behind a flag) any scale-down that would go below 3 masters.
  • One removal at a time for masters, verified against GET /_cluster/state voting config, mirroring what the scaler already does per-pool for data nodes.

Activity

  1. added a commit that references this issue on Aug 6, 2026
    1ff82b4
  2. added 2 commits that reference this issue on Aug 20, 2026
    45c716b
    3c2aca2
  3. added 2 commits that reference this issue on Aug 28, 2026
    767cbe6
    d8a6836
  4. added a commit that references this issue on Sep 10, 2026
    3ffe117
  5. added a commit that references this issue on Sep 11, 2026
    a290317
  6. added a commit that references this issue on Sep 11, 2026
    fb173ee
  7. added 2 commits that reference this issue on Sep 22, 2026
    3b0b2e9
    3299927
  8. added a commit that references this issue on Sep 24, 2026
    8b7e7a9
  9. added 2 commits that reference this issue on Sep 24, 2026
    6fd50b1
    c2d83a2
  10. added a commit that references this issue on Sep 24, 2026
    d33303d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions