opensearch: a single-replica cluster loses quorum for good when the operator removes the bootstrap pod #4230

Describe the bug

With replicas: 1 a managed OpenSearch cluster has exactly two cluster-manager-eligible nodes: the operator's bootstrap pod and the single node pool member. OpenSearch keeps the voting configuration at an odd size, so with two eligible nodes it holds exactly one, and that one is the node that bootstrapped the cluster: the bootstrap pod.

The operator deletes the bootstrap pod as soon as status.initialized flips, which happens the moment every cluster-manager StatefulSet reports its replicas ready. Nothing checks the voting configuration first. That deletion removes the only voter, and the surviving node can no longer hold an election. There is no path back: changing the voting configuration itself needs a quorum.

The cluster is dead about a minute after it finishes installing, and nothing says so.

Environment

• Cozystack: main
• opensearch-k8s-operator 2.8.0 (the same code path is unchanged in 3.0.2)
• OpenSearch 2.11.1
• Reproduced both in the in-tree e2e lane and standalone in Docker

To Reproduce

In a cluster:

1. Create an OpenSearch release with replicas: 1.
2. Wait for the node pod to go ready and for the operator to delete <release>-bootstrap-0.
3. kubectl -n <ns> exec <release>-nodes-0 -- curl -sk https://localhost:9200/_cluster/health hangs.
4. Delete the node pod, or change anything that rolls it. It never comes back ready.

Standalone, no Kubernetes needed, which isolates it from everything Cozystack adds:

docker network create vc-net
for n in osA osB; do
  docker run -d --name $n --network vc-net \
    -e node.name=$n -e DISABLE_SECURITY_PLUGIN=true \
    -e OPENSEARCH_JAVA_OPTS="-Xms512m -Xmx512m" \
    -e cluster.name=vc -e discovery.seed_hosts=osA,osB \
    -e cluster.initial_master_nodes=osA -e network.bind_host=0.0.0.0 \
    opensearchproject/opensearch:2.11.1
done
# wait for both to join, then:
docker exec osA curl -s "localhost:9200/_cluster/state/metadata?filter_path=metadata.cluster_coordination"
docker rm -f osA          # what the operator does to the bootstrap pod
docker logs osB | tail    # osB can no longer elect a cluster-manager

osA stands in for the bootstrap pod, osB for the single node pool member, and the env vars are the ones the operator sets on both (pkg/builders/cluster.go, the node StatefulSet at the cluster.initial_master_nodes entry and NewBootstrapPod).

Expected behaviour

Removing the bootstrap pod leaves a cluster that can still elect a cluster-manager.

Actual behaviour

The voting configuration names only the bootstrap node, so removing it strands the survivor:

{"metadata": {"cluster_coordination": {
  "term": 1,
  "last_committed_config": ["A-0DkgYIR-WxgFmv4oQtrg"],
  "last_accepted_config": ["A-0DkgYIR-WxgFmv4oQtrg"],
  "voting_config_exclusions": []
}}}

A-0DkgYIR-WxgFmv4oQtrg is osA, the bootstrapper. osB joined the cluster and is cluster-manager-eligible, and it is still not a voter.

Logs

From the e2e run, the node pod after the bootstrap pod was deleted:

[WARN ][o.o.c.c.ClusterFormationFailureHelper] [opensearch-test-nodes-0] cluster-manager not discovered or elected yet,
an election requires a node with id [XJOEIRauRceKqYSbEdvGoQ],
have discovered [{opensearch-test-nodes-0}{MqEXb6CkQW-sPtQYWayKiA}...] which is not a quorum

The standalone reproduction prints the same sentence, with its own ids:

[WARN ][o.o.c.c.ClusterFormationFailureHelper] [osB] cluster-manager not discovered or elected yet,
an election requires a node with id [A-0DkgYIR-WxgFmv4oQtrg],
have discovered [{osB}{govFgV6FTGSntR5qsd_Mdg}...]

Timeline from the e2e run, which is where the ordering is visible:

Additional context

It is invisible until something restarts the node. A node with no cluster-manager still answers GET / with 200, which is what the operator's readiness probe asks it. The security plugin already has its configuration in memory, so the probe keeps passing and the pod stays Ready over a cluster that can no longer serve a single cluster-state request. Only after a restart does the node fail to load its security configuration, start answering the probe with an error, and reveal the state it has been in since the bootstrap pod went away.

Why this is only surfacing now. Until #4228 the securityconfig job was OOMKilled, status.initialized never flipped, and the bootstrap pod therefore lived forever. The suite failed earlier for that reason and never reached this. Fixing the OOM lets initialization complete, which is what triggers the deletion.

Why replicas: 1 specifically. The count of cluster-manager-eligible nodes is the bootstrap pod plus the node pool members carrying the role. At replicas: 1 that is 2, so the voting configuration holds 1, and it is the bootstrap pod. At replicas: 2 it is 3, the voting configuration holds 3, and losing the bootstrap pod leaves 2 of 3, which is still a quorum. So replicas: 1 is the broken case and the chart's own default of 3 is not affected.

Upstream already knows. opensearch-k8s-operator#1448 is open, labelled bug and release-blocker, and its third point describes this exact case: with a 1-master pool the bootstrap pod is the sole voter, and deleting it strands the survivor. Nothing there needs filing from us, and the analysis above was reached independently of it, which is some comfort that both are right.

What is in flight there, and why I would not wait for it:

• #1483 adds a voting-config exclusion before the bootstrap pod is deleted, with the comment "so small master pools do not lose quorum" and a test for the POST-delete-DELETE sequence. It is the fix for this. Last touched 2026-09-11, and currently carries changes-requested from a maintainer, a merge conflict, and a red golangci lint.
• #1150 approaches it from the other side: for a single node it creates no bootstrap pod at all, sets discovery.type: single-node with an empty cluster.initial_master_nodes, and drops --fail from the node probe. Open since 2025-10-29, untouched since 2026-07-07.
• #1430 removes the bootstrap pod entirely. Untouched since 2026-07-07.
• #1504 is merged, but it keeps the bootstrap pod alive longer rather than making its removal safe, so it does not cover this.

What we could do here. The choice looks like a product decision rather than a mechanical one, so I am listing rather than recommending:

• Refuse replicas: 1 for OpenSearch in values.schema.json. It fixes the product by removing the configuration that cannot work, and it is a breaking change for anyone running one replica today, who is in any case running a cluster that died shortly after it was installed. The restriction can come back off once the operator carries the upstream fix.
• Move the e2e suite to a replica count that survives the teardown. On its own this only hides the defect, so it is worth doing only alongside the point above, and with a comment naming this issue and Feat/webhook workload monitors #1448.
• Wait for upstream and bump the operator. Given the state of the three PRs above, that is not a date anyone can put in a plan.

Checklist

• I have checked the documentation
• I have searched for similar issues
• I have included all required information
• I have provided clear steps to reproduce
• I have included relevant logs