Skip to content

fix: keep bootstrap pod until a real node has joined the cluster - #1504

Merged
synhershko merged 4 commits into
opensearch-project:mainfrom
data-ops-pulse:fix/1486-keep-bootstrap-until-cluster-joins
Sep 4, 2026
Merged

synhershko merged 4 commits into
opensearch-project:mainfrom
data-ops-pulse:fix/1486-keep-bootstrap-until-cluster-joins

Conversation

@josedev-union

Copy link
Copy Markdown
Collaborator

Description

status.initialized was derived from Kubernetes pod readiness alone, so the bootstrap pod was deleted before cluster state recovered and nodes stayed seeded to -bootstrap-0

fix #1486

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

Check List

  • Commits are signed per the DCO using --signoff
  • Unittest added for the new/changed functionality and all unit tests are successful
  • Customer-visible features documented
  • No linter warnings (make lint)

If CRDs are changed:

  • CRD YAMLs updated (make manifests) and also copied into the helm chart
  • Changes to CRDs documented

Please refer to the PR guidelines before submitting this pull request.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@josedev-union
josedev-union force-pushed the fix/1486-keep-bootstrap-until-cluster-joins branch from 17957cf to 0bc4697 Compare September 1, 2026 12:09
@josedev-union
josedev-union marked this pull request as ready for review September 1, 2026 12:10

@synhershko synhershko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is the right direction: deciding on _cat/nodes membership instead of pod readiness is the correct signal for removing the bootstrap node. I merged main into the branch to resolve the helpers_test.go conflict; all unit tests pass.

Requesting one change so this fixes #1486 in full, plus a few small cleanups.

1. Require all master pods to have joined, not just one

ClusterHasNonBootstrapNode returns true as soon as any non-bootstrap node is listed. With the security plugin disabled (the exact config in #1486), the readiness probe is curl /, which returns 200 as soon as HTTP binds and before the node has joined anything. So with a 3-master pool, all three pods can be Ready while only one has actually joined. At that point the check passes, Initialized flips, and the bootstrap pod is deleted while the cluster has two master-eligible nodes. OpenSearch keeps an odd voting configuration, so that is a single voter, and it is the bootstrap node (current cluster manager). Deleting it loses quorum permanently, which is the same silent failure #1486 describes.

The fix is small: require every expected master pod name to be present in _cat/nodes. AllMastersReady already walks the master pools and knows the StatefulSet names, so something like:

// ExpectedMasterNodeNames returns the pod names of all cluster-manager-eligible replicas.
func ExpectedMasterNodeNames(cr *opensearchv1.OpenSearchCluster) []string {
	masterRole := helpers.ResolveClusterManagerRole(cr.Spec.General.Version)
	var names []string
	for _, pool := range cr.Spec.NodePools {
		if !helpers.ContainsString(helpers.MapClusterRoles(pool.Roles, cr.Spec.General.Version), masterRole) {
			continue
		}
		for i := int32(0); i < pool.Replicas; i++ {
			names = append(names, fmt.Sprintf("%s-%d", StsName(cr, &pool), i))
		}
	}
	return names
}

and in util:

func AllMastersJoinedCluster(nodes []responses.CatNodesResponse, expected []string) bool {
	joined := map[string]bool{}
	for _, n := range nodes {
		joined[n.Name] = true
	}
	for _, name := range expected {
		if !joined[name] {
			return false
		}
	}
	return len(expected) > 0
}

Node names default to the pod hostname, so this matches _cat/nodes output. Please also add a controller-level test for it: the reconciler tests in pkg/reconcilers already use httpmock, so registering a _cat/nodes responder that returns only the bootstrap node (then bootstrap + all masters) and asserting Status.Initialized and the bootstrap pod's presence would pin this down properly.

2. Drop the desiredReplicas <= 0 guard in AllMastersReady

That is a behavior change: a master pool with replicas: 0 used to be ignored and now blocks initialization forever. readyReplicas != desiredReplicas already handles it.

3. The bootstrap-name exclusion in CountRunningPodsForNodePool is not needed

The bootstrap pod only carries the cluster label, never the node-pool label (unless a user adds it via spec.bootstrap.labels). This helper is also used by the scaler, rolling restart and emptyDir recovery, so I would rather not add a name-based special case there for a scenario that does not occur. Same for the "mislabeled bootstrap pod" test. The checkedMasterPool return is fine to keep.

4. CI

The CVE scan failure is golang.org/x/crypto v0.54.0 (CVE-2026-56854), which main is also on, so not caused by this PR.

Related, for awareness rather than this PR: with a single-master pool, bootstrap + master is two voters and the bootstrap node holds the voting slot, so removing it strands the remaining master even when it has joined. That is #1448 and needs a voting-config exclusion before deletion (or #1430, dropping the bootstrap pod altogether). This PR is still the right interim fix for pools of two or more.

status.initialized was derived from Kubernetes pod readiness alone, so the bootstrap pod was deleted before cluster state recovered and nodes stayed seeded to <name>-bootstrap-0

fix opensearch-project#1486

Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
…strap pod

With the security plugin disabled the readiness probe passes as soon as HTTP
binds, before the node has joined anything, so a pool of Ready masters can
still have only one real cluster member. Deleting the bootstrap pod at that
point loses quorum. Gate Status.Initialized on every expected cluster-manager
pod name appearing in _cat/nodes instead of on any non-bootstrap node.

Also drop the replicas<=0 guard in AllMastersReady (a zero-replica master
pool must not block initialization) and the bootstrap-name special case in
CountRunningPodsForNodePool (the bootstrap pod never carries the node-pool
label).

Adds a controller-level test that mocks _cat/nodes via httpmock: the bootstrap
pod is kept while only some masters are listed and removed once all are.

Signed-off-by: Itamar Syn-Hershko <[email protected]>

@synhershko synhershko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review comments addressed: all-masters-joined gate via _cat/nodes with a controller-level test, replicas<=0 guard dropped, bootstrap-name exclusion removed. CI green.

@synhershko
synhershko merged commit cb621e5 into opensearch-project:main Sep 4, 2026
17 checks passed
@github-project-automation github-project-automation Bot moved this from 👀 In Review to ✅ Done in Engineering Effectiveness Board Sep 4, 2026
@synhershko
synhershko deleted the fix/1486-keep-bootstrap-until-cluster-joins branch September 4, 2026 14:51
monotek pushed a commit to monotek/opensearch-k8s-operator that referenced this pull request Oct 6, 2026
…nsearch-project#1504)

### Description
status.initialized was derived from Kubernetes pod readiness alone, so
the bootstrap pod was deleted before cluster state recovered and nodes
stayed seeded to <name>-bootstrap-0

fix opensearch-project#1486

### Issues Resolved
_List any issues this PR will resolve, e.g. Closes [...]._ 

### Check List
- [ ] Commits are signed per the DCO using --signoff 
- [ ] Unittest added for the new/changed functionality and all unit
tests are successful
- [ ] Customer-visible features documented
- [ ] No linter warnings (`make lint`)

If CRDs are changed:
- [ ] CRD YAMLs updated (`make manifests`) and also copied into the helm
chart
- [ ] Changes to CRDs documented

Please refer to the [PR
guidelines](https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/developing.md#submitting-a-pr)
before submitting this pull request.

By submitting this pull request, I confirm that my contribution is made
under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and
signing off your commits, please check
[here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin).

---------

Signed-off-by: josedev-union <[email protected]>
Signed-off-by: Itamar Syn-Hershko <[email protected]>
Co-authored-by: Itamar Syn-Hershko <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

Development

Successfully merging this pull request may close these issues.

Cluster never forms when security.tls.http.enabled=false: bootstrap pod is never created and nodes stay seeded to <name>-bootstrap-0

2 participants