Repository navigation
fix: keep bootstrap pod until a real node has joined the cluster - #1504
Conversation
17957cf to
0bc4697
Compare
synhershko
left a comment
There was a problem hiding this comment.
Thanks, this is the right direction: deciding on _cat/nodes membership instead of pod readiness is the correct signal for removing the bootstrap node. I merged main into the branch to resolve the helpers_test.go conflict; all unit tests pass.
Requesting one change so this fixes #1486 in full, plus a few small cleanups.
1. Require all master pods to have joined, not just one
ClusterHasNonBootstrapNode returns true as soon as any non-bootstrap node is listed. With the security plugin disabled (the exact config in #1486), the readiness probe is curl /, which returns 200 as soon as HTTP binds and before the node has joined anything. So with a 3-master pool, all three pods can be Ready while only one has actually joined. At that point the check passes, Initialized flips, and the bootstrap pod is deleted while the cluster has two master-eligible nodes. OpenSearch keeps an odd voting configuration, so that is a single voter, and it is the bootstrap node (current cluster manager). Deleting it loses quorum permanently, which is the same silent failure #1486 describes.
The fix is small: require every expected master pod name to be present in _cat/nodes. AllMastersReady already walks the master pools and knows the StatefulSet names, so something like:
// ExpectedMasterNodeNames returns the pod names of all cluster-manager-eligible replicas.
func ExpectedMasterNodeNames(cr *opensearchv1.OpenSearchCluster) []string {
masterRole := helpers.ResolveClusterManagerRole(cr.Spec.General.Version)
var names []string
for _, pool := range cr.Spec.NodePools {
if !helpers.ContainsString(helpers.MapClusterRoles(pool.Roles, cr.Spec.General.Version), masterRole) {
continue
}
for i := int32(0); i < pool.Replicas; i++ {
names = append(names, fmt.Sprintf("%s-%d", StsName(cr, &pool), i))
}
}
return names
}and in util:
func AllMastersJoinedCluster(nodes []responses.CatNodesResponse, expected []string) bool {
joined := map[string]bool{}
for _, n := range nodes {
joined[n.Name] = true
}
for _, name := range expected {
if !joined[name] {
return false
}
}
return len(expected) > 0
}Node names default to the pod hostname, so this matches _cat/nodes output. Please also add a controller-level test for it: the reconciler tests in pkg/reconcilers already use httpmock, so registering a _cat/nodes responder that returns only the bootstrap node (then bootstrap + all masters) and asserting Status.Initialized and the bootstrap pod's presence would pin this down properly.
2. Drop the desiredReplicas <= 0 guard in AllMastersReady
That is a behavior change: a master pool with replicas: 0 used to be ignored and now blocks initialization forever. readyReplicas != desiredReplicas already handles it.
3. The bootstrap-name exclusion in CountRunningPodsForNodePool is not needed
The bootstrap pod only carries the cluster label, never the node-pool label (unless a user adds it via spec.bootstrap.labels). This helper is also used by the scaler, rolling restart and emptyDir recovery, so I would rather not add a name-based special case there for a scenario that does not occur. Same for the "mislabeled bootstrap pod" test. The checkedMasterPool return is fine to keep.
4. CI
The CVE scan failure is golang.org/x/crypto v0.54.0 (CVE-2026-56854), which main is also on, so not caused by this PR.
Related, for awareness rather than this PR: with a single-master pool, bootstrap + master is two voters and the bootstrap node holds the voting slot, so removing it strands the remaining master even when it has joined. That is #1448 and needs a voting-config exclusion before deletion (or #1430, dropping the bootstrap pod altogether). This PR is still the right interim fix for pools of two or more.
status.initialized was derived from Kubernetes pod readiness alone, so the bootstrap pod was deleted before cluster state recovered and nodes stayed seeded to <name>-bootstrap-0 fix opensearch-project#1486 Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
9742ac8 to
52d7b6f
Compare
Signed-off-by: Itamar Syn-Hershko <[email protected]>
…strap pod With the security plugin disabled the readiness probe passes as soon as HTTP binds, before the node has joined anything, so a pool of Ready masters can still have only one real cluster member. Deleting the bootstrap pod at that point loses quorum. Gate Status.Initialized on every expected cluster-manager pod name appearing in _cat/nodes instead of on any non-bootstrap node. Also drop the replicas<=0 guard in AllMastersReady (a zero-replica master pool must not block initialization) and the bootstrap-name special case in CountRunningPodsForNodePool (the bootstrap pod never carries the node-pool label). Adds a controller-level test that mocks _cat/nodes via httpmock: the bootstrap pod is kept while only some masters are listed and removed once all are. Signed-off-by: Itamar Syn-Hershko <[email protected]>
synhershko
left a comment
There was a problem hiding this comment.
Review comments addressed: all-masters-joined gate via _cat/nodes with a controller-level test, replicas<=0 guard dropped, bootstrap-name exclusion removed. CI green.
…nsearch-project#1504) ### Description status.initialized was derived from Kubernetes pod readiness alone, so the bootstrap pod was deleted before cluster state recovered and nodes stayed seeded to <name>-bootstrap-0 fix opensearch-project#1486 ### Issues Resolved _List any issues this PR will resolve, e.g. Closes [...]._ ### Check List - [ ] Commits are signed per the DCO using --signoff - [ ] Unittest added for the new/changed functionality and all unit tests are successful - [ ] Customer-visible features documented - [ ] No linter warnings (`make lint`) If CRDs are changed: - [ ] CRD YAMLs updated (`make manifests`) and also copied into the helm chart - [ ] Changes to CRDs documented Please refer to the [PR guidelines](https://github.com/opensearch-project/opensearch-k8s-operator/blob/main/docs/developing.md#submitting-a-pr) before submitting this pull request. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check [here](https://github.com/opensearch-project/OpenSearch/blob/main/CONTRIBUTING.md#developer-certificate-of-origin). --------- Signed-off-by: josedev-union <[email protected]> Signed-off-by: Itamar Syn-Hershko <[email protected]> Co-authored-by: Itamar Syn-Hershko <[email protected]>
Description
status.initialized was derived from Kubernetes pod readiness alone, so the bootstrap pod was deleted before cluster state recovered and nodes stayed seeded to -bootstrap-0
fix #1486
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
Check List
make lint)If CRDs are changed:
make manifests) and also copied into the helm chartPlease refer to the PR guidelines before submitting this pull request.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.