Skip to content

Remove bootstrap pod - #1430

Open
josedev-union wants to merge 15 commits into
opensearch-project:mainfrom
data-ops-pulse:remove-bootstrap-pod
Open

josedev-union wants to merge 15 commits into
opensearch-project:mainfrom
data-ops-pulse:remove-bootstrap-pod

Conversation

@josedev-union

Copy link
Copy Markdown
Collaborator

Description

This PR aims to remove bootstrap pod from cluster lifecycle and just use cluster manager nodes for bootstrapping the cluster.

Issues Resolved

List any issues this PR will resolve, e.g. Closes [...].

Check List

  • Commits are signed per the DCO using --signoff
  • Unittest added for the new/changed functionality and all unit tests are successful
  • Customer-visible features documented
  • No linter warnings (make lint)

If CRDs are changed:

  • CRD YAMLs updated (make manifests) and also copied into the helm chart
  • Changes to CRDs documented

Please refer to the PR guidelines before submitting this pull request.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@synhershko synhershko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the right direction and I'd like to get it in. Removing the bootstrap pod fixes three long-standing problems by construction: re-applying a CR over existing PVCs can no longer mint a new cluster UUID (#965), there's no transient master whose deletion can strand the voting config on small clusters, and the bare-pod recreate loop under mutating admission webhooks (#1364) is gone entirely. The supporting pieces are also correct: pointing the securityconfig job at the discovery service works because it publishes not-ready addresses, and the TCP startup probe is necessary now that no node can answer authenticated HTTP before securityadmin runs. Tests updated, CI green.

A few things need to be addressed before merge though:

  1. Migration for existing clusters. The old operator deleted <cluster>-bootstrap-0 and its PVC via the Status.Initialized branch; this PR removes that branch without any cleanup. Any cluster mid-bootstrap at operator upgrade (or where the old operator never got to delete the pod) keeps a master-eligible bare pod in the voting config forever, plus an orphaned PVC. Please add a best-effort one-time deletion of the legacy bootstrap pod/PVC (ideally with a voting-config exclusion first), kept for a couple of releases.
  2. Rolling restart on upgrade. cluster.initial_master_nodes changes value on every STS, so every existing cluster goes through a full (managed) rolling restart when the operator upgrades. That's safe via the rollingRestart reconciler, but it needs to be called out in the PR description, release notes, and docs.
  3. Operator chart CRDs. charts/opensearch-operator/files/*.yaml still contain bootstrap — needs make manifests + copy per the checklist.
  4. Rebase. Main split docs/designs/crd.md into docs/designs/crd/*.md, so this conflicts; after rebasing, the regenerated split docs should drop BootstrapConfig too.
  5. User guide. docs/userguide/main.md still documents the bootstrap pod, bootstrap.pluginsList, bootstrap.keystore, and bootstrap hostAliases.
  6. probes.startup.command is now silently ignored — the field stays in the CRD but the code reading it was removed with the exec probe. Either keep honoring a user-supplied command or deprecate the field explicitly.

Two smaller notes, non-blocking: the comment at pkg/reconcilers/cluster.go:438 still references the bootstrap pod; and since initial_master_nodes remains permanently set (now to pool-0 names), a follow-up could blank it once Status.Initialized to fully match OpenSearch guidance — much lower risk than before, so fine as a follow-up issue.

@synhershko

Copy link
Copy Markdown
Collaborator

@josedev-union let's wait with this following v3 release, this is too big of a change to handle now

IvanHunters added a commit to cozystack/cozystack that referenced this pull request Sep 14, 2026
…er bootstrap (#4231)

## What this PR does

Builds the OpenSearch operator in-tree, the way kamaji and
redis-operator already are, and carries one source patch.

The operator deletes its bootstrap pod the moment `status.initialized`
flips, without looking at the voting configuration. With `replicas: 1`
there are exactly two cluster-manager-eligible nodes, OpenSearch keeps
the voting configuration at an odd size of one, and the voter it keeps
is the bootstrap node that formed the cluster. Deleting that pod takes
away the only vote there was, and the survivor can never hold an
election again, because changing the voting configuration itself needs a
quorum. Full write-up in #4230.

The patch excludes the bootstrap node from the voting configuration
before the pod goes, and confirms the exclusion committed before
deleting anything:

1. Pod already gone: nothing to do.
2. Build a cluster client. Cannot: requeue, keep the pod.
3. `POST
/_cluster/voting_config_exclusions?node_names=<release>-bootstrap-0`.
Fails: requeue, keep the pod.
4. Read the voting configuration back. Still a voter: requeue, keep the
pod.
5. Delete the pod.
6. `DELETE /_cluster/voting_config_exclusions`. Fails: log and carry on,
the pod is already gone and the list is only capped, not load-bearing.

Step 4 is the one place this differs from the upstream attempt, which
relies on the POST being blocking. I did not verify that property, and
polling the committed configuration is correct whether or not it holds.

Every failure to reach the cluster keeps the bootstrap pod and requeues.
That is a deliberate trade: a bootstrap pod outliving its purpose costs
one pod, a cluster that cannot elect a cluster-manager needs a human.

### How the values were arrived at

Not from the documentation. Reproduced outside Kubernetes first, two
containers with the env the operator sets,
`cluster.initial_master_nodes` pointing at the first:

```text
last_committed_config: ["D6Fijf-SRN-FwIXVGNehPw"]        # the bootstrapper, alone
POST .../voting_config_exclusions?node_names=osA  ->  200
last_committed_config: ["oy8DPOV-SKKildaIZpgoPg"]        # moved to the survivor
```

Without the exclusion, removing the first container leaves the second
printing `cluster-manager not discovered or elected yet, an election
requires a node with id [...]` forever, which is the same sentence the
e2e run produced.

### Why carry it here rather than wait

Upstream knows:
[opensearch-k8s-operator#1448](opensearch-project/opensearch-k8s-operator#1448)
is open with `release-blocker`, and its third point is this exact case.
Three PRs address it and none looks close:
[#1483](opensearch-project/opensearch-k8s-operator#1483)
has changes-requested, a merge conflict and a red lint;
[#1150](opensearch-project/opensearch-k8s-operator#1150)
and
[#1430](opensearch-project/opensearch-k8s-operator#1430)
have not moved since July. When a release carries the fix, this patch
and this image build come back out.

The alternative considered and rejected was refusing `replicas: 1` in
`values.schema.json`. It is a breaking change for existing releases and
removes a configuration rather than fixing it.

### Before this can merge

`values.yaml` deliberately carries no `manager.image` yet. The image
does not exist, and committing a first-party reference without a digest
is exactly what `docs/agents/image-refs.md` forbids. Once the first
build here publishes it, I will add `repository` plus `tag@sha256:...`
in a second commit and take the PR out of draft.

### Note on CI cost

Adding the build line to the root `Makefile` puts it in
`full_rebuild_pattern`, so every push to this branch rebuilds all 28
units rather than one. Verified with `hack/build-matrix.sh`.
Unavoidable: the line is what puts the image in the matrix at all.

### What was checked locally

- The image builds and the patch applies cleanly; `/manager --help` runs
from the built image.
- `go build ./...`, `go vet` and the operator's own `pkg/reconcilers`
tests pass with the patch applied.
- `gofmt` clean, `helm unittest` for the package green, `pre-commit`
green on the changed files.
- `hack/build-matrix.sh` selects the package for a package-only change.

Not checked locally, and the reason this needs the e2e lane: that the
fix actually keeps a real cluster electable. The `opensearch` chainsaw
suite is the test for it.

### Screenshots

Not a UI change.

### Downstream repositories


- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note


```release-note
fix(opensearch-operator): exclude the bootstrap node from the voting configuration before removing it, so a single-replica OpenSearch cluster stays able to elect a cluster-manager
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

- **New Features**
- Added support for building and publishing the OpenSearch operator
image as part of the standard build workflow.
- Operator image versions now stay aligned automatically with the
packaged source version.
- Added a packaged operator image configuration for consistent
deployment.
- Improved cluster initialization by safely excluding the bootstrap node
from voting before removal and clearing the exclusion afterward.
- Added retry handling when cluster state is unavailable or voting
changes are not yet committed.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

@synhershko synhershko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My earlier points 1-6 are addressed.

Note: the one-time restart on upgrade will hang on single-master clusters because of the rolling restart quorum guard (#1449). That is not caused by this PR, but #1449 should land before this ships.

Comment thread docs/userguide/migration-guide.md Resolved
Comment thread charts/opensearch-operator/CHANGELOG.md Resolved

@synhershko synhershko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My three inline points from the last review (initial_master_nodes on every STS, keeping spec.bootstrap as a deprecated field, the broken ### Fixed heading in the CHANGELOG) are still open at this commit.

Comment thread docs/userguide/main.md Resolved
@josedev-union

Copy link
Copy Markdown
Collaborator Author

yeah, sorry, i forgot to set this as draft

@josedev-union
josedev-union marked this pull request as draft October 9, 2026 07:42
@josedev-union

Copy link
Copy Markdown
Collaborator Author

as we missed the change of major version upgrade, need to support bootstrap fields in CR and make them deprecated.

@josedev-union josedev-union left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@josedev-union
josedev-union marked this pull request as ready for review October 21, 2026 15:39

@josedev-union josedev-union left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@josedev-union

Copy link
Copy Markdown
Collaborator Author

lgtm

@synhershko synhershko left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My earlier points are addressed. Withdrawn: I said the setting is ignored once a cluster has formed and that blanking it could be a follow-up. Both are wrong for a pod with an empty data dir, see the inline comment on cluster.go.

func ResolveInitialMasterNodes(cr *opensearchv1.OpenSearchCluster, existingValue string, hasExistingValue bool, bootstrapPodExists bool) (string, bool) {
if cr.Status.Initialized {
if hasExistingValue {
return existingValue, true

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On clusters created by this operator the kept value names pool pod-0 and stays on the STS forever, so a pod-0 that returns with an empty data dir can bootstrap a second cluster. With the one-name list of a single manager pool, ClusterBootstrapService.onFoundPeersUpdated matches the local node and Coordinator.setInitialConfiguration generates a new cluster UUID and starts an election, racing the join to the real leader. This hits every pod-0 replacement on an emptyDir manager pool (rolling restart, upgrade, eviction) and a lost PVC. The old bootstrap-0 value never matched a pool pod, so this is new. From reading the source, not reproduced. OpenSearch docs say the list should be empty for nodes joining an existing cluster: https://docs.opensearch.org/latest/tuning-your-cluster/discovery-cluster-formation/settings. Fix: read the value from an optional configMapKeyRef and delete that ConfigMap once status.initialized is true (recreate it on emptyDir recovery), so later pods get no setting and the template does not change.

Comment thread docs/userguide/main.md
```

Then run `kubectl apply -f cluster.yaml`. If you watch the cluster (e.g. `watch -n 2 kubectl get pods`), you will see that after a few seconds the Operator will create several pods. First, a bootstrap pod will be created (`my-first-cluster-bootstrap-0`) that helps with initial master discovery. Then three pods for the OpenSearch cluster will be created (`my-first-cluster-nodes-0/1/2`), and one pod for the dashboards instance. After the pods are appearing as ready, which normally takes about 1-2 minutes, you can connect to your cluster using port-forwarding.
Then run `kubectl apply -f cluster.yaml`. If you watch the cluster (e.g. `watch -n 2 kubectl get pods`), you will see that after a few seconds the Operator will create several pods. The cluster-manager nodes themselves bootstrap the cluster (via `cluster.initial_master_nodes` pointing at pod-0 of each manager pool that has at least one replica). Three pods for the OpenSearch cluster will be created (`my-first-cluster-masters-0/1/2`), and one pod for the dashboards instance. After the pods are appearing as ready, which normally takes about 1-2 minutes, you can connect to your cluster using port-forwarding.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong pod names: the example above uses component: nodes, so the pods are my-first-cluster-nodes-0/1/2, not masters.

@josedev-union
josedev-union force-pushed the remove-bootstrap-pod branch 2 times, most recently from 7a6094f to bd26f85 Compare November 11, 2026 07:14
psmitj and others added 14 commits November 11, 2026 08:52
Signed-off-by: josedev-union <[email protected]>
- restore nodeAttributes rbac
- clean up leftover bootstrap pods/PVCs with voting-config exclusion
- honor probes.startup.command
- sync CRDs in the chart and bundle and document the upgrade rolling restart

Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: 👀 In Review

Development

Successfully merging this pull request may close these issues.

3 participants