Repository navigation
Remove bootstrap pod - #1430
Remove bootstrap pod#1430josedev-union wants to merge 15 commits into
Conversation
0cf16e5 to
45b4ba0
Compare
45b4ba0 to
5ef5983
Compare
synhershko
left a comment
There was a problem hiding this comment.
This is the right direction and I'd like to get it in. Removing the bootstrap pod fixes three long-standing problems by construction: re-applying a CR over existing PVCs can no longer mint a new cluster UUID (#965), there's no transient master whose deletion can strand the voting config on small clusters, and the bare-pod recreate loop under mutating admission webhooks (#1364) is gone entirely. The supporting pieces are also correct: pointing the securityconfig job at the discovery service works because it publishes not-ready addresses, and the TCP startup probe is necessary now that no node can answer authenticated HTTP before securityadmin runs. Tests updated, CI green.
A few things need to be addressed before merge though:
- Migration for existing clusters. The old operator deleted
<cluster>-bootstrap-0and its PVC via theStatus.Initializedbranch; this PR removes that branch without any cleanup. Any cluster mid-bootstrap at operator upgrade (or where the old operator never got to delete the pod) keeps a master-eligible bare pod in the voting config forever, plus an orphaned PVC. Please add a best-effort one-time deletion of the legacy bootstrap pod/PVC (ideally with a voting-config exclusion first), kept for a couple of releases. - Rolling restart on upgrade.
cluster.initial_master_nodeschanges value on every STS, so every existing cluster goes through a full (managed) rolling restart when the operator upgrades. That's safe via the rollingRestart reconciler, but it needs to be called out in the PR description, release notes, and docs. - Operator chart CRDs.
charts/opensearch-operator/files/*.yamlstill containbootstrap— needsmake manifests+ copy per the checklist. - Rebase. Main split
docs/designs/crd.mdintodocs/designs/crd/*.md, so this conflicts; after rebasing, the regenerated split docs should dropBootstrapConfigtoo. - User guide.
docs/userguide/main.mdstill documents the bootstrap pod,bootstrap.pluginsList,bootstrap.keystore, and bootstrap hostAliases. probes.startup.commandis now silently ignored — the field stays in the CRD but the code reading it was removed with the exec probe. Either keep honoring a user-supplied command or deprecate the field explicitly.
Two smaller notes, non-blocking: the comment at pkg/reconcilers/cluster.go:438 still references the bootstrap pod; and since initial_master_nodes remains permanently set (now to pool-0 names), a follow-up could blank it once Status.Initialized to fully match OpenSearch guidance — much lower risk than before, so fine as a follow-up issue.
|
@josedev-union let's wait with this following v3 release, this is too big of a change to handle now |
…er bootstrap (#4231) ## What this PR does Builds the OpenSearch operator in-tree, the way kamaji and redis-operator already are, and carries one source patch. The operator deletes its bootstrap pod the moment `status.initialized` flips, without looking at the voting configuration. With `replicas: 1` there are exactly two cluster-manager-eligible nodes, OpenSearch keeps the voting configuration at an odd size of one, and the voter it keeps is the bootstrap node that formed the cluster. Deleting that pod takes away the only vote there was, and the survivor can never hold an election again, because changing the voting configuration itself needs a quorum. Full write-up in #4230. The patch excludes the bootstrap node from the voting configuration before the pod goes, and confirms the exclusion committed before deleting anything: 1. Pod already gone: nothing to do. 2. Build a cluster client. Cannot: requeue, keep the pod. 3. `POST /_cluster/voting_config_exclusions?node_names=<release>-bootstrap-0`. Fails: requeue, keep the pod. 4. Read the voting configuration back. Still a voter: requeue, keep the pod. 5. Delete the pod. 6. `DELETE /_cluster/voting_config_exclusions`. Fails: log and carry on, the pod is already gone and the list is only capped, not load-bearing. Step 4 is the one place this differs from the upstream attempt, which relies on the POST being blocking. I did not verify that property, and polling the committed configuration is correct whether or not it holds. Every failure to reach the cluster keeps the bootstrap pod and requeues. That is a deliberate trade: a bootstrap pod outliving its purpose costs one pod, a cluster that cannot elect a cluster-manager needs a human. ### How the values were arrived at Not from the documentation. Reproduced outside Kubernetes first, two containers with the env the operator sets, `cluster.initial_master_nodes` pointing at the first: ```text last_committed_config: ["D6Fijf-SRN-FwIXVGNehPw"] # the bootstrapper, alone POST .../voting_config_exclusions?node_names=osA -> 200 last_committed_config: ["oy8DPOV-SKKildaIZpgoPg"] # moved to the survivor ``` Without the exclusion, removing the first container leaves the second printing `cluster-manager not discovered or elected yet, an election requires a node with id [...]` forever, which is the same sentence the e2e run produced. ### Why carry it here rather than wait Upstream knows: [opensearch-k8s-operator#1448](opensearch-project/opensearch-k8s-operator#1448) is open with `release-blocker`, and its third point is this exact case. Three PRs address it and none looks close: [#1483](opensearch-project/opensearch-k8s-operator#1483) has changes-requested, a merge conflict and a red lint; [#1150](opensearch-project/opensearch-k8s-operator#1150) and [#1430](opensearch-project/opensearch-k8s-operator#1430) have not moved since July. When a release carries the fix, this patch and this image build come back out. The alternative considered and rejected was refusing `replicas: 1` in `values.schema.json`. It is a breaking change for existing releases and removes a configuration rather than fixing it. ### Before this can merge `values.yaml` deliberately carries no `manager.image` yet. The image does not exist, and committing a first-party reference without a digest is exactly what `docs/agents/image-refs.md` forbids. Once the first build here publishes it, I will add `repository` plus `tag@sha256:...` in a second commit and take the PR out of draft. ### Note on CI cost Adding the build line to the root `Makefile` puts it in `full_rebuild_pattern`, so every push to this branch rebuilds all 28 units rather than one. Verified with `hack/build-matrix.sh`. Unavoidable: the line is what puts the image in the matrix at all. ### What was checked locally - The image builds and the patch applies cleanly; `/manager --help` runs from the built image. - `go build ./...`, `go vet` and the operator's own `pkg/reconcilers` tests pass with the patch applied. - `gofmt` clean, `helm unittest` for the package green, `pre-commit` green on the changed files. - `hack/build-matrix.sh` selects the package for a package-only change. Not checked locally, and the reason this needs the e2e lane: that the fix actually keeps a real cluster electable. The `opensearch` chainsaw suite is the test for it. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): exclude the bootstrap node from the voting configuration before removing it, so a single-replica OpenSearch cluster stays able to elect a cluster-manager ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added support for building and publishing the OpenSearch operator image as part of the standard build workflow. - Operator image versions now stay aligned automatically with the packaged source version. - Added a packaged operator image configuration for consistent deployment. - Improved cluster initialization by safely excluding the bootstrap node from voting before removal and clearing the exclusion afterward. - Added retry handling when cluster state is unavailable or voting changes are not yet committed. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
5ef5983 to
8a2af51
Compare
963ef11 to
79e9d6a
Compare
synhershko
left a comment
There was a problem hiding this comment.
My three inline points from the last review (initial_master_nodes on every STS, keeping spec.bootstrap as a deprecated field, the broken ### Fixed heading in the CHANGELOG) are still open at this commit.
|
yeah, sorry, i forgot to set this as draft |
|
as we missed the change of major version upgrade, need to support bootstrap fields in CR and make them deprecated. |
fa69e7a to
0901a7d
Compare
48839fd to
43e88ea
Compare
480f8bf to
ec9412a
Compare
|
lgtm |
synhershko
left a comment
There was a problem hiding this comment.
My earlier points are addressed. Withdrawn: I said the setting is ignored once a cluster has formed and that blanking it could be a follow-up. Both are wrong for a pod with an empty data dir, see the inline comment on cluster.go.
| func ResolveInitialMasterNodes(cr *opensearchv1.OpenSearchCluster, existingValue string, hasExistingValue bool, bootstrapPodExists bool) (string, bool) { | ||
| if cr.Status.Initialized { | ||
| if hasExistingValue { | ||
| return existingValue, true |
There was a problem hiding this comment.
On clusters created by this operator the kept value names pool pod-0 and stays on the STS forever, so a pod-0 that returns with an empty data dir can bootstrap a second cluster. With the one-name list of a single manager pool, ClusterBootstrapService.onFoundPeersUpdated matches the local node and Coordinator.setInitialConfiguration generates a new cluster UUID and starts an election, racing the join to the real leader. This hits every pod-0 replacement on an emptyDir manager pool (rolling restart, upgrade, eviction) and a lost PVC. The old bootstrap-0 value never matched a pool pod, so this is new. From reading the source, not reproduced. OpenSearch docs say the list should be empty for nodes joining an existing cluster: https://docs.opensearch.org/latest/tuning-your-cluster/discovery-cluster-formation/settings. Fix: read the value from an optional configMapKeyRef and delete that ConfigMap once status.initialized is true (recreate it on emptyDir recovery), so later pods get no setting and the template does not change.
| ``` | ||
|
|
||
| Then run `kubectl apply -f cluster.yaml`. If you watch the cluster (e.g. `watch -n 2 kubectl get pods`), you will see that after a few seconds the Operator will create several pods. First, a bootstrap pod will be created (`my-first-cluster-bootstrap-0`) that helps with initial master discovery. Then three pods for the OpenSearch cluster will be created (`my-first-cluster-nodes-0/1/2`), and one pod for the dashboards instance. After the pods are appearing as ready, which normally takes about 1-2 minutes, you can connect to your cluster using port-forwarding. | ||
| Then run `kubectl apply -f cluster.yaml`. If you watch the cluster (e.g. `watch -n 2 kubectl get pods`), you will see that after a few seconds the Operator will create several pods. The cluster-manager nodes themselves bootstrap the cluster (via `cluster.initial_master_nodes` pointing at pod-0 of each manager pool that has at least one replica). Three pods for the OpenSearch cluster will be created (`my-first-cluster-masters-0/1/2`), and one pod for the dashboards instance. After the pods are appearing as ready, which normally takes about 1-2 minutes, you can connect to your cluster using port-forwarding. |
There was a problem hiding this comment.
Wrong pod names: the example above uses component: nodes, so the pods are my-first-cluster-nodes-0/1/2, not masters.
7a6094f to
bd26f85
Compare
Signed-off-by: psmit <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
- restore nodeAttributes rbac - clean up leftover bootstrap pods/PVCs with voting-config exclusion - honor probes.startup.command - sync CRDs in the chart and bundle and document the upgrade rolling restart Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
bd26f85 to
c9b7b72
Compare
Signed-off-by: josedev-union <[email protected]>
Description
This PR aims to remove bootstrap pod from cluster lifecycle and just use cluster manager nodes for bootstrapping the cluster.
Issues Resolved
List any issues this PR will resolve, e.g. Closes [...].
Check List
make lint)If CRDs are changed:
make manifests) and also copied into the helm chartPlease refer to the PR guidelines before submitting this pull request.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.