Repository navigation
fix: use voting config exclusions when removing master-eligible nodes - #1483
Conversation
e0c0689 to
a572a44
Compare
a572a44 to
ba18ecf
Compare
ba18ecf to
baa5e79
Compare
7c9209c to
7ad0950
Compare
synhershko
left a comment
There was a problem hiding this comment.
Thanks for taking this on - the approach is the right one and it addresses all three parts of #1448 (master scale-down, removing a whole master pool, and the bootstrap pod), plus the webhook guard. A few things need to change before this can go in though.
Blocking
Nil-pointer panic when smartScaler: false and the pool is master-eligible (the default config). In decreaseOneNode, clusterClient is only created inside if smartDecrease { ... }, but the new code then calls services.RemoveExcludeNodeHost(clusterClient, ...) and services.ClearVotingConfigExclusions(clusterClient, ...) whenever isMaster is true. With SmartScaler off, reconcileNodePool routes master pools through exclude → drain → decreaseOneNode(..., smartDecrease=false, isMaster=true), and that dereferences a nil client.
I reproduced it on this branch with a small Ginkgo test that calls decreaseOneNode(status, sts, "masters", false, true) with a mocked k8s client:
runtime error: invalid memory address or nil pointer dereference
Because the StatefulSet has already been shrunk and the scaler status removed by that point, nothing retries: the allocation exclusion and the voting-config exclusion for the removed node are never cleared. Voting exclusions then accumulate on every master scale-down until the default cluster.max_voting_config_exclusions (10) is hit, after which every further master removal fails.
Fix: create the client when smartDecrease || isMaster.
Should fix
-
The "retry without wait" fallback is unsafe. On every clear path (
decreaseOneNode, both branches ofremoveStatefulSet,removeBootstrapPod), ifDELETE _cluster/voting_config_exclusions?wait_for_removal=truefails, the code immediately retries withwait_for_removal=false. That clears the exclusion while the node may still be alive, which lets OpenSearch put it back into the voting configuration right before the pod is killed - i.e. the original bug, re-introduced on a timeout. Lingering exclusions are harmless; please just return an error / requeue and retry the waiting DELETE later. Never clear without wait. -
Client and server timeouts collide. The operator's HTTP transport has
ResponseHeaderTimeout: 30s, and OpenSearch's default server-sidetimeoutfor both the POST and the DELETE is also 30s. The client will usually give up first, which makes the fallback in (1) more likely to trigger. Pass an explicit shortertimeout=query parameter (e.g. 10s) on both requests and rely on requeue. -
removeBootstrapPoderrors now short-circuit the whole reconciler chain. If the OpenSearch client can't be built or the POST fails,ClusterReconciler.Reconcilereturns an error andopensearchControllerstops before running scaler/upgrade/restart/etc. Bootstrap deletion could not fail this way before. Either log and continue, or explain why blocking here is acceptable. -
No tests for the new behaviour. The added tests cover
CountMasterEligibleReplicas,IsMasterStatefulSet, the URL builder and the webhook, but neither the master scale-down path in the scaler norremoveBootstrapPodis exercised - which is how the panic above got through.scaler_test.goalready has httpmock helpers (registerClusterSettingsRespondersetc.); one test per path that asserts the voting-config POST/DELETE calls happen in the right order would be enough.
Minor
if smartDecrease || isMaster {directly afterif !smartDecrease && !isMaster { return ... }is always true.- In
decreaseOneNode, returningrequeue=truewhen the clear fails does nothing useful: the scaler status has already been removed, so the next reconcile seesdesireReplicaDiff == 0. - The webhook now rejects any cluster with no
master/cluster_managerpool, on create as well as update. That's reasonable, but the chart enables the webhook withfailurePolicy: Failby default, so it's a user-visible change and should be mentioned in the changelog and the PR description. - The PR description is still the empty template; please fill in the description and tick the checklist.
- The CVE Scan failure is unrelated - the branch is behind
mainand is missing the CVE fix that landed after this was opened. A rebase should clear it.
Once the nil-client panic and the no-wait fallback are addressed I'm happy to take another look.
7ad0950 to
ad3a7a1
Compare
f05ce72 to
09868b6
Compare
|
Thanks @josedev-union, f05ce72 addresses everything from my review: the nil client is fixed, the no-wait fallback is gone, both requests pass I pushed two things on top:
Unit tests and the controllers envtest suite pass locally. One item from the review is still open: the PR description is the empty template. Could you fill it in? |
09868b6 to
e29d3ca
Compare
…er bootstrap (#4231) ## What this PR does Builds the OpenSearch operator in-tree, the way kamaji and redis-operator already are, and carries one source patch. The operator deletes its bootstrap pod the moment `status.initialized` flips, without looking at the voting configuration. With `replicas: 1` there are exactly two cluster-manager-eligible nodes, OpenSearch keeps the voting configuration at an odd size of one, and the voter it keeps is the bootstrap node that formed the cluster. Deleting that pod takes away the only vote there was, and the survivor can never hold an election again, because changing the voting configuration itself needs a quorum. Full write-up in #4230. The patch excludes the bootstrap node from the voting configuration before the pod goes, and confirms the exclusion committed before deleting anything: 1. Pod already gone: nothing to do. 2. Build a cluster client. Cannot: requeue, keep the pod. 3. `POST /_cluster/voting_config_exclusions?node_names=<release>-bootstrap-0`. Fails: requeue, keep the pod. 4. Read the voting configuration back. Still a voter: requeue, keep the pod. 5. Delete the pod. 6. `DELETE /_cluster/voting_config_exclusions`. Fails: log and carry on, the pod is already gone and the list is only capped, not load-bearing. Step 4 is the one place this differs from the upstream attempt, which relies on the POST being blocking. I did not verify that property, and polling the committed configuration is correct whether or not it holds. Every failure to reach the cluster keeps the bootstrap pod and requeues. That is a deliberate trade: a bootstrap pod outliving its purpose costs one pod, a cluster that cannot elect a cluster-manager needs a human. ### How the values were arrived at Not from the documentation. Reproduced outside Kubernetes first, two containers with the env the operator sets, `cluster.initial_master_nodes` pointing at the first: ```text last_committed_config: ["D6Fijf-SRN-FwIXVGNehPw"] # the bootstrapper, alone POST .../voting_config_exclusions?node_names=osA -> 200 last_committed_config: ["oy8DPOV-SKKildaIZpgoPg"] # moved to the survivor ``` Without the exclusion, removing the first container leaves the second printing `cluster-manager not discovered or elected yet, an election requires a node with id [...]` forever, which is the same sentence the e2e run produced. ### Why carry it here rather than wait Upstream knows: [opensearch-k8s-operator#1448](opensearch-project/opensearch-k8s-operator#1448) is open with `release-blocker`, and its third point is this exact case. Three PRs address it and none looks close: [#1483](opensearch-project/opensearch-k8s-operator#1483) has changes-requested, a merge conflict and a red lint; [#1150](opensearch-project/opensearch-k8s-operator#1150) and [#1430](opensearch-project/opensearch-k8s-operator#1430) have not moved since July. When a release carries the fix, this patch and this image build come back out. The alternative considered and rejected was refusing `replicas: 1` in `values.schema.json`. It is a breaking change for existing releases and removes a configuration rather than fixing it. ### Before this can merge `values.yaml` deliberately carries no `manager.image` yet. The image does not exist, and committing a first-party reference without a digest is exactly what `docs/agents/image-refs.md` forbids. Once the first build here publishes it, I will add `repository` plus `tag@sha256:...` in a second commit and take the PR out of draft. ### Note on CI cost Adding the build line to the root `Makefile` puts it in `full_rebuild_pattern`, so every push to this branch rebuilds all 28 units rather than one. Verified with `hack/build-matrix.sh`. Unavoidable: the line is what puts the image in the matrix at all. ### What was checked locally - The image builds and the patch applies cleanly; `/manager --help` runs from the built image. - `go build ./...`, `go vet` and the operator's own `pkg/reconcilers` tests pass with the patch applied. - `gofmt` clean, `helm unittest` for the package green, `pre-commit` green on the changed files. - `hack/build-matrix.sh` selects the package for a package-only change. Not checked locally, and the reason this needs the e2e lane: that the fix actually keeps a real cluster electable. The `opensearch` chainsaw suite is the test for it. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): exclude the bootstrap node from the voting configuration before removing it, so a single-replica OpenSearch cluster stays able to elect a cluster-manager ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added support for building and publishing the OpenSearch operator image as part of the standard build workflow. - Operator image versions now stay aligned automatically with the packaged source version. - Added a packaged operator image configuration for consistent deployment. - Improved cluster initialization by safely excluding the bootstrap node from voting before removal and clearing the exclusion afterward. - Added retry handling when cluster state is unavailable or voting changes are not yet committed. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
Took another pass over this at One blocker, though, and it comes straight out of the timeout item from my last review. Blocker: every clear call returns 400
ClearVotingConfigExclusionsRequest req = new ClearVotingConfigExclusionsRequest();
if (request.hasParam("wait_for_removal")) {
req.setWaitForRemoval(request.paramAsBoolean("wait_for_removal", true));
}
Reproduced against a real The same DELETE without So the PR can add voting exclusions and never remove any of them. What that does at each call site:
They then accumulate against The fix is to drop Which puts the timeout collision backMy earlier point about the client/server deadlines can't be solved with a query parameter on this endpoint. Once Also worth fixing before merge
TestsThe set of paths covered is the right one — master scale-down with SmartScaler off, failed clear retaining status, reverted scale-down, pending-cleanup resume, whole-STS removal, bootstrap removal — and asserting call ordering and that no Status of my earlier reviewFixed: the nil-client panic with Since I authored the merge commit at the head and two of the commits here, this should get a second approver before it goes in. |
7e5867f to
0e64afc
Compare
0e64afc to
c985a16
Compare
38f4ebe to
6fd50b1
Compare
6fd50b1 to
c2d83a2
Compare
Removing a master-eligible node never touched the voting configuration, so scaling a master pool down, removing a master pool from spec.nodePools or deleting the bootstrap pod could permanently remove voters and lose quorum. Master-eligible nodes are now removed one at a time following the documented sequence: POST /_cluster/voting_config_exclusions for the node (blocks until it has left the voting configuration), shrink the StatefulSet or delete the pod, and once the node has left the cluster clear the exclusions with wait_for_removal=true. The scaler keeps its Drained status until the clear succeeds. The waiting clear is never issued while the node is still a member, since it would sit on OpenSearch's 30s wait_for_removal deadline, which is also the operator's client deadline. This applies regardless of SmartScaler; with SmartScaler off only the voting step runs and the shard drain stays opt-in. A reverted master scale-down clears the list without waiting, the one case where that is correct because the node stays. The scaler also sweeps the exclusions list on every reconcile of an initialized cluster: exclusions whose nodes have all left are cleared with wait, and a live pool member nothing is removing that is still excluded is cleared without wait with a Warning event. The POST is skipped for a node that is not a cluster member, since OpenSearch would otherwise record an _absent_ tombstone that only counts toward cluster.max_voting_config_exclusions. The DELETE endpoint only consumes wait_for_removal; it rejects a timeout parameter with a 400, so the timeout is only sent on the POST. IsMasterStatefulSet falls back to the container's node.roles env when the opensearch.role label is missing or overridden. Closes opensearch-project#1448 Signed-off-by: josedev-union <[email protected]> Signed-off-by: Itamar Syn-Hershko <[email protected]>
c2d83a2 to
d33303d
Compare
Signed-off-by: josedev-union <[email protected]>
Signed-off-by: josedev-union <[email protected]>
Description
Removing a master-eligible node never touched the voting configuration: the scaler treated master pools like data pools (allocation exclusion, wait for shards, delete the pod), so scaling a 3-master pool to 1 removed two voters back to back and could leave the cluster without quorum. The same gap existed when a whole master pool was removed from
spec.nodePoolsand when the bootstrap pod was deleted after initialization.Master-eligible nodes now follow the documented sequence, one node at a time:
POST /_cluster/voting_config_exclusions?node_names=<pod>&timeout=10s, which blocks until the node has left the voting configuration. The POST is skipped for a node that is not a cluster member, since OpenSearch would otherwise record an_absent_tombstone that only counts towardcluster.max_voting_config_exclusions._cat/nodes),DELETE /_cluster/voting_config_exclusions?wait_for_removal=true. The scaler keeps theDrainedstatus until this succeeds. The waiting clear is never issued while the node is still a member: it would sit on OpenSearch's 30swait_for_removaldeadline, which is also the operator's client deadline, and fail.This applies whether or not SmartScaler is enabled. With SmartScaler off only the voting step is performed; the shard drain stays opt-in. The same steps run in
removeStatefulSetfor a removed master pool and in the bootstrap teardown.Other behaviour:
wait_for_removal=false, the one case where a non-waiting clear is correct, since the node stays.ScalerReconcilersweeps the exclusions list on every reconcile of an initialized cluster. Exclusions whose nodes have all left (a clear that failed, an operator restart mid-removal, the bootstrap pod) are cleared with wait. A live pool member that nothing is removing but is still excluded is cleared without wait with aWarningevent, because such a node is silently out of the voting configuration and a later restart of another master could lose quorum. While any excluded node is still leaving, the sweep does nothing.IsMasterStatefulSetfalls back to the container'snode.rolesenv when theopensearch.rolelabel is missing or overridden by a user label.removeBootstrapPodfailures returnRequeueAfterwithout an error so the reconciler chain keeps running.The
DELETEendpoint only consumeswait_for_removal; passingtimeouton it is rejected with a 400, so the timeout is only sent on thePOST.Docs:
docs/userguide/main.md(SmartScaler section) describes the master-removal sequence and the cluster-wide clear;docs/userguide/webhooks.mdnotes the existing guard against clusters without a master-eligible pool.Issues Resolved
Closes #1448
Check List
make lint)If CRDs are changed:
make manifests) and also copied into the helm chartPlease refer to the PR guidelines before submitting this pull request.
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.