Skip to content

Commit 79602e9

Browse files
authored
fix(opensearch-operator): keep a single-replica cluster electable after bootstrap (#4231)
## What this PR does Builds the OpenSearch operator in-tree, the way kamaji and redis-operator already are, and carries one source patch. The operator deletes its bootstrap pod the moment `status.initialized` flips, without looking at the voting configuration. With `replicas: 1` there are exactly two cluster-manager-eligible nodes, OpenSearch keeps the voting configuration at an odd size of one, and the voter it keeps is the bootstrap node that formed the cluster. Deleting that pod takes away the only vote there was, and the survivor can never hold an election again, because changing the voting configuration itself needs a quorum. Full write-up in #4230. The patch excludes the bootstrap node from the voting configuration before the pod goes, and confirms the exclusion committed before deleting anything: 1. Pod already gone: nothing to do. 2. Build a cluster client. Cannot: requeue, keep the pod. 3. `POST /_cluster/voting_config_exclusions?node_names=<release>-bootstrap-0`. Fails: requeue, keep the pod. 4. Read the voting configuration back. Still a voter: requeue, keep the pod. 5. Delete the pod. 6. `DELETE /_cluster/voting_config_exclusions`. Fails: log and carry on, the pod is already gone and the list is only capped, not load-bearing. Step 4 is the one place this differs from the upstream attempt, which relies on the POST being blocking. I did not verify that property, and polling the committed configuration is correct whether or not it holds. Every failure to reach the cluster keeps the bootstrap pod and requeues. That is a deliberate trade: a bootstrap pod outliving its purpose costs one pod, a cluster that cannot elect a cluster-manager needs a human. ### How the values were arrived at Not from the documentation. Reproduced outside Kubernetes first, two containers with the env the operator sets, `cluster.initial_master_nodes` pointing at the first: ```text last_committed_config: ["D6Fijf-SRN-FwIXVGNehPw"] # the bootstrapper, alone POST .../voting_config_exclusions?node_names=osA -> 200 last_committed_config: ["oy8DPOV-SKKildaIZpgoPg"] # moved to the survivor ``` Without the exclusion, removing the first container leaves the second printing `cluster-manager not discovered or elected yet, an election requires a node with id [...]` forever, which is the same sentence the e2e run produced. ### Why carry it here rather than wait Upstream knows: [opensearch-k8s-operator#1448](opensearch-project/opensearch-k8s-operator#1448) is open with `release-blocker`, and its third point is this exact case. Three PRs address it and none looks close: [#1483](opensearch-project/opensearch-k8s-operator#1483) has changes-requested, a merge conflict and a red lint; [#1150](opensearch-project/opensearch-k8s-operator#1150) and [#1430](opensearch-project/opensearch-k8s-operator#1430) have not moved since July. When a release carries the fix, this patch and this image build come back out. The alternative considered and rejected was refusing `replicas: 1` in `values.schema.json`. It is a breaking change for existing releases and removes a configuration rather than fixing it. ### Before this can merge `values.yaml` deliberately carries no `manager.image` yet. The image does not exist, and committing a first-party reference without a digest is exactly what `docs/agents/image-refs.md` forbids. Once the first build here publishes it, I will add `repository` plus `tag@sha256:...` in a second commit and take the PR out of draft. ### Note on CI cost Adding the build line to the root `Makefile` puts it in `full_rebuild_pattern`, so every push to this branch rebuilds all 28 units rather than one. Verified with `hack/build-matrix.sh`. Unavoidable: the line is what puts the image in the matrix at all. ### What was checked locally - The image builds and the patch applies cleanly; `/manager --help` runs from the built image. - `go build ./...`, `go vet` and the operator's own `pkg/reconcilers` tests pass with the patch applied. - `gofmt` clean, `helm unittest` for the package green, `pre-commit` green on the changed files. - `hack/build-matrix.sh` selects the package for a package-only change. Not checked locally, and the reason this needs the e2e lane: that the fix actually keeps a real cluster electable. The `opensearch` chainsaw suite is the test for it. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): exclude the bootstrap node from the voting configuration before removing it, so a single-replica OpenSearch cluster stays able to elect a cluster-manager ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added support for building and publishing the OpenSearch operator image as part of the standard build workflow. - Operator image versions now stay aligned automatically with the packaged source version. - Added a packaged operator image configuration for consistent deployment. - Improved cluster initialization by safely excluding the bootstrap node from voting before removal and clearing the exclusion afterward. - Added retry handling when cluster state is unavailable or voting changes are not yet committed. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents f9ea0da + b222cb6 commit 79602e9

5 files changed

Lines changed: 342 additions & 1 deletion

File tree

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ build: build-deps
3535
make -C packages/system/securitygroup-controller image
3636
make -C packages/system/grafana-operator image
3737
make -C packages/system/redis-operator image
38+
make -C packages/system/opensearch-operator image
3839
make -C packages/core/testing image
3940
make -C packages/core/talos image
4041
make -C packages/core/platform image
Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,32 @@
1+
# Single source for the version: the image's ARG VERSION. The chart and the
2+
# binary have to move together, because the chart ships the CRDs the binary is
3+
# compiled against.
4+
OPENSEARCH_OPERATOR_TAG=$(shell grep -F 'ARG VERSION=' images/opensearch-operator/Dockerfile | cut -f2 -d=)
5+
16
export NAME=opensearch-operator
27
export NAMESPACE=cozy-$(NAME)
38

9+
include ../../../hack/common-envs.mk
410
include ../../../hack/package.mk
511

612
update:
713
rm -rf charts
814
helm repo add opensearch-operator https://opensearch-project.github.io/opensearch-k8s-operator/
915
helm repo update opensearch-operator
10-
helm pull opensearch-operator/opensearch-operator --version 2.8.0 --untar --untardir charts
16+
helm pull opensearch-operator/opensearch-operator --version $(patsubst v%,%,$(OPENSEARCH_OPERATOR_TAG)) --untar --untardir charts
1117
patch --no-backup-if-mismatch -p1 < patches/leaderElection.diff
1218

19+
image:
20+
docker buildx build images/opensearch-operator \
21+
$(call image-tags,opensearch-operator,$(OPENSEARCH_OPERATOR_TAG)) \
22+
$(call cache-args,opensearch-operator) \
23+
--metadata-file images/opensearch-operator.json \
24+
$(BUILDX_ARGS)
25+
REPOSITORY="$(REGISTRY)/opensearch-operator" \
26+
yq -i '.opensearch-operator.manager.image.repository = strenv(REPOSITORY)' values.yaml
27+
TAG=$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/opensearch-operator.json -o json -r) \
28+
yq -i '.opensearch-operator.manager.image.tag = strenv(TAG)' values.yaml
29+
rm -f images/opensearch-operator.json
30+
1331
test:
1432
helm unittest .
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Build the manager binary from the upstream sources with the Cozystack patch
2+
# applied. The patch keeps a single-replica cluster electable when the operator
3+
# retires its bootstrap pod; see patches/bootstrap-voting-exclusion.diff and
4+
# https://github.com/opensearch-project/opensearch-k8s-operator/issues/1448.
5+
FROM golang:1.26@sha256:079e59808d2d252516e27e3f3a9c003740dee7f75e55aa71528766d52bcfc16a AS builder
6+
7+
# Keep in step with the chart version pulled by `make update`: the chart's
8+
# appVersion is what picks the operator image upstream, so a mismatch here would
9+
# run one version's binary against another's CRDs.
10+
ARG VERSION=v2.8.0
11+
12+
ARG TARGETOS
13+
ARG TARGETARCH
14+
15+
WORKDIR /workspace
16+
17+
RUN curl -sSL https://github.com/opensearch-project/opensearch-k8s-operator/archive/refs/tags/${VERSION}.tar.gz | tar -xzvf- --strip=1
18+
19+
COPY patches /patches
20+
RUN git apply /patches/*.diff
21+
22+
WORKDIR /workspace/opensearch-operator
23+
24+
RUN GOOS=$TARGETOS GOARCH=$TARGETARCH CGO_ENABLED=0 go build -a -o manager main.go
25+
26+
FROM gcr.io/distroless/static:nonroot@sha256:d29e660cc75a5b6b1334e03c5c81ccf9bc0884a002c6000dbf0fb96034814478
27+
WORKDIR /
28+
COPY --from=builder /workspace/opensearch-operator/manager .
29+
USER 65532:65532
30+
31+
ENTRYPOINT ["/manager"]
Lines changed: 282 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,282 @@
1+
diff --git a/opensearch-operator/opensearch-gateway/services/os_client.go b/opensearch-operator/opensearch-gateway/services/os_client.go
2+
index 2ec30c5..a33fa11 100644
3+
--- a/opensearch-operator/opensearch-gateway/services/os_client.go
4+
+++ b/opensearch-operator/opensearch-gateway/services/os_client.go
5+
@@ -4,6 +4,7 @@ import (
6+
"context"
7+
"crypto/tls"
8+
"encoding/json"
9+
+ "fmt"
10+
"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/helpers"
11+
"io"
12+
"k8s.io/utils/ptr"
13+
@@ -535,3 +536,115 @@ func generateAPIPathSnapshotUpdatePolicies(resource, name string, seqno, primary
14+
path.WriteString(strconv.Itoa(primaryterm))
15+
return path
16+
}
17+
+
18+
+// votingConfigState is the slice of the cluster state that says who may vote.
19+
+// The coordination metadata names voters by node id, so the node list travels
20+
+// in the same response to resolve an id back to a name.
21+
+type votingConfigState struct {
22+
+ Nodes map[string]struct {
23+
+ Name string `json:"name"`
24+
+ } `json:"nodes"`
25+
+ Metadata struct {
26+
+ ClusterCoordination struct {
27+
+ LastCommittedConfig []string `json:"last_committed_config"`
28+
+ VotingConfigExclusions []struct {
29+
+ NodeName string `json:"node_name"`
30+
+ } `json:"voting_config_exclusions"`
31+
+ } `json:"cluster_coordination"`
32+
+ } `json:"metadata"`
33+
+}
34+
+
35+
+// ExcludeNodeFromVotingConfig commits a voting configuration that leaves
36+
+// nodeName out, which is what makes the node safe to remove: a node still in
37+
+// the configuration is a vote the cluster keeps counting on.
38+
+func (client *OsClusterClient) ExcludeNodeFromVotingConfig(ctx context.Context, nodeName string) error {
39+
+ path := generateAPIPathVotingConfigExclusions(nodeName)
40+
+ res, err := doHTTPPost(ctx, client.client, path, nil)
41+
+ if err != nil {
42+
+ return err
43+
+ }
44+
+ defer helpers.SafeClose(res.Body)
45+
+ if res.IsError() {
46+
+ return fmt.Errorf("excluding %s from the voting configuration returned %s", nodeName, res.Status())
47+
+ }
48+
+ return nil
49+
+}
50+
+
51+
+// ClearVotingConfigExclusions empties the exclusion list, which is capped and
52+
+// would otherwise keep entries for nodes that are long gone. wait_for_removal
53+
+// stays at its default: clearing without it can re-admit a node that is still
54+
+// running, and the caller has already removed the one it excluded.
55+
+func (client *OsClusterClient) ClearVotingConfigExclusions(ctx context.Context) error {
56+
+ var path strings.Builder
57+
+ path.WriteString("/_cluster/voting_config_exclusions")
58+
+ res, err := doHTTPDelete(ctx, client.client, path)
59+
+ if err != nil {
60+
+ return err
61+
+ }
62+
+ defer helpers.SafeClose(res.Body)
63+
+ if res.IsError() {
64+
+ return fmt.Errorf("clearing the voting configuration exclusions returned %s", res.Status())
65+
+ }
66+
+ return nil
67+
+}
68+
+
69+
+func (client *OsClusterClient) readVotingConfigState(ctx context.Context) (votingConfigState, error) {
70+
+ var state votingConfigState
71+
+ var path strings.Builder
72+
+ path.WriteString("/_cluster/state/nodes,metadata")
73+
+ path.WriteString("?filter_path=nodes.*.name,metadata.cluster_coordination.last_committed_config")
74+
+ path.WriteString(",metadata.cluster_coordination.voting_config_exclusions")
75+
+ res, err := doHTTPGet(ctx, client.client, path)
76+
+ if err != nil {
77+
+ return state, err
78+
+ }
79+
+ defer helpers.SafeClose(res.Body)
80+
+ if res.IsError() {
81+
+ return state, fmt.Errorf("reading the voting configuration returned %s", res.Status())
82+
+ }
83+
+ err = json.NewDecoder(res.Body).Decode(&state)
84+
+ return state, err
85+
+}
86+
+
87+
+// IsInVotingConfig reports whether nodeName is still one of the nodes whose
88+
+// vote counts. An exclusion is committed through the cluster state rather than
89+
+// applied by the time the request returns, so this is what tells a caller the
90+
+// exclusion has actually taken.
91+
+func (client *OsClusterClient) IsInVotingConfig(ctx context.Context, nodeName string) (bool, error) {
92+
+ state, err := client.readVotingConfigState(ctx)
93+
+ if err != nil {
94+
+ return false, err
95+
+ }
96+
+
97+
+ for _, id := range state.Metadata.ClusterCoordination.LastCommittedConfig {
98+
+ if node, ok := state.Nodes[id]; ok && node.Name == nodeName {
99+
+ return true, nil
100+
+ }
101+
+ }
102+
+ return false, nil
103+
+}
104+
+
105+
+// IsVotingConfigExcluded reports whether nodeName is still named in the
106+
+// exclusion list. The list is capped, so an entry left behind for a node that
107+
+// no longer exists costs capacity a later membership change may need.
108+
+func (client *OsClusterClient) IsVotingConfigExcluded(ctx context.Context, nodeName string) (bool, error) {
109+
+ state, err := client.readVotingConfigState(ctx)
110+
+ if err != nil {
111+
+ return false, err
112+
+ }
113+
+
114+
+ for _, ex := range state.Metadata.ClusterCoordination.VotingConfigExclusions {
115+
+ if ex.NodeName == nodeName {
116+
+ return true, nil
117+
+ }
118+
+ }
119+
+ return false, nil
120+
+}
121+
+
122+
+func generateAPIPathVotingConfigExclusions(nodeName string) strings.Builder {
123+
+ var path strings.Builder
124+
+ path.Grow(len("/_cluster/voting_config_exclusions?node_names=") + len(nodeName))
125+
+ path.WriteString("/_cluster/voting_config_exclusions?node_names=")
126+
+ path.WriteString(nodeName)
127+
+ return path
128+
+}
129+
diff --git a/opensearch-operator/pkg/reconcilers/cluster.go b/opensearch-operator/pkg/reconcilers/cluster.go
130+
index 710f585..c199fa0 100644
131+
--- a/opensearch-operator/pkg/reconcilers/cluster.go
132+
+++ b/opensearch-operator/pkg/reconcilers/cluster.go
133+
@@ -4,7 +4,9 @@ import (
134+
"context"
135+
"fmt"
136+
"strings"
137+
+ "time"
138+
139+
+ "github.com/Opster/opensearch-k8s-operator/opensearch-operator/opensearch-gateway/services"
140+
"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s"
141+
"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/util"
142+
policyv1 "k8s.io/api/policy/v1"
143+
@@ -17,6 +19,8 @@ import (
144+
"github.com/go-logr/logr"
145+
"github.com/samber/lo"
146+
appsv1 "k8s.io/api/apps/v1"
147+
+ corev1 "k8s.io/api/core/v1"
148+
+ k8serrors "k8s.io/apimachinery/pkg/api/errors"
149+
"k8s.io/apimachinery/pkg/api/resource"
150+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
151+
"k8s.io/client-go/tools/record"
152+
@@ -25,6 +29,12 @@ import (
153+
"sigs.k8s.io/controller-runtime/pkg/log"
154+
)
155+
156+
+// How long to wait before trying again when the cluster cannot be reached or
157+
+// has not yet committed the exclusion. Short enough that a healthy cluster
158+
+// sheds its bootstrap pod promptly, long enough not to spin on an API that is
159+
+// still coming up.
160+
+const bootstrapRetireRetry = 10 * time.Second
161+
+
162+
type ClusterReconciler struct {
163+
client k8s.K8sClient
164+
ctx context.Context
165+
@@ -98,7 +108,7 @@ func (r *ClusterReconciler) Reconcile() (ctrl.Result, error) {
166+
bootstrapPod := builders.NewBootstrapPod(r.instance, r.reconcilerContext.Volumes, r.reconcilerContext.VolumeMounts)
167+
result.CombineErr(ctrl.SetControllerReference(r.instance, bootstrapPod, r.client.Scheme()))
168+
if r.instance.Status.Initialized {
169+
- result.Combine(r.client.ReconcileResource(bootstrapPod, reconciler.StateAbsent))
170+
+ result.Combine(r.retireBootstrapPod(bootstrapPod))
171+
} else {
172+
result.Combine(r.client.ReconcileResource(bootstrapPod, reconciler.StatePresent))
173+
}
174+
@@ -130,6 +140,108 @@ func (r *ClusterReconciler) Reconcile() (ctrl.Result, error) {
175+
return result.Result, result.Err
176+
}
177+
178+
+// retireBootstrapPod takes the bootstrap node out of the voting configuration
179+
+// before deleting its pod.
180+
+//
181+
+// The bootstrap node is the one that formed the cluster, so it is the voter the
182+
+// cluster keeps while the voting configuration holds a single node, which is
183+
+// what it holds whenever exactly two nodes are cluster_manager-eligible: the
184+
+// bootstrap pod and one node pool replica. Deleting the pod then takes away the
185+
+// only vote there was, and the survivor can never hold an election again,
186+
+// because changing the voting configuration itself needs a quorum. Excluding
187+
+// the node first moves the configuration onto the surviving node, which is what
188+
+// makes the deletion safe.
189+
+//
190+
+// Every failure to reach the cluster leaves the pod alone and requeues. A
191+
+// bootstrap pod that outlives its purpose costs a pod; a cluster that cannot
192+
+// elect a cluster-manager does not recover without a human.
193+
+func (r *ClusterReconciler) retireBootstrapPod(bootstrapPod *corev1.Pod) (*ctrl.Result, error) {
194+
+ podGone := false
195+
+ if _, err := r.client.GetPod(bootstrapPod.Name, bootstrapPod.Namespace); err != nil {
196+
+ if !k8serrors.IsNotFound(err) {
197+
+ return &ctrl.Result{}, err
198+
+ }
199+
+ podGone = true
200+
+ }
201+
+
202+
+ annotations := map[string]string{"cluster-name": r.instance.GetName()}
203+
+ nodeName := builders.BootstrapPodName(r.instance)
204+
+
205+
+ clusterClient, err := util.CreateClientForCluster(r.client, r.ctx, r.instance, nil)
206+
+ if err != nil {
207+
+ r.logger.Error(err, "Failed to create os client to retire the bootstrap node", "node", nodeName)
208+
+ r.recorder.AnnotatedEventf(r.instance, annotations, "Warning", "Bootstrap",
209+
+ "Cannot reach the cluster to exclude %s from voting; keeping the bootstrap pod", nodeName)
210+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
211+
+ }
212+
+
213+
+ // Deleting the pod is a request, not a wait, so the clearing step below can
214+
+ // find the node still in the cluster and fail. Retrying it here is what
215+
+ // keeps a stale entry from sitting in the capped exclusion list forever,
216+
+ // where it would cost a later membership change the capacity it needs.
217+
+ if podGone {
218+
+ return r.clearBootstrapExclusion(clusterClient, nodeName)
219+
+ }
220+
+
221+
+ if err := clusterClient.ExcludeNodeFromVotingConfig(r.ctx, nodeName); err != nil {
222+
+ r.logger.Error(err, "Failed to exclude the bootstrap node from the voting configuration", "node", nodeName)
223+
+ r.recorder.AnnotatedEventf(r.instance, annotations, "Warning", "Bootstrap",
224+
+ "Could not exclude %s from voting; keeping the bootstrap pod", nodeName)
225+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
226+
+ }
227+
+
228+
+ // The exclusion travels through the cluster state, so it is committed a
229+
+ // moment after the request returns rather than by the time it does.
230+
+ stillVoting, err := clusterClient.IsInVotingConfig(r.ctx, nodeName)
231+
+ if err != nil {
232+
+ r.logger.Error(err, "Failed to read the voting configuration", "node", nodeName)
233+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
234+
+ }
235+
+ if stillVoting {
236+
+ r.logger.Info("Waiting for the bootstrap node to leave the voting configuration", "node", nodeName)
237+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
238+
+ }
239+
+
240+
+ result, err := r.client.ReconcileResource(bootstrapPod, reconciler.StateAbsent)
241+
+ if err != nil {
242+
+ return result, err
243+
+ }
244+
+
245+
+ r.recorder.AnnotatedEventf(r.instance, annotations, "Normal", "Bootstrap",
246+
+ "Excluded %s from voting and removed the bootstrap pod", nodeName)
247+
+
248+
+ if cleared, err := r.clearBootstrapExclusion(clusterClient, nodeName); err != nil || cleared.RequeueAfter > 0 {
249+
+ return cleared, err
250+
+ }
251+
+ return result, nil
252+
+}
253+
+
254+
+// clearBootstrapExclusion drops the exclusion this reconciler added, once the
255+
+// node it named is really gone.
256+
+//
257+
+// Clearing empties the whole list rather than one entry, so it is left at the
258+
+// API's default wait_for_removal=true: that is what stops it from re-admitting
259+
+// a node some other operation excluded and is still draining. A failure is
260+
+// therefore expected while the bootstrap pod is terminating, and is answered
261+
+// with a requeue rather than a log line, because nothing else will come back to
262+
+// it.
263+
+func (r *ClusterReconciler) clearBootstrapExclusion(clusterClient *services.OsClusterClient, nodeName string) (*ctrl.Result, error) {
264+
+ excluded, err := clusterClient.IsVotingConfigExcluded(r.ctx, nodeName)
265+
+ if err != nil {
266+
+ r.logger.Error(err, "Failed to read the voting configuration exclusions", "node", nodeName)
267+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
268+
+ }
269+
+ if !excluded {
270+
+ return &ctrl.Result{}, nil
271+
+ }
272+
+
273+
+ if err := clusterClient.ClearVotingConfigExclusions(r.ctx); err != nil {
274+
+ r.logger.Error(err, "Failed to clear the voting configuration exclusions", "node", nodeName)
275+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
276+
+ }
277+
+ return &ctrl.Result{}, nil
278+
+}
279+
+
280+
func (r *ClusterReconciler) reconcileNodeStatefulSet(nodePool opsterv1.NodePool, username string) (*ctrl.Result, error) {
281+
found, nodePoolConfig := r.reconcilerContext.fetchNodePoolHash(nodePool.Component)
282+

‎packages/system/opensearch-operator/values.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,15 @@ opensearch-operator:
88
replicaCount: 1
99
# Enable cluster-wide mode to watch all namespaces
1010
watchNamespace: ""
11+
# Built in-tree from images/opensearch-operator, which carries the
12+
# bootstrap voting-exclusion patch. Without this block the vendored chart
13+
# falls back to opensearchproject/opensearch-operator at the chart's
14+
# appVersion, which is the unpatched upstream build, and the reference stays
15+
# outside the first-party set hack/lib/image-refs.sh collects for promote,
16+
# retag and mirror. `make image` rewrites both keys on every build.
17+
image:
18+
repository: ghcr.io/cozystack/cozystack/opensearch-operator
19+
tag: v2.8.0@sha256:cbe6b50f13aa5abf338209d38dfcd80e73296e97510465b5fca73a0130bfaad7
1120
kubeRbacProxy:
1221
image:
1322
# gcr.io/kubebuilder/kube-rbac-proxy was sunset; use the original upstream

0 commit comments

Comments
 (0)