Skip to content

Commit b222cb6

Browse files
committed
fix(opensearch-operator): keep a single-replica cluster electable after bootstrap
The operator deletes the bootstrap pod as soon as initialization completes and never looks at the voting configuration first. With one cluster_manager replica there are exactly two eligible nodes, OpenSearch keeps the configuration at an odd size of one, and the voter it keeps is the bootstrap node that formed the cluster. Deleting that pod takes away the only vote there was, and the survivor can never hold an election again, because changing the configuration itself needs a quorum. The cluster is unusable about a minute after it installs, and a node with no cluster-manager still answers the readiness probe, so nothing says so until something restarts the pod. Build the operator in-tree, the way kamaji and redis-operator already are, and carry a patch that excludes the bootstrap node from the voting configuration and waits for that to commit before deleting the pod. Reaching the cluster is a precondition rather than a nicety: every failure keeps the pod and requeues, because a bootstrap pod that outlives its purpose costs a pod, while a cluster that cannot elect does not recover without a human. Deleting a pod is a request rather than a wait, so clearing the exclusion afterwards can find the node still in the cluster and fail. That path requeues and is retried once the pod is really gone, which keeps a stale entry out of the capped exclusion list; clearing stays at the API default wait_for_removal=true so it cannot re-admit a node some other operation is still draining. Upstream carries the same defect, tracked as opensearch-project/opensearch-k8s-operator#1448. Assisted-by: LLM Signed-off-by: IvanHunters <[email protected]>
1 parent f9ea0da commit b222cb6

5 files changed

Lines changed: 342 additions & 1 deletion

File tree

‎Makefile‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ build: build-deps
3535
make -C packages/system/securitygroup-controller image
3636
make -C packages/system/grafana-operator image
3737
make -C packages/system/redis-operator image
38+
make -C packages/system/opensearch-operator image
3839
make -C packages/core/testing image
3940
make -C packages/core/talos image
4041
make -C packages/core/platform image
Lines changed: 19 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,32 @@
1+
# Single source for the version: the image's ARG VERSION. The chart and the
2+
# binary have to move together, because the chart ships the CRDs the binary is
3+
# compiled against.
4+
OPENSEARCH_OPERATOR_TAG=$(shell grep -F 'ARG VERSION=' images/opensearch-operator/Dockerfile | cut -f2 -d=)
5+
16
export NAME=opensearch-operator
27
export NAMESPACE=cozy-$(NAME)
38

9+
include ../../../hack/common-envs.mk
410
include ../../../hack/package.mk
511

612
update:
713
rm -rf charts
814
helm repo add opensearch-operator https://opensearch-project.github.io/opensearch-k8s-operator/
915
helm repo update opensearch-operator
10-
helm pull opensearch-operator/opensearch-operator --version 2.8.0 --untar --untardir charts
16+
helm pull opensearch-operator/opensearch-operator --version $(patsubst v%,%,$(OPENSEARCH_OPERATOR_TAG)) --untar --untardir charts
1117
patch --no-backup-if-mismatch -p1 < patches/leaderElection.diff
1218

19+
image:
20+
docker buildx build images/opensearch-operator \
21+
$(call image-tags,opensearch-operator,$(OPENSEARCH_OPERATOR_TAG)) \
22+
$(call cache-args,opensearch-operator) \
23+
--metadata-file images/opensearch-operator.json \
24+
$(BUILDX_ARGS)
25+
REPOSITORY="$(REGISTRY)/opensearch-operator" \
26+
yq -i '.opensearch-operator.manager.image.repository = strenv(REPOSITORY)' values.yaml
27+
TAG=$(IMAGE_TAG)@$$(yq e '."containerimage.digest"' images/opensearch-operator.json -o json -r) \
28+
yq -i '.opensearch-operator.manager.image.tag = strenv(TAG)' values.yaml
29+
rm -f images/opensearch-operator.json
30+
1331
test:
1432
helm unittest .
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Build the manager binary from the upstream sources with the Cozystack patch
2+
# applied. The patch keeps a single-replica cluster electable when the operator
3+
# retires its bootstrap pod; see patches/bootstrap-voting-exclusion.diff and
4+
# https://github.com/opensearch-project/opensearch-k8s-operator/issues/1448.
5+
FROM golang:1.26@sha256:079e59808d2d252516e27e3f3a9c003740dee7f75e55aa71528766d52bcfc16a AS builder
6+
7+
# Keep in step with the chart version pulled by `make update`: the chart's
8+
# appVersion is what picks the operator image upstream, so a mismatch here would
9+
# run one version's binary against another's CRDs.
10+
ARG VERSION=v2.8.0
11+
12+
ARG TARGETOS
13+
ARG TARGETARCH
14+
15+
WORKDIR /workspace
16+
17+
RUN curl -sSL https://github.com/opensearch-project/opensearch-k8s-operator/archive/refs/tags/${VERSION}.tar.gz | tar -xzvf- --strip=1
18+
19+
COPY patches /patches
20+
RUN git apply /patches/*.diff
21+
22+
WORKDIR /workspace/opensearch-operator
23+
24+
RUN GOOS=$TARGETOS GOARCH=$TARGETARCH CGO_ENABLED=0 go build -a -o manager main.go
25+
26+
FROM gcr.io/distroless/static:nonroot@sha256:d29e660cc75a5b6b1334e03c5c81ccf9bc0884a002c6000dbf0fb96034814478
27+
WORKDIR /
28+
COPY --from=builder /workspace/opensearch-operator/manager .
29+
USER 65532:65532
30+
31+
ENTRYPOINT ["/manager"]
Lines changed: 282 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,282 @@
1+
diff --git a/opensearch-operator/opensearch-gateway/services/os_client.go b/opensearch-operator/opensearch-gateway/services/os_client.go
2+
index 2ec30c5..a33fa11 100644
3+
--- a/opensearch-operator/opensearch-gateway/services/os_client.go
4+
+++ b/opensearch-operator/opensearch-gateway/services/os_client.go
5+
@@ -4,6 +4,7 @@ import (
6+
"context"
7+
"crypto/tls"
8+
"encoding/json"
9+
+ "fmt"
10+
"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/helpers"
11+
"io"
12+
"k8s.io/utils/ptr"
13+
@@ -535,3 +536,115 @@ func generateAPIPathSnapshotUpdatePolicies(resource, name string, seqno, primary
14+
path.WriteString(strconv.Itoa(primaryterm))
15+
return path
16+
}
17+
+
18+
+// votingConfigState is the slice of the cluster state that says who may vote.
19+
+// The coordination metadata names voters by node id, so the node list travels
20+
+// in the same response to resolve an id back to a name.
21+
+type votingConfigState struct {
22+
+ Nodes map[string]struct {
23+
+ Name string `json:"name"`
24+
+ } `json:"nodes"`
25+
+ Metadata struct {
26+
+ ClusterCoordination struct {
27+
+ LastCommittedConfig []string `json:"last_committed_config"`
28+
+ VotingConfigExclusions []struct {
29+
+ NodeName string `json:"node_name"`
30+
+ } `json:"voting_config_exclusions"`
31+
+ } `json:"cluster_coordination"`
32+
+ } `json:"metadata"`
33+
+}
34+
+
35+
+// ExcludeNodeFromVotingConfig commits a voting configuration that leaves
36+
+// nodeName out, which is what makes the node safe to remove: a node still in
37+
+// the configuration is a vote the cluster keeps counting on.
38+
+func (client *OsClusterClient) ExcludeNodeFromVotingConfig(ctx context.Context, nodeName string) error {
39+
+ path := generateAPIPathVotingConfigExclusions(nodeName)
40+
+ res, err := doHTTPPost(ctx, client.client, path, nil)
41+
+ if err != nil {
42+
+ return err
43+
+ }
44+
+ defer helpers.SafeClose(res.Body)
45+
+ if res.IsError() {
46+
+ return fmt.Errorf("excluding %s from the voting configuration returned %s", nodeName, res.Status())
47+
+ }
48+
+ return nil
49+
+}
50+
+
51+
+// ClearVotingConfigExclusions empties the exclusion list, which is capped and
52+
+// would otherwise keep entries for nodes that are long gone. wait_for_removal
53+
+// stays at its default: clearing without it can re-admit a node that is still
54+
+// running, and the caller has already removed the one it excluded.
55+
+func (client *OsClusterClient) ClearVotingConfigExclusions(ctx context.Context) error {
56+
+ var path strings.Builder
57+
+ path.WriteString("/_cluster/voting_config_exclusions")
58+
+ res, err := doHTTPDelete(ctx, client.client, path)
59+
+ if err != nil {
60+
+ return err
61+
+ }
62+
+ defer helpers.SafeClose(res.Body)
63+
+ if res.IsError() {
64+
+ return fmt.Errorf("clearing the voting configuration exclusions returned %s", res.Status())
65+
+ }
66+
+ return nil
67+
+}
68+
+
69+
+func (client *OsClusterClient) readVotingConfigState(ctx context.Context) (votingConfigState, error) {
70+
+ var state votingConfigState
71+
+ var path strings.Builder
72+
+ path.WriteString("/_cluster/state/nodes,metadata")
73+
+ path.WriteString("?filter_path=nodes.*.name,metadata.cluster_coordination.last_committed_config")
74+
+ path.WriteString(",metadata.cluster_coordination.voting_config_exclusions")
75+
+ res, err := doHTTPGet(ctx, client.client, path)
76+
+ if err != nil {
77+
+ return state, err
78+
+ }
79+
+ defer helpers.SafeClose(res.Body)
80+
+ if res.IsError() {
81+
+ return state, fmt.Errorf("reading the voting configuration returned %s", res.Status())
82+
+ }
83+
+ err = json.NewDecoder(res.Body).Decode(&state)
84+
+ return state, err
85+
+}
86+
+
87+
+// IsInVotingConfig reports whether nodeName is still one of the nodes whose
88+
+// vote counts. An exclusion is committed through the cluster state rather than
89+
+// applied by the time the request returns, so this is what tells a caller the
90+
+// exclusion has actually taken.
91+
+func (client *OsClusterClient) IsInVotingConfig(ctx context.Context, nodeName string) (bool, error) {
92+
+ state, err := client.readVotingConfigState(ctx)
93+
+ if err != nil {
94+
+ return false, err
95+
+ }
96+
+
97+
+ for _, id := range state.Metadata.ClusterCoordination.LastCommittedConfig {
98+
+ if node, ok := state.Nodes[id]; ok && node.Name == nodeName {
99+
+ return true, nil
100+
+ }
101+
+ }
102+
+ return false, nil
103+
+}
104+
+
105+
+// IsVotingConfigExcluded reports whether nodeName is still named in the
106+
+// exclusion list. The list is capped, so an entry left behind for a node that
107+
+// no longer exists costs capacity a later membership change may need.
108+
+func (client *OsClusterClient) IsVotingConfigExcluded(ctx context.Context, nodeName string) (bool, error) {
109+
+ state, err := client.readVotingConfigState(ctx)
110+
+ if err != nil {
111+
+ return false, err
112+
+ }
113+
+
114+
+ for _, ex := range state.Metadata.ClusterCoordination.VotingConfigExclusions {
115+
+ if ex.NodeName == nodeName {
116+
+ return true, nil
117+
+ }
118+
+ }
119+
+ return false, nil
120+
+}
121+
+
122+
+func generateAPIPathVotingConfigExclusions(nodeName string) strings.Builder {
123+
+ var path strings.Builder
124+
+ path.Grow(len("/_cluster/voting_config_exclusions?node_names=") + len(nodeName))
125+
+ path.WriteString("/_cluster/voting_config_exclusions?node_names=")
126+
+ path.WriteString(nodeName)
127+
+ return path
128+
+}
129+
diff --git a/opensearch-operator/pkg/reconcilers/cluster.go b/opensearch-operator/pkg/reconcilers/cluster.go
130+
index 710f585..c199fa0 100644
131+
--- a/opensearch-operator/pkg/reconcilers/cluster.go
132+
+++ b/opensearch-operator/pkg/reconcilers/cluster.go
133+
@@ -4,7 +4,9 @@ import (
134+
"context"
135+
"fmt"
136+
"strings"
137+
+ "time"
138+
139+
+ "github.com/Opster/opensearch-k8s-operator/opensearch-operator/opensearch-gateway/services"
140+
"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/k8s"
141+
"github.com/Opster/opensearch-k8s-operator/opensearch-operator/pkg/reconcilers/util"
142+
policyv1 "k8s.io/api/policy/v1"
143+
@@ -17,6 +19,8 @@ import (
144+
"github.com/go-logr/logr"
145+
"github.com/samber/lo"
146+
appsv1 "k8s.io/api/apps/v1"
147+
+ corev1 "k8s.io/api/core/v1"
148+
+ k8serrors "k8s.io/apimachinery/pkg/api/errors"
149+
"k8s.io/apimachinery/pkg/api/resource"
150+
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
151+
"k8s.io/client-go/tools/record"
152+
@@ -25,6 +29,12 @@ import (
153+
"sigs.k8s.io/controller-runtime/pkg/log"
154+
)
155+
156+
+// How long to wait before trying again when the cluster cannot be reached or
157+
+// has not yet committed the exclusion. Short enough that a healthy cluster
158+
+// sheds its bootstrap pod promptly, long enough not to spin on an API that is
159+
+// still coming up.
160+
+const bootstrapRetireRetry = 10 * time.Second
161+
+
162+
type ClusterReconciler struct {
163+
client k8s.K8sClient
164+
ctx context.Context
165+
@@ -98,7 +108,7 @@ func (r *ClusterReconciler) Reconcile() (ctrl.Result, error) {
166+
bootstrapPod := builders.NewBootstrapPod(r.instance, r.reconcilerContext.Volumes, r.reconcilerContext.VolumeMounts)
167+
result.CombineErr(ctrl.SetControllerReference(r.instance, bootstrapPod, r.client.Scheme()))
168+
if r.instance.Status.Initialized {
169+
- result.Combine(r.client.ReconcileResource(bootstrapPod, reconciler.StateAbsent))
170+
+ result.Combine(r.retireBootstrapPod(bootstrapPod))
171+
} else {
172+
result.Combine(r.client.ReconcileResource(bootstrapPod, reconciler.StatePresent))
173+
}
174+
@@ -130,6 +140,108 @@ func (r *ClusterReconciler) Reconcile() (ctrl.Result, error) {
175+
return result.Result, result.Err
176+
}
177+
178+
+// retireBootstrapPod takes the bootstrap node out of the voting configuration
179+
+// before deleting its pod.
180+
+//
181+
+// The bootstrap node is the one that formed the cluster, so it is the voter the
182+
+// cluster keeps while the voting configuration holds a single node, which is
183+
+// what it holds whenever exactly two nodes are cluster_manager-eligible: the
184+
+// bootstrap pod and one node pool replica. Deleting the pod then takes away the
185+
+// only vote there was, and the survivor can never hold an election again,
186+
+// because changing the voting configuration itself needs a quorum. Excluding
187+
+// the node first moves the configuration onto the surviving node, which is what
188+
+// makes the deletion safe.
189+
+//
190+
+// Every failure to reach the cluster leaves the pod alone and requeues. A
191+
+// bootstrap pod that outlives its purpose costs a pod; a cluster that cannot
192+
+// elect a cluster-manager does not recover without a human.
193+
+func (r *ClusterReconciler) retireBootstrapPod(bootstrapPod *corev1.Pod) (*ctrl.Result, error) {
194+
+ podGone := false
195+
+ if _, err := r.client.GetPod(bootstrapPod.Name, bootstrapPod.Namespace); err != nil {
196+
+ if !k8serrors.IsNotFound(err) {
197+
+ return &ctrl.Result{}, err
198+
+ }
199+
+ podGone = true
200+
+ }
201+
+
202+
+ annotations := map[string]string{"cluster-name": r.instance.GetName()}
203+
+ nodeName := builders.BootstrapPodName(r.instance)
204+
+
205+
+ clusterClient, err := util.CreateClientForCluster(r.client, r.ctx, r.instance, nil)
206+
+ if err != nil {
207+
+ r.logger.Error(err, "Failed to create os client to retire the bootstrap node", "node", nodeName)
208+
+ r.recorder.AnnotatedEventf(r.instance, annotations, "Warning", "Bootstrap",
209+
+ "Cannot reach the cluster to exclude %s from voting; keeping the bootstrap pod", nodeName)
210+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
211+
+ }
212+
+
213+
+ // Deleting the pod is a request, not a wait, so the clearing step below can
214+
+ // find the node still in the cluster and fail. Retrying it here is what
215+
+ // keeps a stale entry from sitting in the capped exclusion list forever,
216+
+ // where it would cost a later membership change the capacity it needs.
217+
+ if podGone {
218+
+ return r.clearBootstrapExclusion(clusterClient, nodeName)
219+
+ }
220+
+
221+
+ if err := clusterClient.ExcludeNodeFromVotingConfig(r.ctx, nodeName); err != nil {
222+
+ r.logger.Error(err, "Failed to exclude the bootstrap node from the voting configuration", "node", nodeName)
223+
+ r.recorder.AnnotatedEventf(r.instance, annotations, "Warning", "Bootstrap",
224+
+ "Could not exclude %s from voting; keeping the bootstrap pod", nodeName)
225+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
226+
+ }
227+
+
228+
+ // The exclusion travels through the cluster state, so it is committed a
229+
+ // moment after the request returns rather than by the time it does.
230+
+ stillVoting, err := clusterClient.IsInVotingConfig(r.ctx, nodeName)
231+
+ if err != nil {
232+
+ r.logger.Error(err, "Failed to read the voting configuration", "node", nodeName)
233+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
234+
+ }
235+
+ if stillVoting {
236+
+ r.logger.Info("Waiting for the bootstrap node to leave the voting configuration", "node", nodeName)
237+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
238+
+ }
239+
+
240+
+ result, err := r.client.ReconcileResource(bootstrapPod, reconciler.StateAbsent)
241+
+ if err != nil {
242+
+ return result, err
243+
+ }
244+
+
245+
+ r.recorder.AnnotatedEventf(r.instance, annotations, "Normal", "Bootstrap",
246+
+ "Excluded %s from voting and removed the bootstrap pod", nodeName)
247+
+
248+
+ if cleared, err := r.clearBootstrapExclusion(clusterClient, nodeName); err != nil || cleared.RequeueAfter > 0 {
249+
+ return cleared, err
250+
+ }
251+
+ return result, nil
252+
+}
253+
+
254+
+// clearBootstrapExclusion drops the exclusion this reconciler added, once the
255+
+// node it named is really gone.
256+
+//
257+
+// Clearing empties the whole list rather than one entry, so it is left at the
258+
+// API's default wait_for_removal=true: that is what stops it from re-admitting
259+
+// a node some other operation excluded and is still draining. A failure is
260+
+// therefore expected while the bootstrap pod is terminating, and is answered
261+
+// with a requeue rather than a log line, because nothing else will come back to
262+
+// it.
263+
+func (r *ClusterReconciler) clearBootstrapExclusion(clusterClient *services.OsClusterClient, nodeName string) (*ctrl.Result, error) {
264+
+ excluded, err := clusterClient.IsVotingConfigExcluded(r.ctx, nodeName)
265+
+ if err != nil {
266+
+ r.logger.Error(err, "Failed to read the voting configuration exclusions", "node", nodeName)
267+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
268+
+ }
269+
+ if !excluded {
270+
+ return &ctrl.Result{}, nil
271+
+ }
272+
+
273+
+ if err := clusterClient.ClearVotingConfigExclusions(r.ctx); err != nil {
274+
+ r.logger.Error(err, "Failed to clear the voting configuration exclusions", "node", nodeName)
275+
+ return &ctrl.Result{RequeueAfter: bootstrapRetireRetry}, nil
276+
+ }
277+
+ return &ctrl.Result{}, nil
278+
+}
279+
+
280+
func (r *ClusterReconciler) reconcileNodeStatefulSet(nodePool opsterv1.NodePool, username string) (*ctrl.Result, error) {
281+
found, nodePoolConfig := r.reconcilerContext.fetchNodePoolHash(nodePool.Component)
282+

‎packages/system/opensearch-operator/values.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,15 @@ opensearch-operator:
88
replicaCount: 1
99
# Enable cluster-wide mode to watch all namespaces
1010
watchNamespace: ""
11+
# Built in-tree from images/opensearch-operator, which carries the
12+
# bootstrap voting-exclusion patch. Without this block the vendored chart
13+
# falls back to opensearchproject/opensearch-operator at the chart's
14+
# appVersion, which is the unpatched upstream build, and the reference stays
15+
# outside the first-party set hack/lib/image-refs.sh collects for promote,
16+
# retag and mirror. `make image` rewrites both keys on every build.
17+
image:
18+
repository: ghcr.io/cozystack/cozystack/opensearch-operator
19+
tag: v2.8.0@sha256:cbe6b50f13aa5abf338209d38dfcd80e73296e97510465b5fca73a0130bfaad7
1120
kubeRbacProxy:
1221
image:
1322
# gcr.io/kubebuilder/kube-rbac-proxy was sunset; use the original upstream

0 commit comments

Comments
 (0)