Skip to content

Commit 325a4e3

Browse files
authored
fix(opensearch-operator): let a permanently yellow cluster restart (cozystack#4317)
## What this PR does A single-replica OpenSearch release stops rolling once any index on it asks for a replica. The copy has nowhere to go on one data node, so the cluster stays yellow for good. The operator's restart gate at v2.8.0 opens on green, or on yellow only when `.opensearch-observability` is the yellow index, and at OpenSearch 2.11.1 that index is `auto_expand_replicas: 0-2`, green on one node. So the gate never opens. Every release gets such an index by itself. The operator sets `plugins.security.audit.type: internal_opensearch`, and the first audited event creates a daily `security-auditlog-*` index with one replica. After that nothing that needs a restart is rolled out, a version upgrade or the move of the HTTP listener onto the cert-manager chain included. This is what timed out the `opensearch` e2e suite on some runs. Both failed runs I read show the same thing in the OpenSearch log: the bootstrap node leaves, `Connection reset` on the transport layer, and `[security-auditlog-<date>] creating index ... shards [1]/[1]` within 40 ms. The security plugin audits that exception (`SecuritySSLNettyTransport` hands it to `AuditLogSslExceptionHandler`), so the reset is what creates the index. Then the operator logs `waiting for health to be green` every ten seconds until the assert gives up. Not sure why the green runs had no audit index before the roll, a green run keeps no snapshot. The fix carries opensearch-project/opensearch-k8s-operator#1569 back to v2.8.0 as a second patch on the in-tree operator image, with the upstream tests. The new gate lets a yellow cluster through once allocation is enabled, every data node has joined and no shard is moving, and `PreparePodForDelete` refuses to restart a node that holds the only active copy of a shard meant to have others. A lone data node is exempt. I left out the third commit of that PR (tiered data roles in `HasDataRole`), the chart renders only the `data` role. A bump would not help: the only operator build after v2.8.0 is `3.0.0-alpha` from January, and its gate holds this case the same way. The patch goes away on the first release that includes cozystack#1569. The suite now creates an index with an unassignable replica before it moves the listener, so the roll always runs against a yellow cluster. The catch block also reads cluster health and the shard table from the node, because the operator logs that health held a restart back and never what the health was. The image digest in `values.yaml` stays as it is. Closes cozystack#4259 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff: an e2e suite, the operator image's Dockerfile header and patch directory, one comment in the operator package's `values.yaml`. No values key, schema, make target or documented workaround changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): a single-replica OpenSearch release no longer gets stuck ahead of every rolling restart once an index with a replica, such as the security audit log or any index created with default settings, leaves the cluster permanently yellow ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved rolling restart handling for clusters that remain yellow, allowing restarts to proceed safely once shard activity has settled. - Prevented restart operations from becoming blocked by temporary shard-allocation settings. - Added safeguards to ensure data is not disrupted while replicas are relocating or initializing. - **Diagnostics** - Added automatic cluster health and shard diagnostics when restart-related test operations time out or encounter issues. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 74c8724 + d9fbeab commit 325a4e3

4 files changed

Lines changed: 441 additions & 5 deletions

File tree

‎hack/e2e-chainsaw/opensearch/chainsaw-test.yaml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,21 @@ spec:
5353
apiVersion: v1
5454
kind: Service
5555
format: yaml
56+
# The operator logs that cluster health held a restart back, never what the
57+
# health was or which shards kept it there, and neither is a Kubernetes object.
58+
- script:
59+
# Three reads of at most 30s each. The op timeout sits above their sum so
60+
# that an inner budget is what fires and the later reads still run.
61+
timeout: 2m
62+
content: |
63+
for path in '_cluster/health?level=indices&pretty' '_cat/shards?v' '_cluster/allocation/explain?pretty'; do
64+
echo "== $path"
65+
timeout 30 kubectl -n "$NAMESPACE" exec opensearch-test-nodes-0 -c opensearch -- /bin/bash -c '
66+
curl -k --silent --show-error --max-time 20 \
67+
-u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" \
68+
"https://localhost:9200/$1"' os_api "$path" || true
69+
echo
70+
done
5671
steps:
5772
- name: create-opensearch
5873
try:
@@ -125,6 +140,48 @@ spec:
125140
status:
126141
readyReplicas: 1
127142

143+
# On one data node an index that asks for a replica leaves the cluster yellow
144+
# for good, because the copy has no second node to land on, and the operator
145+
# restarts a pod only once its health gate opens. A release gets there without
146+
# anyone asking for it: the security audit log and any index a tenant creates
147+
# with the default replica count both do it. Whether one exists by the time the
148+
# listener moves is a matter of timing, so the suite creates one itself and the
149+
# roll below always has to get through a yellow cluster.
150+
- name: make-the-cluster-permanently-yellow
151+
try:
152+
- script:
153+
# Above the two in-pod budgets below, so one of them is what fires.
154+
timeout: 3m
155+
content: |
156+
set -eu
157+
158+
# The credentials and the form of the operator's own readiness probe,
159+
# so a pod that reports Ready answers this as well.
160+
os_api() {
161+
kubectl -n "$NAMESPACE" exec opensearch-test-nodes-0 -c opensearch -- /bin/bash -c '
162+
curl -k --silent --show-error --max-time 75 \
163+
-u "$(cat /mnt/admin-credentials/username):$(cat /mnt/admin-credentials/password)" \
164+
-H "Content-Type: application/json" "$@"' os_api "$@"
165+
}
166+
167+
os_api --fail -X PUT "https://localhost:9200/unassignable-replica" \
168+
--data '{"settings":{"index":{"number_of_shards":1,"number_of_replicas":1}}}'
169+
170+
# wait_for_status is satisfied by green as well, so the answer is read
171+
# rather than trusted: a fixture that stopped holding the cluster
172+
# yellow would leave the roll below proving nothing about the gate.
173+
# No --fail here: a wait that times out answers 408 with the health in
174+
# its body, and that body is what the message below has to print.
175+
health=$(os_api "https://localhost:9200/_cluster/health/unassignable-replica?wait_for_status=yellow&timeout=60s")
176+
case "$health" in
177+
*'"status":"yellow"'*) ;;
178+
*)
179+
echo "an index with an unassignable replica did not leave the cluster yellow" >&2
180+
echo "$health" >&2
181+
exit 1
182+
;;
183+
esac
184+
128185
- name: move-onto-cert-manager-issuer
129186
try:
130187
# A JSON-patch remove rather than an apply of a second fixture: what an

‎packages/system/opensearch-operator/images/opensearch-operator/Dockerfile‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,18 @@
1-
# Build the manager binary from the upstream sources with the Cozystack patch
2-
# applied. The patch keeps a single-replica cluster electable when the operator
3-
# retires its bootstrap pod; see patches/bootstrap-voting-exclusion.diff and
1+
# Build the manager binary from the upstream sources with the Cozystack patches
2+
# applied.
3+
#
4+
# patches/bootstrap-voting-exclusion.diff keeps a single-replica cluster
5+
# electable when the operator retires its bootstrap pod; see
46
# https://github.com/opensearch-project/opensearch-k8s-operator/issues/1448.
7+
#
8+
# patches/restart-on-permanently-yellow.diff carries
9+
# https://github.com/opensearch-project/opensearch-k8s-operator/pull/1569 back to
10+
# this version: a cluster whose replicas can never be assigned stays yellow for
11+
# good, and the restart gate held every rolling restart and upgrade on it. The
12+
# part of that change that counts tiered data roles (data_hot and the like) as
13+
# data nodes is left out, because packages/apps/opensearch renders the plain
14+
# data role only; carry it as well if that ever changes. Drop the patch on the
15+
# first operator release that includes the change.
516
FROM golang:1.26@sha256:079e59808d2d252516e27e3f3a9c003740dee7f75e55aa71528766d52bcfc16a AS builder
617

718
# Keep in step with the chart version pulled by `make update`: the chart's

0 commit comments

Comments
 (0)