Repository navigation
fix: let rolling restarts and upgrades proceed on a permanently yellow cluster - #1569
Conversation
|
@synhershko as mentioned in my PR, the test with your branch also worked for me |
There was a problem hiding this comment.
lgtm.
btw, i have one thing to clarify here
DataNodesCount only counts the literal data role while number_of_data_nodes also includes tiered roles such as hot, warm, etc.
Tiered node pools will be skipped for the wait check (tiered only roles which has no data in the role list)
|
Good catch, and it is worse than a skipped wait - thanks for spotting it. Fixed in 5c0c2db (also merged current You are right that
I fixed it in var dataRoles = []string{
"data", "data_content", "data_hot", "data_warm", "data_cold", "data_frozen", "search", "warm",
}That also corrects The role list is the part worth a second pair of eyes. I took it as every Added tests covering each tiered role and a negative case for |
…w cluster A cluster whose replicas can never be assigned (e.g. number_of_replicas higher than the number of other data nodes) stays yellow forever, and the restart health gate only let yellow through when every unassigned replica was blocked by the node_version decider mid-upgrade. Rolling restarts and version upgrades therefore stalled indefinitely. The health gate now accepts a yellow cluster once shard allocation is enabled and nothing is moving (no relocating, initializing or delayed shards and no in-flight shard fetches). At that point the allocator has already decided the remaining replicas cannot be placed. The operator's own "primaries" allocation throttle is lifted first, for both drain modes, so it can't be mistaken for an unassignable replica. To keep this safe, PreparePodForDelete (without drain) now refuses to delete a pod holding the only active copy of a shard that should have replicas, so a restart never takes a shard below one active copy. A cluster with a single data node is exempt, as there is nowhere else a copy could live (same as a green cluster with zero replicas). CheckClusterRestartOnYellow is removed; its node_version case is covered by the new rules. Resolves opensearch-project#1424 Signed-off-by: Itamar Syn-Hershko <[email protected]>
…s on yellow - Run the sole-copy check for drain mode too: with 2 data nodes the drain path only checks system-index primaries, so a non-system shard whose only active copy is on the node could still be restarted. - On a yellow cluster, only proceed once number_of_data_nodes reaches the expected data node count. A node that hasn't rejoined leaves no trace in the activity counters once its delayed allocation expires, and a pod can report Ready before it joins. - Ignore search replicas when counting copies; they can't be promoted. - Scope the docs note to drainDataNodes: false. Signed-off-by: Itamar Syn-Hershko <[email protected]>
HasDataRole only matched the literal "data" role, but OpenSearch counts any node whose role can contain data. A pool with only data_content, data_hot/warm/cold/frozen, search or warm was therefore invisible to every caller that asks "does this pool hold data". The new yellow-cluster gate made that visible: it compares health.number_of_data_nodes, which includes those roles, against DataNodesCount, which did not. On a cluster with tiered pools the count was always too low, so "waiting for all data nodes to join" never fired and a restart could proceed with a data node still missing. The same undercount skipped the sole-active-copy guard on a tiered-only cluster, where DataNodesCount returned 0 and the nodeCount > 1 check was false. Widening the helper fixes the other callers too: upgrade ordering no longer treats a tiered pool as a non-data pool, cluster reconcile no longer lets its env vars change mid-upgrade, and the emptyDir data-loss accounting now counts tiered pods, which are just as exposed to losing their volume. Signed-off-by: Itamar Syn-Hershko <[email protected]>
5c0c2db to
c2ed9e4
Compare
…4317) ## What this PR does A single-replica OpenSearch release stops rolling once any index on it asks for a replica. The copy has nowhere to go on one data node, so the cluster stays yellow for good. The operator's restart gate at v2.8.0 opens on green, or on yellow only when `.opensearch-observability` is the yellow index, and at OpenSearch 2.11.1 that index is `auto_expand_replicas: 0-2`, green on one node. So the gate never opens. Every release gets such an index by itself. The operator sets `plugins.security.audit.type: internal_opensearch`, and the first audited event creates a daily `security-auditlog-*` index with one replica. After that nothing that needs a restart is rolled out, a version upgrade or the move of the HTTP listener onto the cert-manager chain included. This is what timed out the `opensearch` e2e suite on some runs. Both failed runs I read show the same thing in the OpenSearch log: the bootstrap node leaves, `Connection reset` on the transport layer, and `[security-auditlog-<date>] creating index ... shards [1]/[1]` within 40 ms. The security plugin audits that exception (`SecuritySSLNettyTransport` hands it to `AuditLogSslExceptionHandler`), so the reset is what creates the index. Then the operator logs `waiting for health to be green` every ten seconds until the assert gives up. Not sure why the green runs had no audit index before the roll, a green run keeps no snapshot. The fix carries opensearch-project/opensearch-k8s-operator#1569 back to v2.8.0 as a second patch on the in-tree operator image, with the upstream tests. The new gate lets a yellow cluster through once allocation is enabled, every data node has joined and no shard is moving, and `PreparePodForDelete` refuses to restart a node that holds the only active copy of a shard meant to have others. A lone data node is exempt. I left out the third commit of that PR (tiered data roles in `HasDataRole`), the chart renders only the `data` role. A bump would not help: the only operator build after v2.8.0 is `3.0.0-alpha` from January, and its gate holds this case the same way. The patch goes away on the first release that includes #1569. The suite now creates an index with an unassignable replica before it moves the listener, so the roll always runs against a yellow cluster. The catch block also reads cluster health and the shard table from the node, because the operator logs that health held a restart back and never what the health was. The image digest in `values.yaml` stays as it is. Closes #4259 ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map against the diff: an e2e suite, the operator image's Dockerfile header and patch directory, one comment in the operator package's `values.yaml`. No values key, schema, make target or documented workaround changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(opensearch-operator): a single-replica OpenSearch release no longer gets stuck ahead of every rolling restart once an index with a replica, such as the security audit log or any index created with default settings, leaves the cluster permanently yellow ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved rolling restart handling for clusters that remain yellow, allowing restarts to proceed safely once shard activity has settled. - Prevented restart operations from becoming blocked by temporary shard-allocation settings. - Added safeguards to ensure data is not disrupted while replicas are relocating or initializing. - **Diagnostics** - Added automatic cluster health and shard diagnostics when restart-related test operations time out or encounter issues. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
…it (#4279) ## What this is `Refs #4259`. A diagnosis of the opensearch suite timeout plus the one in-repo change it warrants: making the failing step say which of its two candidates actually happened. It does **not** claim to fix the root cause — see "What this does not do". ## What the run actually showed The `verify-statefulset-rolled-onto-the-new-mount-plan` assert erred with exactly two field failures, both under `status`: `(currentRevision == updateRevision): false` and `updatedReplicas: field not found`. No volume field appears in the error list, and `readyReplicas: 1` shows as an unchanged line, so the pod was Ready at the deadline and nothing had replaced it. Per e2e-testing.md ("Reading a failure diff: the actual side is a projection"), the real cause is the error line, not the diff — and a Chainsaw diff projects the live object onto the expected fields, so the three `http-ca` / `http-cert` / `http-key` lines showing in the diff are the expected tree, not proof the volumes are absent. Read that way, the volume assertions matched: the three volumes were present on the live template. The "volumes missing" reading is the projection trap the doc warns against. ## Which candidate Candidate (b): the template was rewritten onto the subPath mount plan, and the operator has not restarted the pod onto it. This is consistent with the source, traced against opensearch-k8s-operator v2.8.0 as vendored: - The chart renders the CR correctly — `generate` dropped, `secret` and `caSecret` named — proven by `packages/apps/opensearch/tests/tls_upgrade_test.yaml`. - Given that CR, `pkg/reconcilers/tls.go` `handleHttp` appends exactly the three volumes the step asserts (`http-ca` from `opensearch-test.http-ca`, `http-key` and `http-cert` from `opensearch-test-http-server`), and `pkg/reconcilers/cluster.go` `reconcileNodeStatefulSet` writes that template onto the StatefulSet. Three branches return ahead of that write — no node-pool config hash, the `opensearch.role` recreate, and parallel recovery — but the parallel-recovery gate is `ReadyReplicas < replicas-1`, unreachable at one replica, so the write happens. The reconcilers that run ahead of the cluster reconciler do not stop it either: the TLS reconciler runs first and returns the admin-certificate path's result, empty in the ordinary case, and `securityconfig.Reconcile` short-circuits to nil when the security-config checksum is unchanged, which the move does not change. So the operator does write the volumes. - The nodePool StatefulSet is `OnDelete` (`pkg/builders/cluster.go` sets `UpdateStrategy.Type: OnDelete` alongside `PodManagementPolicy: OrderedReady`), so the StatefulSet controller never deletes the pod to apply a template change: `podManagementPolicy` orders creation and deletion during scaling, `updateStrategy` decides the update. The roll is the operator's rolling-restart reconciler's to drive (`pkg/reconcilers/rollingRestart.go`) — it deletes the pod itself. It needs `status.Initialized`, it needs `ReadyReplicas == replicas` (a Ready pod is a precondition, not the blocker), then it opens a client to the OpenSearch API and calls `CheckClusterStatusForRestart`, which holds the restart while the cluster is red; on yellow the gate now enables shard allocation, waits for every data node to join, waits for shard activity to settle, then proceeds (#4317 deleted the old `.opensearch-observability` carve-out, and this suite builds the operator with that patch). Any of those gates can sit indefinitely. `status.currentRevision` never advances by itself under `OnDelete`; the operator patches it once `updatedReplicas == replicas`, so the second assert's own condition cannot go true unless that reconciler ran to the end. - The pod's startup probe is a TCP check on 9200 (`builders/cluster.go`), so "connection refused" is OpenSearch not listening, not a probe-scheme mismatch. The plaintext `GET / HTTP/1.1` from `curl/8.5.0` on the TLS listener is neither the startup/liveness probe (TCP) nor the readiness probe (`curl -k https://localhost:9200`, localhost, HTTPS) — it is external traffic to the LoadBalancer address, a red herring for the stall. ## What this changes The step now runs two asserts instead of one. The first waits for the operator to rewrite the template onto the three volumes, on a reconcile-scale budget (6m) — a failure here is the operator not writing them (candidate a). The second waits for the roll to complete onto that template, on the pod-lifetime budget (12m) — a failure here with the first green is the roll not completing (candidate b): the operator never restarting the pod, which the rolling-restart reconciler holds while `CheckClusterStatusForRestart` reports the cluster red — on yellow the gate enables shard allocation, waits for every data node to join, waits for shard activity to settle, then proceeds — or a replacement that was deleted but never came Ready. The second assert carries the three volume projections and `observedGeneration == metadata.generation` as well, so the roll is pinned to the template the first assert checked rather than to whatever revision is current when the status converges. The suite `catch` also captures the StatefulSet as YAML and as a describe, and the operator's own log, because `updateRevision`, `currentRevision`, `generation` and `updatedReplicas` are what separate the two halves — describe alone omits them — and the restart decision is logged in the operator, not in the cluster pods. The reconciler logs `Starting rolling restart of the StatefulSet` (`rollingRestart.go:155`) once its early gates (`Initialized`, ready-replica, update-pending) pass; it builds its OpenSearch client just before, at `:138`, and that client pings the cluster — so a run whose log lacks `Starting rolling restart` either had an earlier gate hold it silently or had the client ping fail loudly, not only the former, while that line present followed by a `Couldn't proceed with rolling restart ... because <reason>` line names a health-stage hold (the health gate is `CheckClusterStatusForRestart`, and it is not the last: `PreparePodForDelete` follows it). The operator-log `tail` was raised so it spans the whole assert window rather than a third of it. ## What this does not do It does not fix the stall; #4317 does, and is already merged. The run that captured this (`Run E2E tests`, job 104576486900) shows the rolling-restart reconciler holding on health: the manager logged `Starting rolling restart of the StatefulSet opensearch-test-nodes` (`rollingRestart.go:155`, printed only after the `Initialized` and ready-replica gates passed) and then, every ten seconds to the deadline, `Couldn't proceed with rolling restart ... waiting for health to be green` (`os_data_service.go:180`), which is reached only after `GetFlatClusterSettings` returned 2xx (`os_client.go:224` makes anything else an error). So the operator reached the cluster and the cluster answered; health held the restart. #4259 established why, and the CA move plays no part: on a single data node the bootstrap node leaving is audited into a fresh `security-auditlog-*` index with one replica, that replica has nowhere to go, the cluster stays yellow for good, and the v2.8.0 restart gate opens only on green (or on a yellow `.opensearch-observability`) so it never opens. #4317 backports opensearch-project/opensearch-k8s-operator#1569 to fix that gate; this PR is only the diagnostic split that names which half of the roll a failure is. The split content itself is not the failure: on `49f3f059f` both asserts passed, the second after 1m47s, and this change touches only the step's comments and the operator-log tail, not the assertions. With this change in place, a future occurrence names the half, and the catch carries the roll's own fields and the operator log, so the crust-gather can be read against a specific claim rather than an ambiguous one. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Improved OpenSearch StatefulSet failure diagnostics by capturing YAML, detailed resource descriptions, and the latest operator pod logs. * Expanded rolling-restart verification to check volume projections, template updates, rollout readiness, observed and metadata generations, and revision consistency. * Separated template validation from rollout completion checks with dedicated six- and twelve-minute time limits. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
Description
Rolling restarts and version upgrades stall forever on a cluster that is permanently yellow, e.g. a single data node with
number_of_replicas >= 1, or more replicas than the other data nodes can hold (the same applies to allocation awareness, allocation filters and disk watermarks).CheckClusterStatusForRestartonly lets a yellow cluster through when every unassigned replica is blocked by thenode_versiondecider, which only happens mid-upgrade. In every other case it waits for green, which never comes.This PR fixes it with two small rules in
os_data_service.goand no new API surface:cluster.routing.allocation.enableisall(lifting its ownprimariesthrottle, in both drain modes). It then proceeds only once every expected data node has joined (number_of_data_nodes≥ the data node count from the node pools) and there is no shard activity: no relocating, initializing or delayed-unassigned shards and no in-flight shard fetches. With all data nodes present, allocation enabled and nothing moving, the allocator has already decided the remaining replicas cannot be placed, so waiting will never turn the cluster green. Yellow already guarantees every primary is active. The data node check matters because a node that hasn't rejoined leaves no trace in the activity counters once its delayed allocation expires, and a pod can report Ready before it has joined.PreparePodForDeletenow refuses to delete a pod that holds the only active (STARTED/RELOCATING) copy of a shard that is supposed to have other copies, in both drain modes. With drain, this runs after the node is excluded and its drain checks pass, which matters for 2-data-node clusters, where the drain path only waits for system-index primaries. A restart therefore never takes a shard below one active copy, the same floor a green cluster provides. Search replicas are not counted as copies, since they can't be promoted. A cluster with a single data node is exempt: there is nowhere else a copy could live, which is exactly what the operator already does for a green single-node cluster with zero replicas.CheckClusterRestartOnYellowis removed; itsnode_versioncase is covered by the rules above.Why this approach instead of #1446 (and #1426)
healthGatePolicyto document, migrate or get wrong. Users hitting Add policy to continue rolling restart on safely recoverable yellow health #1424 are fixed without changing their specs.drainDataNodes: falseit can restart the node holding the only started copy of a shard whose replica is unassigned, taking the index red for the whole restart, and losing data on emptyDir storage. Here, that pod is never deleted.allonly runs when drain is off, so replicas held back by aprimaries/nonesetting can be misclassified as recoverable._cluster/healthplus one settings GET per poll, and one trimmed_cat/shardsper pod deletion. The old path ran a per-index_cat/shardsplus an allocation-explain call per unassigned replica on every poll; feat: add recoverable yellow health gate for rolling restarts #1446 fetches all shards, twice with the policy on.Edge cases:
CheckPodSafeToDelete), so replicas that have nowhere to go still block the restart, as they do today.drainDataNodesis the remedy.Issues Resolved
Resolves #1424. Resolves #1570 (the health gate no longer queries system indices by name, so
system_indices.permission.enabledcannot 403 it; see #1579 (comment)). Supersedes #1446, #1426 and #1579.Check List
make lint) (go vetis clean)Tests added (
opensearch-gateway/services/restart_gate_test.go):soleActiveCopyOnNode: primary with unassigned replica, replica started elsewhere, zero replicas, initializing copy on candidate, relocating source on either side, master-only candidate, search replicas not counted as copies.CheckClusterStatusForRestart(httpmock): settled yellow with drain on/off, allocation throttle lifted first, waits on a missing data node / initializing / delayed / in-flight fetch, red never proceeds, and the health gate never lists shards.PreparePodForDelete: sole copy blocks deletion on a multi-node cluster; single data node is exempt; with drain and 2 data nodes a non-system sole copy blocks deletion while a copy elsewhere proceeds.The full
opensearch-gateway,pkgandcontrollerssuites (including envtest) pass.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.