Skip to content

fix: let rolling restarts and upgrades proceed on a permanently yellow cluster - #1569

Merged
synhershko merged 3 commits into
opensearch-project:mainfrom
data-ops-pulse:fix/1424-yellow-restart-deadlock
Sep 18, 2026
Merged

synhershko merged 3 commits into
opensearch-project:mainfrom
data-ops-pulse:fix/1424-yellow-restart-deadlock

Conversation

@synhershko

@synhershko synhershko commented Sep 11, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Rolling restarts and version upgrades stall forever on a cluster that is permanently yellow, e.g. a single data node with number_of_replicas >= 1, or more replicas than the other data nodes can hold (the same applies to allocation awareness, allocation filters and disk watermarks). CheckClusterStatusForRestart only lets a yellow cluster through when every unassigned replica is blocked by the node_version decider, which only happens mid-upgrade. In every other case it waits for green, which never comes.

This PR fixes it with two small rules in os_data_service.go and no new API surface:

  1. Proceed on a settled yellow cluster. When health is yellow, the operator first makes sure transient cluster.routing.allocation.enable is all (lifting its own primaries throttle, in both drain modes). It then proceeds only once every expected data node has joined (number_of_data_nodes ≥ the data node count from the node pools) and there is no shard activity: no relocating, initializing or delayed-unassigned shards and no in-flight shard fetches. With all data nodes present, allocation enabled and nothing moving, the allocator has already decided the remaining replicas cannot be placed, so waiting will never turn the cluster green. Yellow already guarantees every primary is active. The data node check matters because a node that hasn't rejoined leaves no trace in the activity counters once its delayed allocation expires, and a pod can report Ready before it has joined.
  2. Vet the specific pod before every restart. PreparePodForDelete now refuses to delete a pod that holds the only active (STARTED/RELOCATING) copy of a shard that is supposed to have other copies, in both drain modes. With drain, this runs after the node is excluded and its drain checks pass, which matters for 2-data-node clusters, where the drain path only waits for system-index primaries. A restart therefore never takes a shard below one active copy, the same floor a green cluster provides. Search replicas are not counted as copies, since they can't be promoted. A cluster with a single data node is exempt: there is nowhere else a copy could live, which is exactly what the operator already does for a green single-node cluster with zero replicas.

CheckClusterRestartOnYellow is removed; its node_version case is covered by the rules above.

Why this approach instead of #1446 (and #1426)

  • No new CRD field. The behaviour is correct by default, so there is no opt-in healthGatePolicy to document, migrate or get wrong. Users hitting Add policy to continue rolling restart on safely recoverable yellow health #1424 are fixed without changing their specs.
  • Safe per node, not just per cluster. feat: add recoverable yellow health gate for rolling restarts #1446 checks cluster-wide health only. With drainDataNodes: false it can restart the node holding the only started copy of a shard whose replica is unassigned, taking the index red for the whole restart, and losing data on emptyDir storage. Here, that pod is never deleted.
  • Allocation throttle handled in both drain modes. In feat: add recoverable yellow health gate for rolling restarts #1446 the reset to all only runs when drain is off, so replicas held back by a primaries/none setting can be misclassified as recoverable.
  • Cheaper. The yellow gate is one _cluster/health plus one settings GET per poll, and one trimmed _cat/shards per pod deletion. The old path ran a per-index _cat/shards plus an allocation-explain call per unassigned replica on every poll; feat: add recoverable yellow health gate for rolling restarts #1446 fetches all shards, twice with the policy on.
  • No extra status writes, so no additional reconcile churn while the gate is waiting.
  • Small: touches only the gateway service, the health response type and the two gate call sites.

Edge cases:

  • Red: unchanged, never proceeds.
  • A data node missing from the cluster (e.g. a restarted pod that is Ready but hasn't rejoined): the yellow gate waits.
  • Drain on, more than 2 data nodes: the node is emptied before deletion (CheckPodSafeToDelete), so replicas that have nowhere to go still block the restart, as they do today.
  • Drain on, 2 data nodes: after system-index primaries are drained, the sole-copy check still blocks deletion if the node holds the only active copy of any other shard.
  • Dedicated master nodes hold no shards, so they are no longer blocked by a permanently yellow cluster.
  • Awareness with an empty zone and drain off: every primary is a sole copy, so the restart blocks with a clear log line; enabling drainDataNodes is the remedy.
  • Single data node with drain on still cannot drain (nowhere to move shards); that is pre-existing behaviour on green clusters too.

Issues Resolved

Resolves #1424. Resolves #1570 (the health gate no longer queries system indices by name, so system_indices.permission.enabled cannot 403 it; see #1579 (comment)). Supersedes #1446, #1426 and #1579.

Check List

  • Commits are signed per the DCO using --signoff
  • Unittest added for the new/changed functionality and all unit tests are successful
  • Customer-visible features documented
  • No linter warnings (make lint) (go vet is clean)

Tests added (opensearch-gateway/services/restart_gate_test.go):

  • soleActiveCopyOnNode: primary with unassigned replica, replica started elsewhere, zero replicas, initializing copy on candidate, relocating source on either side, master-only candidate, search replicas not counted as copies.
  • CheckClusterStatusForRestart (httpmock): settled yellow with drain on/off, allocation throttle lifted first, waits on a missing data node / initializing / delayed / in-flight fetch, red never proceeds, and the health gate never lists shards.
  • PreparePodForDelete: sole copy blocks deletion on a multi-node cluster; single data node is exempt; with drain and 2 data nodes a non-system sole copy blocks deletion while a copy elsewhere proceeds.

The full opensearch-gateway, pkg and controllers suites (including envtest) pass.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@robertpaschedag

Copy link
Copy Markdown
Contributor

@synhershko as mentioned in my PR, the test with your branch also worked for me

@josedev-union josedev-union left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm.
btw, i have one thing to clarify here
DataNodesCount only counts the literal data role while number_of_data_nodes also includes tiered roles such as hot, warm, etc.
Tiered node pools will be skipped for the wait check (tiered only roles which has no data in the role list)

@synhershko

Copy link
Copy Markdown
Collaborator Author

Good catch, and it is worse than a skipped wait - thanks for spotting it. Fixed in 5c0c2db (also merged current main).

You are right that DataNodesCount only matched the literal data role while number_of_data_nodes counts every node whose role can contain data. Two consequences, both from this PR:

  1. The new health.NumberOfDataNodes < dataNodes gate compares OpenSearch's count against ours. With any tiered pool ours is too low, the comparison is trivially satisfied, and "waiting for all data nodes to join the cluster" never fires - so a restart could proceed with a data node still absent, which is the case that gate exists for.
  2. Worse, on a cluster whose data pools are all tiered (no literal data anywhere), DataNodesCount returns 0, so nodeCount > 1 is false and the new sole-active-copy guard in PreparePodForDelete is skipped entirely. That is the guard that stops us taking down the last active copy of a shard.

I fixed it in HasDataRole rather than at the call site, since all five callers mean "this pool holds data" and were all wrong in the same direction:

var dataRoles = []string{
	"data", "data_content", "data_hot", "data_warm", "data_cold", "data_frozen", "search", "warm",
}

That also corrects findNextNodePoolForUpgrade (a tiered pool was ordered as a non-data pool), the mid-upgrade env-var freeze in cluster.go, and the emptyDir data-loss accounting, where tiered pods are just as exposed to losing their volume as data ones.

The role list is the part worth a second pair of eyes. I took it as every DiscoveryNodeRole with canContainData() == true, which is what getDataNodes() and therefore number_of_data_nodes uses, and restricted it to roles already in the operator's ValidNodeRoles. If search or warm is not counted there in some version, the error flips direction and becomes a hang rather than a skipped wait, so please say if either looks wrong to you.

Added tests covering each tiered role and a negative case for ingest/ml/transform/remote_cluster_client/master. Full suite passes locally.

…w cluster

A cluster whose replicas can never be assigned (e.g. number_of_replicas
higher than the number of other data nodes) stays yellow forever, and the
restart health gate only let yellow through when every unassigned replica
was blocked by the node_version decider mid-upgrade. Rolling restarts and
version upgrades therefore stalled indefinitely.

The health gate now accepts a yellow cluster once shard allocation is
enabled and nothing is moving (no relocating, initializing or delayed
shards and no in-flight shard fetches). At that point the allocator has
already decided the remaining replicas cannot be placed. The operator's own
"primaries" allocation throttle is lifted first, for both drain modes, so it
can't be mistaken for an unassignable replica.

To keep this safe, PreparePodForDelete (without drain) now refuses to
delete a pod holding the only active copy of a shard that should have
replicas, so a restart never takes a shard below one active copy. A cluster
with a single data node is exempt, as there is nowhere else a copy could
live (same as a green cluster with zero replicas).

CheckClusterRestartOnYellow is removed; its node_version case is covered by
the new rules.

Resolves opensearch-project#1424

Signed-off-by: Itamar Syn-Hershko <[email protected]>
…s on yellow

- Run the sole-copy check for drain mode too: with 2 data nodes the drain
  path only checks system-index primaries, so a non-system shard whose only
  active copy is on the node could still be restarted.
- On a yellow cluster, only proceed once number_of_data_nodes reaches the
  expected data node count. A node that hasn't rejoined leaves no trace in
  the activity counters once its delayed allocation expires, and a pod can
  report Ready before it joins.
- Ignore search replicas when counting copies; they can't be promoted.
- Scope the docs note to drainDataNodes: false.

Signed-off-by: Itamar Syn-Hershko <[email protected]>
HasDataRole only matched the literal "data" role, but OpenSearch counts
any node whose role can contain data. A pool with only data_content,
data_hot/warm/cold/frozen, search or warm was therefore invisible to every
caller that asks "does this pool hold data".

The new yellow-cluster gate made that visible: it compares
health.number_of_data_nodes, which includes those roles, against
DataNodesCount, which did not. On a cluster with tiered pools the count was
always too low, so "waiting for all data nodes to join" never fired and a
restart could proceed with a data node still missing. The same undercount
skipped the sole-active-copy guard on a tiered-only cluster, where
DataNodesCount returned 0 and the nodeCount > 1 check was false.

Widening the helper fixes the other callers too: upgrade ordering no longer
treats a tiered pool as a non-data pool, cluster reconcile no longer lets
its env vars change mid-upgrade, and the emptyDir data-loss accounting now
counts tiered pods, which are just as exposed to losing their volume.

Signed-off-by: Itamar Syn-Hershko <[email protected]>
@josedev-union
josedev-union force-pushed the fix/1424-yellow-restart-deadlock branch from 5c0c2db to c2ed9e4 Compare September 18, 2026 04:23

@josedev-union josedev-union left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@synhershko
synhershko merged commit 1bd5b8a into opensearch-project:main Sep 18, 2026
24 of 25 checks passed
@github-project-automation github-project-automation Bot moved this from 👀 In Review to ✅ Done in Engineering Effectiveness Board Sep 18, 2026
@synhershko
synhershko deleted the fix/1424-yellow-restart-deadlock branch September 18, 2026 08:04
lexfrei added a commit to cozystack/cozystack that referenced this pull request Sep 18, 2026
…4317)

## What this PR does

A single-replica OpenSearch release stops rolling once any index on it
asks for a replica. The copy has nowhere to go on one data node, so the
cluster stays yellow for good. The operator's restart gate at v2.8.0
opens on green, or on yellow only when `.opensearch-observability` is
the yellow index, and at OpenSearch 2.11.1 that index is
`auto_expand_replicas: 0-2`, green on one node. So the gate never opens.

Every release gets such an index by itself. The operator sets
`plugins.security.audit.type: internal_opensearch`, and the first
audited event creates a daily `security-auditlog-*` index with one
replica. After that nothing that needs a restart is rolled out, a
version upgrade or the move of the HTTP listener onto the cert-manager
chain included.

This is what timed out the `opensearch` e2e suite on some runs. Both
failed runs I read show the same thing in the OpenSearch log: the
bootstrap node leaves, `Connection reset` on the transport layer, and
`[security-auditlog-<date>] creating index ... shards [1]/[1]` within 40
ms. The security plugin audits that exception
(`SecuritySSLNettyTransport` hands it to `AuditLogSslExceptionHandler`),
so the reset is what creates the index. Then the operator logs `waiting
for health to be green` every ten seconds until the assert gives up. Not
sure why the green runs had no audit index before the roll, a green run
keeps no snapshot.

The fix carries opensearch-project/opensearch-k8s-operator#1569 back to
v2.8.0 as a second patch on the in-tree operator image, with the
upstream tests. The new gate lets a yellow cluster through once
allocation is enabled, every data node has joined and no shard is
moving, and `PreparePodForDelete` refuses to restart a node that holds
the only active copy of a shard meant to have others. A lone data node
is exempt. I left out the third commit of that PR (tiered data roles in
`HasDataRole`), the chart renders only the `data` role.

A bump would not help: the only operator build after v2.8.0 is
`3.0.0-alpha` from January, and its gate holds this case the same way.
The patch goes away on the first release that includes #1569.

The suite now creates an index with an unassignable replica before it
moves the listener, so the roll always runs against a yellow cluster.
The catch block also reads cluster health and the shard table from the
node, because the operator logs that health held a restart back and
never what the health was.

The image digest in `values.yaml` stays as it is.

Closes #4259

### Screenshots

Not a UI change.

### Downstream repositories

Walked the trigger map against the diff: an e2e suite, the operator
image's Dockerfile header and patch directory, one comment in the
operator package's `values.yaml`. No values key, schema, make target or
documented workaround changes.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(opensearch-operator): a single-replica OpenSearch release no longer gets stuck ahead of every rolling restart once an index with a replica, such as the security audit log or any index created with default settings, leaves the cluster permanently yellow
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Improved rolling restart handling for clusters that remain yellow,
allowing restarts to proceed safely once shard activity has settled.
- Prevented restart operations from becoming blocked by temporary
shard-allocation settings.
- Added safeguards to ensure data is not disrupted while replicas are
relocating or initializing.

- **Diagnostics**
- Added automatic cluster health and shard diagnostics when
restart-related test operations time out or encounter issues.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
lexfrei added a commit to cozystack/cozystack that referenced this pull request Sep 21, 2026
…it (#4279)

## What this is

`Refs #4259`. A diagnosis of the opensearch suite timeout plus the one
in-repo change it warrants: making the failing step say which of its two
candidates actually happened. It does **not** claim to fix the root
cause — see "What this does not do".

## What the run actually showed

The `verify-statefulset-rolled-onto-the-new-mount-plan` assert erred
with exactly two field failures, both under `status`: `(currentRevision
== updateRevision): false` and `updatedReplicas: field not found`. No
volume field appears in the error list, and `readyReplicas: 1` shows as
an unchanged line, so the pod was Ready at the deadline and nothing had
replaced it. Per e2e-testing.md ("Reading a failure diff: the actual
side is a projection"), the real cause is the error line, not the diff —
and a Chainsaw diff projects the live object onto the expected fields,
so the three `http-ca` / `http-cert` / `http-key` lines showing in the
diff are the expected tree, not proof the volumes are absent. Read that
way, the volume assertions matched: the three volumes were present on
the live template. The "volumes missing" reading is the projection trap
the doc warns against.

## Which candidate

Candidate (b): the template was rewritten onto the subPath mount plan,
and the operator has not restarted the pod onto it. This is consistent
with the source, traced against opensearch-k8s-operator v2.8.0 as
vendored:

- The chart renders the CR correctly — `generate` dropped, `secret` and
`caSecret` named — proven by
`packages/apps/opensearch/tests/tls_upgrade_test.yaml`.
- Given that CR, `pkg/reconcilers/tls.go` `handleHttp` appends exactly
the three volumes the step asserts (`http-ca` from
`opensearch-test.http-ca`, `http-key` and `http-cert` from
`opensearch-test-http-server`), and `pkg/reconcilers/cluster.go`
`reconcileNodeStatefulSet` writes that template onto the StatefulSet.
Three branches return ahead of that write — no node-pool config hash,
the `opensearch.role` recreate, and parallel recovery — but the
parallel-recovery gate is `ReadyReplicas < replicas-1`, unreachable at
one replica, so the write happens. The reconcilers that run ahead of the
cluster reconciler do not stop it either: the TLS reconciler runs first
and returns the admin-certificate path's result, empty in the ordinary
case, and `securityconfig.Reconcile` short-circuits to nil when the
security-config checksum is unchanged, which the move does not change.
So the operator does write the volumes.
- The nodePool StatefulSet is `OnDelete` (`pkg/builders/cluster.go` sets
`UpdateStrategy.Type: OnDelete` alongside `PodManagementPolicy:
OrderedReady`), so the StatefulSet controller never deletes the pod to
apply a template change: `podManagementPolicy` orders creation and
deletion during scaling, `updateStrategy` decides the update. The roll
is the operator's rolling-restart reconciler's to drive
(`pkg/reconcilers/rollingRestart.go`) — it deletes the pod itself. It
needs `status.Initialized`, it needs `ReadyReplicas == replicas` (a
Ready pod is a precondition, not the blocker), then it opens a client to
the OpenSearch API and calls `CheckClusterStatusForRestart`, which holds
the restart while the cluster is red; on yellow the gate now enables
shard allocation, waits for every data node to join, waits for shard
activity to settle, then proceeds (#4317 deleted the old
`.opensearch-observability` carve-out, and this suite builds the
operator with that patch). Any of those gates can sit indefinitely.
`status.currentRevision` never advances by itself under `OnDelete`; the
operator patches it once `updatedReplicas == replicas`, so the second
assert's own condition cannot go true unless that reconciler ran to the
end.
- The pod's startup probe is a TCP check on 9200
(`builders/cluster.go`), so "connection refused" is OpenSearch not
listening, not a probe-scheme mismatch. The plaintext `GET / HTTP/1.1`
from `curl/8.5.0` on the TLS listener is neither the startup/liveness
probe (TCP) nor the readiness probe (`curl -k https://localhost:9200`,
localhost, HTTPS) — it is external traffic to the LoadBalancer address,
a red herring for the stall.

## What this changes

The step now runs two asserts instead of one. The first waits for the
operator to rewrite the template onto the three volumes, on a
reconcile-scale budget (6m) — a failure here is the operator not writing
them (candidate a). The second waits for the roll to complete onto that
template, on the pod-lifetime budget (12m) — a failure here with the
first green is the roll not completing (candidate b): the operator never
restarting the pod, which the rolling-restart reconciler holds while
`CheckClusterStatusForRestart` reports the cluster red — on yellow the
gate enables shard allocation, waits for every data node to join, waits
for shard activity to settle, then proceeds — or a replacement that was
deleted but never came Ready. The second assert carries the three volume
projections and `observedGeneration == metadata.generation` as well, so
the roll is pinned to the template the first assert checked rather than
to whatever revision is current when the status converges. The suite
`catch` also captures the StatefulSet as YAML and as a describe, and the
operator's own log, because `updateRevision`, `currentRevision`,
`generation` and `updatedReplicas` are what separate the two halves —
describe alone omits them — and the restart decision is logged in the
operator, not in the cluster pods. The reconciler logs `Starting rolling
restart of the StatefulSet` (`rollingRestart.go:155`) once its early
gates (`Initialized`, ready-replica, update-pending) pass; it builds its
OpenSearch client just before, at `:138`, and that client pings the
cluster — so a run whose log lacks `Starting rolling restart` either had
an earlier gate hold it silently or had the client ping fail loudly, not
only the former, while that line present followed by a `Couldn't proceed
with rolling restart ... because <reason>` line names a health-stage
hold (the health gate is `CheckClusterStatusForRestart`, and it is not
the last: `PreparePodForDelete` follows it). The operator-log `tail` was
raised so it spans the whole assert window rather than a third of it.

## What this does not do

It does not fix the stall; #4317 does, and is already merged. The run
that captured this (`Run E2E tests`, job 104576486900) shows the
rolling-restart reconciler holding on health: the manager logged
`Starting rolling restart of the StatefulSet opensearch-test-nodes`
(`rollingRestart.go:155`, printed only after the `Initialized` and
ready-replica gates passed) and then, every ten seconds to the deadline,
`Couldn't proceed with rolling restart ... waiting for health to be
green` (`os_data_service.go:180`), which is reached only after
`GetFlatClusterSettings` returned 2xx (`os_client.go:224` makes anything
else an error). So the operator reached the cluster and the cluster
answered; health held the restart. #4259 established why, and the CA
move plays no part: on a single data node the bootstrap node leaving is
audited into a fresh `security-auditlog-*` index with one replica, that
replica has nowhere to go, the cluster stays yellow for good, and the
v2.8.0 restart gate opens only on green (or on a yellow
`.opensearch-observability`) so it never opens. #4317 backports
opensearch-project/opensearch-k8s-operator#1569 to fix that gate; this
PR is only the diagnostic split that names which half of the roll a
failure is. The split content itself is not the failure: on `49f3f059f`
both asserts passed, the second after 1m47s, and this change touches
only the step's comments and the operator-log tail, not the assertions.
With this change in place, a future occurrence names the half, and the
catch carries the roll's own fields and the operator log, so the
crust-gather can be read against a specific claim rather than an
ambiguous one.

### Release note

```release-note
NONE
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Tests**
* Improved OpenSearch StatefulSet failure diagnostics by capturing YAML,
detailed resource descriptions, and the latest operator pod logs.
* Expanded rolling-restart verification to check volume projections,
template updates, rollout readiness, observed and metadata generations,
and revision consistency.
* Separated template validation from rollout completion checks with
dedicated six- and twelve-minute time limits.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: ✅ Done

3 participants