Skip to content

Commit 36bbc62

Browse files
committed
test(opensearch): separate the mount-plan rewrite from the roll onto it
The verify-statefulset-rolled step folded two outcomes with two owners into one assert: that the operator rewrote the nodes template onto the subPath mount plan, and that the StatefulSet finished rolling onto it. When the roll stalled, the timeout could not say which half did not happen. The failure surfaced only as the status mismatch, while the volume projections in the diff read as absent even though they were present -- the projection trap e2e-testing.md documents, where the "actual" side of a Chainsaw diff is a projection and the real cause is the error line above it, not the diff. Split into two asserts so each candidate fails on its own line: the template rewrite on a reconcile-scale budget, the roll on the pod-lifetime one. The nodePool StatefulSet is OnDelete (opensearch-k8s-operator v2.8.0), so the roll is the operator's rolling-restart reconciler's to drive, not the StatefulSet controller's -- it deletes the pod itself, and currentRevision advances only once that reconciler patches it. A failure on the first assert is the operator not writing the volumes; a failure on the second with the first green is the roll not completing. The catch captures the StatefulSet as YAML and as a describe and the operator's own log, because updateRevision, currentRevision, generation and updatedReplicas are what separate the two halves -- describe omits them -- and the restart decision is logged in the operator, not in the cluster pods. The roll-gate comment ties each log signature to a v2.8.0 source line: the reconciler builds its OpenSearch client at rollingRestart.go:138, before it logs "Starting rolling restart" at :155, and that client pings the cluster, so the absence of the log line means an early gate held silently or the client ping failed loudly -- both, not just the first. The stall itself is fixed in #4317: a single-replica cluster is left permanently yellow by an index whose replica cannot be assigned. This change only makes the suite name which half of the roll a future failure stalled on. Refs #4259 Signed-off-by: Alexey Artamonov <[email protected]> Assisted-by: LLM
1 parent 301a7fe commit 36bbc62

1 file changed

Lines changed: 89 additions & 0 deletions

File tree

‎hack/e2e-chainsaw/opensearch/chainsaw-test.yaml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,23 @@ spec:
3232
apiVersion: opensearch.opster.io/v1
3333
kind: OpenSearchCluster
3434
name: opensearch-test
35+
# The roll's own fields — updateRevision, currentRevision, generation,
36+
# updatedReplicas — are what separate a template that was never rewritten from
37+
# one the operator has not restarted the pod onto (#4259). The nodePool
38+
# StatefulSet is OnDelete (opensearch-k8s-operator v2.8.0, builders/cluster.go
39+
# sets UpdateStrategy.Type OnDelete), so the roll is the operator's
40+
# rolling-restart reconciler's to drive — it deletes the pod itself — not the
41+
# StatefulSet controller's. describe omits those fields, so the StatefulSet is
42+
# captured both ways: get -o yaml carries the revisions and the rendered volume
43+
# list, describe carries the pod template and the controller's events.
44+
- get:
45+
apiVersion: apps/v1
46+
kind: StatefulSet
47+
format: yaml
48+
- describe:
49+
apiVersion: apps/v1
50+
kind: StatefulSet
51+
name: opensearch-test-nodes
3552
# cert-manager records a refusal to issue in the Certificate's conditions.
3653
# Without this an unissued chain shows up only as an absent Secret, and the
3754
# CR then names a Secret that is never going to appear.
@@ -46,6 +63,37 @@ spec:
4663
- podLogs:
4764
selector: opster.io/opensearch-cluster=opensearch-test
4865
tail: 100
66+
# The restart is the operator's rolling-restart reconciler's to start (see the
67+
# StatefulSet catch note above). It logs "Starting rolling restart of the
68+
# StatefulSet" at rollingRestart.go:155 (opensearch-k8s-operator v2.8.0) once
69+
# the early gates pass — cluster Initialized (:123), every replica ready (:94),
70+
# an update actually pending (:150-154). Just before that line, at :138, it
71+
# builds its OpenSearch client, which pings the cluster
72+
# (NewOsClusterClientFromConfig, os_client.go:104-112) and returns the error
73+
# from the reconcile at :139-141 when the cluster is unreachable. So the line's
74+
# absence reads two ways, not one: an early gate held silently, or that client
75+
# ping failed — loudly, as a reconcile error — before the line was reached. The
76+
# line present and followed by "Couldn't proceed with rolling restart ...
77+
# because <reason>" (:182) means the roll reached the health/pod-delete stage,
78+
# where CheckClusterStatusForRestart (:177) holds it until cluster health is
79+
# green: a yellow cluster no longer holds under an exception — it enables shard
80+
# allocation, waits for every data node to join, waits for shard activity to
81+
# settle, then proceeds — so only red reaches the health hold, and <reason>
82+
# names it "waiting for health to be green" for the mount-plan incident.
83+
# That "Couldn't proceed ... because" line is the health hold's signal, one per
84+
# ~10s requeue and what the tail below is sized for, and
85+
# CheckClusterStatusForRestart is not the last gate: PreparePodForDelete (:195)
86+
# follows it. The verdict is in the operator, not the cluster pods above, and it
87+
# runs in its own namespace.
88+
- podLogs:
89+
namespace: cozy-opensearch-operator
90+
selector: control-plane=controller-manager
91+
# tail spans the whole assert window. The manager logs ~4-5 lines per ~10s
92+
# requeue, not one, so 100 lines reach back only about a third of a 12m roll
93+
# and can lose a one-shot cause. 2000 is effectively everything the container
94+
# has logged, not a sized window — on run 105251129282 it returned all 457
95+
# lines since start.
96+
tail: 2000
4997
# The external Service is a LoadBalancer, so an address pool with nothing left
5098
# in it holds the HelmRelease short of Ready with nothing in the HelmRelease
5199
# itself naming the cause.
@@ -291,13 +339,54 @@ spec:
291339
# same reason — the replacement pod pulls and starts OpenSearch from scratch.
292340
- name: verify-statefulset-rolled-onto-the-new-mount-plan
293341
try:
342+
# Two asserts, not one, so a timeout names which half did not happen: the
343+
# template rewrite and the roll onto it are separate outcomes with separate
344+
# owners (#4259).
345+
#
346+
# First half: the operator rewrote the template onto the subPath mount plan.
347+
# That is the operator's to do the moment the CR names the cert-manager
348+
# Secrets, which verify-cr-dropped-generate and the certificate steps already
349+
# waited for, so it lands within a reconcile or two rather than a pod
350+
# lifetime. A shorter budget than the roll below, and a failure here is the
351+
# operator not writing the volumes — not a pod that will not start.
352+
- assert:
353+
timeout: 6m
354+
resource:
355+
apiVersion: apps/v1
356+
kind: StatefulSet
357+
metadata:
358+
name: opensearch-test-nodes
359+
(spec.template.spec.volumes[?name == 'http-cert'].secret.secretName):
360+
- opensearch-test-http-server
361+
(spec.template.spec.volumes[?name == 'http-key'].secret.secretName):
362+
- opensearch-test-http-server
363+
(spec.template.spec.volumes[?name == 'http-ca'].secret.secretName):
364+
- opensearch-test.http-ca
365+
# Second half: the operator restarted the pod onto that template and the
366+
# StatefulSet finished rolling. currentRevision only advances here once the
367+
# operator's rolling-restart reconciler patches it (see the StatefulSet catch
368+
# note above), so equal revisions plus a ready updated replica is that
369+
# reconciler having run to the end. This half also asserts readyReplicas, so a
370+
# failure with the assert above green is the roll not completing — either the
371+
# operator never restarted the pod (the reconciler holds on the health gate,
372+
# per the podLogs note) or a replacement was deleted but never came Ready — not
373+
# a template that was never written.
374+
#
375+
# The three volume projections are carried in here too, so the roll is pinned
376+
# to the template the first half checked and not to whatever revision happens
377+
# to be current when the status converges. The CR moves only
378+
# once in this suite, but the split otherwise lets a rewrite landing after the
379+
# first assert satisfy this half on its own. observedGeneration == generation
380+
# is compared at the resource root because a status-scoped expression cannot
381+
# reach metadata from its own node.
294382
- assert:
295383
timeout: 12m
296384
resource:
297385
apiVersion: apps/v1
298386
kind: StatefulSet
299387
metadata:
300388
name: opensearch-test-nodes
389+
(status.observedGeneration == metadata.generation): true
301390
(spec.template.spec.volumes[?name == 'http-cert'].secret.secretName):
302391
- opensearch-test-http-server
303392
(spec.template.spec.volumes[?name == 'http-key'].secret.secretName):

0 commit comments

Comments
 (0)