Skip to content

Commit 04fb916

Browse files
scooby87scooby87
committed
test(opensearch): separate the mount-plan rewrite from the roll onto it
The verify-statefulset-rolled step folded two outcomes with two owners into one assert: that the operator rewrote the nodes template onto the subPath mount plan, and that the StatefulSet finished rolling onto it. When the roll stalled, the timeout could not say which half did not happen. The failure surfaced only as the status mismatch, while the volume projections in the diff read as absent even though they were present -- the projection trap e2e-testing.md documents, where the "actual" side of a Chainsaw diff is a projection and the real cause is the error line above it, not the diff. Split into two asserts so each candidate fails on its own line: the template rewrite on a reconcile-scale budget, the roll on the pod-lifetime one. The nodePool StatefulSet is OnDelete (opensearch-k8s-operator v2.8.0), so the roll is the operator's rolling-restart reconciler's to drive, not the StatefulSet controller's -- it deletes the pod itself, and currentRevision advances only once that reconciler patches it. A failure on the first assert is the operator not writing the volumes; a failure on the second with the first green is the roll not completing, which the rolling-restart reconciler holds until CheckClusterStatusForRestart reports cluster health green. The second assert carries the three volume projections and observedGeneration == generation too, so the roll is pinned to the template the first checked rather than to whatever revision is current when the status converges. The catch now also captures the StatefulSet as YAML and as a describe, and the operator's own log, because updateRevision, currentRevision, generation and updatedReplicas are what separate the two halves -- describe omits them -- and the restart decision is logged in the operator, not in the cluster pods. Refs #4259 Signed-off-by: Alexey Artamonov <[email protected]>
1 parent 6818795 commit 04fb916

1 file changed

Lines changed: 75 additions & 0 deletions

File tree

‎hack/e2e-chainsaw/opensearch/chainsaw-test.yaml‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,23 @@ spec:
3232
apiVersion: opensearch.opster.io/v1
3333
kind: OpenSearchCluster
3434
name: opensearch-test
35+
# The roll's own fields — updateRevision, currentRevision, generation,
36+
# updatedReplicas — are what separate a template that was never rewritten from
37+
# one the operator has not restarted the pod onto (#4259). The nodePool
38+
# StatefulSet is OnDelete (opensearch-k8s-operator v2.8.0, builders/cluster.go
39+
# sets UpdateStrategy.Type OnDelete), so the roll is the operator's
40+
# rolling-restart reconciler's to drive — it deletes the pod itself — not the
41+
# StatefulSet controller's. describe omits those fields, so the StatefulSet is
42+
# captured both ways: get -o yaml carries the revisions and the rendered volume
43+
# list, describe carries the pod template and the controller's events.
44+
- get:
45+
apiVersion: apps/v1
46+
kind: StatefulSet
47+
format: yaml
48+
- describe:
49+
apiVersion: apps/v1
50+
kind: StatefulSet
51+
name: opensearch-test-nodes
3552
# cert-manager records a refusal to issue in the Certificate's conditions.
3653
# Without this an unissued chain shows up only as an absent Secret, and the
3754
# CR then names a Secret that is never going to appear.
@@ -46,6 +63,20 @@ spec:
4663
- podLogs:
4764
selector: opster.io/opensearch-cluster=opensearch-test
4865
tail: 100
66+
# The restart is the operator's rolling-restart reconciler's to start, and its
67+
# last gate holds until CheckClusterStatusForRestart reports cluster health
68+
# green (or yellow under its .opensearch-observability exception) — a client
69+
# that cannot reach the cluster errors out of that gate, it does not hold it.
70+
# Only that gate logs, so a run whose log lacks "Starting rolling restart of
71+
# the StatefulSet" had an earlier gate in the reconciler hold it silently. The
72+
# verdict is in the operator, not the cluster pods above, and it runs in its
73+
# own namespace.
74+
- podLogs:
75+
namespace: cozy-opensearch-operator
76+
selector: control-plane=controller-manager
77+
# tail spans the whole assert window: at one health log per 10s over a 12m
78+
# roll, 100 lines reach back only a third of it and lose a one-shot cause.
79+
tail: 2000
4980
# The external Service is a LoadBalancer, so an address pool with nothing left
5081
# in it holds the HelmRelease short of Ready with nothing in the HelmRelease
5182
# itself naming the cause.
@@ -234,13 +265,57 @@ spec:
234265
# same reason — the replacement pod pulls and starts OpenSearch from scratch.
235266
- name: verify-statefulset-rolled-onto-the-new-mount-plan
236267
try:
268+
# Two asserts, not one, so a timeout names which half did not happen: the
269+
# template rewrite and the roll onto it are separate outcomes with separate
270+
# owners (#4259).
271+
#
272+
# First half: the operator rewrote the template onto the subPath mount plan.
273+
# That is the operator's to do the moment the CR names the cert-manager
274+
# Secrets, which verify-cr-dropped-generate and the certificate steps already
275+
# waited for, so it lands within a reconcile or two rather than a pod
276+
# lifetime. A shorter budget than the roll below, and a failure here is the
277+
# operator not writing the volumes — not a pod that will not start.
278+
- assert:
279+
timeout: 6m
280+
resource:
281+
apiVersion: apps/v1
282+
kind: StatefulSet
283+
metadata:
284+
name: opensearch-test-nodes
285+
(spec.template.spec.volumes[?name == 'http-cert'].secret.secretName):
286+
- opensearch-test-http-server
287+
(spec.template.spec.volumes[?name == 'http-key'].secret.secretName):
288+
- opensearch-test-http-server
289+
(spec.template.spec.volumes[?name == 'http-ca'].secret.secretName):
290+
- opensearch-test.http-ca
291+
# Second half: the operator restarted the pod onto that template and the
292+
# StatefulSet finished rolling. currentRevision only advances here once the
293+
# operator's rolling-restart reconciler patches it (rollingRestart.go), so
294+
# equal revisions plus a ready updated replica is that reconciler having run
295+
# to the end. This half also asserts readyReplicas, so a failure with the
296+
# assert above green is the roll not completing: either the operator never
297+
# restarted the pod — the rolling-restart reconciler holds until
298+
# CheckClusterStatusForRestart reports cluster health green — or a
299+
# replacement was deleted but never came Ready, not a template that was
300+
# never written. Same 12m as the first boot, for the same reason: the
301+
# operator deletes the pod and the replacement pulls and starts OpenSearch
302+
# from scratch.
303+
#
304+
# The three volume projections are carried in here too, so the roll is pinned
305+
# to the template the first half checked and not to whatever revision happens
306+
# to be current when the status converges. The CR moves only
307+
# once in this suite, but the split otherwise lets a rewrite landing after the
308+
# first assert satisfy this half on its own. observedGeneration == generation
309+
# is compared at the resource root because a status-scoped expression cannot
310+
# reach metadata from its own node.
237311
- assert:
238312
timeout: 12m
239313
resource:
240314
apiVersion: apps/v1
241315
kind: StatefulSet
242316
metadata:
243317
name: opensearch-test-nodes
318+
(status.observedGeneration == metadata.generation): true
244319
(spec.template.spec.volumes[?name == 'http-cert'].secret.secretName):
245320
- opensearch-test-http-server
246321
(spec.template.spec.volumes[?name == 'http-key'].secret.secretName):

0 commit comments

Comments
 (0)