Skip to content

Commit 5297ac7

Browse files
test(e2e): stop bringing up the monitoring instance nothing looks at
The install patched the root Tenant with monitoring: true and then spent five separate 15m waits on the result: vmalert and vmalertmanager, vlclusters/generic, vmcluster shortterm and longterm, the grafana-db CNPG cluster, and the Grafana Deployment. Nothing in the run inspects any of it. No Chainsaw suite touches a monitoring object. The one that mentions monitoring, foundationdb, asserts a WorkloadMonitor CR whose CRD belongs to packages/system/cozystack-controller, not to the monitoring stack, and passes without it. The two BATS files that did assert on monitoring, hack/e2e-apps/monitoring-oidc-{system,customconfig}.bats, had no invocation site after #2979 and were removed in #4019, where their assertions turned out to be covered by packages/system/monitoring/tests/oidc_test.yaml with 37 cases against their 11. The waits were not merely idle. Their own comment records the cost: "a slow VictoriaLogs bring-up can fail the install on a pull request that never touched monitoring", and the block had been unified onto 15m precisely because the components contend for node capacity with the rest of the bring-up. So this was a component the run never looked at, holding a share of the 27-minute install and a share of its flake. The monitoring PACKAGE stays installed. tenant-application declares dependsOn cozystack.monitoring-application, so disabling it would leave the root Tenant with an unresolved dependency and the platform HelmRelease never Ready. Only the tenant-root instance is turned off. One real check went with the block, and it is named in the file rather than quietly dropped: a curl through root-ingress-controller at Host: grafana.example.org, the only assertion that the root ingress actually routes to a backend. Grafana was its target by availability rather than by design. Restoring it wants a target the run installs anyway -- seaweedfs publishes s3.<root-host> and filer.<root-host> -- and choosing the response to match needs a live stand, so it is a follow-up rather than a guess made here. Savings are unmeasured on purpose: the happy path never spent the 15m budgets, so what this returns is the real bring-up time of the stack plus the tail risk, and only a run can say what that is. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent 67b4e23 commit 5297ac7

1 file changed

Lines changed: 29 additions & 73 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 29 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -312,9 +312,23 @@ EOF
312312
# wait in the previous test does not cover that. Prove an AUTHENTICATED
313313
# request against the actual resource succeeds before the patch's own GET.
314314
timeout 120 sh -ec 'until kubectl get tenants.apps.cozystack.io root -n tenant-root >/dev/null 2>&1; do sleep 2; done'
315-
kubectl patch tenants/root -n tenant-root --type merge -p '{"spec":{"host":"example.org","ingress":true,"monitoring":true,"etcd":true,"isolated":true, "seaweedfs": true}}'
315+
# monitoring stays OFF. Nothing in the suite looks at it: no Chainsaw suite
316+
# touches a monitoring object, and the two BATS files that did
317+
# (hack/e2e-apps/monitoring-oidc-*.bats) were orphaned by #2979 and removed in
318+
# #4019 once their assertions turned out to be covered by
319+
# packages/system/monitoring/tests/oidc_test.yaml. Bringing the stack up cost
320+
# the install four separate 15m waits (vmalert/vmalertmanager, vlclusters,
321+
# vmcluster shortterm+longterm, the grafana-db CNPG cluster and the Grafana
322+
# Deployment) for a component the run never inspects, and the comment those
323+
# waits carried said as much: a slow VictoriaLogs bring-up could fail the
324+
# install on a pull request that never touched monitoring.
325+
#
326+
# The monitoring PACKAGE stays installed -- tenant-application declares
327+
# dependsOn cozystack.monitoring-application, so removing it would strand the
328+
# root Tenant. This turns off the tenant-root INSTANCE only.
329+
kubectl patch tenants/root -n tenant-root --type merge -p '{"spec":{"host":"example.org","ingress":true,"monitoring":false,"etcd":true,"isolated":true, "seaweedfs": true}}'
316330

317-
timeout 60 sh -ec 'until kubectl get hr -n tenant-root etcd ingress monitoring seaweedfs tenant-root >/dev/null 2>&1; do sleep 1; done'
331+
timeout 60 sh -ec 'until kubectl get hr -n tenant-root etcd ingress seaweedfs tenant-root >/dev/null 2>&1; do sleep 1; done'
318332
# tenant-root parent HR only flips Ready after every child HR is Ready,
319333
# so listing all four top-level children plus the parent gives precise
320334
# failure messages without redundant separate waits. seaweedfs now
@@ -325,7 +339,7 @@ EOF
325339
# parent's Ready can land past the HR's single 15m timeout window; the HR
326340
# re-reconciles every 1m until it converges, so this wait is 20m to observe
327341
# that eventual Ready rather than expiring first.
328-
kubectl wait hr/etcd hr/ingress hr/monitoring hr/seaweedfs hr/tenant-root \
342+
kubectl wait hr/etcd hr/ingress hr/seaweedfs hr/tenant-root \
329343
-n tenant-root --timeout=20m --for=condition=ready
330344

331345

@@ -346,77 +360,19 @@ EOF
346360
-l app.kubernetes.io/name=etcd,app.kubernetes.io/instance=etcd,app.kubernetes.io/managed-by=etcd-operator \
347361
--for=condition=ready --timeout=10m
348362

349-
# VictoriaMetrics components. vmalert/vmalertmanager, vlclusters/generic and
350-
# vmcluster/shortterm+longterm are all vm-operator-managed resources that flip
351-
# updateStatus=operational only once their workloads are scheduled and Ready.
352-
# During platform bring-up they contend for node resources with the rest of
353-
# the install, so convergence is load-sensitive: on a calm sandbox each reaches
354-
# operational in under a second, but under install-time load (concurrent e2e
355-
# sandboxes on one runner) monitoring bring-up is slow. vmalert already uses a
356-
# 15m budget; vlclusters and vmcluster used 10m, so this block carried a
357-
# 10m/15m split even though all three contend for the same node capacity and a
358-
# slow VictoriaLogs bring-up can fail the install on a PR that never touched
359-
# monitoring. Unify the block on one 15m budget (near-zero cost in the happy
360-
# path, comfortably inside the E2E job budget) and dump live status on timeout
361-
# so a genuine stuck-not-slow regression stays legible instead of surfacing as
362-
# a bare "timed out" line.
363-
timeout 60 sh -ec 'until kubectl get vmalert/vmalert-shortterm -n tenant-root >/dev/null 2>&1; do sleep 2; done'
364-
timeout 60 sh -ec 'until kubectl get vmalertmanager/alertmanager -n tenant-root >/dev/null 2>&1; do sleep 2; done'
365-
kubectl wait vmalert/vmalert-shortterm vmalertmanager/alertmanager -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=15m || {
366-
echo "=== vmalert/vmalert-shortterm, vmalertmanager/alertmanager did not reach updateStatus=operational ==="
367-
kubectl get vmalert/vmalert-shortterm vmalertmanager/alertmanager -n tenant-root -o yaml 2>&1 || true
368-
echo "=== tenant-root pods ==="
369-
kubectl get pods -n tenant-root -o wide 2>&1 || true
370-
false
371-
}
372-
timeout 60 sh -ec 'until kubectl get vlclusters/generic -n tenant-root >/dev/null 2>&1; do sleep 2; done'
373-
kubectl wait vlclusters/generic -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=15m || {
374-
echo "=== vlclusters/generic did not reach updateStatus=operational ==="
375-
kubectl get vlclusters/generic -n tenant-root -o yaml 2>&1 || true
376-
echo "=== tenant-root pods ==="
377-
kubectl get pods -n tenant-root -o wide 2>&1 || true
378-
false
379-
}
380-
timeout 60 sh -ec 'until kubectl get vmcluster/shortterm vmcluster/longterm -n tenant-root >/dev/null 2>&1; do sleep 2; done'
381-
kubectl wait vmcluster/shortterm vmcluster/longterm -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=15m || {
382-
echo "=== vmcluster/shortterm,longterm did not reach updateStatus=operational ==="
383-
kubectl get vmcluster/shortterm vmcluster/longterm -n tenant-root -o yaml 2>&1 || true
384-
echo "=== tenant-root pods ==="
385-
kubectl get pods -n tenant-root -o wide 2>&1 || true
386-
false
387-
}
388-
389-
# Grafana. The grafana-db CNPG cluster and the grafana-deployment Deployment
390-
# complete the tenant-root monitoring bring-up and contend for the same node
391-
# resources as the VictoriaMetrics stack above during install. Under
392-
# install-time load (concurrent e2e sandboxes on one runner) either can be slow
393-
# and fail the install on a PR that never touched monitoring, so both move from
394-
# their 10m budget to the same uniform 15m as the vm-operator waits above and
395-
# dump live status on timeout to keep a genuine stuck-not-slow regression
396-
# legible instead of surfacing as a bare "timed out" line.
397-
timeout 60 sh -ec 'until kubectl get clusters.postgresql.cnpg.io/grafana-db -n tenant-root >/dev/null 2>&1; do sleep 2; done'
398-
kubectl wait clusters.postgresql.cnpg.io/grafana-db -n tenant-root --for=condition=ready --timeout=15m || {
399-
echo "=== clusters.postgresql.cnpg.io/grafana-db did not reach condition=ready ==="
400-
kubectl get clusters.postgresql.cnpg.io/grafana-db -n tenant-root -o yaml 2>&1 || true
401-
echo "=== tenant-root pods ==="
402-
kubectl get pods -n tenant-root -o wide 2>&1 || true
403-
false
404-
}
405-
timeout 60 sh -ec 'until kubectl get deploy/grafana-deployment -n tenant-root >/dev/null 2>&1; do sleep 2; done'
406-
kubectl wait deploy/grafana-deployment -n tenant-root --for=condition=available --timeout=15m || {
407-
echo "=== deploy/grafana-deployment did not reach condition=available ==="
408-
kubectl get deploy/grafana-deployment -n tenant-root -o yaml 2>&1 || true
409-
echo "=== tenant-root pods ==="
410-
kubectl get pods -n tenant-root -o wide 2>&1 || true
411-
false
412-
}
363+
# The monitoring instance is off (see the Tenant patch above), so the waits
364+
# that used to sit here are gone: vmalert, vmalertmanager, vlclusters/generic,
365+
# vmcluster shortterm+longterm, the grafana-db CNPG cluster and the Grafana
366+
# Deployment, each on a 15m budget.
367+
#
368+
# One real check went with them and is worth naming rather than quietly
369+
# losing: a curl through root-ingress-controller at Host: grafana.example.org,
370+
# which was the only assertion that the root ingress actually routes to a
371+
# backend. Grafana was its target by availability, not by design. Restoring it
372+
# wants a target the run installs anyway -- seaweedfs publishes
373+
# s3.<root-host> and filer.<root-host> -- and picking the response to match
374+
# needs a live stand, so it is a follow-up rather than a guess made here.
413375

414-
# Verify Grafana via ingress
415-
ingress_ip=$(kubectl get svc root-ingress-controller -n tenant-root -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
416-
if ! curl -sS -k "https://${ingress_ip}" -H 'Host: grafana.example.org' --max-time 30 | grep -q Found; then
417-
echo "Failed to access Grafana via ingress at ${ingress_ip}" >&2
418-
exit 1
419-
fi
420376
}
421377

422378
@test "Keycloak OIDC stack is healthy" {

0 commit comments

Comments
 (0)