Skip to content

Commit a31a73c

Browse files
lexfreilexfrei
authored
fix(tenant): gate VM-resource HelmReleases on victoria-metrics-operator readiness (#3141)
## What this PR does The tenant `etcd`, `ingress` and `monitoring` HelmReleases create VictoriaMetrics custom resources — `VMPodScrape` directly in `etcd` and `ingress`, and `VMCluster` / `VMAlert` / `VLCluster` / `VMAlertmanager` / `VMServiceScrape` via the `monitoring-system` child that `monitoring` creates. All of these are intercepted by the `victoria-metrics-operator` validating webhook, which is configured with `failurePolicy: Fail`. On a cold install the operator pod can be briefly unavailable on `:9443` (for example a transient API-server blip during startup leaves the webhook backend not yet serving). In that window the tenant releases reconcile, the webhook rejects their VM* resources with `connection refused`, and the releases churn through failed-install → rollback → retry. They carry `install.remediation.retries: -1` and recover eventually, but the churn can outlast a cold-install readiness deadline and surfaces as spurious install failures. This adds a Flux `spec.dependsOn` from each of those three tenant releases to the `victoria-metrics-operator` HelmRelease in `cozy-victoria-metrics-operator`, so they wait for the operator to be Ready before creating any VM* resource. The system monitoring components already gate on the operator the same way (a variant-level `dependsOn` in the platform package sources); this brings the tenant releases to parity. `failurePolicy: Fail` is preserved — validation is unchanged. The tenant releases are sharded (`sharding.fluxcd.io/key`) while the operator release is not. helm-controller resolves `dependsOn` through an uncached API read that bypasses the shard's watch-label-selector cache (since v1.1.0, PR fluxcd/helm-controller#1070), so the cross-shard reference is visible to the tenant shard's controller; this repo ships helm-controller v1.5.0. A helm-unittest suite pins the gate on all three releases and asserts that the tenant releases creating no VM* resources (`seaweedfs`, `gateway`) do not carry it, documenting the intentional narrowness of the gate. ### Release note ```release-note fix(tenant): gate tenant etcd, ingress and monitoring HelmReleases on victoria-metrics-operator readiness so they no longer fail to install when the operator's admission webhook is briefly unavailable on a cold install ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved tenant installation reliability by adding readiness gating: etcd, ingress, and monitoring Helm releases now wait for the Victoria Metrics operator to become Ready before applying dependent resources. * Prevents install/rollback failures caused by temporary admission webhook unavailability during cold starts. * **Tests** * Added Helm-unittest coverage to confirm the dependency gate is present only for templates that create VM* resources (and omitted for templates that don’t). <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 03435dc + 6db5ae9 commit a31a73c

4 files changed

Lines changed: 109 additions & 0 deletions

File tree

‎packages/apps/tenant/templates/etcd.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,15 @@ metadata:
1313
apps.cozystack.io/application.group: apps.cozystack.io
1414
apps.cozystack.io/application.name: etcd
1515
spec:
16+
# This release creates a VMPodScrape resource guarded by the
17+
# victoria-metrics-operator admission webhook (failurePolicy: Fail). On a cold
18+
# install the operator can be briefly unavailable on :9443, so gate on its
19+
# HelmRelease becoming Ready to avoid an install failure → rollback. The system
20+
# monitoring components gate on the operator the same way, via a variant-level
21+
# dependsOn in packages/core/platform/sources/monitoring.yaml.
22+
dependsOn:
23+
- name: victoria-metrics-operator
24+
namespace: cozy-victoria-metrics-operator
1625
chartRef:
1726
kind: ExternalArtifact
1827
name: cozystack-etcd-application-default-etcd

‎packages/apps/tenant/templates/ingress.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,15 @@ metadata:
1313
apps.cozystack.io/application.group: apps.cozystack.io
1414
apps.cozystack.io/application.name: ingress
1515
spec:
16+
# This release creates VMPodScrape resources guarded by the
17+
# victoria-metrics-operator admission webhook (failurePolicy: Fail). On a cold
18+
# install the operator can be briefly unavailable on :9443, so gate on its
19+
# HelmRelease becoming Ready to avoid an install failure → rollback. The system
20+
# monitoring components gate on the operator the same way, via a variant-level
21+
# dependsOn in packages/core/platform/sources/monitoring.yaml.
22+
dependsOn:
23+
- name: victoria-metrics-operator
24+
namespace: cozy-victoria-metrics-operator
1625
chartRef:
1726
kind: ExternalArtifact
1827
name: cozystack-ingress-application-default-ingress

‎packages/apps/tenant/templates/monitoring.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,17 @@ metadata:
1313
apps.cozystack.io/application.group: apps.cozystack.io
1414
apps.cozystack.io/application.name: monitoring
1515
spec:
16+
# This release (via its monitoring-system child) creates VM* custom resources
17+
# guarded by the victoria-metrics-operator admission webhook (failurePolicy:
18+
# Fail). On a cold install the operator can be briefly unavailable on :9443, so
19+
# gate on its HelmRelease becoming Ready to avoid an install failure → rollback.
20+
# Gating the parent keeps the monitoring-system child (which emits the VM* CRs)
21+
# from being created until the operator is Ready. The system monitoring
22+
# components gate on the operator the same way, via a variant-level dependsOn in
23+
# packages/core/platform/sources/monitoring.yaml.
24+
dependsOn:
25+
- name: victoria-metrics-operator
26+
namespace: cozy-victoria-metrics-operator
1627
chartRef:
1728
kind: ExternalArtifact
1829
name: cozystack-monitoring-application-default-monitoring

‎packages/apps/tenant/tests/vmo_webhook_dependson_test.yaml‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
suite: tenant HelmReleases gate on the victoria-metrics-operator
2+
3+
# The victoria-metrics-operator admission webhook is configured with
4+
# failurePolicy: Fail and rejects the VM* custom resources these tenant
5+
# releases create (etcd/ingress: VMPodScrape; monitoring via its
6+
# monitoring-system child: VMCluster/VMAlert/VLCluster/VMAlertmanager/
7+
# VMServiceScrape) whenever the operator pod is not yet serving on :9443.
8+
# On a cold install the operator can be briefly unavailable, so every tenant
9+
# release that creates VM* resources must depend on the operator HelmRelease
10+
# becoming Ready — otherwise the release reconciles too early, the webhook
11+
# rejects with connection refused, the install fails and rolls back. Pin the
12+
# dependsOn so the gate cannot regress.
13+
14+
# Restrict rendering to the gated HelmRelease templates so the chart's
15+
# namespace.yaml `lookup` (which errors under helm-unittest) is not evaluated.
16+
templates:
17+
- templates/etcd.yaml
18+
- templates/ingress.yaml
19+
- templates/monitoring.yaml
20+
- templates/seaweedfs.yaml
21+
- templates/gateway.yaml
22+
23+
release:
24+
name: tenant-test
25+
namespace: tenant-test
26+
27+
set:
28+
_cluster:
29+
expose-ingress: tenant-root
30+
expose-external-ips: "192.0.2.10"
31+
gateway-enabled: "true"
32+
etcd: true
33+
ingress: true
34+
monitoring: true
35+
seaweedfs: true
36+
gateway: true
37+
host: ""
38+
39+
tests:
40+
- it: etcd HR depends on the victoria-metrics-operator
41+
asserts:
42+
- template: templates/etcd.yaml
43+
equal:
44+
path: spec.dependsOn
45+
value:
46+
- name: victoria-metrics-operator
47+
namespace: cozy-victoria-metrics-operator
48+
49+
- it: ingress HR depends on the victoria-metrics-operator
50+
asserts:
51+
- template: templates/ingress.yaml
52+
equal:
53+
path: spec.dependsOn
54+
value:
55+
- name: victoria-metrics-operator
56+
namespace: cozy-victoria-metrics-operator
57+
58+
- it: monitoring HR depends on the victoria-metrics-operator
59+
asserts:
60+
- template: templates/monitoring.yaml
61+
equal:
62+
path: spec.dependsOn
63+
value:
64+
- name: victoria-metrics-operator
65+
namespace: cozy-victoria-metrics-operator
66+
67+
# The gate is intentionally narrow: only the releases that create VM* custom
68+
# resources carry it. Tenant releases that create none (seaweedfs, gateway)
69+
# must not be needlessly blocked on the operator.
70+
- it: seaweedfs HR carries no operator gate
71+
asserts:
72+
- template: templates/seaweedfs.yaml
73+
notExists:
74+
path: spec.dependsOn
75+
76+
- it: gateway HR carries no operator gate
77+
asserts:
78+
- template: templates/gateway.yaml
79+
notExists:
80+
path: spec.dependsOn

0 commit comments

Comments
 (0)