Chart name and version
chart: victoria-metrics-k8s-stack
version: 0.55.2
Describe the bug
During the initial install of the stack chart (using FluxCD), results in the following error:
Helm install failed for release observability/victoria-metrics with chart [email protected]+6420ec535fe3:
2 errors occurred:
* Internal error occurred: failed calling webhook "vmrules.operator.victoriametrics.com": failed to call webhook: Post "https://victoria-metrics-victoria-metrics-operator.observability.svc:9443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s": no endpoints available for service "victoria-metrics-victoria-metrics-operator"
* Internal error occurred: failed calling webhook "vmrules.operator.victoriametrics.com": failed to call webhook: Post "https://victoria-metrics-victoria-metrics-operator.observability.svc:9443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s": no endpoints available for service "victoria-metrics-victoria-metrics-operator"
I notice that during the install, the pods get created, some become healthy, and before all of them can become healthy, this error causes them to get destroyed and the helm install to be considered failed. It happens within a span of maybe ~5 seconds. I suspect the pods can't get healthy fast enough (seems like at least 10s or so before all pods become healthy), and thus the validating webhook config fails because the backends aren't available yet.
I had to comment out almost all of the config (except things like disabling grafana), let it run the initial install (which worked perfectly fine), then uncomment out the config. After the initial deploy, the full config below works perfectly fine (that I can notice).
Seems like there might be an order of operations issue, where you wouldn't be able to re-deploy this in a brand new environment in 1 go, because creation of vmrules can't be done until the VM pods are online and healthy.
Custom values
Please provide only custom values (excluding default ones):
fullnameOverride: stack
victoria-metrics-operator:
env:
- name: VM_VMALERTDEFAULT_CONFIGRELOADERCPU
value: 0
- name: VM_VMAGENTDEFAULT_CONFIGRELOADERCPU
value: 0
- name: VM_VMALERTMANAGER_CONFIGRELOADERCPU
value: 0
operator:
enable_converter_ownership: true # Required to allow VM to remove VM rules it imports if a prometheus rule is deleted
defaultDashboards:
enabled: true
defaultRules:
rules:
groups:
etcd:
create: false
kubernetesSystemControllerManager:
create: false
kubernetesSystemScheduler:
create: false
vmsingle:
spec:
retentionPeriod: "1y"
replicaCount: 1
extraArgs:
maxLabelsPerTimeseries: "50"
storage:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 100Gi
resources:
limits:
memory: 6Gi
requests:
cpu: 500m
memory: 2Gi
alertmanager:
enabled: true
spec:
externalURL: https://alertmanager.${cluster_internal_domain}
useManagedConfig: true
config:
route:
group_by: ["alertname", "job"]
group_interval: 10m
group_wait: 1m
receiver: discord
repeat_interval: 12h
routes:
- receiver: "blackhole"
matchers:
- alertname=~"InfoInhibitor|Watchdog"
- receiver: discord
group_interval: 5m
group_wait: 30s
repeat_interval: 12h
matchers:
- severity=~"warning|critical"
inhibit_rules:
- source_matchers:
- severity = "critical"
target_matchers:
- severity = "warning"
equal: ["alertname", "namespace"]
receivers:
- name: "blackhole"
- name: discord
discord_configs:
- webhook_url_secret:
name: alertmanager-env-vars
key: DISCORD_WEBHOOK_URL
vmalert:
enabled: true
spec:
selectAllByDefault: false # Skip vmalerts for vmlog rules
ruleSelector:
matchExpressions:
- key: vmalert-logs.io/enabled
operator: NotIn
values: ["true"]
ruleNamespaceSelector:
matchExpressions:
- key: somekey
operator: NotIn
values: ["never-used-value"]
extraArgs:
external.url: https://vm.${cluster_internal_domain}
vmagent:
spec:
extraArgs:
promscrape.maxScrapeSize: 50MiB
promscrape.streamParse: "true"
promscrape.dropOriginalLabels: "true"
resources:
limits:
memory: 1Gi
requests:
cpu: 50m
memory: 500Mi
grafana:
enabled: false
prometheus-node-exporter:
vmScrape:
spec:
endpoints:
- port: metrics
relabelConfigs:
- source_labels:
- __meta_kubernetes_endpoint_node_name
target_label: node
kubelet:
vmScrape:
spec:
# drop high cardinality label and useless metrics for cadvisor and kubelet
metricRelabelConfigs:
# Drop less useful container CPU metrics.
- sourceLabels: [__name__]
action: drop
regex: "container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)"
# Drop less useful / always zero container memory metrics.
- sourceLabels: [__name__]
action: drop
regex: "container_memory_(failures_total|mapped_file|swap)"
# Drop less useful container process metrics.
- sourceLabels: [__name__]
action: drop
# regex: 'container_(file_descriptors|tasks_state|threads_max)'
regex: "container_(tasks_state|threads_max)"
# Drop less useful container filesystem metrics.
- sourceLabels: [__name__]
action: drop
regex: "container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)"
# Drop less useful container blkio metrics.
- sourceLabels: [__name__]
action: drop
regex: "container_blkio_device_usage_total"
# Drop container spec metrics that overlap with kube-state-metrics.
- sourceLabels: [__name__]
action: drop
regex: "container_spec.*"
# Drop cgroup metrics with no pod.
- sourceLabels: [id, pod]
action: drop
regex: ".+;"
- action: drop
sourceLabels: [__name__]
regex: prober_probe_duration_seconds_bucket
# Drop high-cardinality labels.
- action: labeldrop
regex: (uid|id|pod_uid|interface)
- action: drop
sourceLabels: [__name__]
regex: (rest_client_request_duration_seconds_bucket|rest_client_request_duration_seconds_sum|rest_client_request_duration_seconds_count)
kubeControllerManager:
enabled: false
kubeEtcd:
enabled: false
kubeScheduler:
enabled: false
kubeProxy:
enabled: false
additionalVictoriaMetricsMap:
dockerhub-rules:
create: true
groups:
- name: dockerhub
rules:
- alert: DockerhubRateLimitRisk
annotations:
summary: Kubernetes cluster Dockerhub rate limit risk
expr: count(time() - container_last_seen{image=~"(docker.io).*",container!=""} < 30) > 100
labels:
severity: critical
oom-rules:
create: true
groups:
- name: oom
rules:
- alert: OomKilled
annotations:
summary: Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }} has been OOMKilled {{ $value }} times in the last 10 minutes.
expr: (kube_pod_container_status_restarts_total - kube_pod_container_status_restarts_total offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m]) == 1
labels:
severity: critical
Chart name and version
chart: victoria-metrics-k8s-stack
version: 0.55.2
Describe the bug
During the initial install of the stack chart (using FluxCD), results in the following error:
I notice that during the install, the pods get created, some become healthy, and before all of them can become healthy, this error causes them to get destroyed and the helm install to be considered failed. It happens within a span of maybe ~5 seconds. I suspect the pods can't get healthy fast enough (seems like at least 10s or so before all pods become healthy), and thus the validating webhook config fails because the backends aren't available yet.
I had to comment out almost all of the config (except things like disabling grafana), let it run the initial install (which worked perfectly fine), then uncomment out the config. After the initial deploy, the full config below works perfectly fine (that I can notice).
Seems like there might be an order of operations issue, where you wouldn't be able to re-deploy this in a brand new environment in 1 go, because creation of vmrules can't be done until the VM pods are online and healthy.
Custom values
Please provide only custom values (excluding default ones):