Skip to content

bug: potential order of operations issue with initial install of stack chart #2288

Description

@lrstanley

Chart name and version
chart: victoria-metrics-k8s-stack
version: 0.55.2

Describe the bug

During the initial install of the stack chart (using FluxCD), results in the following error:

Helm install failed for release observability/victoria-metrics with chart [email protected]+6420ec535fe3:
  2 errors occurred:
    * Internal error occurred: failed calling webhook "vmrules.operator.victoriametrics.com": failed to call webhook: Post "https://victoria-metrics-victoria-metrics-operator.observability.svc:9443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s": no endpoints available for service "victoria-metrics-victoria-metrics-operator"
    * Internal error occurred: failed calling webhook "vmrules.operator.victoriametrics.com": failed to call webhook: Post "https://victoria-metrics-victoria-metrics-operator.observability.svc:9443/validate-operator-victoriametrics-com-v1beta1-vmrule?timeout=10s": no endpoints available for service "victoria-metrics-victoria-metrics-operator"

I notice that during the install, the pods get created, some become healthy, and before all of them can become healthy, this error causes them to get destroyed and the helm install to be considered failed. It happens within a span of maybe ~5 seconds. I suspect the pods can't get healthy fast enough (seems like at least 10s or so before all pods become healthy), and thus the validating webhook config fails because the backends aren't available yet.

I had to comment out almost all of the config (except things like disabling grafana), let it run the initial install (which worked perfectly fine), then uncomment out the config. After the initial deploy, the full config below works perfectly fine (that I can notice).

Seems like there might be an order of operations issue, where you wouldn't be able to re-deploy this in a brand new environment in 1 go, because creation of vmrules can't be done until the VM pods are online and healthy.

Custom values
Please provide only custom values (excluding default ones):

fullnameOverride: stack
victoria-metrics-operator:
  env:
    - name: VM_VMALERTDEFAULT_CONFIGRELOADERCPU
      value: 0
    - name: VM_VMAGENTDEFAULT_CONFIGRELOADERCPU
      value: 0
    - name: VM_VMALERTMANAGER_CONFIGRELOADERCPU
      value: 0
  operator:
    enable_converter_ownership: true # Required to allow VM to remove VM rules it imports if a prometheus rule is deleted
defaultDashboards:
  enabled: true
defaultRules:
  rules:
    groups:
      etcd:
        create: false
      kubernetesSystemControllerManager:
        create: false
      kubernetesSystemScheduler:
        create: false
vmsingle:
  spec:
    retentionPeriod: "1y"
    replicaCount: 1
    extraArgs:
      maxLabelsPerTimeseries: "50"
    storage:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
    resources:
      limits:
        memory: 6Gi
      requests:
        cpu: 500m
        memory: 2Gi
alertmanager:
  enabled: true
  spec:
    externalURL: https://alertmanager.${cluster_internal_domain}
  useManagedConfig: true
  config:
    route:
      group_by: ["alertname", "job"]
      group_interval: 10m
      group_wait: 1m
      receiver: discord
      repeat_interval: 12h
      routes:
        - receiver: "blackhole"
          matchers:
            - alertname=~"InfoInhibitor|Watchdog"
        - receiver: discord
          group_interval: 5m
          group_wait: 30s
          repeat_interval: 12h
          matchers:
            - severity=~"warning|critical"
    inhibit_rules:
      - source_matchers:
          - severity = "critical"
        target_matchers:
          - severity = "warning"
        equal: ["alertname", "namespace"]
    receivers:
      - name: "blackhole"
      - name: discord
        discord_configs:
          - webhook_url_secret:
              name: alertmanager-env-vars
              key: DISCORD_WEBHOOK_URL
vmalert:
  enabled: true
  spec:
    selectAllByDefault: false # Skip vmalerts for vmlog rules
    ruleSelector:
      matchExpressions:
        - key: vmalert-logs.io/enabled
          operator: NotIn
          values: ["true"]
    ruleNamespaceSelector:
      matchExpressions:
        - key: somekey
          operator: NotIn
          values: ["never-used-value"]
    extraArgs:
      external.url: https://vm.${cluster_internal_domain}
vmagent:
  spec:
    extraArgs:
      promscrape.maxScrapeSize: 50MiB
      promscrape.streamParse: "true"
      promscrape.dropOriginalLabels: "true"
    resources:
      limits:
        memory: 1Gi
      requests:
        cpu: 50m
        memory: 500Mi
grafana:
  enabled: false
prometheus-node-exporter:
  vmScrape:
    spec:
      endpoints:
        - port: metrics
          relabelConfigs:
            - source_labels:
                - __meta_kubernetes_endpoint_node_name
              target_label: node
kubelet:
  vmScrape:
    spec:
      # drop high cardinality label and useless metrics for cadvisor and kubelet
      metricRelabelConfigs:
        # Drop less useful container CPU metrics.
        - sourceLabels: [__name__]
          action: drop
          regex: "container_cpu_(cfs_throttled_seconds_total|load_average_10s|system_seconds_total|user_seconds_total)"
        # Drop less useful / always zero container memory metrics.
        - sourceLabels: [__name__]
          action: drop
          regex: "container_memory_(failures_total|mapped_file|swap)"
        # Drop less useful container process metrics.
        - sourceLabels: [__name__]
          action: drop
          # regex: 'container_(file_descriptors|tasks_state|threads_max)'
          regex: "container_(tasks_state|threads_max)"
        # Drop less useful container filesystem metrics.
        - sourceLabels: [__name__]
          action: drop
          regex: "container_fs_(io_current|io_time_seconds_total|io_time_weighted_seconds_total|reads_merged_total|sector_reads_total|sector_writes_total|writes_merged_total)"
        # Drop less useful container blkio metrics.
        - sourceLabels: [__name__]
          action: drop
          regex: "container_blkio_device_usage_total"
        # Drop container spec metrics that overlap with kube-state-metrics.
        - sourceLabels: [__name__]
          action: drop
          regex: "container_spec.*"
        # Drop cgroup metrics with no pod.
        - sourceLabels: [id, pod]
          action: drop
          regex: ".+;"
        - action: drop
          sourceLabels: [__name__]
          regex: prober_probe_duration_seconds_bucket
        # Drop high-cardinality labels.
        - action: labeldrop
          regex: (uid|id|pod_uid|interface)
        - action: drop
          sourceLabels: [__name__]
          regex: (rest_client_request_duration_seconds_bucket|rest_client_request_duration_seconds_sum|rest_client_request_duration_seconds_count)
kubeControllerManager:
  enabled: false
kubeEtcd:
  enabled: false
kubeScheduler:
  enabled: false
kubeProxy:
  enabled: false
additionalVictoriaMetricsMap:
  dockerhub-rules:
    create: true
    groups:
      - name: dockerhub
        rules:
          - alert: DockerhubRateLimitRisk
            annotations:
              summary: Kubernetes cluster Dockerhub rate limit risk
            expr: count(time() - container_last_seen{image=~"(docker.io).*",container!=""} < 30) > 100
            labels:
              severity: critical
  oom-rules:
    create: true
    groups:
      - name: oom
        rules:
          - alert: OomKilled
            annotations:
              summary: Container {{ $labels.container }} in pod {{ $labels.namespace }}/{{ $labels.pod }} has been OOMKilled {{ $value }} times in the last 10 minutes.
            expr: (kube_pod_container_status_restarts_total - kube_pod_container_status_restarts_total offset 10m >= 1) and ignoring (reason) min_over_time(kube_pod_container_status_last_terminated_reason{reason="OOMKilled"}[10m]) == 1
            labels:
              severity: critical

Metadata

Metadata

Assignees

No one assigned

    Labels

    wontfixThis will not be worked on

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions