Skip to content

fix(kafka-operator): Kafka alert rules match only the old broker pod and PVC names #5303

Open

Description

The Kafka alert rules in the kafka-operator package only know the broker pool named kafka, so they miss every broker of a cluster with a release-hashed pool. Today the rules are not deployed at all, so this breaks the moment someone turns them back on.

The template that loads them is commented out. packages/system/kafka-operator/templates/alerts.yaml wraps its whole .Files.Glob "alerts/*.yaml" loop in a template comment since #790, and rendering the package gives no PrometheusRule. The rules still sit in packages/system/kafka-operator/alerts/prometheus-rules.yaml.

The rules select broker pods with .+-kafka-[0-9]+ and data PVCs with data(-[0-9]+)?-(.+)-kafka-[0-9]+. Only a cluster migrated from ZooKeeper keeps a broker pool named kafka. Every other cluster gets a broker pool named b-<hash> (see kafka.brokerPoolName in packages/apps/kafka/templates/_versions.tpl). Its pods are <cluster>-b-<hash>-N and its data PVCs are data-0-<cluster>-b-<hash>-N, and neither pattern matches them. The controller pool c-<hash> is not matched either.

With the rules enabled again, this is what happens:

  • KafkaRunningOutOfSpace and ScrapeProblem never fire for such a cluster.
  • KafkaBrokerContainersDown is absent(container_last_seen{container="kafka",pod=~".+-kafka-[0-9]+"}), so it fires all the time on a platform where no cluster has a kafka pool.
  • KafkaContainerRestartedInTheLast5Minutes compares all kafka containers with twice the matched ones, and the ratio is wrong once both pool styles exist.

This was already broken before the Strimzi 1.2 upgrade. The hashed pools came with support for several KRaft clusters per namespace, and the rules file did not change then.

Widening the broker patterns to (kafka|b-[0-9a-f]{8}) would fix the names. Matching on Strimzi labels may work better where the metric carries them. Turning the rules back on is a separate question from #790.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    triage/needs-triageIndicates an issue needs triage by a maintainer

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions