feat(monitoring): scrape KubeVirt VMI metrics from virt-handler - #4219

Thanks for chasing this down, Andrei Kvapil (@kvaps). The diagnosis is right: no Monitoring receives kubevirt_vmi_* today. The tenant-subtree semantics are carefully worked out too. Since this is still a draft, I'd like to raise a placement concern before more goes into it.

Core monitoring shouldn't know about pluggable components

system/monitoring is the chart every Monitoring application renders. With this PR it hard-codes KubeVirt's namespace, pod label, port and TLS quirks. The test I'd apply: if the FoundationDB app, or an app in an external repository, wanted its metrics collected, would its author have to patch system/monitoring? Today the answer is no. Each package ships its own scrape objects and rules, and the platform picks them up by convention:

• apps: kafka (templates/podscrape.yaml), redis and valkey (templates/servicescrape.yaml), clickhouse (keeper scrape in templates/chkeeper.yaml), postgres (CNPG enablePodMonitor: true, converted by vm-operator);
• tenant modules: extra/etcd (scrape plus rules), extra/ingress (nginx-scrape.yaml);
• system components: etcd-operator, linstor, kubeovn-plunger, opencost, and rules in kafka-operator, postgres-operator, keda, seaweedfs, piraeus-operator;
• system/monitoring itself scrapes only its own VictoriaMetrics components (vmcluster-scrape.yaml, vmalert-scrape.yaml).

The selection side is generic. A scrape object in a tenant namespace is picked up by that tenant's vmagent through the namespace.cozystack.io/monitoring namespace label, and the cluster-wide vmagent (selectAllByDefault: true) picks up everything. Neither needs to know what it is scraping. KubeVirt should stay on that contract rather than become the first component whose scrape lives inside the monitoring chart.

KubeVirt used to ship its own scrape

system/kubevirt-operator shipped a ServiceMonitor on KubeVirt's kubevirt-prometheus-metrics Service from #573 onwards. The v1.8.2 refresh (6af0bf7) dropped it, because the update recipe at the time ran rm -rf templates. The alert rule came back a few days later as templates/prometheusrule.yaml (7106f07), but the scrape did not. So what this PR fixes is a regression, and the scrape's natural home is next to the rule it feeds.

That rule is worth looking at in the same change:

• It matches exported_namespace, which only exists when the scrape runs without honorLabels. With this PR's honorLabels: true it still never fires.
• kubevirt_vm_info comes from virt-controller, which this PR doesn't scrape.
• The kubevirt/kubevirt-control-plane dashboard in dashboards-infra.list also reads virt-controller and virt-api series, such as the workqueue metrics. A virt-handler-only scrape leaves it empty.

Two correctness issues with the current shape

• Duplicates in root storage. The cluster-wide vmagent has selectAllByDefault: true and no namespace selector, so it also selects every tenant's copy of kubevirt-virt-handler. With T tenant Monitorings it scrapes each virt-handler T+1 times. Root storage gets each tenant's VMI series once from the tenant-root copy and again from the tenant's own, and once more per nesting level. vm-operator's default job is per scrape object (<namespace>/<name>, as in prometheus-operator), so the copies shouldn't dedup and root-side sums would double-count; I haven't confirmed that on a live cluster. cozystack/community#94 reads VM series from tenant-root and assumes they are scraped once.
• The cozy-monitoring release. The monitoring source also installs this chart as monitoring in cozy-monitoring, which vmagent.yaml and dashboards.yaml treat as root-equivalent. There this template renders keep with cozy-monitoring(-.*)?, which drops every series.

A smaller point: tenant-a(-.*)? also keeps the series of a sub-tenant that has its own Monitoring. A tenant vmagent otherwise skips such a namespace, so the filter doesn't match the label semantics described in the template comment.

Proposal

1. Move the scrape to system/kubevirt-operator, next to prometheusrule.yaml. Make it a VMServiceScrape, or a ServiceMonitor since the converter is on, selecting prometheus.kubevirt.io: "true" in cozy-kubevirt: port metrics, https, insecureSkipVerify as upstream's own ServiceMonitor does, and honorLabels: true. That keeps namespace and name on the VMI, which is what dashboards/vm/virtual-machines.json already queries. It covers virt-handler, virt-controller and virt-api in one object.

2. Fix the alerts in the same move. Switch the PrometheusRule from exported_namespace to namespace, and pin the scrape in tests/vendored_manifest_test.yaml the way prometheusrule.yaml already is, so a future refresh can't drop it again.

3. Drop the per-tenant fan-out from this PR. The cluster-wide vmagent then scrapes each virt-handler exactly once into tenant-root. Scoping a shared platform exporter's series to a tenant is a read-path problem, and cozystack/community#94 designs that for exactly these series, with a forced namespace matcher in front of tenant-root's vmselect.

4. If tenant stores must hold these series, for example for tenant vmalert rules on VM metrics, make that a generic monitoring capability rather than a KubeVirt one. For instance, a tenant Monitoring could also select scrape objects that carry an opt-in label from platform namespaces, and drop any series whose namespace is a tenant namespace outside its subtree. KubeVirt would opt in by labelling its own scrape. Two caveats:

   • vm-operator ANDs a VMAgent's selector and namespace selector per kind, so the opt-in channel needs its own kind (for example VMScrapeConfig through scrapeConfigSelector) or a second agent.
   • Every tenant vmagent would pull each virt-handler's full payload. virt-handler gathers domain stats from the virt-launchers at scrape time, so the cost grows with tenants × nodes.

   That seems worth a short community proposal rather than this PR. The tenant chart's egress-virt-handler policy (#2199) couples core to KubeVirt in the same way. If the fan-out does happen, the allow rule could move into the KubeVirt package as an ingress rule on virt-handler.

Writing the contract down

As far as I can find, the "ship your own scrape objects and rules" convention isn't documented in this repository. Writing it down would make the FoundationDB question above easy to answer for the next contributor. It would cover:

• where scrape objects and rules live;
• that one in a tenant namespace reaches both the tenant's Monitoring and root, and one in a system namespace reaches root only;
• that series should keep namespace naming the workload's namespace.

Dashboards are the one place a package still has to edit core: dashboards.list and dashboards-infra.list, plus the JSON under dashboards/ baked into the grafana-dashboards image. That's a separate follow-up, but it belongs on the same list.

Happy to help with the relocation if useful.