Guest cluster monitoringAgents send metrics to an unresolvable target (vmagent.remoteWrite.url vs .urls) #5258

What happened

A managed Kubernetes (guest) cluster in a tenant, with the monitoringAgents addon enabled, ships logs to its ancestor tenant correctly but metrics never arrive. The guest's vmagent tries to remote-write to vminsert-shortterm.cozy-monitoring.svc / vminsert-longterm.cozy-monitoring.svc, which do not exist from inside the guest:

couldn't send a block ... Post "http://vminsert-shortterm.cozy-monitoring.svc:8480/insert/0/prometheus":
  dial tcp4: lookup vminsert-shortterm.cozy-monitoring.svc on 10.x.0.10:53: no such host

cozy-monitoring is a host-platform namespace; a guest cluster cannot resolve or reach it, and it is not where the tenant's vminsert lives.

Root cause

packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml sets the vmagent target under the key vmagent.remoteWrite.url (singular string):

    vmagent:
      remoteWrite:
        url: http://vminsert-shortterm.{{ $targetTenant }}.svc.{{ $clusterDomain }}:8480/insert/0/prometheus

But the cozy-monitoring-agents chart reads vmagent.remoteWrite.urls (plural list). packages/system/monitoring-agents/templates/vmagent.yaml:

  remoteWrite:
  {{- range .Values.vmagent.remoteWrite.urls }}
  - url: {{ tpl . $ | quote }}
  {{- end }}

and packages/system/monitoring-agents/values.yaml defaults those urls to global.target, which is cozy-monitoring:

  remoteWrite:
    urls:
      - http://vminsert-shortterm.{{ .Values.global.target }}.svc:8480/insert/0/prometheus
      - http://vminsert-longterm.{{ .Values.global.target }}.svc:8480/insert/0/prometheus

So the app's vmagent.remoteWrite.url override is silently ignored (wrong key), and the chart falls back to the cozy-monitoring default. The logs path is unaffected because Fluent Bit's output host is set directly and correctly to the ancestor tenant in the same template.

Impact

Metrics from every guest cluster that uses monitoringAgents to ship to an ancestor tenant are dropped. The HelmRelease still reports Ready, hiding the failure. There is no runtime workaround: the generated HelmRelease / VMAgent CR is reconciled, so patching either is reverted, and the app template does not merge monitoringAgents.valuesOverride for this field.

Suggested fix

In packages/apps/kubernetes/templates/helmreleases/monitoring-agents.yaml, set vmagent.remoteWrite.urls (the key the chart reads) as a list targeting the ancestor tenant's ingest with the cluster domain, covering both retention tiers:

    vmagent:
      remoteWrite:
        urls:
          - http://vminsert-shortterm.{{ $targetTenant }}.svc.{{ $clusterDomain }}:8480/insert/0/prometheus
          - http://vminsert-longterm.{{ $targetTenant }}.svc.{{ $clusterDomain }}:8480/insert/0/prometheus

Related

Counterpart of #5256 (the logs / tenant-NetworkPolicy side of the same guest-cluster monitoring flow).