Skip to content

fix(chart): emit prometheus.io/scheme=https across all annotation paths - #3055

Merged
fedepaol merged 2 commits into
metallb:mainfrom
lexfrei:fix/chart-prometheus-scheme-https
May 27, 2026
Merged

fedepaol merged 2 commits into
metallb:mainfrom
lexfrei:fix/chart-prometheus-scheme-https

Conversation

@lexfrei

@lexfrei lexfrei commented May 21, 2026 •

Copy link
Copy Markdown
Contributor

Is this a BUG FIX or a FEATURE ?:

/kind bug

What this PR does / why we need it:

Since v0.16.0 kube-rbac-proxy is gone and both controller and speaker serve /metrics directly over HTTPS on port 9120. Three different scrape-discovery surfaces in the chart still pointed at HTTP, all of which now fail with TLS handshake errors on the v0.16 endpoint:

  1. Pod-level prometheus.io/scrape annotations on the controller Deployment and speaker DaemonSet. Annotation-based scrapers default to HTTP without an explicit prometheus.io/scheme.
  2. PodMonitor.spec.podMetricsEndpoints for both controller and speaker in templates/podmonitor.yaml. PodMonitor endpoints default to HTTP when scheme: is unset.
  3. The annotation-only Service objects bundled alongside the ServiceMonitors in templates/servicemonitor.yaml (speaker-monitor-service and controller-monitor-service). These services exist to be discovered via prometheus.io/scrape annotations by scrapers that do not use the Operator's ServiceMonitor CRD; they were missing the matching prometheus.io/scheme annotation.

Emit prometheus.io/scheme: "https" / scheme: https in each of those three paths.

The chart no longer exposes a knob to disable TLS on metrics (tls.controllerMetricsTLSSecret / tls.speakerMetricsTLSSecret are optional only in the sense that a self-signed cert is auto-generated when unset). The chart's own ServiceMonitor spec.endpoints[].scheme is already hardcoded to "https", confirming the project's stance that the endpoint is HTTPS-only. The annotation and PodMonitor paths were simply out of sync.

The new prometheus.io/scheme annotation on the speaker pod template sits inside the existing {{- if not .Values.speaker.frr.enabled }} block, matching the guard already used for prometheus.io/port (legacy FRR mode runs multiple metrics endpoints and a single annotation pair cannot cover both — users on FRR mode are expected to use ServiceMonitor anyway).

Special notes for your reviewer:

Verified locally (helm v3.18, v3.21):

  • Default mode (frr-k8s): controller and speaker pods both emit scrape: true + scheme: https + port: 9120 annotations.
  • Legacy FRR mode (speaker.frr.enabled=true, frrk8s.enabled=false): speaker emits only scrape: true; controller emits the full triple.
  • prometheus.podMonitor.enabled=true: both PodMonitors render with podMetricsEndpoints[].scheme: https.
  • prometheus.serviceMonitor.enabled=true: both monitor-services render with prometheus.io/scrape: "true" AND prometheus.io/scheme: "https".
  • Without prometheus.scrapeAnnotations: zero prometheus.io/* annotations on the pods.
  • helm lint clean.

ServiceMonitor users are unaffected by the pod-annotation change (they declare scheme: https explicitly in the ServiceMonitor spec). The PodMonitor and Service-annotation changes round out the missing pieces in the same area.

Release note:

Helm chart: pod-level `prometheus.io/scrape` annotations, the `PodMonitor` endpoints, and the annotation-only Service objects bundled with the ServiceMonitor now all emit `scheme: https` to match the HTTPS-only metrics endpoint introduced in v0.16.0. Annotation-based and PodMonitor-based Prometheus scrapers no longer fail with TLS handshake errors against the metrics port.

@lexfrei
lexfrei requested review from fedepaol and oribon as code owners May 21, 2026 00:01

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the MetalLB Helm chart by adding the prometheus.io/scheme: "https" annotation to the controller and speaker pod templates to support HTTPS metrics scraping. The reviewer pointed out that the changes are incomplete, as podmonitor.yaml and servicemonitor.yaml also require updates to their schemes and TLS configurations to prevent Prometheus scraping failures.

@fedepaol

Copy link
Copy Markdown
Member

@lexfrei please fill the release notes and kind as per the template you overrode

@fedepaol

Copy link
Copy Markdown
Member

and thanks for the PR!

@lexfrei

lexfrei commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

@fedepaol thanks for the heads-up! I missed the template on the first pass; the body now follows the /kind + release-note structure. I also updated the other four sibling PRs I have open with the same fix:

Rebasing each branch onto current main now.

@lexfrei
lexfrei force-pushed the fix/chart-prometheus-scheme-https branch from 01409cd to c77fa0a Compare May 21, 2026 10:16
Since v0.16.0 kube-rbac-proxy is gone and both controller and speaker serve
`/metrics` directly over HTTPS on port 9120 (`tls.controllerMetricsTLSSecret`
/ `tls.speakerMetricsTLSSecret` provide certs; a self-signed cert is
auto-generated if neither is set). Three different scrape-discovery
surfaces in the chart still pointed at HTTP, all of which now fail with
TLS handshake errors on the v0.16 endpoint:

1. Pod-level `prometheus.io/scrape` annotations on the controller
   Deployment and speaker DaemonSet. Annotation-based scrapers default to
   HTTP without an explicit `prometheus.io/scheme`.
2. `PodMonitor.spec.podMetricsEndpoints` for both controller and speaker
   in `templates/podmonitor.yaml`. PodMonitor endpoints default to HTTP
   when `scheme:` is unset.
3. The annotation-only `Service` objects bundled alongside the
   ServiceMonitors in `templates/servicemonitor.yaml` (speaker-monitor-
   service and controller-monitor-service). These services exist to be
   discovered via `prometheus.io/scrape` annotations by scrapers that do
   not use the Operator's ServiceMonitor CRD; they were missing the
   matching `prometheus.io/scheme` annotation.

Emit `prometheus.io/scheme: "https"` / `scheme: https` in each of those
three paths.

Why unconditional `https`:

The chart no longer exposes a knob to disable TLS on metrics
(`tls.controllerMetricsTLSSecret` / `tls.speakerMetricsTLSSecret` are
optional only in the sense that a self-signed cert is auto-generated
when unset). The chart's own ServiceMonitor `spec.endpoints[].scheme` is
already hardcoded to `"https"`, confirming the project's stance that the
endpoint is HTTPS-only. The annotation and PodMonitor paths were simply
out of sync.

Speaker FRR-mode guard (pod-level annotations only):

The new `prometheus.io/scheme` annotation on the speaker pod template
sits inside the existing `{{- if not .Values.speaker.frr.enabled }}`
block. Legacy FRR mode runs multiple metrics endpoints (speaker on
9120, frr sidecar on 9121) and a single annotation pair cannot cover
both — the chart already suppresses `prometheus.io/port` in that case
and emits only `prometheus.io/scrape: "true"`. Suppressing the new
`scheme` annotation under the same guard keeps the annotation set
internally consistent; users on legacy FRR mode are expected to use
ServiceMonitor.

Verified locally (helm v3.18, v3.21):

- Default mode (frr-k8s): controller and speaker pods both emit
  `scrape: true` + `scheme: https` + `port: 9120` annotations.
- Legacy FRR mode (`speaker.frr.enabled=true`, `frrk8s.enabled=false`):
  speaker emits only `scrape: true`; controller emits the full triple.
- `prometheus.podMonitor.enabled=true`: both PodMonitors render with
  `podMetricsEndpoints[].scheme: https`.
- `prometheus.serviceMonitor.enabled=true`: both monitor-services render
  with `prometheus.io/scrape: "true"` AND `prometheus.io/scheme: "https"`.
- Without `prometheus.scrapeAnnotations`: zero `prometheus.io/*`
  annotations on the pods.
- `helm lint` clean.

ServiceMonitor users are unaffected by the pod-annotation change (they
declare `scheme: https` explicitly in the ServiceMonitor spec). The
PodMonitor and Service-annotation changes round out the missing pieces
in the same area.

Signed-off-by: Aleksei Sviridkin <[email protected]>
@lexfrei
lexfrei force-pushed the fix/chart-prometheus-scheme-https branch from c77fa0a to 8bdda33 Compare May 21, 2026 10:22
@lexfrei lexfrei changed the title fix(chart): emit prometheus.io/scheme=https on scrape annotations fix(chart): emit prometheus.io/scheme=https across all annotation paths May 21, 2026
@fedepaol
fedepaol merged commit af69ff7 into metallb:main May 27, 2026
28 of 30 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants