feat(helm): add metrics endpoint and optional ServiceMonitor support - #1802

…agent-dev#2401)

## Description

This adds native
[ServiceMonitor](https://prometheus-operator.dev/docs/api-reference/api/#monitoring.coreos.com/v1.ServiceMonitor)
support to the kagent Helm chart so controller metrics are collected by
[Prometheus Operator](https://prometheus-operator.dev/) out of the box.

This is the follow up that
[kagent-dev#1803](kagent-dev#1803) deferred. That
PR landed the metrics Service and the scrape RBAC and closed
[kagent-dev#1369](kagent-dev#1369), but left the
ServiceMonitor out of scope.
[kagent-dev#1802](kagent-dev#1802) carried one and
was closed as a duplicate of kagent-dev#1803, so the ServiceMonitor half of kagent-dev#1369
was never reviewed on its own and no template for it exists in the chart
today. This PR is additive on top of what kagent-dev#1803 merged and changes none
of it.

Today controller.metrics.enabled provisions the metrics Service and the
ClusterRoles for authenticated scrapes, but not the resource that makes
Prometheus actually scrape them. Every user running
[kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack)
has to hand write a ServiceMonitor outside the chart and keep its port
name and namespace in sync with the chart by hand. The bundled
[kagent-tools](https://github.com/kagent-dev/tools) subchart already
ships one, so this closes the gap for the controller.

New values live under controller.metrics.serviceMonitor, disabled by
default and gated on controller.metrics.enabled. Interval, scrape
timeout, honorLabels, relabelings, metric relabelings, extra labels,
annotations and a namespace override are exposed. With secure serving
on, the endpoint defaults to https with the ServiceAccount token and
insecureSkipVerify, which matches the self signed certificate the
controller serves, and both are overridable.

The metrics Service port name moved into a shared helper used by both
the Service and the ServiceMonitor endpoint, so the two cannot drift
when secure serving is flipped. Rendered output for existing installs is
unchanged.

## Changelog

Added an optional Prometheus Operator ServiceMonitor for the controller
metrics endpoint, configurable under controller.metrics.serviceMonitor.

## Testing

1. helm unittest helm/kagent passes with 298 tests, 15 of them new in
helm/kagent/tests/controller-servicemonitor_test.yaml.
2. Nothing renders unless both controller.metrics.enabled and
controller.metrics.serviceMonitor.enabled are set, and the bindAddress
disable sentinel still suppresses it.
3. Rendered the secure and plaintext variants with helm template, then
validated them against a live cluster with kubectl apply
--dry-run=server so the manifest is checked against the real
ServiceMonitor CRD rather than only the chart.

## Additional Notes

Rendering the ServiceMonitor does not authorize the scrape on its own.
With secure serving, the metrics reader ClusterRole still has to be
bound to the Prometheus ServiceAccount, which the values comment calls
out.

The chart does not gate on the monitoring.coreos.com API being present,
matching what kagent-tools does. The explicit opt in flag is the gate,
so a missing CRD fails loudly at install time instead of silently
dropping the resource during helm template.

---------

Signed-off-by: younsl <[email protected]>