Repository navigation
System components use different metrics paths #2067
Description
What happened?
The repo gives no rule for how a substrate component emits metrics. The components use different paths, and the documentation does not agree with the code.
The components on main use three setups:
| Component | OTLP push | Prometheus /metrics |
Setup call |
|---|---|---|---|
| ateapi, atelet, atenet-router, credential-provider (kubernetes-secrets), glutton | yes | :9090 |
InitMetrics + StartMetricsServer |
| atecontroller | yes | :8080 serves only the controller-runtime registry. It does not serve the OTel instruments. |
InitMetricsPushOnly + Prometheus bridge |
| ateom-gvisor, ateom-microvm | yes, through the atelet relay | none | InitMetricsPushOnlyVia |
This mix causes these problems:
- Each environment uses a different path. kind scrapes ateapi, atelet, and atenet-router on
:9090. It also filters the collector to prevent a second copy. GKE uses only the OTLP push. The HPA in theautoscaled-workerpooldemo reads a scrape of ateapi. - Each metric arrives two times. A cluster that scrapes the components gets every metric two times, with two names. The push sends
ate_actor_crashes, and the scrape sendsate_actor_crashes_total. No setting stops the push ([Bug]: OTEL_METRICS_EXPORTER=none does not stop the OTLP metric export #1877). - Scrapes occur that the operator did not ask for. The base manifests put
prometheus.io/scrape: "true"on ateapi, atelet, and atenet-router. Thus, a cluster that runs the usualkubernetes-podsjob scrapes them. The operator does not need to do anything. - The documentation does not agree with itself. Section 2 of
docs/observability.mdsays that every metric goes over OTLP. Section 4 says that kind reads metrics from the collector on:8889. But the kind Prometheus job reads the component metrics from:9090. - A new component has no rule.
docs/dev/best-practices/metrics.mdlists the setup functions. It does not say which function a new component must call. It does not give a reason. - The health probes use the metrics server. ateapi, atelet, and atenet-router serve
/readyzand/healthzon:9090. Thus, a change to the metrics endpoint also changes the health checks. - Two open pull requests use different policies. atecontroller: the :8080 metrics listener follows OTEL_METRICS_EXPORTER #1976 stops the atecontroller listener on
:8080. fix(serverboot): honor OTEL_METRICS_EXPORTER=none #1878 adds a mode that has a scrape and no push.
Expected Behavior
Every component obeys one policy, and OTEL_METRICS_EXPORTER selects the path.
The policy. By default, each component pushes over OTLP and serves /metrics. The operator can select one path, both paths, or no path:
OTEL_METRICS_EXPORTER |
OTLP push | /metrics |
|---|---|---|
| not set (the default) | yes | yes |
otlp |
yes | no |
prometheus |
no | yes |
otlp,prometheus |
yes | yes (the same as not set) |
none |
no | no |
The default is different from the spec, where not set means otlp. The docs state this difference. /metrics stays on :9090 (:8080 for atecontroller), not on localhost:9464 as in Go autoexport.
One parser for the exporter variables. serverboot reads the variables itself and does not use autoexport. OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER (#2197) use the same parser, so two variables in the same ConfigMap use the same syntax.
Tasks
- Manifests and kind. Remove the scrape annotations from the base manifests, and add
scrape: "true"to the agentgateway patch. Make the kind Prometheus read all component metrics from the collector on:8889, and remove the direct scrape and the keep filter. Update the HPA demo, with a note that the push interval controls how fast the HPA reacts. Updatedocs/observability.mdanddocs/dev/best-practices/metrics.md. - Health probes. Move
/readyzand/healthzto a server that is always on, separate from/metrics. RemovehostPort: 9090from atelet. - The variable. Add the shared parser to
serverboot, apply the policy table, and make atecontroller's:8080obey it. This changes the meaning ofnonefrom fix(serverboot): honor OTEL_METRICS_EXPORTER=none #1878, so it waits for step 2. It replaces atecontroller: the :8080 metrics listener follows OTEL_METRICS_EXPORTER #1976.
Options considered
- Option 1:
/metricsalways on, and the variable controls only the push. The thread selected it first. It changed to the policy above, becausenoneandprometheusmust keep their meaning in the spec. - Option 2: push by default, and
/metricsonly when the operator selects it. Rejected: it removescurl /metricsby default, and it breaks clusters that scrape today. - Option 3: Option 2 with the list form. The policy above keeps the list form, but
/metricsstays on by default. - Option 4: push only, with no
/metrics. Rejected: no comparable Kubernetes project removes the endpoint. - Option 5: documentation only. Rejected: the unrequested scrapes stay.
Activity
JeffLuoo commentedon Oct 1, 2026
Da Huang (@git286) Julian Gutierrez Oschmann (@juli4n) Tim Bai (@baizhenyu) Krisztian F (@krisztianfekete) please review and comments are welcome.
I can make the discussion into a public google doc if find it easier to comment.
git286 commentedon Oct 1, 2026
I am OK with option1 but we need to consider following:
Maybe option 3 is also more reasonable given that no one is really scraping the endpoints (except Kind?)
JeffLuoo commentedon Oct 1, 2026
Yes. It's the goal of this issue.
Yes, we can do it.
Double write -> duplicated data
Some downsides are:
curl /metrics needsa change of the setting.Also I am not sure does anyone scrape the components outside of kind today in real cluster? If so, this will be a breaking change.
krisztianfekete commentedon Oct 5, 2026
Note that
OTEL_METRICS_EXPORTER=noneshould mean no exporter at all. The spec treats the Prometheus pull reader as an exporter too, and it already has a value for scrape only (prometheus). So the switch should manage both readers, not just the push./metrics(our default, we document that it differs from the spec's otlp)Other things I'd add:
/metricson every node IP._totalvs. no suffix is a backend setting, not a push vs. scrape thing.ate_*in the keep regex. Then kind works like GKE and the annotations can really go away. The HPA demo reads :8889 too, with a note that the push interval sets how fast the HPA reacts.prometheus.io/port: 15020on atenet-router and depends on the basescrape: "true". Let's not break this.OTEL_METRICS_EXPORTERto ateom pods, they always push via the relay.JeffLuoo commentedon Oct 5, 2026
Thank you for the detailed review.
OTEL_METRICS_EXPORTERmanages both readers. Agreed.noneshould mean no exporter, andprometheusis the value for scrape only.One change: no list form. The spec allows a list, but it is optional ("MAY"). Most Go applications take one value only. The Go contrib package
autoexportlooks upotlp,prometheusas one name and returns an error, and so do buildkit, filebeat, and other Go projects that read the variable. Unset already gives both readers, so the list adds nothing. I propose:/metrics(our default; the docs say that it differs from the spec)otlp: push onlyprometheus:/metricsonlynone: no push and no/metricsAny other value logs a warning and keeps the default.
Health probes first, and remove atelet's
hostPort: 9090. Agreed.otlpandnoneturn off/metrics, so the probes need their own server before the variable changes._total. Agreed. The backend adds the suffix, not the path. I will correct the issue text.Kind. Agreed
#1976. Agreed. atecontroller's
:8080should obey the variable, not turn off the listener.agentgateway. Good catch. I will update the issue text.
ateom. Agreed, and it is true today. atecontroller copies the OTLP endpoint, the export interval and timeout, and the sampler variables to the worker pods, but not
OTEL_METRICS_EXPORTER. I will update the issue text.Does it sound good to not accept a list form? Krisztian F (@krisztianfekete)
JeffLuoo commentedon Oct 5, 2026
Similar discussion, but for logs in #2197
krisztianfekete commentedon Oct 6, 2026
Thanks, all good except the list. You're right that autoexport doesn't take it, but we don't use autoexport, since serverboot parses the variable itself. #2197 needs the list anyway. Logs go to stdout only by default, so
otlp,consoleis the only way to get both copies.I'd rather have one small parser in serverboot that both
OTEL_METRICS_EXPORTERandOTEL_LOGS_EXPORTERuse. That way two variables in the same ConfigMap don't take different syntax. For metrics,otlp,prometheusthen just means the same as unset, which is fine. For an unknown value, warn and skip it but keep the rest, as the spec says.JeffLuoo commentedon Oct 6, 2026
SGTM, I will update the description to reflect it.
JeffLuoo commentedon Oct 6, 2026
Quentin Bisson (@QuentinBisson) Tim Bai (@baizhenyu) #2190 and #2193 both add exporter parsing in
serverboot, with different rules. This issue agreed on one parser for everyOTEL_*_EXPORTERvariable:nonewith other items, the same for every signal (actorevent: write each actor event to one place, with a shared OTEL_*_EXPORTER parser #2193 rejects the value; please confirm that we keep that).otlp,prometheus, orconsolein the same way.Proposal: land the parser first (from the loop in #2193, made general). Then #2190 uses it for metrics and traces instead of
otlpExportEnabled, and #2189 rebases on that. The behavior in #2190 (prometheusstops the push) is correct for now. Only the syntax must change.baizhenyu commentedon Oct 6, 2026
The discussion sounds good to me. I will update #2193 to reflect the decisions.
actorevent: write each actor event to one place, with a shared OTEL_*…
Components honor OTEL_TRACES_EXPORTER=none and treat OTEL_METRICS_EXP…
Merge upstream main through 25a0b88 (#67)
atecontroller: address review on the :8080 listener
Sync upstream sandbox, snapshot, and HTTPS egress updates (#69)
atecontroller: the :8080 metrics listener follows OTEL_METRICS_EXPORT…
Health probes move to their own server on :9091, and :9090 serves onl…