ateworker no longer exports to localhost:4317 without a relay or endpoint, and takes its own exporter settings by QuentinBisson · Pull Request #2189 · agent-substrate/substrate

Quentin Bisson (QuentinBisson) · 2026-10-05T07:36:36Z

Follow-up to #1878 (#1877).

ateworker exports through atelet's relay, or directly to its own OTLP endpoint when there is no relay. With neither, it used to export to localhost:4317, where no collector runs. It now skips the trace, metric and log exporters and logs a warning per signal with reason="no OTLP relay or endpoint".

ate-controller now always sets the worker pod identity (OTEL_RESOURCE_ATTRIBUTES), metric export tuning and sampler, so telemetry sent through the relay carries pod identity even without an endpoint. The worker exporters come from new worker-only ATE_WORKER_OTEL_{LOGS,TRACES,METRICS}_EXPORTER keys in ate-otel-config (flags --otel-{logs,traces,metrics}-exporter). The controller's own values don't fit the workers: prometheus means scraped, and ateworker serves no /metrics.

Breaking change

Please add the breaking-change label.

Worker pods no longer take OTEL_LOGS_EXPORTER from ate-controller. If you set it to send worker usage records over OTLP, also set ATE_WORKER_OTEL_LOGS_EXPORTER to the same value in ate-otel-config. The kind overlay already does.
On installs where ate-controller has no OTLP endpoint, WorkerPool Deployments roll once on upgrade because the worker env changes. Installs that set the endpoint in ate-otel-config (the default) don't roll.
Tests pass
Appropriate changes to documentation are included in the PR

Max Smythe (maxsmythe) · 2026-10-06T02:00:20Z

Jeff Luo (@JeffLuoo) It looks like you may have context for this PR (built on a previous PR you reviewed)?

Jeff Luo (JeffLuoo) · 2026-10-08T15:29:33Z

Can you rebase to resolve conflict? #2190 and #2193 are merged.

Quentin Bisson (QuentinBisson) · 2026-10-08T20:06:07Z

Rebased on main, conflict resolved.

Jeff Luo (JeffLuoo)

Needs a rebase onto main to resolve conflicts with #1976 in internal/serverboot/serverboot.go and docs/observability.md.

Also in internal/serverboot/serverboot_test.go (line 242, outside the diff): TestInitMetricsPushOnlyHasNoPrometheusSurface calls InitMetricsPushOnlyVia(context.Background(), "test-pushonly", nil) with no OTEL_EXPORTER_OTLP_ENDPOINT. With hasOTLPDestination, a nil conn and no endpoint no longer installs the OTLP periodic reader, so the test passes without exercising a push reader (unless OTEL_EXPORTER_OTLP_ENDPOINT happens to be set in the caller's environment). Please pass lazyConn(t) there so the test still builds a provider with a push reader.

Jeff Luo (JeffLuoo) · 2026-10-13T18:06:35Z

+	// OTEL_LOGS_EXPORTER, OTEL_TRACES_EXPORTER and OTEL_METRICS_EXPORTER set on
+	// ateom, passed through untouched. They are worker-only values rather than
+	// the controller's own: on the control plane, metrics none means scraped
+	// instead of pushed, and ateom serves no /metrics. Empty keeps ateom's


After #1976 on main, scraped control-plane components use OTEL_METRICS_EXPORTER=prometheus rather than none (none now disables both OTLP push and atecontroller's :8080 scrape listener). Please update this comment during the rebase (prometheus means scraped instead of pushed on the control plane, while ateom serves no /metrics).

Jeff Luo (JeffLuoo) · 2026-10-13T18:06:35Z

 ### Scraping instead of pushing

-ateapi, atelet, atenet-router and the credential provider also serve every instrument on their Prometheus `/metrics` endpoint. A cluster that scrapes those endpoints sets `OTEL_METRICS_EXPORTER=none` on the components, so each series reaches the backend once. With `none` the components install no OTLP metric reader and keep the Prometheus one; traces and logs are unaffected. atecontroller then registers its instruments (`ate.workerpool.*`) on controller-runtime's registry, so the manager's `:8080` serves them next to the controller-runtime families. ateom serves no endpoint of its own, so leave the variable unset on the worker pods, or it exports no metrics at all. The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`. `otlp` (the default) keeps the push, and `prometheus` or `none` stops it. `/metrics` stays on for each value. ateom serves no `/metrics`, so there `prometheus` is skipped as unknown with a warning, and the push stays on unless the variable is `none`.
+ateapi, atelet, atenet-router and the credential provider also serve every instrument on their Prometheus `/metrics` endpoint. A cluster that scrapes those endpoints sets `OTEL_METRICS_EXPORTER=none` on the components, so each series reaches the backend once. With `none` the components install no OTLP metric reader and keep the Prometheus one; traces and logs are unaffected. atecontroller then registers its instruments (`ate.workerpool.*`) on controller-runtime's registry, so the manager's `:8080` serves them next to the controller-runtime families. ateom serves no endpoint of its own, so `ate-controller` does not propagate its own value to the worker pods, where `none` would export no metrics at all. To set the variable on ateom, set `ATEOM_OTEL_METRICS_EXPORTER` in `ate-otel-config` (or the controller's `--otel-metrics-exporter` flag). The variable takes a comma-separated list, with the same rules as `OTEL_LOGS_EXPORTER`. `otlp` (the default) keeps the push, and `prometheus` or `none` stops it. `/metrics` stays on for each value. ateom serves no `/metrics`, so there `prometheus` is skipped as unknown with a warning, and the push stays on unless the variable is `none`.


This paragraph conflicts with #1976 on main, where the scrape-only setting changed from OTEL_METRICS_EXPORTER=none to OTEL_METRICS_EXPORTER=prometheus (and atecontroller's :8080 listener was moved to its own paragraph). When rebasing onto main, please keep the OTEL_METRICS_EXPORTER=prometheus wording from #1976 while adding the ATEOM_OTEL_METRICS_EXPORTER note.

Jeff Luo (JeffLuoo) · 2026-10-13T18:06:35Z

@@ -327,6 +348,7 @@ func newMeterProvider(ctx context.Context, serviceName string, push, relayCapabl
 	}


nit: When a control-plane binary runs with OTEL_METRICS_EXPORTER=prometheus, export is false even though the Prometheus reader is active (and after rebasing onto main, metricsExporters also logs "Metrics exporters resolved"). Consider naming this key otlp_export (or push) so false is not misread as all metrics export being off.

Jeff Luo (JeffLuoo) · 2026-10-13T18:06:35Z

+	}
+	if lp != nil {
+		t.Error("InitLogging returned a provider with nowhere to export")
+	}


Only InitLogging is tested here for the no-destination case (RelayCapable: true, conn == nil, and no endpoint env), and only InitTracing is tested above for exporting through a relay conn without an endpoint.

Please also add tests covering:

InitTracing and InitMetricsPushOnlyVia with RelayCapable: true, conn == nil, and no OTLP endpoint (verifying no spans/metrics are pushed and no OTLP reader is installed).

InitMetricsPushOnlyVia with a non-nil conn and empty OTEL_EXPORTER_OTLP_ENDPOINT (verifying metrics export through the connection).

…endpoint A relay-capable component with neither a relay connection nor an OTLP endpoint would export to the SDK default, localhost:4317, where a worker pod runs no collector. Traces, metrics and logs now skip the exporter in that case. A relay connection is enough on its own: atelet only serves the socket when it has a collector.

…oint ateworker exports through atelet's relay whenever atelet has a collector, whether or not ate-controller has an endpoint to propagate. The pod identity in OTEL_RESOURCE_ATTRIBUTES, the metric export tuning and the sampler now reach the worker pods either way; the endpoint is still set only when configured. Worker pools on installs without an endpoint roll once when the controller is upgraded.

ate-controller propagated the endpoint, sampler and metric tuning to the worker pods but not the trace and metric exporter selection, so turning traces or metrics off for the workers was not possible: the relay forwards ateworker's telemetry as it is, and atelet's own values do not apply to it. All three worker exporters now come from worker-only variables in ate-otel-config: ATE_WORKER_OTEL_LOGS_EXPORTER, ATE_WORKER_OTEL_TRACES_EXPORTER and ATE_WORKER_OTEL_METRICS_EXPORTER, the defaults of --otel-logs-exporter, --otel-traces-exporter and --otel-metrics-exporter. On the control plane OTEL_METRICS_EXPORTER=prometheus means the metrics are scraped from /metrics instead of pushed, and ateworker serves no /metrics, so passing that value on would drop ateworker's metrics. Logs and traces follow the same rule so every worker exporter is set the same way. The kind overlay sets ATE_WORKER_OTEL_LOGS_EXPORTER to otlp,console, as it does OTEL_LOGS_EXPORTER. Unset, ateworker keeps its defaults.

Quentin Bisson (QuentinBisson) · 2026-10-19T10:11:55Z

Jeff Luo (@JeffLuoo) can you take another look?

Quentin Bisson (QuentinBisson) force-pushed the fix/otel-exporter-followups branch from ea1b67e to 7e6810e Compare October 5, 2026 08:08

Quentin Bisson (QuentinBisson) changed the title ~~Components honor OTEL_TRACES_EXPORTER=none and treat OTEL_METRICS_EXPORTER=prometheus as none~~ Components honor OTEL_TRACES_EXPORTER=none and OTEL_METRICS_EXPORTER=prometheus; ateom exports only through the relay or its own endpoint Oct 5, 2026

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 5, 2026

docs(fork): record the upstream status of the telemetry rows giantswarm/substrate#117

Merged

Quentin Bisson (QuentinBisson) force-pushed the fix/otel-exporter-followups branch 2 times, most recently from 401be34 to 9a7133f Compare October 5, 2026 08:40

Quentin Bisson (QuentinBisson) changed the title ~~Components honor OTEL_TRACES_EXPORTER=none and OTEL_METRICS_EXPORTER=prometheus; ateom exports only through the relay or its own endpoint~~ ateom exports only through the relay or its own endpoint, and always carries its pod identity Oct 5, 2026

Quentin Bisson (QuentinBisson) force-pushed the fix/otel-exporter-followups branch 3 times, most recently from 9a7133f to ba090f3 Compare October 5, 2026 09:05

Quentin Bisson (QuentinBisson) marked this pull request as ready for review October 5, 2026 09:08

Quentin Bisson (QuentinBisson) changed the title ~~ateom exports only through the relay or its own endpoint, and always carries its pod identity~~ ateom exports only through the relay or its own endpoint, with its pod identity and its own exporter selection Oct 5, 2026

Max Smythe (maxsmythe) requested a review from Jeff Luo (JeffLuoo) October 6, 2026 01:58

Max Smythe (maxsmythe) added area/observability kind/bug Something isn't working / bugfixes labels Oct 6, 2026

Jeff Luo (JeffLuoo) mentioned this pull request Oct 6, 2026

System components use different metrics paths #2067

Open

Quentin Bisson (QuentinBisson) force-pushed the fix/otel-exporter-followups branch 2 times, most recently from a7aff9f to d4fb680 Compare October 8, 2026 08:44

Quentin Bisson (QuentinBisson) force-pushed the fix/otel-exporter-followups branch from d4fb680 to 1a61b57 Compare October 8, 2026 19:56

Quentin Bisson (QuentinBisson) force-pushed the fix/otel-exporter-followups branch from 1a61b57 to 6dba59b Compare October 13, 2026 13:31

Jeff Luo (JeffLuoo) reviewed Oct 13, 2026

View reviewed changes

Quentin Bisson (QuentinBisson) force-pushed the fix/otel-exporter-followups branch 2 times, most recently from 2dbeb99 to a950697 Compare October 13, 2026 19:39

Golden Garlic (garlicKim21) mentioned this pull request Oct 14, 2026

[Bug]: Chart leaves OTLP export on when no endpoint is set, so components push to localhost:4317 kagent-dev/substrate#76

Closed

2 tasks done

Quentin Bisson (QuentinBisson) force-pushed the fix/otel-exporter-followups branch from a950697 to 43c72a2 Compare October 16, 2026 05:13

Quentin Bisson (QuentinBisson) changed the title ~~ateom exports only through the relay or its own endpoint, with its pod identity and its own exporter selection~~ ateworker exports only through the relay or its own endpoint, with its pod identity and its own exporter selection Oct 16, 2026

Quentin Bisson (QuentinBisson) added 2 commits October 18, 2026 14:14

Quentin Bisson (QuentinBisson) force-pushed the fix/otel-exporter-followups branch from 43c72a2 to edf3370 Compare October 18, 2026 12:24

Quentin Bisson (QuentinBisson) changed the title ~~ateworker exports only through the relay or its own endpoint, with its pod identity and its own exporter selection~~ ateworker no longer exports to localhost:4317 without a relay or endpoint, and takes its own exporter settings Oct 19, 2026

Jeff Luo (JeffLuoo) added the breaking-change PRs with breaking changes label Oct 19, 2026

Jeff Luo (JeffLuoo) approved these changes Oct 19, 2026

View reviewed changes

Jeff Luo (JeffLuoo) enabled auto-merge October 19, 2026 14:22

Jeff Luo (JeffLuoo) added this pull request to the merge queue Oct 19, 2026

Merged via the queue into agent-substrate:main with commit a62e898 Oct 19, 2026
15 checks passed

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 20, 2026

docs(fork): the ledger follows the upstream moves of 2026-W43 giantswarm/substrate#497

Merged

2 tasks

Suresh Attaluri (whysosuresh) mentioned this pull request Oct 25, 2026

KubernetesAgentSubstrate: install Agent Substrate v0.6.0 plantonhq/planton#638

Closed

Navigation Menu

Repository navigation

ateworker no longer exports to localhost:4317 without a relay or endpoint, and takes its own exporter settings - #2189

ateworker no longer exports to localhost:4317 without a relay or endpoint, and takes its own exporter settings#2189
Jeff Luo (JeffLuoo) merged 3 commits into
agent-substrate:mainfrom
QuentinBisson:fix/otel-exporter-followups

Quentin Bisson (QuentinBisson) commented Oct 5, 2026 •

edited

Loading

Uh oh!

Max Smythe (maxsmythe) commented Oct 6, 2026

Uh oh!

Jeff Luo (JeffLuoo) commented Oct 8, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 8, 2026

Uh oh!

Jeff Luo (JeffLuoo) left a comment

Uh oh!

Jeff Luo (JeffLuoo) Oct 13, 2026

Uh oh!

Jeff Luo (JeffLuoo) Oct 13, 2026

Uh oh!

Jeff Luo (JeffLuoo) Oct 13, 2026

Uh oh!

Jeff Luo (JeffLuoo) Oct 13, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -327,6 +348,7 @@ func newMeterProvider(ctx context.Context, serviceName string, push, relayCapabl
		}

Conversation

Quentin Bisson (QuentinBisson) commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Breaking change

Uh oh!

Max Smythe (maxsmythe) commented Oct 6, 2026

Uh oh!

Jeff Luo (JeffLuoo) commented Oct 8, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 8, 2026

Uh oh!

Jeff Luo (JeffLuoo) left a comment

Choose a reason for hiding this comment

Uh oh!

Jeff Luo (JeffLuoo) Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Jeff Luo (JeffLuoo) Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Jeff Luo (JeffLuoo) Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Jeff Luo (JeffLuoo) Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 19, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Quentin Bisson (QuentinBisson) commented Oct 5, 2026 •

edited

Loading