otlprelay: forward log records by baizhenyu · Pull Request #1800 · agent-substrate/substrate

Tim Bai (baizhenyu) · 2026-09-22T18:47:43Z

Step 1 of #1748. The relay forwards LogsService the way it forwards traces and metrics: source gate, verbatim payload, atelet's own OTEL_EXPORTER_OTLP_LOGS_HEADERS upstream.

No content check on the records. Identity spoofing through the relay is handled in #1748 by the consumer rule (a lifecycle record counts only under ateapi's resource, which the source gate already enforces) and per-pod sockets (#741), not by a denylist here.

Endpoint and compression resolution (second commit, from review). The relay read the generic, traces, and metrics OTLP variables and ignored the logs ones, so a logs-specific collector or compression setting was silently overridden. All three signals now go through one resolver: each falls back to the generic, and every resolved value must agree, since the relay has one upstream connection. The conflict error names the variable each value came from; a signal that falls back names the generic variable, not its own unset one.

One behavior change: traces=A, metrics=A, generic=B with logs unset used to pass and now fails at atelet startup, because logs falls back to B and that is a two-collector configuration.

Tests: an ateom log batch arrives with the ateom's resource; the source gate, mixed-batch, header-isolation, and per-signal header tests cover logs; resolver tables gained logs rows including the fallback case. Docs say the relay carries logs, traces, and metrics.

The relay carried traces and metrics. ateom is about to emit log records over it, so it forwards LogsService the same way: source gate, verbatim payload, atelet's own headers upstream.

Krisztian F (krisztianfekete)

Added some questions/comments!

Krisztian F (krisztianfekete) · 2026-09-23T13:10:30Z

 	headersEnv        = "OTEL_EXPORTER_OTLP_HEADERS"
 	tracesHeadersEnv  = "OTEL_EXPORTER_OTLP_TRACES_HEADERS"
 	metricsHeadersEnv = "OTEL_EXPORTER_OTLP_METRICS_HEADERS"
+	logsHeadersEnv    = "OTEL_EXPORTER_OTLP_LOGS_HEADERS"


What happens if atelet also sets OTEL_EXPORTER_OTLP_LOGS_ENDPOINT? I think upstreamTarget() (L548) only reads the generic, traces and metrics variables, so logs would quietly go to the traces/metrics collector. Same question for LOGS_COMPRESSION in upstreamCompression() (L508).

Right, both resolvers ignored the logs variable, so logs would have followed the traces/metrics collector silently. Both now read it and apply the same conflict rule, and the error names the variables the two values came from, since a fallback pulls the generic in under a signal that is not itself set. One behavior change falls out: a config that sets the traces and metrics overrides to one collector and leaves logs unset now fails at startup, because logs falls back to the generic and that is a two-collector configuration. The old test row asserting the lenient result is updated and a new row pins the refusal.

Krisztian F (krisztianfekete) · 2026-09-23T13:11:18Z

Suggested change

// traces, metrics, and logs may legitimately differ and are resolved separately.

Krisztian F (krisztianfekete) · 2026-09-23T13:12:05Z

logRelay shares the gate too now, want to update this?

Krisztian F (krisztianfekete) · 2026-09-23T13:14:19Z

+	collogspb.RegisterLogsServiceServer(s.grpc, &logRelay{
+		upstream: collogspb.NewLogsServiceClient(upstream),
+		headers:  logHeaders,
+		gate:     gate,
+	})


Just a forward thinking question for later steps: a new ateom that talks to an old atelet gets Unimplemented on logs. The SDK drops that batch without retrying, and Dial won't fall back because the socket exists I think. How were you planning for RelayCapable to handle that?

It won't, and we don't plan to make it. An old atelet returns Unimplemented, the SDK doesn't retry it, and Dial only falls back when the socket is missing, so a new ateom on an old atelet loses its log batches for the rollout window. We accept that: the project is early and doesn't carry backward compatibility across a version skew (#1644 broke the discovery response shape in place the same way, and #809 landed the relay with the same atelet-before-ateom ordering). The window is bounded, self-heals, and costs telemetry, not the actor; RelayCapable stays a resource attribute.

Krisztian F (krisztianfekete) · 2026-09-23T13:15:40Z

+	logs     []*collogspb.ExportLogsServiceRequest
 	traceMD  []metadata.MD
 	metricMD []metadata.MD
+	logMD    []metadata.MD


Are you planning to extend exportBoth and TestExportAttachesAteletHeaders with a logs coverage? That's the path where we most want proof that an ateom can't pick atelet's credentials.

Yes. exportBoth is now exportAll with a log export, TestExportDropsClientMetadata checks the log leg too, and TestExportAttachesAteletHeaders sets OTEL_EXPORTER_OTLP_LOGS_HEADERS and asserts the log export carries it and nothing of the client's.

Krisztian F (krisztianfekete) · 2026-09-23T13:17:22Z

+	}
+}
+
+func TestRelayRefusesNonAteomLogs(t *testing.T) {


Wdyt about doing this as a third Export in the TestRelayRefusesNonAteomSource table instead? Then logs also get the empty service.name case for free. Same idea for TestRelayAcceptsEveryAteomService and TestRelayRefusesMixedBatch.

Done for all three; the separate logs test is gone, and the empty service.name case now covers logs.

Krisztian F (krisztianfekete) · 2026-09-23T13:18:30Z

+func (l *logsSink) Export(ctx context.Context, req *collogspb.ExportLogsServiceRequest) (*collogspb.ExportLogsServiceResponse, error) {
+	l.parent.mu.Lock()
+	l.parent.logs = append(l.parent.logs, req)
+	if md, ok := metadata.FromIncomingContext(ctx); ok {
+		l.parent.logMD = append(l.parent.logMD, md.Copy())
+	}
+	l.parent.mu.Unlock()
+	l.parent.got <- struct{}{}
+	return &collogspb.ExportLogsServiceResponse{}, nil
+}


If you add a LOGS_ENDPOINT conflict check, would a logs row in TestUpstreamTarget be the natural place to check this?

Yes; three logs rows there (agrees, conflicts with generic, conflicts with traces) plus the row where logs is unset and falls back to the generic, and two in the compression table.

Krisztian F (krisztianfekete) · 2026-09-23T13:19:20Z

+carries logs, traces, and metrics, so a worker pod exports log records the same
+way it exports the other two signals.


Technically, worker pods will only export logs later, right?

Right. Split into two facts: the relay carries logs, traces, and metrics; worker pods do not emit log records yet.

The relay read the generic, traces, and metrics OTLP variables and ignored the logs ones, so a logs-specific collector or compression setting was silently overridden by whatever traces and metrics resolved to. All three signals now go through one resolver that requires every resolved value to agree, and its conflict error names the variable each value came from, since a signal that falls back to the generic pulls it in under a variable that is not itself set. Also extends the source-gate, header, and mixed-batch tests to the log path.

Krisztian F (krisztianfekete)

This LGTM, can you please update the PR description to reflect the new changes?

Tim Bai (baizhenyu) · 2026-09-23T15:30:23Z

This LGTM, can you please update the PR description to reflect the new changes?

Thanks! Description updated.

…bstrate#1857) Step 2 of agent-substrate#1748, plus the consumer rule from step 1b. `LoggingOptions` gains `ExporterConn` and `RelayCapable`, mirroring `TracingOptions`. Both ateoms initialize logging after metrics with the relay connection and defer a nil-guarded shutdown. `OTEL_LOGS_EXPORTER` still defaults to `none`, so no ateom emits a record; this is the provider the actor events land on when emission moves. A relay-capable log resource carries `ate.otlp.relay` like traces and metrics do. `newLoggerProvider` is split from `InitLogging` so the test asserts that on an emitted record's resource, the way `TestMeterProviderRelayAttribute` does for metrics. `docs/observability.md` states the consumer rule that replaced the relay denylist dropped in agent-substrate#1800: a lifecycle record is authoritative only under ateapi's resource, because the relay admits only ateom resources. Follow-up for the emission step, not here: the controller does not inject `OTEL_LOGS_EXPORTER` into worker pods, so no environment exercises log-over-relay yet.

otlprelay: forward log records

6f8db75

The relay carried traces and metrics. ateom is about to emit log records over it, so it forwards LogsService the same way: source gate, verbatim payload, atelet's own headers upstream.

Tim Bai (baizhenyu) requested review from Jeff Luo (JeffLuoo), Da Huang (git286) and Krisztian F (krisztianfekete) September 22, 2026 19:03

Jeff Luo (JeffLuoo) added the area/observability label Sep 22, 2026

Krisztian F (krisztianfekete) reviewed Sep 23, 2026

View reviewed changes

Krisztian F (krisztianfekete) approved these changes Sep 23, 2026

View reviewed changes

Jeff Luo (JeffLuoo) added this pull request to the merge queue Sep 23, 2026

Merged via the queue into agent-substrate:main with commit 35dde2e Sep 23, 2026
11 checks passed

Tim Bai (baizhenyu) mentioned this pull request Sep 24, 2026

serverboot: give the ateoms a LoggerProvider over the relay #1857

Merged

Bowei Du (bowei) added area/node kind/feature An enhancement / feature request or implementation labels Sep 29, 2026


	// traces, metrics, and logs may legitimately differ and are resolved separately.

		carries logs, traces, and metrics, so a worker pod exports log records the same
		way it exports the other two signals.

Conversation

Tim Bai (baizhenyu) commented Sep 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Krisztian F (krisztianfekete) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Krisztian F (krisztianfekete) left a comment

Choose a reason for hiding this comment

Uh oh!

Tim Bai (baizhenyu) commented Sep 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Tim Bai (baizhenyu) commented Sep 22, 2026 •

edited

Loading