[Design] Actor event routing: stdout, OTLP, or both #2197

Problem

Substrate emits two kinds of actor events through internal/actorevent:

On main, an event always goes to stdout. With OTEL_LOGS_EXPORTER=otlp, it also goes out as an OTLP event. A cluster whose logging agent also collects pod stdout, as GKE's does, then stores every event twice.

A simple fix is to send each event to one place. That removes the duplicates, but it also removes the stdout copy that protects the lifecycle stream. We need agreement on the questions below before we change the routing.

Current behavior and a proposal

OTEL_LOGS_EXPORTER controls the routing. The default is none. An OTLP endpoint alone does not turn on the OTLP copy.

In the proposal, the variable takes a comma-separated list, as the OTel spec allows, and console keeps the stdout copy beside the OTLP event. The default GKE install sets only the endpoint, so it writes stdout only, today and in the proposal. kind sets otlp today; the proposal changes it to otlp,console, so kind keeps both copies.

ateapi reads OTEL_LOGS_EXPORTER from the ate-otel-config ConfigMap. The ateoms get it from atecontroller, which copies its --otel-logs-exporter flag into each worker pod. The flag defaults to the controller's own OTEL_LOGS_EXPORTER, from the same ConfigMap. So today one key controls both streams.

Open questions

1. Is a lost lifecycle record acceptable with otlp alone?

A consumer takes the last lifecycle event for an actor, so a lost record shows a stale state with no signal. With otlp alone, the OTLP batch processor drops records with no signal when:

• its queue is full (2048 records),
• the collector stays unreachable after retries,
• the process exits ungracefully.

Graceful shutdown is safe: records written after shutdown go to stdout.

Options:

• (a) Accept it and document it. Operators who need a backup use otlp,console.
• (b) Always write lifecycle events to stdout. Only usage events go to one place. A GKE-style agent then stores lifecycle events twice again, but that stream is small.
• (c) Let operators choose per stream (question 2).

Proposal: (a). With otlp, each lifecycle event goes to OTLP only, and the docs state the loss cases above. Operators who cannot accept a lost lifecycle record set OTEL_LOGS_EXPORTER=otlp,console. Then each event also goes to stdout, and a cluster that collects pod stdout stores it twice. (b) and (c) stay open if reviewers disagree.

2. Should each stream have its own setting?

The two streams come from different processes, so they can have different values without new emitter code. For example, ateapi could use otlp,console (lifecycle events kept on stdout) and the worker pods otlp (usage events in one place).

Options:

• (a) Use the existing controller flag. Set --otel-logs-exporter in the ate-controller Deployment args. This needs no code, but the admin edits two files with two methods, and the flag is hard to find.

• (b) Add an optional ConfigMap key that only atecontroller reads, for example ATEOM_OTEL_LOGS_EXPORTER. The controller uses the flag first, then this key, then OTEL_LOGS_EXPORTER. If the key is not set, the behavior is the same as today. This is about 5 lines in cmd/atecontroller/main.go, plus docs. The worker pod still gets an env var named OTEL_LOGS_EXPORTER, so the ateoms do not change.

• (c) Keep one setting. OTEL_LOGS_EXPORTER in ate-otel-config stays the single source of truth for both streams, as today. No code change.

If (b): name the key by component (ATEOM_...) or by stream (usage events)? A component name describes the mechanism. A stream name matches how admins think, but it becomes wrong if a stream moves to a different component.

A change to the worker pods' value needs a controller restart, which rolls every WorkerPool Deployment and replaces the workers.

Proposal: (c). One key in one file is the easiest setting to find, change, and audit. With the proposal for question 1, an operator who cannot accept a lost lifecycle record sets otlp,console; usage events then also go to stdout, which is the cost of the single setting. (a) or (b) can follow later if operators need a different value per stream.

3. Is console the right name?

In the proposal, console means the component's own JSON log line, not the OTel SDK's console exporter format. It follows the OTel spec's list syntax, but an operator can expect the SDK format. Alternatives: keep console and document the difference, or use a Substrate-specific setting.

Related

• [Design] Actor lifecycle events #1683: design of the actor lifecycle events.
• Collect component stdout logs with a filelog receiver #1741: collect component stdout with a filelog receiver. If that lands, stdout and OTLP feed one pipeline, and the duplicate question applies there too.
• atelet: drop its actor usage records and count each activation's CPU once #1984: the ateoms emit the usage events.