Skip to content

Commit d64a905

Browse files
committed
actorevent: write each actor event to one place
An actor event went both to stdout and, with OTEL_LOGS_EXPORTER=otlp, as an OTLP event. On a cluster whose logging agent also collects pod stdout, as GKE's does, that stored every record twice. Now a record goes to OTLP when the logs exporter takes it, and to stdout otherwise. With the exporter unset or none, the default everywhere but kind, nothing changes. OTEL_LOGS_EXPORTER now takes a comma-separated list, as the spec allows, of otlp and console. Adding console to otlp keeps the stdout record beside the OTLP event, for operators who cannot lose a lifecycle record and for debugging. console is the component's own JSON log line, not the OTel SDK's console exporter format. InitLogging applies the setting, so ateapi and the ateoms follow it alike. kind sets otlp,console, and atecontroller copies the value into the worker pods, so kind keeps the actor events in kubectl logs. The list is read by a parser meant for every OTEL_*_EXPORTER variable, so the other signals can share its rules: names are trimmed, lowercased, and counted once; an unknown or empty item is skipped with a warning and the rest apply; none with an exporter, or a value with no known name, keeps the signal's default. It returns the set of selected exporters, which replaces the fixed logs exporter values. The record is built from the same call either way, so its fields cannot drift between the two forms. The lint exception that recorded the double write is removed. The docs state when otlp alone can lose a record, and that a collector reading both forms drops one by event.name.
1 parent fa858d0 commit d64a905

19 files changed

Lines changed: 588 additions & 336 deletions

File tree

‎cmd/ateapi/internal/controlapi/crash_test.go‎

Lines changed: 49 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ import (
2323
"slices"
2424
"strings"
2525
"sync"
26+
"sync/atomic"
2627
"testing"
2728
"time"
2829

@@ -695,12 +696,9 @@ func crashRecords(t *testing.T) *[]stdoutRecord {
695696
return logRecords(t, actorevent.Crashed.Body)
696697
}
697698

698-
// stdoutRecord is one captured record from the stdout copy. It keeps the level
699-
// and the time, not just the attributes, so a test can hold those against the
700-
// OTLP copy. The message is whatever logRecords filtered on.
699+
// stdoutRecord is the attributes of one captured stdout record. The message is
700+
// whatever logRecords filtered on.
701701
type stdoutRecord struct {
702-
level slog.Level
703-
time time.Time
704702
attrs map[string]string
705703
}
706704

@@ -714,7 +712,7 @@ func logRecords(t *testing.T, msg string) *[]stdoutRecord {
714712
if r.Message != msg {
715713
return
716714
}
717-
rec := stdoutRecord{level: r.Level, time: r.Time, attrs: map[string]string{}}
715+
rec := stdoutRecord{attrs: map[string]string{}}
718716
r.Attrs(func(a slog.Attr) bool {
719717
rec.attrs[a.Key] = a.Value.String()
720718
return true
@@ -725,33 +723,42 @@ func logRecords(t *testing.T, msg string) *[]stdoutRecord {
725723
return &records
726724
}
727725

728-
// otlpEvent is one captured record from the OTLP copy of a log record.
726+
// otlpEvent is one captured OTLP log record.
729727
type otlpEvent struct {
730-
name string
731-
body string
732-
severity otellog.Severity
733-
timestamp time.Time
734-
attrs map[string]string
728+
name string
729+
body string
730+
severity otellog.Severity
731+
attrs map[string]string
735732
}
736733

737734
var (
738735
otlpSinkOnce sync.Once
739736
otlpSinkMu sync.Mutex
740737
otlpSink []otlpEvent
738+
// otlpOn is whether the one global provider takes records. Off, the
739+
// actor events go to stdout, as when no logs exporter is configured.
740+
otlpOn atomic.Bool
741741
)
742742

743+
// switchProcessor takes records only while otlpOn is set, so one global
744+
// provider can stand for the logs exporter being on or off.
745+
type switchProcessor struct{ sdklog.Processor }
746+
747+
func (p switchProcessor) Enabled(ctx context.Context, param sdklog.EnabledParameters) bool {
748+
return otlpOn.Load() && p.Processor.Enabled(ctx, param)
749+
}
750+
743751
type otlpSinkExporter struct{}
744752

745753
func (otlpSinkExporter) Export(_ context.Context, records []sdklog.Record) error {
746754
otlpSinkMu.Lock()
747755
defer otlpSinkMu.Unlock()
748756
for _, r := range records {
749757
e := otlpEvent{
750-
name: r.EventName(),
751-
body: r.Body().String(),
752-
severity: r.Severity(),
753-
timestamp: r.Timestamp(),
754-
attrs: map[string]string{},
758+
name: r.EventName(),
759+
body: r.Body().String(),
760+
severity: r.Severity(),
761+
attrs: map[string]string{},
755762
}
756763
r.WalkAttributes(func(kv attribute.KeyValue) bool {
757764
e.attrs[string(kv.Key)] = kv.Value.String()
@@ -765,16 +772,17 @@ func (otlpSinkExporter) Export(_ context.Context, records []sdklog.Record) error
765772
func (otlpSinkExporter) Shutdown(context.Context) error { return nil }
766773
func (otlpSinkExporter) ForceFlush(context.Context) error { return nil }
767774

768-
// otlpEvents captures the events emitted while a test runs, so a test can assert
769-
// the OTLP copy beside the stdout one. The global logger provider only ever
770-
// delegates once, so one provider serves the whole binary and each call clears
771-
// the sink. Like logRecords, this makes the caller non-parallel.
775+
// otlpEvents turns the logs exporter on for the rest of the test and captures
776+
// the events emitted, which then go to OTLP instead of stdout. The global logger
777+
// provider only ever delegates once, so one provider serves the whole binary,
778+
// switched off between tests, and each call clears the sink. Like logRecords,
779+
// this makes the caller non-parallel.
772780
func otlpEvents(t *testing.T) func() []otlpEvent {
773781
t.Helper()
774782

775783
otlpSinkOnce.Do(func() {
776784
global.SetLoggerProvider(sdklog.NewLoggerProvider(
777-
sdklog.WithProcessor(sdklog.NewSimpleProcessor(otlpSinkExporter{}))))
785+
sdklog.WithProcessor(switchProcessor{sdklog.NewSimpleProcessor(otlpSinkExporter{})})))
778786
})
779787

780788
clear := func() {
@@ -783,7 +791,11 @@ func otlpEvents(t *testing.T) func() []otlpEvent {
783791
otlpSink = nil
784792
}
785793
clear()
786-
t.Cleanup(clear)
794+
otlpOn.Store(true)
795+
t.Cleanup(func() {
796+
otlpOn.Store(false)
797+
clear()
798+
})
787799

788800
return func() []otlpEvent {
789801
otlpSinkMu.Lock()
@@ -802,25 +814,20 @@ func (f slogHandlerFunc) Handle(_ context.Context, r slog.Record) error {
802814
func (f slogHandlerFunc) WithAttrs([]slog.Attr) slog.Handler { return f }
803815
func (f slogHandlerFunc) WithGroup(string) slog.Handler { return f }
804816

805-
// assertCopiesAgree checks the fields that no longer live at a call site. Both
806-
// copies take their severity and body from ev, so neither can hold its own. A
807-
// stdout body that drifted fails earlier, when logRecords matches nothing.
808-
func assertCopiesAgree(t *testing.T, stdout stdoutRecord, otlp otlpEvent, ev actorevent.Event) {
817+
// assertEvent checks the fields that no longer live at a call site: the event
818+
// takes its severity and body from ev, so it cannot hold its own.
819+
func assertEvent(t *testing.T, otlp otlpEvent, ev actorevent.Event) {
809820
t.Helper()
810821

811-
if stdout.level != ev.Level() {
812-
t.Errorf("stdout level = %v, want %v", stdout.level, ev.Level())
822+
if otlp.name != ev.Name {
823+
t.Errorf("event name = %q, want %q", otlp.name, ev.Name)
813824
}
814825
if otlp.severity != ev.Severity {
815826
t.Errorf("OTLP severity = %v, want %v", otlp.severity, ev.Severity)
816827
}
817828
if otlp.body != ev.Body {
818829
t.Errorf("OTLP body = %q, want %q", otlp.body, ev.Body)
819830
}
820-
// One time.Now() serves both, so a consumer can join them on it.
821-
if !stdout.time.Equal(otlp.timestamp) {
822-
t.Errorf("timestamps differ: stdout %v, OTLP %v", stdout.time, otlp.timestamp)
823-
}
824831
}
825832

826833
// The crash record is the only signal carrying actor identity, so it must fire
@@ -848,11 +855,17 @@ func TestCrashActor_RecordAndCounterAgree(t *testing.T) {
848855
if err := crashActor(ctx, st, actorRef, ateattr.OperationResume, "test crash"); err != nil {
849856
t.Fatalf("crashActor: %v", err)
850857
}
851-
if len(*records) != 1 {
852-
t.Fatalf("got %d crash records, want 1", len(*records))
858+
// The logs exporter is on, so the record goes to OTLP only.
859+
if len(*records) != 0 {
860+
t.Errorf("got %d stdout crash records with the exporter on, want 0", len(*records))
861+
}
862+
gotEvents := events()
863+
if len(gotEvents) != 1 {
864+
t.Fatalf("got %d crash events, want 1: %v", len(gotEvents), gotEvents)
853865
}
866+
assertEvent(t, gotEvents[0], actorevent.Crashed)
854867

855-
got := (*records)[0].attrs
868+
got := gotEvents[0].attrs
856869
stored, err := st.GetActor(ctx, actorRef)
857870
if err != nil {
858871
t.Fatalf("GetActor: %v", err)
@@ -873,27 +886,10 @@ func TestCrashActor_RecordAndCounterAgree(t *testing.T) {
873886
t.Error("crash record carries no ate.actor.uid; it cannot survive a name reuse")
874887
}
875888

876-
// The OTLP copy is the same record under an event name. One call writes both,
877-
// so anything either copy holds alone is a bug in actorevent.Log.
878-
gotEvents := events()
879-
if len(gotEvents) != 1 {
880-
t.Fatalf("got %d crash events, want 1: %v", len(gotEvents), gotEvents)
881-
}
882-
if gotEvents[0].name != actorevent.Crashed.Name {
883-
t.Errorf("event name = %q, want %q", gotEvents[0].name, actorevent.Crashed.Name)
884-
}
885-
if !maps.Equal(gotEvents[0].attrs, got) {
886-
t.Errorf("crash event attributes = %v, want the stdout record's %v", gotEvents[0].attrs, got)
887-
}
888-
assertCopiesAgree(t, (*records)[0], gotEvents[0], actorevent.Crashed)
889-
890889
// Re-crashing an already-crashed actor must move neither signal.
891890
if err := crashActor(ctx, st, actorRef, ateattr.OperationResume, "test crash"); err != nil {
892891
t.Fatalf("second crashActor: %v", err)
893892
}
894-
if len(*records) != 1 {
895-
t.Errorf("got %d crash records after re-crashing, want 1", len(*records))
896-
}
897893
if gotEvents := events(); len(gotEvents) != 1 {
898894
t.Errorf("got %d crash events after re-crashing, want 1", len(gotEvents))
899895
}

‎cmd/ateapi/internal/controlapi/workflow_test.go‎

Lines changed: 9 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,6 @@ package controlapi
1616

1717
import (
1818
"context"
19-
"maps"
2019
"testing"
2120

2221
"github.com/agent-substrate/substrate/cmd/ateapi/internal/store"
@@ -115,10 +114,16 @@ func TestActorStateChangeRecords(t *testing.T) {
115114
w := &ActorWorkflow{store: persistence}
116115
tt.transition(t, w, actorRef, actor, tmpl)
117116

118-
if len(*records) != 1 {
119-
t.Fatalf("got %d state records, want 1: %v", len(*records), *records)
117+
// The logs exporter is on, so the record goes to OTLP only.
118+
if len(*records) != 0 {
119+
t.Errorf("got %d stdout state records with the exporter on, want 0: %v", len(*records), *records)
120120
}
121-
got := (*records)[0].attrs
121+
gotEvents := events()
122+
if len(gotEvents) != 1 {
123+
t.Fatalf("got %d state events, want 1: %v", len(gotEvents), gotEvents)
124+
}
125+
assertEvent(t, gotEvents[0], actorevent.StateChanged)
126+
got := gotEvents[0].attrs
122127
want := map[string]string{
123128
string(ateattr.AtespaceKey): actorRef.Atespace,
124129
string(ateattr.ActorNameKey): actorRef.Name,
@@ -137,20 +142,6 @@ func TestActorStateChangeRecords(t *testing.T) {
137142
t.Errorf("got %d attributes, want %d: %v", len(got), len(want), got)
138143
}
139144

140-
// The OTLP copy is the same record under an event name. One call writes
141-
// both, so anything either copy holds alone is a bug in actorevent.Log.
142-
gotEvents := events()
143-
if len(gotEvents) != 1 {
144-
t.Fatalf("got %d state events, want 1: %v", len(gotEvents), gotEvents)
145-
}
146-
if gotEvents[0].name != actorevent.StateChanged.Name {
147-
t.Errorf("event name = %q, want %q", gotEvents[0].name, actorevent.StateChanged.Name)
148-
}
149-
if !maps.Equal(gotEvents[0].attrs, got) {
150-
t.Errorf("state event attributes = %v, want the stdout record's %v", gotEvents[0].attrs, got)
151-
}
152-
assertCopiesAgree(t, (*records)[0], gotEvents[0], actorevent.StateChanged)
153-
154145
stored, err := persistence.GetActor(ctx, actorRef)
155146
if err != nil {
156147
t.Fatalf("reload actor: %v", err)

‎cmd/ateapi/main.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -151,12 +151,12 @@ func main() {
151151

152152
lp, err := serverboot.InitLogging(ctx, serverboot.LoggingOptions{
153153
ServiceName: "ateapi",
154-
Exporter: serverboot.ResolveLogsExporter(ctx, serverboot.LogsExporterNone),
154+
Exporter: serverboot.ResolveLogsExporter(ctx),
155155
})
156156
if err != nil {
157157
serverboot.Fatal(ctx, "Failed to initialize logging", err)
158158
}
159-
// Nil when the exporter is none.
159+
// Nil when the exporter does not include otlp.
160160
if lp != nil {
161161
defer serverboot.ShutdownProvider("LoggerProvider", lp.Shutdown)
162162
}

‎cmd/atecontroller/internal/controllers/workerpool_apply.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,8 @@ type ateomOTelSettings struct {
7979
// default and drops the arg, which is dead config on its own.
8080
TracesSampler string
8181
TracesSamplerArg string
82-
// LogsExporter is the raw OTEL_LOGS_EXPORTER value. It turns on the OTLP
83-
// copy of the usage records; empty keeps ateom's default, none.
82+
// LogsExporter is the raw OTEL_LOGS_EXPORTER value. otlp sends the usage
83+
// records over OTLP instead of stdout; empty keeps ateom's default, none.
8484
LogsExporter string
8585
}
8686

‎cmd/ateom-gvisor/main.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -160,14 +160,14 @@ func do(ctx context.Context) error {
160160

161161
lp, err := serverboot.InitLogging(ctx, serverboot.LoggingOptions{
162162
ServiceName: serviceName,
163-
Exporter: serverboot.ResolveLogsExporter(ctx, serverboot.LogsExporterNone),
163+
Exporter: serverboot.ResolveLogsExporter(ctx),
164164
ExporterConn: relayConn,
165165
RelayCapable: true,
166166
})
167167
if err != nil {
168168
serverboot.Fatal(ctx, "Failed to initialize logging", err)
169169
}
170-
// Nil when the exporter is none.
170+
// Nil when the exporter does not include otlp.
171171
if lp != nil {
172172
defer serverboot.ShutdownProvider("LoggerProvider", lp.Shutdown)
173173
}

‎cmd/ateom-microvm/main.go‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -147,14 +147,14 @@ func do(ctx context.Context) error {
147147

148148
lp, err := serverboot.InitLogging(ctx, serverboot.LoggingOptions{
149149
ServiceName: serviceName,
150-
Exporter: serverboot.ResolveLogsExporter(ctx, serverboot.LogsExporterNone),
150+
Exporter: serverboot.ResolveLogsExporter(ctx),
151151
ExporterConn: relayConn,
152152
RelayCapable: true,
153153
})
154154
if err != nil {
155155
serverboot.Fatal(ctx, "Failed to initialize logging", err)
156156
}
157-
// Nil when the exporter is none.
157+
// Nil when the exporter does not include otlp.
158158
if lp != nil {
159159
defer serverboot.ShutdownProvider("LoggerProvider", lp.Shutdown)
160160
}

‎docs/dev/best-practices/otel-collector.md‎

Lines changed: 3 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -435,8 +435,9 @@ for a worked example.
435435

436436
**Substrate exports one thing over OTLP: the actor events**, ateapi's
437437
lifecycle records and the ateoms' usage samples, through
438-
`serverboot.InitLogging`. They are off unless
439-
`OTEL_LOGS_EXPORTER=otlp` is set, which only the kind overlay does today. See
438+
`serverboot.InitLogging`. `OTEL_LOGS_EXPORTER=otlp` sends them over OTLP
439+
instead of stdout, and `otlp,console` to both; only the kind overlay sets it, to
440+
`otlp,console`. See
440441
[the same records over OTLP](../../observability.md#the-same-records-over-otlp).
441442

442443
Everything else is stdout. `serverboot.InitLogger` writes structured JSON there,
@@ -462,14 +463,6 @@ onto the log record's own trace fields with no transformation.
462463
Those logs are collected by whatever agent already reads container stdout on
463464
your nodes. The collector is not in that path.
464465

465-
If you do add such an agent, note that the actor events, ateapi's lifecycle
466-
records and the ateoms' usage samples, go to both stdout and OTLP. Reading
467-
stdout as well gives you each record twice. ateapi runs in `ate-system` and the
468-
ateoms run in the worker pool namespaces, so a namespace exclusion does not
469-
cover both: drop the records by `event.name` (`ate.actor.state_changed`,
470-
`ate.actor.crashed`, `ate.actor.usage_sampled`), or exclude the
471-
`ate-api-server` and `ateom` containers by name.
472-
473466
The logs pipeline also serves **your** workloads: actors or services
474467
instrumented with the OpenTelemetry logs SDK that push OTLP log records to the
475468
same endpoint. It needs no collector change beyond being configured.

‎docs/metrics/substrate.yaml‎

Lines changed: 0 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -193,20 +193,6 @@ lint_exceptions:
193193
This label is older than the naming convention of the router. A change to
194194
ate.router.park.outcome needs a change to the dashboards at the same time.
195195
atenet.router.route.duration had the same change before.
196-
- attribute: none
197-
metric: none
198-
rule: One record goes to one place.
199-
brief: >
200-
ateapi writes each actor lifecycle record, and the ateoms each usage
201-
sample, twice: to stdout, and as an OTLP event. Nothing reads both
202-
today. No collector config in this repository declares a filelog
203-
receiver, and neither enterprise collector is a
204-
DaemonSet, thus no agent reads the stdout copy into the same pipeline.
205-
The stdout copy is what kubectl logs shows. The OTLP copy is what a
206-
backend queries and needs no knowledge of the stdout envelope.
207-
An agent that reads pod stdout makes this a duplicate. Then drop one:
208-
drop the records by event name, or leave the ate-api-server and ateom
209-
containers out of the globs of that agent.
210196
211197
# -----------------------------------------------------------------------------
212198
# The subsystems with no metrics. An agent that reads only the metrics gives the

0 commit comments

Comments
 (0)