actorevent: define the actor usage event and the activation epoch by baizhenyu · Pull Request #1881 · agent-substrate/substrate

Tim Bai (baizhenyu) · 2026-09-25T14:30:28Z

Step 3 of #1748. Vocabulary only: nothing emits the event or sets the epoch yet.

Event. ate.actor.usage_sampled in actorevent, sixteen required keys via ateattr: identity, pool, sandbox class, source, ate.stats.kind (periodic, initial, final), the four measurements, observation time, epoch. Measurement names follow the ate.actor.stats.* instruments; units are in the registry briefs. Body is Actor usage sampled, the accepted stdout shape change.

Scope. The usage stream has its own instrumentation scope, the lifecycle scope with /usage appended, so a consumer can sample or drop it without touching lifecycle records. NewEmitter takes the scope; the package-level Log keeps the lifecycle one, so ateapi is unchanged.

Epoch. ate.actor.epoch and WorkloadStatsSample.epoch_unix_nano: the unix-nano time an activation began. CPU time is per activation for every source; lifetime is the sum over epochs of each epoch's highest value. Memory usage and working set are absolute; peak is as the source reports it. An ateom that leaves the epoch at zero still sends the raw guest counter. The measurement comments on fields 8 to 11 say all of this.

Registry. Event group in events.yaml, seven attributes in metrics.yaml, ate.actor.epoch in the no-actor-identity forbidden list. The one-record-one-place rule and the collector guide now name both scopes.

Left for step 4: the event's code anchor moves to the ateom emit site, atelet's poller comments and the guide's stdout usage section get rewritten, and the controller starts passing OTEL_LOGS_EXPORTER to worker pods.

Krisztian F (krisztianfekete) · 2026-09-25T17:08:59Z

Wdyt about taking the event time instead? For a usage sample, that should be when we read it, not when we write it. Then ate.stats.observed_at is not even needed, right?

Agreed. LogAt takes the time; the step 4 emitter passes the read time. ate.stats.observed_at is gone.

Krisztian F (krisztianfekete) · 2026-09-25T17:10:08Z

+        stability: development
+        type: int
+        brief: When the measurement was read, as the unix time in nanoseconds.
+        note: Logs only. The record timestamp is when the record was written, which can be later.


The OTel log model has this in reverse. Timestamp is when the thing happened and ObservedTimestamp is when we saw it. The read time fits Timestamp exactly, so I'd drop this key, see my other comment above.

Dropped. The read time is the record Timestamp.

Krisztian F (krisztianfekete) · 2026-09-25T17:11:22Z

+      - id: ate.stats.cpu.time
+        stability: development
+        type: int
+        brief: The CPU time used since the activation began, in microseconds.


Same name as ate.actor.stats.cpu.time, but that one is in seconds (so are semconv's container.cpu.time and process.cpu.time). Can we emit a double in seconds so a query works the same on both signals?

Done: double, seconds. The wire stays microseconds; the step 4 emitter converts.

Krisztian F (krisztianfekete) · 2026-09-25T17:12:33Z

+// NewEmitter writes under scope, one of the scope names above.
+func NewEmitter(lp log.LoggerProvider, scope string) *Emitter {


Do we even need a second scope at all? event_name already distinguishes two streams.

Agreed, one scope. NewEmitter lost the parameter and the docs select by event.name.

Krisztian F (krisztianfekete) · 2026-09-25T17:14:16Z

Can we clamp this like addSat in statspoller.go and add a math.MaxUint64 test case?

Done, with a MaxUint64 case.

Krisztian F (krisztianfekete) · 2026-09-25T17:17:15Z

+  // every source. The cgroup source restarts on its own; the guest-agent
+  // source is rebased on its first read after a restore, since the guest's
+  // counters survive in guest RAM. An ateom that leaves epoch_unix_nano at zero


This is coming later only, right?

Yes, step 4. Until then an ateom leaves the epoch at zero and sends the raw counter, as the next sentence says.

Krisztian F (krisztianfekete) · 2026-09-25T17:18:41Z

+      - ref: ate.stats.memory.usage
+        requirement_level: required


How about conditionally_required when the source isn't unspecified, so pending actors don't send fake zeros that skew the avg? Same for memory.peak on old kernels.

Agreed. The four measurements are conditional on the source not being unspecified, and peak also on the source reporting one. A pending record carries identity, pool, class, source, kind and epoch only. Event gained a Conditional list and the registry test reads both levels.

Krisztian F (krisztianfekete) · 2026-09-25T17:19:25Z

+      - ref: ate.workerpool.namespace
+        requirement_level: required


Should we match this with:

substrate/docs/metrics/registry/metrics.yaml

Lines 897 to 899 in 54cba01

- ref: ate.workerpool.namespace

requirement_level:

conditionally_required: The pool of the worker is known.

?

I would keep them required here. The instrument's condition exists because atelet resolves the pool from a pod list that can miss. The step 4 emitter gets both from the downward API, metadata.namespace and the ate.dev/worker-pool label, so it always knows them.

Krisztian F (krisztianfekete) · 2026-09-25T17:22:14Z

+        brief: >
+          The activation of the actor that a record belongs to, as the unix
+          time in nanoseconds when it began.


This is basically OTel's cumulative StartTimeUnixNano, right?

Exactly that. The note now says so.

Krisztian F (krisztianfekete) · 2026-09-25T17:23:44Z

+
+// TestEmitterScope pins that records land under the scope the emitter was
+// built with, so the usage stream cannot share the lifecycle scope by accident.
+func TestEmitterScope(t *testing.T) {


Nit: if we do this as a table with t.Run it would match the other tests here. But it goes away anyway if we drop the second scope, see my comment above.

Gone with the second scope.

ate.actor.usage_sampled is the record an ateom will emit per actor: identity, pool, sandbox class, source, kind, the activation epoch, and, once the actor is measurable, four measurements. The measurements are conditional so a pending actor sends no fake zeros; absence means not measured. They share the names and units of the ate.actor.stats.* instruments, so one query serves both signals. The record's timestamp is the read time, which LogAt now takes from the caller. A uint64 attribute above MaxInt64 now clamps rather than wraps, as the metric side does. The epoch identifies one activation of an actor, a Run or Restore, as the unix-nano time it began: the StartTimeUnixNano of a cumulative metric point, carried as an attribute. It is what makes CPU time summable: within an epoch the counter is monotonic from zero, so lifetime CPU is the sum over epochs of each epoch's highest value. Without it, no comparison of consecutive samples finds every boundary, since a restored guest can resume its counter above the last reported value. WorkloadStatsSample carries the epoch too, and its measurement comments now say which figures are per activation, which are absolute, and which are raw. The registry declares the event and its attributes, all logs-only, and bars the epoch from metric labels alongside the other per-actor keys. Nothing emits the event or sets the epoch yet.

Tim Bai (baizhenyu) force-pushed the actor-usage-event branch 3 times, most recently from 366f313 to 1d0a91f Compare September 25, 2026 15:05

Tim Bai (baizhenyu) requested review from Jeff Luo (JeffLuoo), Da Huang (git286) and Krisztian F (krisztianfekete) and removed request for Krisztian F (krisztianfekete) September 25, 2026 15:59

Krisztian F (krisztianfekete) reviewed Sep 25, 2026

View reviewed changes

Tim Bai (baizhenyu) added kind/feature An enhancement / feature request or implementation area/observability labels Sep 25, 2026

Tim Bai (baizhenyu) force-pushed the actor-usage-event branch from 1d0a91f to f30634d Compare September 25, 2026 19:28

Tim Bai (baizhenyu) requested a review from Krisztian F (krisztianfekete) September 25, 2026 19:33

Krisztian F (krisztianfekete) approved these changes Sep 25, 2026

View reviewed changes

Jeff Luo (JeffLuoo) added this pull request to the merge queue Sep 25, 2026

Merged via the queue into agent-substrate:main with commit 03ba582 Sep 25, 2026
11 checks passed

		// NewEmitter writes under scope, one of the scope names above.
		func NewEmitter(lp log.LoggerProvider, scope string) *Emitter {

	- ref: ate.workerpool.namespace
	requirement_level:
	conditionally_required: The pool of the worker is known.

Conversation

Tim Bai (baizhenyu) commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Tim Bai (baizhenyu) commented Sep 25, 2026 •

edited

Loading