Repository navigation
Record actor state changes from ateapi - #1638
Jeff Luo (JeffLuoo) merged 3 commits into
Conversation
27cb0c1 to
647bdab
Compare
647bdab to
e67104f
Compare
Krisztian F (krisztianfekete)
changed the title
Jeff Luo (JeffLuoo)
left a comment
Jeff Luo (JeffLuoo)
left a comment
There was a problem hiding this comment.
Have you looked at OTel entity events? This record is effectively an entity.state stream for ate.actor — same identity/description/relationship split, same last-write-wins contract. Not a blocker for this PR given entity event is still in development stage.
Two comments -
-
There's no terminal record.
deletingis the last state a deleted actor ever reports, so "gone" and "stuck deleting" look identical to a consumer. Can we log the actual delete as well? -
Transition-only emission has no cold-start or expiry story: an actor that hasn't transitioned inside the retention window has no state, and a missed record is indistinguishable from steady state. Just document it somewhere.
Yes, that's exactly the what I am designing this around.
Thanks, added these! |
| // | ||
| // Crashes go through logActorCrashed instead, so read the state off | ||
| // ate.actor.state rather than off the message. | ||
| func logActorStateChanged(ctx context.Context, actor *ateapipb.Actor, opName, state string) { |
There was a problem hiding this comment.
I think "creation" should also be covered. When an actor is created, it holds a suspended stage: https://github.com/agent-substrate/substrate/blob/main/cmd/ateapi/internal/controlapi/actor.go#L112
An actor created and never resumed has no state at all in the stream — not "unknown since retention", genuinely never emitted, even with infinite retention. If we have creation event, it can at least record something in logs for this actor.
My understanding of this logs like "logs for all lifecycle of an actor".
There was a problem hiding this comment.
Added. CreateActor logs now with operation=create, reporting suspended since that's what a new actor actually is.
| // | ||
| // Crashes go through logActorCrashed instead, so read the state off | ||
| // ate.actor.state rather than off the message. | ||
| func logActorStateChanged(ctx context.Context, actor *ateapipb.Actor, opName, state string) { |
There was a problem hiding this comment.
Regarding the state field in the parameters.
substrate/internal/ateattr/ateattr.go
Lines 285 to 300 in 44f4000
The function takes the state as a literal, but storedActor is the committed record and already carries it. That means each of the eight call sites hand-pairs a constant with whatever the mutate closure assigned, and nothing catches a mismatch. A wrong literal produces a record that confidently reports a state the store never held.
ateattr already has the pattern in SnapshotScopeValue as attached above. Can you update the function to grep the state from actor *ateapipb.Actor?
There was a problem hiding this comment.
Added ActorStateValue so the helper reads state off the committed record. Only the delete-terminal one still names it by hand, since the row is gone by then.
Record actor state changes from ateapi
ateapi now writes a log record every time an actor changes state.
Before
Nothing tells you what state an actor is in. Metrics can't carry actor identity, and the control plane samples traces at 10%, so most transitions leave nothing behind at all. If you want to know whether an agent is running, paused or suspended, there's nowhere to look.
After
One filter:
Last record wins. That's the state it's in now, and the timestamp is when it got there.
The record:
{ "time": "…", "level": "INFO", "msg": "Actor state changed", "ate.atespace": "ate-demo-counter", "ate.actor.name": "counter-1", "ate.actor.uid": "8f2a…", "ate.template.atespace": "ate-demo-counter", "ate.template.name": "counter", "ate.actor.operation.name": "suspend", "ate.actor.state": "suspended" }The two new keys
ate.actor.stateis theateapipb.ActorStatevalues lowercased, so the log wording and the state machine can't drift apart.ate.actor.operation.namesays what caused the change. The state on its own doesn't tell you: an actor lands insuspendedfrom a suspend andpausedfrom a pause, and only one of those gives the worker back.Actor crashedpicks up the same state key. A crash is a state change too, and it's the one an actor can reach without any operation finishing.Notes for review
Why ateapi. It owns the state machine. It also sees transitions that never reach a worker, like deleting an actor that was already suspended, or a resume that fails in the scheduler.
It can't log a state the store never had. The record goes out after the store commit, and every state commit has a version precondition, so a losing writer in a concurrent update writes nothing.