Skip to content

Lifecycle metrics omit empty template labels, and a crash during a delete reports the delete operation - #2356

Merged
Krisztian F (krisztianfekete) merged 4 commits into
agent-substrate:mainfrom
JeffLuoo:docs/update-metrics-registry
Oct 13, 2026
Merged

Krisztian F (krisztianfekete) merged 4 commits into
agent-substrate:mainfrom
JeffLuoo:docs/update-metrics-registry

Conversation

@JeffLuoo

@JeffLuoo Jeff Luo (JeffLuoo) commented Oct 8, 2026 •

Copy link
Copy Markdown
Collaborator

Behavior changes

  • ate.actor.crashes: when ateapi loses the worker of an actor in DELETING (possible with any_state), the crash now has ate.actor.operation.name=delete. Before this change it had unknown.
  • The crash message in the actor status now starts with delete failed: in that case. Before this change it had no prefix.
  • ate.actor.lifecycle.operation.duration: when resume, suspend, pause, or revert fails before ateapi loads the actor, the series now omits ate.template.name and ate.template.atespace. Before this change it had them as the empty string.
  • rpc.server.call.duration and rpc.client.call.duration now list rpc.system.name in the registry, and the client histogram also lists server.address and server.port. This is only a change to the docs. otelgrpc already sent these keys.

@baizhenyu Tim Bai (baizhenyu) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the sweep. I checked each changed claim against main, and most hold: the reverting state, revert ending in SUSPENDED, the RESOURCE_EXHAUSTED form from otelgrpc's canonicalString, the ateoms having no meter beyond the server handler, the final record on shutdown and micro-VM re-host with the pending fallback, and both new code anchors. hack/verify/metrics.sh passes.

A few places still differ from the code, inline:

  1. Pool keys on a delete with any_state.
  2. unknown also covers a lost worker during DELETING.
  3. The empty template strings on early failures (a code fix would let the registry say "omitted").
  4. credprovider names only the kubernetes-secrets binary; the GCP Secret Manager plugin emits nothing.
  5. The histogram note now repeats the per-attribute conditions.

Comment thread docs/metrics/registry/metrics.yaml Resolved
Comment thread docs/metrics/registry/metrics.yaml Resolved
Comment thread docs/metrics/registry/metrics.yaml Resolved
Comment thread docs/metrics/registry/metrics.yaml Resolved
Comment thread docs/metrics/registry/metrics.yaml Resolved
- Add reverting to ate.actor.state to match ateapipb.ActorState and RevertActor state-change events.
- Update ate.actor.lifecycle.operation.duration attribute requirements and notes for create and delete, which do not resolve ate.sandbox.class or snapshot attributes and omit template keys on a failed delete.
- Update rpc.* examples, emitters, and code anchors, plus ate.stats.kind=final and ate.actor.operation.name=unknown descriptions to match current ateapi, ateom, and credprovider behavior.
…riggers

- Clarify ate.actor.lifecycle.operation.duration label sets for suspend, pause, and revert, empty template labels on pre-load failures, and missing ate.sandbox.class when revert runs without a template.
- Add the ateom-microvm re-host trigger and unread-teardown case to ate.stats.kind=final, and update revert/reverting briefs to not assume an existing snapshot.
- Align rpc.server.call.duration emitters and lifecycle snapshot labels in docs/observability.md.
@JeffLuoo
Jeff Luo (JeffLuoo) force-pushed the docs/update-metrics-registry branch from 0ddc8c9 to dd3eaff Compare October 9, 2026 17:16
…provider

- crashBoundActor records a lost worker during DELETING as the delete operation, not unknown. A delete with any_state can run on an actor that still has a worker.
- lifecycleOpAttrs omits the template keys when the workflow holds no actor. Before this change, a failure before or in the load step recorded the template keys as the empty string.
- The lifecycle histogram note no longer repeats each attribute condition. The template keys now state that a failed load omits them. The pool keys now state that a delete never records them.
- The rpc.server.call.duration note names the kubernetes-secrets binary as credprovider. blind_spots now lists the GCP Secret Manager plugin, which has no metrics.
- The lifecycle row in docs/observability.md names ate.snapshot.fidelity, which replaced the snapshot scope.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, two things before merge:

  1. The rpc entries still miss keys otelgrpc emits: rpc.system.name on both, + server.address and server.port on rpc.client.call.duration. Please add the refs. server.address sets the ateom socket path, which has the pod UID, so maybe file a followup for a View that drops it?
  2. This now changes behavior (crash op label for DELETING, the delete failed: crash message, no empty template series), so please retitle it away from docs/ and list those changes in the description.

otelgrpc v0.70.0 sets rpc.system.name on rpc.server.call.duration and on rpc.client.call.duration. On the client histogram it also sets server.address and server.port from the remote address of the connection. The registry did not list these keys.

The registry now defines the three keys and the two histograms reference them. server.port is absent on a call over a unix socket. The brief of server.address says that atelet calls each ateom on a socket path that holds the worker pod UID, so on atelet the key has one value for each worker pod.
@JeffLuoo Jeff Luo (JeffLuoo) changed the title docs/metrics: align registry attributes and metadata with implementation Lifecycle metrics omit empty template labels, and a crash during a delete reports the delete operation Oct 13, 2026
@JeffLuoo

Copy link
Copy Markdown
Collaborator Author

Updated, and created #2556

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for #2556.

Non-blocking: rpc.system.name and the other borrowed rpc.*/file.name keys are release_candidate/development upstream, not stable, and metrics.go:164 still leaves revert out of the instrument description.

@JeffLuoo

Copy link
Copy Markdown
Collaborator Author

Thanks for your review Krisztian F (@krisztianfekete) I will work on updating them in a follow-up PR. Can you help add this change to the merge queue?

@krisztianfekete
Krisztian F (krisztianfekete) added this pull request to the merge queue Oct 13, 2026
Merged via the queue into agent-substrate:main with commit 8a5b3a4 Oct 13, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/observability kind/docs Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants