Repository navigation
Lifecycle metrics omit empty template labels, and a crash during a delete reports the delete operation - #2356
Conversation
Tim Bai (baizhenyu)
left a comment
There was a problem hiding this comment.
Thanks for the sweep. I checked each changed claim against main, and most hold: the reverting state, revert ending in SUSPENDED, the RESOURCE_EXHAUSTED form from otelgrpc's canonicalString, the ateoms having no meter beyond the server handler, the final record on shutdown and micro-VM re-host with the pending fallback, and both new code anchors. hack/verify/metrics.sh passes.
A few places still differ from the code, inline:
- Pool keys on a delete with
any_state. unknownalso covers a lost worker during DELETING.- The empty template strings on early failures (a code fix would let the registry say "omitted").
credprovidernames only the kubernetes-secrets binary; the GCP Secret Manager plugin emits nothing.- The histogram note now repeats the per-attribute conditions.
- Add reverting to ate.actor.state to match ateapipb.ActorState and RevertActor state-change events. - Update ate.actor.lifecycle.operation.duration attribute requirements and notes for create and delete, which do not resolve ate.sandbox.class or snapshot attributes and omit template keys on a failed delete. - Update rpc.* examples, emitters, and code anchors, plus ate.stats.kind=final and ate.actor.operation.name=unknown descriptions to match current ateapi, ateom, and credprovider behavior.
…riggers - Clarify ate.actor.lifecycle.operation.duration label sets for suspend, pause, and revert, empty template labels on pre-load failures, and missing ate.sandbox.class when revert runs without a template. - Add the ateom-microvm re-host trigger and unread-teardown case to ate.stats.kind=final, and update revert/reverting briefs to not assume an existing snapshot. - Align rpc.server.call.duration emitters and lifecycle snapshot labels in docs/observability.md.
0ddc8c9 to
dd3eaff
Compare
…provider - crashBoundActor records a lost worker during DELETING as the delete operation, not unknown. A delete with any_state can run on an actor that still has a worker. - lifecycleOpAttrs omits the template keys when the workflow holds no actor. Before this change, a failure before or in the load step recorded the template keys as the empty string. - The lifecycle histogram note no longer repeats each attribute condition. The template keys now state that a failed load omits them. The pool keys now state that a delete never records them. - The rpc.server.call.duration note names the kubernetes-secrets binary as credprovider. blind_spots now lists the GCP Secret Manager plugin, which has no metrics. - The lifecycle row in docs/observability.md names ate.snapshot.fidelity, which replaced the snapshot scope.
dd3eaff to
f520175
Compare
Krisztian F (krisztianfekete)
left a comment
There was a problem hiding this comment.
Thanks, two things before merge:
- The rpc entries still miss keys otelgrpc emits:
rpc.system.nameon both, +server.addressandserver.portonrpc.client.call.duration. Please add the refs.server.addresssets the ateom socket path, which has the pod UID, so maybe file a followup for a View that drops it? - This now changes behavior (crash op label for DELETING, the
delete failed:crash message, no empty template series), so please retitle it away fromdocs/and list those changes in the description.
otelgrpc v0.70.0 sets rpc.system.name on rpc.server.call.duration and on rpc.client.call.duration. On the client histogram it also sets server.address and server.port from the remote address of the connection. The registry did not list these keys. The registry now defines the three keys and the two histograms reference them. server.port is absent on a call over a unix socket. The brief of server.address says that atelet calls each ateom on a socket path that holds the worker pod UID, so on atelet the key has one value for each worker pod.
|
Updated, and created #2556 |
Krisztian F (krisztianfekete)
left a comment
There was a problem hiding this comment.
LGTM, thanks for #2556.
Non-blocking: rpc.system.name and the other borrowed rpc.*/file.name keys are release_candidate/development upstream, not stable, and metrics.go:164 still leaves revert out of the instrument description.
|
Thanks for your review Krisztian F (@krisztianfekete) I will work on updating them in a follow-up PR. Can you help add this change to the merge queue? |
Behavior changes
ate.actor.crashes: when ateapi loses the worker of an actor in DELETING (possible withany_state), the crash now hasate.actor.operation.name=delete. Before this change it hadunknown.delete failed:in that case. Before this change it had no prefix.ate.actor.lifecycle.operation.duration: when resume, suspend, pause, or revert fails before ateapi loads the actor, the series now omitsate.template.nameandate.template.atespace. Before this change it had them as the empty string.rpc.server.call.durationandrpc.client.call.durationnow listrpc.system.namein the registry, and the client histogram also listsserver.addressandserver.port. This is only a change to the docs. otelgrpc already sent these keys.