Skip to content

Drop per-pod server.address label from rpc.client.call.duration metrics - #2576

Merged
Jeff Luo (JeffLuoo) merged 2 commits into
agent-substrate:mainfrom
JeffLuoo:rpc-client-drop-server-address
Oct 14, 2026
Merged

Jeff Luo (JeffLuoo) merged 2 commits into
agent-substrate:mainfrom
JeffLuoo:rpc-client-drop-server-address

Conversation

@JeffLuoo

Copy link
Copy Markdown
Collaborator

otelgrpc sets server.address on rpc.client.call.duration from the remote address of the client connection: atelet dials each ateom over a Unix socket path containing the worker pod UID, and TCP clients record the IP address of an individual server pod. This produced one server.address value per pod, causing metric series cardinality to grow with worker pod count and churn.

  • Add a metric View in serverboot.newMeterProvider that strips server.address from all instruments in the otelgrpc scope while keeping server.port (one value per service).
  • Remove server.address from docs/metrics/registry/metrics.yaml and docs/metrics/substrate.yaml, and document the view on rpc.client.call.duration.

Fixes #2556

  • Tests pass
  • Appropriate changes to documentation are included in the PR

otelgrpc sets server.address on rpc.client.call.duration from the remote address of the client connection. atelet calls each ateom on a socket path that holds the worker pod UID, and a TCP client gets the IP address of one server pod. Thus the key had one value for each pod, and the number of series grew with the number of worker pods.

newMeterProvider now adds a View that removes server.address from each instrument of the otelgrpc scope. Every component builds its MeterProvider there, so the View covers each client. The scope match also covers the older histogram that otelgrpc records when OTEL_SEMCONV_STABILITY_OPT_IN asks for it. server.port stays, because it has one value for each service.

The registry no longer lists server.address on rpc.client.call.duration, and the note of the histogram says why the key is absent.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM, once the lint error is fixed!

attribute.Value.Emit is deprecated, and staticcheck fails on it.
@JeffLuoo

Copy link
Copy Markdown
Collaborator Author

fixed

@JeffLuoo
Jeff Luo (JeffLuoo) added this pull request to the merge queue Oct 14, 2026
Merged via the queue into agent-substrate:main with commit 94fc611 Oct 14, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

rpc.client.call.duration has one server.address value for each worker pod

2 participants