Repository navigation
Many dimensions with http.client.open_connections #122752
Description
Description
See https://www.reddit.com/r/dotnet/comments/1pwkqfl/looking_for_example_code_of_opentelemetry_tracing/nw4k1wy/ and Azure/azure-sdk-for-net#52674
The tags on the metric should be low cardinality. High cardinality tags creates many dimensions.
Many dimensions are bad:
- Some tooling will stop recording values when dimension limit is reached (prevent excessive memory use)
- Observability platforms can charge per-dimension stored, so many dimensions could create a large bill
Reproduction Steps
Send requests to many servers. Each server adds multiple dimensions to the metric because "server.address", "server.port" and "network.peer.address" tags will be different for each server.
Expected behavior
Constrained number of dimensions.
Actual behavior
Unconstrained number of dimensions.
Regression?
No response
Known Workarounds
No response
Configuration
No response
Other information
No response
Reacted by Armin Shoeibi, kevbry and Erwin
Activity
dotnet-policy-service commentedon Dec 27, 2025
Tagging subscribers to this area: @dotnet/ncl
See info in area-owners.md if you want to be subscribed.
kevbry commentedon Dec 27, 2025
Thanks for creating this!
IMO the issue with http.client.open_connections isn't necessarily that there are too many dimensions, it's that it continues to emit measurements even after those measurements become meaningless. For http.client.open_connections, measurements are emitted when both the idle and active counts for a given endpoint are zero. In other words, making a single http request to a given endpoint starts two new endless streams of measurements (one for each of the possible values of http.connection.state), all but the first dozen or so of which always have a count of zero.
Reducing the cardinality by clipping out the remote IP is the only way to reduce the noise right now because of the lack of filtering APIs in otel, but a better solution would be to stop emitting measurements once the connection left the pool and stopped being an open_connection.
Clockwork-Muse commentedon Dec 27, 2025
It's true that there isn't a "standard" API, for this, but this is one of the things you can do if you deploy a collector; eg configure the filter processor.
For cloud tracing backends, it may also be possible to separately configure them to drop or otherwise exclude the records in such a way that they aren't counted as a chargeable event.
kevbry commentedon Dec 27, 2025
Sorry, I should have been more specific - I meant in the .net otel library, which doesn't have APIs allowing filtering of metrics.
With a quick read through the docs on filter processor I don't see how to express what I need to do here, which is omit measurements which match on all tags aside from http.connection.state and where both counts are zero. Even if it is possible, I'd imagine it's going to be computationally expensive to do so.
ManickaP commentedon Jan 7, 2026
So the tags are defined by OTel: https://opentelemetry.io/docs/specs/semconv/dotnet/dotnet-http-metrics/#metric-httpclientopen_connections
Address is only "recommended", otoh port is mandatory if it's not the default value and having port specified without address seems wrong to me.
@noahfalk is there a way to actually stop emitting the counter once it's been started?
If so, I don't see an issue with us doing that. If not, maybe we should explore a way how to do that.
Disclaimer, I was not present during the OTel discussions, so feel free to correct me on anything.
noahfalk commentedon Jan 8, 2026
One option is to switch from UpDownCounter to ObservableUpDownCounter. This will give you a callback every time the metric is going to be emitted. The http client code needs to provide the value of the metric for each tag combination you want to report. Any combinations you don't emit during the callback (for example all the logically zero-valued ones for old connections) shouldn't be transmitted or stored.
The other option, more of a workaround IMO, would be encouraging users to use cardinality limits to ensure that not too many different tag combinations are sending data at the same time. I don't recall if the OTel SDK is sophisticated enough to prioritize the more recently added tag combinations which users would care about but at minimum it can ensure the total amount of data is constrained.
HTH!
JamesNK commentedon Jan 8, 2026
The semantic conventions aren't perfect. They might not be aware this is causing issues. I suggest you create an issue in their GH repository to discuss the problem.
ManickaP commentedon Jan 8, 2026
So the OTel states which type of instrument to use: https://github.com/open-telemetry/semantic-conventions/blob/main/docs/dotnet/dotnet-http-metrics.md#metric-httpclientopen_connections
I do not understand the diffrences enough to be able to say whether we can change it to its "observable" version. Can this have an impact on the listeners? Can this influence the observation tools and visualization?
I still think it would be valuable to be able to "remove" the irrelevant values/dimensions from the reported results / observed counters. @noahfalk you already have an issue on the similar note #83822 it seems.
And I can file this up with OTel, but IMHO the address is valuable dimmension to this counter, so I'd rather see this resolved in less radical way.
Oh and @JamesNK, could the original reporter try out the cardility limits suggestion from Noah?
noahfalk commentedon Jan 8, 2026
The difference between Counter/UpDownCounter/Histogram are important and determines the shape of the emitted data and how down-stream consumers would interpret it. The difference between the Observable and non-Observable forms however are (with some small caveats*) implementation choices that you are free to switch between. I don't expect anyone will complain if you switched this to Observable and I've never interpreted the semantic convention that says UpDownCounter to be mandating the specifically the non-observable form of it. Its an awkward naming issue where the name for the set of {UpDownCounter, ObservableUpDownCounter} is also UpDownCounter so its hard to tell when someone is refering equally to both or specifically to the non-observable form.
(*) The caveats I'm aware of if you are curious:
JamesNK commentedon Jan 8, 2026
You can ping the reporters. No point me sitting in the middle.
I suggest talking with OTEL folks not necessarily to remove the attributes, but to come up with an agreed way to handle this situation. They can put a note on the attribute that details what to do. Then you’d still be following the spec and not doing your own thing.
15 remaining items
cijothomas commentedon Jan 14, 2026
@noahfalk You are right. OTel conventions allow using sync or observable variant
https://github.com/open-telemetry/semantic-conventions/blob/main/docs/general/metrics.md#instrument-types
One possible sequence to address the issue:
acu-kevinb commentedon Jan 14, 2026
This was the only information I could find quickly on how to enable it, and I assumed it would result in the set of reported measurements usually containing only those that had reported a new value in the past interval, but after 15 minutes or so I was still seeing a lot of pairs that were both zero for several intervals.
It's very possible that I'm just configuring it incorrectly or I've made a bad assumption about how it's supposed to work.
victormarante commentedon Feb 20, 2026
Are there any updates on this issue, or potential work arounds? We are currently hitting daily cap on log analytics ingestion, where ~45% ingestion are http.* custom metrics.
kevbry commentedon Feb 20, 2026
@victormarante
I used a metric view to trim out the highest-cardinality dimension of http.client.open_connections here: Azure/azure-sdk-for-net#52674 (comment)
Alternatively, you can use views to snip out entire instruments if you want. Examples here:
https://github.com/acu-kevinb/HttpClientMetricsIssue/blob/afab54b8461f00aa2f4b5318aeaa7a9eb94d2880/HttpClientMetricsWorkerTest/Program.cs#L37
victormarante commentedon Feb 20, 2026
Thanks, will do same as the first comment you posted.
ManickaP commentedon Feb 23, 2026
Just for the info. I opened 2 issues in OTel repos:
There's quite a bit of discussion and even more issues linked.
TLDR this needs to be solved from OTel design perspective first. And since it has many moving parts, there's no final decision yet.
brasky commentedon Mar 13, 2026
For anyone who sees this in the future I got Azure support to refund me much of my bill that was caused due to this issue by linking them the issue and my invoice. Be prepared to wait a long time for them to get back to you, it took me months.
macel94 commentedon Apr 8, 2026
+1, we are hitting this too
kevbry commentedon Apr 8, 2026
Is there some interim fix that can be put in place ahead of something more permanent? Maybe putting a view like this in by default? Azure/azure-sdk-for-net#52674 (comment)
App insights already overrides the default behaviour for logging query strings so this wouldn't be without precedent.
ManickaP commentedon Jun 2, 2026
There's been a small update in OTel that changes
peer.addressto opt-in - open-telemetry/semantic-conventions#3759.We don't have a precedent for opt-in tags in our codebase. I'll discuss this with my team, but most likely we'll just remove
peer.addressfor now.And it looks like that the discussions + OTel specs + necessary changes in counters will not be done in time for us to make it in .NET 11. So I'll also discuss with the team whether it's worth switiching to
Observable*temporarily. It's a non-trivial effort, but doable and we should be able to make this one for .NET 11.Please upvote this comment if you need the switch to observable counters for .NET 11, it will help me prioritize the change accordingly.
[HTTP] Remove peer.address from connection OTel counters (#128940)
[HTTP] Remove peer.address from connection OTel counters (#128940)
[HTTP] High cardinality metrics to observable (#131275)