Skip to content

Commit 6524d48

Browse files
cijothomascijothomas
authored
[concepts] Document metric cardinality limits and overflow (#10630)
1 parent 7fc8a18 commit 6524d48

3 files changed

Lines changed: 95 additions & 3 deletions

File tree

‎.cspell/en-words.txt‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -59,6 +59,9 @@ tabpane
5959
timeframe
6060
uids
6161
unconference
62+
undercount
63+
undercounting
64+
undercounts
6265
unitless
6366
unsampled
6467
unshallow

‎content/en/docs/concepts/glossary.md‎

Lines changed: 13 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,9 +57,19 @@ relationship between events and services. See [baggage spec][baggage].
5757

5858
The number of unique values for a given [Attribute](#attribute) or set of
5959
attributes. High cardinality means many unique values, which can impact the
60-
performance and storage requirements of telemetry backends. For example, a
61-
`user_id` attribute would have high cardinality, while a `status_code` attribute
62-
with values like "200", "404", "500" would have low cardinality.
60+
performance and storage requirements of telemetry backends, and the memory used
61+
by the [Metric](#metric) SDK. For example, a `user_id` attribute would have high
62+
cardinality, while a `status_code` attribute with values like "200", "404",
63+
"500" would have low cardinality.
64+
65+
### Cardinality limit
66+
67+
A configurable maximum on the number of unique attribute combinations a
68+
[Metric](#metric) SDK tracks for a single metric stream, used to bound memory
69+
usage. When the limit is reached, further combinations are aggregated into a
70+
single overflow data point identified by the `otel.metric.overflow=true`
71+
attribute. See
72+
[Cardinality limits](/docs/concepts/signals/metrics/#cardinality-limits).
6373

6474
### Client library
6575

‎content/en/docs/concepts/signals/metrics.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,6 +111,85 @@ by the SDK. You can customize which metric instruments are to be processed or
111111
ignored. You can also customize aggregation and what attributes you want to
112112
report on metrics.
113113

114+
## Cardinality limits
115+
116+
The **cardinality** of a metric is the number of unique attribute combinations
117+
reported for it. Because the SDK keeps a separate aggregation state (a data
118+
point) in memory for each unique combination, cardinality drives the memory cost
119+
of metrics. Unlike logs, this cost scales with the number of distinct attribute
120+
combinations rather than with request volume, so high-cardinality attributes,
121+
such as user IDs or raw URL paths, can cause unbounded memory growth.
122+
123+
To protect applications from this, the OpenTelemetry metrics SDK enforces a
124+
**cardinality limit**: a maximum number of unique attribute combinations tracked
125+
per metric stream. The default is 2000 and can be overridden with a
126+
[View](#views).
127+
128+
When the limit is reached, additional attribute combinations are not dropped
129+
outright. Instead, their measurements are aggregated into a single **overflow
130+
data point** identified by the attribute `otel.metric.overflow=true`. This
131+
design has three important properties:
132+
133+
- **No measurements are lost.** Only the attributes are dropped; the recorded
134+
values are folded into the overflow data point, so the metric's overall total
135+
stays correct.
136+
- **Memory is bounded.** The SDK never tracks more than the configured number of
137+
combinations.
138+
- **Overflow is observable.** Every SDK uses the same
139+
`otel.metric.overflow=true` marker, so a single query can detect overflow
140+
across services, languages, and backends.
141+
142+
The trade-off is that any query that **filters or groups by an attribute** on an
143+
overflowed metric undercounts, because the measurements folded into overflow no
144+
longer carry that attribute.
145+
146+
This is easy to underestimate, because overflow replaces the **entire**
147+
attribute combination, not just its high-cardinality part. Suppose a request
148+
counter records `url.path` (high cardinality) together with `success` (a
149+
boolean). Once the stream overflows, a measurement for
150+
`{url.path=/checkout, success=false}` is folded into the single overflow data
151+
point `{otel.metric.overflow=true}`, dropping `success` along with `url.path`. A
152+
query for `success=false` then misses that measurement, even though `success` on
153+
its own is as low-cardinality as an attribute can be. An error-rate alert built
154+
on `success=false` can therefore stop firing, while the metric's overall total
155+
stays correct.
156+
157+
### What the limit does not apply to
158+
159+
The cardinality limit applies only to attributes supplied when recording
160+
measurements through the metrics API. It does **not** apply to:
161+
162+
- [Resource](/docs/concepts/resources/) attributes, such as `service.name` or
163+
`service.instance.id`.
164+
- Instrumentation scope attributes set when a [Meter](#meter) is created.
165+
166+
Values in Resource and instrumentation scope attributes are recorded on every
167+
data point, including the overflow one, so they remain reliably queryable even
168+
during overflow. This is not a reason to relocate measurement attributes to work
169+
around the limit. Attributes should be placed according to what they describe: a
170+
Resource describes the entity producing telemetry, instrumentation scope
171+
describes the instrumenting library, and measurement attributes describe an
172+
individual measurement. When attributes are modeled this way, context that is
173+
constant for the lifetime of the process, such as service name, environment, or
174+
region, naturally belongs in the Resource, where it also stays queryable under
175+
overflow.
176+
177+
### Temporality and cardinality limits
178+
179+
For synchronous instruments,
180+
[aggregation temporality](/docs/specs/otel/metrics/data-model/#temporality)
181+
determines how quickly the SDK can reclaim aggregation state, and therefore how
182+
quickly the limit is reached:
183+
184+
- With **delta** temporality, the SDK resets state after each cycle, so the
185+
limit bounds only the combinations active within a single cycle.
186+
- With **cumulative** temporality, the SDK retains state across cycles, so once
187+
the limit is reached, new combinations keep overflowing until the process
188+
restarts.
189+
190+
Asynchronous instruments follow different rules that are beyond the scope of
191+
this overview.
192+
114193
## Language Support
115194

116195
Metrics are a [stable](/docs/specs/otel/versioning-and-stability/#stable) signal

0 commit comments

Comments
 (0)