[AGTMETRICS-309] add support for tag cardinality by brianna-wang-dd · Pull Request #327 · DataDog/datadog-go

brianna-wang-dd · 2025-07-07T15:07:19Z

This PR implements tag cardinality for this library by sending a cardinality field for each metric as card:<cardinality> field. Possible tag cardinality values are none, low, orchestrator, or high.
system.mangos:21|g#foo:bar,baz:boo|card:high

The tag cardinality can be specified as a global setting (through the user passing the option to the constructor when setting up Statsd or by reading the value from DD_CARDINALITY or DATADOG_CARDINALITY). This can be overridden for each metric when creating one.

In order to ensure backward compatibility, a variadic parameters are added to each metric function that allow the tag cardinality to be passed in as type CardinalityParameter.

This PR was tested with unit test for tag cardinality specific functionality (such as global initialization from the 3 possible sources, checking the validity of inputs, and getting the global tag cardinality) and general metric functions (which included testing the metric-specific overrides for the tag cardinality).

This reverts commit 57c0ec6.

… names

…adog-go into brianna.wang/tag-cardinality

statsd/aggregator.go

statsd/aggregator_test.go

statsd/tag_cardinality.go

StephenWakely

Nicely done! I just have a few comments which are pretty pedantic so shouldn't block the PR, but feel free to address!

statsd/format.go

statsd/tag_cardinality.go

StephenWakely

Perfect!

AlexanderYastrebov · 2025-12-02T18:41:57Z

statsd/aggregator.go

-func (a *aggregator) count(name string, value int64, tags []string) error {
-	context := getContext(name, tags)
+func (a *aggregator) count(name string, value int64, tags []string, cardinality Cardinality) error {
+	resolvedCardinality := resolveCardinality(cardinality)


This is redundant

AlexanderYastrebov · 2025-12-02T18:49:28Z

statsd/tag_cardinality.go

+func parseTagCardinality(parameters []Parameter) Cardinality {
+	cardinality := CardinalityNotSet
+	for _, o := range parameters {
+		c, ok := o.(Cardinality)
+		if ok {
+			cardinality = c
+		}
+	}
+	return resolveCardinality(cardinality)
+}


This is called in the hot path and has to obtain global lock to get the default value which has high performance hit. Another problem is that second client with different options (o.tagCardinality) would override global default.

A better approach could be to get rid of global value and store default cardinality from environment inside of the client structure:

diff --git a/statsd/aggregator.go b/statsd/aggregator.go index 03629d2..283b39b 100644 --- a/statsd/aggregator.go +++ b/statsd/aggregator.go @@ -224,8 +224,7 @@ func getContextAndTags(name string, tags []string, cardinality Cardinality) (str } func (a *aggregator) count(name string, value int64, tags []string, cardinality Cardinality) error { - resolvedCardinality := resolveCardinality(cardinality) - context := getContext(name, tags, resolvedCardinality) + context := getContext(name, tags, cardinality) a.countsM.RLock() if count, found := a.counts[context]; found { count.sample(value) @@ -234,10 +233,10 @@ func (a *aggregator) count(name string, value int64, tags []string, cardinality } a.countsM.RUnlock() - metric := newCountMetric(name, value, tags, resolvedCardinality) + metric := newCountMetric(name, value, tags, cardinality) a.countsM.Lock() - // Check if another goroutines hasn't created the value betwen the RUnlock and 'Lock' + // Check if another goroutines hasn't created the value between the RUnlock and 'Lock' if count, found := a.counts[context]; found { count.sample(value) a.countsM.Unlock() diff --git a/statsd/statsd_benchmark_test.go b/statsd/statsd_benchmark_test.go index 35ea6ed..f03566f 100644 --- a/statsd/statsd_benchmark_test.go +++ b/statsd/statsd_benchmark_test.go @@ -86,7 +86,7 @@ func benchmarkStatsdDifferentMetrics(b *testing.B, transport string, extraOption name := fmt.Sprintf("test.metric%d", testNumber) tags := []string{"tag:tag"} for pb.Next() { - client.Gauge(name, 1, tags, 1) + client.Count(name, 1, tags, 1) } }) client.Flush() @@ -106,7 +106,7 @@ func benchmarkStatsdSameMetrics(b *testing.B, transport string, extraOptions ... b.RunParallel(func(pb *testing.PB) { for pb.Next() { tags := []string{"tag:tag"} - client.Gauge("test.metric", 1, tags, 1) + client.Count("test.metric", 1, tags, 1) } }) client.Flush() diff --git a/statsd/statsdex.go b/statsd/statsdex.go index d9e9b4e..ffba55f 100644 --- a/statsd/statsdex.go +++ b/statsd/statsdex.go @@ -300,6 +300,7 @@ type ClientEx struct { errorOnBlockedChannel bool errorHandler ErrorHandler originDetection bool + defaultTagCardinality Cardinality } // statsdTelemetry contains telemetry metrics about the client @@ -479,6 +480,7 @@ func newWithWriter(w Transport, o *Options, writerName string) (*ClientEx, error // Initializes the global tag cardinality with either the value passed in by the user or the value from the DD_CARDINALITY/DATADOG_CARDINALITY environment variable. initTagCardinality(o.tagCardinality) + c.defaultTagCardinality = CardinalityNotSet // TODO o.tagCardinality initContainerID(o.containerID, fillInContainerID(o), isHostCgroupNamespace()) isUDS := writerName == writerNameUDS @@ -738,7 +740,7 @@ func (c *ClientEx) Count(name string, value int64, tags []string, rate float64, return ErrNoClient } atomic.AddUint64(&c.telemetry.totalMetricsCount, 1) - cardinality := parseTagCardinality(parameters) + cardinality := parseTagCardinality2(parameters, c.defaultTagCardinality) if c.agg != nil { return c.agg.count(name, value, tags, cardinality) } diff --git a/statsd/tag_cardinality.go b/statsd/tag_cardinality.go index 1289e47..524c013 100644 --- a/statsd/tag_cardinality.go +++ b/statsd/tag_cardinality.go @@ -97,3 +97,13 @@ func resolveCardinality(card Cardinality) Cardinality { } return card } + +func parseTagCardinality2(parameters []Parameter, defaultCardinality Cardinality) Cardinality { + for _, o := range parameters { + c, ok := o.(Cardinality) + if ok { + return c + } + } + return defaultCardinality +}

Here is benchstat for one of the benchmarks for the above:

goos: darwin goarch: arm64 pkg: github.com/DataDog/datadog-go/v5/statsd cpu: Apple M4 Max │ HEAD~1 │ HEAD │ │ sec/op │ sec/op vs base │ StatsdUDPSameMetricMutexAggregation-16 254.8n ± 0% 156.3n ± 0% -38.65% (p=0.000 n=10) │ HEAD~1 │ HEAD │ │ %_dropRate │ %_dropRate vs base │ StatsdUDPSameMetricMutexAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ ¹ all samples are equal

Created #346 to address.

The change DataDog#327 introduced support for tag cardinality with global default value that could be overriden per metric call via parameter. Default cardinality value is stored in a global variable guarded by a mutex which has two downsides: * performance impact - each measure call has to obtain mutex to get the default value * two clients created with different default cardinality override the same global value This change eliminates global default value and uses default value per statsd client specified during client construction with a fallback to environment variables. User supplied cardinality value is validated during client construction and during parameter parsing with a fallback to default value such that all internal components can assume they always receive a valid value. Performance improvement (via `Statsd.*Aggregation` benchmarks): ``` goos: darwin goarch: arm64 pkg: github.com/DataDog/datadog-go/v5/statsd cpu: Apple M4 Max │ master │ HEAD │ │ sec/op │ sec/op vs base │ StatsdUDPSameMetricMutexAggregation-16 248.3n ± 1% 142.3n ± 0% -42.68% (p=0.000 n=10) StatsdUDPSameMetricChannelAggregation-16 252.3n ± 0% 143.8n ± 1% -43.02% (p=0.000 n=10) StatsdUDPDifferentMetricMutexAggregation-16 250.7n ± 1% 148.6n ± 0% -40.71% (p=0.000 n=10) StatsdUDPDifferentMetricChannelAggregation-16 252.5n ± 1% 150.3n ± 0% -40.48% (p=0.000 n=10) StatsdUDSSameMetricMutexAggregation-16 248.9n ± 1% 143.0n ± 1% -42.53% (p=0.000 n=10) StatsdUDSSameMetricChannelAggregation-16 251.9n ± 1% 144.0n ± 1% -42.87% (p=0.000 n=10) StatsdUDPSifferentMetricMutexAggregation-16 252.1n ± 0% 148.8n ± 1% -40.97% (p=0.000 n=10) StatsdUDSDifferentMetricChannelAggregation-16 252.8n ± 1% 149.5n ± 0% -40.85% (p=0.000 n=10) geomean 251.2n 146.3n -41.77% │ master │ HEAD │ │ %_dropRate │ %_dropRate vs base │ StatsdUDPSameMetricMutexAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDPSameMetricChannelAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDPDifferentMetricMutexAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDPDifferentMetricChannelAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDSSameMetricMutexAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDSSameMetricChannelAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDPSifferentMetricMutexAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDSDifferentMetricChannelAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ geomean ² +0.00% ² ¹ all samples are equal ² summaries must be >0 to compute geomean ```

* statsd: remove global cardinality The change #327 introduced support for tag cardinality with global default value that could be overriden per metric call via parameter. Default cardinality value is stored in a global variable guarded by a mutex which has two downsides: * performance impact - each measure call has to obtain mutex to get the default value * two clients created with different default cardinality override the same global value This change eliminates global default value and uses default value per statsd client specified during client construction with a fallback to environment variables. User supplied cardinality value is validated during client construction and during parameter parsing with a fallback to default value such that all internal components can assume they always receive a valid value. Performance improvement (via `Statsd.*Aggregation` benchmarks): ``` goos: darwin goarch: arm64 pkg: github.com/DataDog/datadog-go/v5/statsd cpu: Apple M4 Max │ master │ HEAD │ │ sec/op │ sec/op vs base │ StatsdUDPSameMetricMutexAggregation-16 248.3n ± 1% 142.3n ± 0% -42.68% (p=0.000 n=10) StatsdUDPSameMetricChannelAggregation-16 252.3n ± 0% 143.8n ± 1% -43.02% (p=0.000 n=10) StatsdUDPDifferentMetricMutexAggregation-16 250.7n ± 1% 148.6n ± 0% -40.71% (p=0.000 n=10) StatsdUDPDifferentMetricChannelAggregation-16 252.5n ± 1% 150.3n ± 0% -40.48% (p=0.000 n=10) StatsdUDSSameMetricMutexAggregation-16 248.9n ± 1% 143.0n ± 1% -42.53% (p=0.000 n=10) StatsdUDSSameMetricChannelAggregation-16 251.9n ± 1% 144.0n ± 1% -42.87% (p=0.000 n=10) StatsdUDPSifferentMetricMutexAggregation-16 252.1n ± 0% 148.8n ± 1% -40.97% (p=0.000 n=10) StatsdUDSDifferentMetricChannelAggregation-16 252.8n ± 1% 149.5n ± 0% -40.85% (p=0.000 n=10) geomean 251.2n 146.3n -41.77% │ master │ HEAD │ │ %_dropRate │ %_dropRate vs base │ StatsdUDPSameMetricMutexAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDPSameMetricChannelAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDPDifferentMetricMutexAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDPDifferentMetricChannelAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDSSameMetricMutexAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDSSameMetricChannelAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDPSifferentMetricMutexAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ StatsdUDSDifferentMetricChannelAggregation-16 0.000 ± 0% 0.000 ± 0% ~ (p=1.000 n=10) ¹ geomean ² +0.00% ² ¹ all samples are equal ² summaries must be >0 to compute geomean ``` * statsd: set CardinalityNotSet to zero Set CardinalityNotSet to zero to avoid gap between it and CardinalityNone which equals to one.

brianna-wang-dd added 30 commits July 1, 2025 11:22

added dd_external_env var

7c8d49f

added external env var initialization and sanitization

5995a04

appended external env to metrics

cba778e

added external env paramater for buffer_test

a53ceed

added external env parameter for other tests

83f9a07

added external env parameter to service_check_test

1ebf695

added test for external env in buffer_test

57c0ec6

Revert "added test for external env in buffer_test"

69526cd

This reverts commit 57c0ec6.

moved external env logic to separate file

aed1500

removed external env parameter from function calls

94180da

removed external env as field in metric struct

91f5d11

added unit tests

50857f5

added documentation for functions

068c5d8

unset external env at end of unit test

53753c8

commented out getExternalEnv unit test

91744fc

added defer to unsetEnv()

21b9990

added resetExternaEnv function

a25f1e6

added test to TestBufferGauge

7f922a4

bug fix

6b34540

added more unit tests to buffer_test

a4a3533

removed external env in client struct

665a2c4

removed comment

5927a2e

added tag cardinality basic variables and functions

2ffec24

added documentation

c5a9641

moved tag cardinality to separate file

b86eb61

added tag cardinality functions and append to metrics

28bc15d

added tag cardinality global initiation

9cd4f1e

Merge branch 'master' into brianna.wang/tag-cardinality

c1768d6

added metric function copies, additional env var, and changed new var…

b3cfccc

… names

Merge branch 'brianna.wang/tag-cardinality' of github.com:DataDog/dat…

ebf8f67

…adog-go into brianna.wang/tag-cardinality

brianna-wang-dd added 11 commits July 11, 2025 14:03

removed funky benchmarks

fde2605

moved tag cardinality parsing to separate function

48b7119

use parseTagCardinality() in set()

2923175

draft for adding tag cardinality check to aggregations

3e72f08

added cardinality to new metric structs and aggregation/flush tests

4706d02

unit test edits

ef14b98

added more unit tests for tag cardinality in aggregation

7a5966a

added usage of global setting fallback on aggregated metrics

66e514d

some refactoring and cleanup

aa8d50f

added test for resolveCardinality

0edf74e

cleaned up and added more aggregation unit tests

f0b38af

StephenWakely requested changes Jul 15, 2025

View reviewed changes

statsd/aggregator.go Outdated Show resolved Hide resolved

statsd/aggregator.go Outdated Show resolved Hide resolved

statsd/aggregator_test.go Show resolved Hide resolved

statsd/tag_cardinality.go Outdated Show resolved Hide resolved

brianna-wang-dd added 4 commits July 15, 2025 10:37

updated context formatting

7c3b6db

bug fix

dfb31b7

switch design of cardinality to int and enum

1c17669

removed comments

e65b3a5

StephenWakely previously approved these changes Jul 17, 2025

View reviewed changes

statsd/format.go Show resolved Hide resolved

statsd/format.go Show resolved Hide resolved

statsd/tag_cardinality.go Outdated Show resolved Hide resolved

made separators const

a6268ca

brianna-wang-dd dismissed StephenWakely’s stale review via a6268ca July 17, 2025 13:41

moved resolveCardinality call in parseTagCardinality

78ee96b

StephenWakely approved these changes Jul 17, 2025

View reviewed changes

rayz approved these changes Jul 21, 2025

View reviewed changes

brianna-wang-dd merged commit a6b9cdf into master Jul 21, 2025
35 checks passed

brianna-wang-dd deleted the brianna.wang/tag-cardinality branch July 21, 2025 14:35

eliottness mentioned this pull request Aug 20, 2025

Breaking change in v5.7.0 #332

Closed

AlexanderYastrebov reviewed Dec 2, 2025

View reviewed changes

AlexanderYastrebov mentioned this pull request Dec 19, 2025

statsd: remove global cardinality #346

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AGTMETRICS-309] add support for tag cardinality#327

[AGTMETRICS-309] add support for tag cardinality#327
brianna-wang-dd merged 68 commits intomasterfrom
brianna.wang/tag-cardinality

brianna-wang-dd commented Jul 7, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

StephenWakely left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

StephenWakely left a comment

Uh oh!

Uh oh!

AlexanderYastrebov Dec 2, 2025

Uh oh!

AlexanderYastrebov Dec 2, 2025 •

edited

Loading

Uh oh!

AlexanderYastrebov Dec 2, 2025

Uh oh!

AlexanderYastrebov Jan 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

brianna-wang-dd commented Jul 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

StephenWakely left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

StephenWakely left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AlexanderYastrebov Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

AlexanderYastrebov Dec 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AlexanderYastrebov Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

AlexanderYastrebov Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

brianna-wang-dd commented Jul 7, 2025 •

edited

Loading

AlexanderYastrebov Dec 2, 2025 •

edited

Loading