Skip to content

Separation of tags between metrics logs and traces fix#43370

Merged
dd-mergequeue[bot] merged 10 commits into
mainfrom
lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix
Dec 18, 2025
Merged

Separation of tags between metrics logs and traces fix#43370
dd-mergequeue[bot] merged 10 commits into
mainfrom
lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix

Conversation

@Lewis-E

@Lewis-E Lewis-E commented Nov 21, 2025

Copy link
Copy Markdown
Contributor

What does this PR do?

  • _dd.compute_stats is a marker for agent-side trace stats computation. We should isolate _dd.compute_stats to the serverless-init traces agent.
  • Deal with the fact that we have DD_SERVERLESS_INIT_ENABLE_BACKEND_TRACE_STATS and DD_SERVERLESS_INIT_DISABLE_TRACE_STATS
  • Remove unused BuildTracerTags function for clarity
  • De-duplicated use of configured tags
  • Renames all tag modification code to be a generalized place for agent tag changes, moved relevant functions into setup function.
  • Update comments and tests

Motivation

  • _dd.compute_stats should not show on logs or as a tag available for runtime metrics.
  • If DD_SERVERLESS_INIT_ENABLE_BACKEND_TRACE_STATS is true, _dd.compute_stats should show on traces.
  • If DD_SERVERLESS_INIT_ENABLE_BACKEND_TRACE_STATS is true andDD_SERVERLESS_INIT_DISABLE_TRACE_STATS is false, we should have accurate metric/trace data.
  • If DD_SERVERLESS_INIT_ENABLE_BACKEND_TRACE_STATS is true andDD_SERVERLESS_INIT_DISABLE_TRACE_STATS is false, AND sampling is <1.0, we should have inaccurate metric trace data.
  • Consistent pattern of tag modification

Describe how you validated your changes

  • Updated existing unit test
  • Test In-Container for cloudrun
  • Test In-Container for cloudrunjobs
  • Test In-Container for appservice
  • Test In-Container for containerapp
Screenshot 2025-12-11 at 1 29 08 PM

Using the lewis/SVLS-4573/serverless-init-test-compute-stats self-monitoring branch

  • Test Sidecar for cloudrun in self-monitoring
  • Test Sidecar for appservice in self-monitoring
  • Test Sidecar for containerapp in self-monitoring

Testing changes to the Azure App Services Extension

  • New unit test
  • Test in AAS -- FunctionTags (configured ex: DD_TAG) still working
Screenshot 2025-12-16 at 10 04 44 AM

@github-actions github-actions Bot added the medium review PR review might take time label Nov 21, 2025
@agent-platform-auto-pr

agent-platform-auto-pr Bot commented Nov 21, 2025

Copy link
Copy Markdown
Contributor

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor 2441ad4

Successful checks

Info

Quality gate Delta On disk size (MiB) Delta On wire size (MiB)
agent_deb_amd64 $${0}$$ $${705.41}$$ < $${707.13}$$ $${+0.01}$$ $${172.99}$$ < $${174.16}$$
agent_deb_amd64_fips $${0}$$ $${701.64}$$ < $${702.86}$$ $${-0.05}$$ $${172.21}$$ < $${173.5}$$
agent_heroku_amd64 $${0}$$ $${327.72}$$ < $${327.92}$$ $${-0}$$ $${87.16}$$ < $${88.21}$$
agent_msi $${0}$$ $${571.12}$$ < $${982.08}$$ $${-0.03}$$ $${142.65}$$ < $${143.49}$$
agent_rpm_amd64 $${0}$$ $${705.4}$$ < $${707.12}$$ $${-0}$$ $${176.11}$$ < $${177.29}$$
agent_rpm_amd64_fips $${0}$$ $${701.62}$$ < $${702.85}$$ $${+0.03}$$ $${175.1}$$ < $${176.32}$$
agent_rpm_arm64 $${0}$$ $${687.27}$$ < $${691.67}$$ $${+0.04}$$ $${159.1}$$ < $${160.7}$$
agent_rpm_arm64_fips $${0}$$ $${684.33}$$ < $${687.54}$$ $${-0.01}$$ $${158.77}$$ < $${160.11}$$
agent_suse_amd64 $${0}$$ $${705.4}$$ < $${707.12}$$ $${-0}$$ $${176.11}$$ < $${177.29}$$
agent_suse_amd64_fips $${0}$$ $${701.62}$$ < $${702.85}$$ $${+0.03}$$ $${175.1}$$ < $${176.32}$$
agent_suse_arm64 $${0}$$ $${687.27}$$ < $${691.67}$$ $${+0.04}$$ $${159.1}$$ < $${160.7}$$
agent_suse_arm64_fips $${0}$$ $${684.33}$$ < $${687.54}$$ $${-0.01}$$ $${158.77}$$ < $${160.11}$$
docker_agent_amd64 $${-0}$$ $${767.2}$$ < $${769.38}$$ $${-0}$$ $${260.51}$$ < $${262.06}$$
docker_agent_arm64 $${-0}$$ $${773.64}$$ < $${778.35}$$ $${+0.01}$$ $${249.62}$$ < $${251.72}$$
docker_agent_jmx_amd64 $${+0}$$ $${958.08}$$ < $${960.26}$$ $${-0.01}$$ $${329.14}$$ < $${330.69}$$
docker_agent_jmx_arm64 $${-0}$$ $${953.24}$$ < $${957.95}$$ $${+0}$$ $${314.24}$$ < $${316.35}$$
docker_cluster_agent_amd64 $${-0}$$ $${180.33}$$ < $${181.08}$$ $${-0}$$ $${63.71}$$ < $${64.49}$$
docker_cluster_agent_arm64 $${+0}$$ $${196.19}$$ < $${198.49}$$ $${+0}$$ $${60.0}$$ < $${61.17}$$
docker_cws_instrumentation_amd64 $${-0}$$ $${7.13}$$ < $${7.18}$$ $${-0}$$ $${2.99}$$ < $${3.33}$$
docker_cws_instrumentation_arm64 $${0}$$ $${6.69}$$ < $${6.92}$$ $${+0}$$ $${2.73}$$ < $${3.09}$$
docker_dogstatsd_amd64 $${0}$$ $${38.76}$$ < $${39.38}$$ $${-0}$$ $${15.0}$$ < $${15.82}$$
docker_dogstatsd_arm64 $${0}$$ $${37.06}$$ < $${37.94}$$ $${-0}$$ $${14.33}$$ < $${14.83}$$
dogstatsd_deb_amd64 $${0}$$ $${29.98}$$ < $${30.61}$$ $${-0}$$ $${7.93}$$ < $${8.79}$$
dogstatsd_deb_arm64 $${0}$$ $${28.14}$$ < $${29.11}$$ $${-0}$$ $${6.81}$$ < $${7.71}$$
dogstatsd_rpm_amd64 $${0}$$ $${29.98}$$ < $${30.61}$$ $${-0}$$ $${7.94}$$ < $${8.8}$$
dogstatsd_suse_amd64 $${0}$$ $${29.98}$$ < $${30.61}$$ $${-0}$$ $${7.94}$$ < $${8.8}$$
iot_agent_deb_amd64 $${0}$$ $${42.92}$$ < $${43.29}$$ $${+0}$$ $${11.23}$$ < $${12.04}$$
iot_agent_deb_arm64 $${0}$$ $${40.05}$$ < $${40.92}$$ $${-0}$$ $${9.61}$$ < $${10.45}$$
iot_agent_deb_armhf $${0}$$ $${40.63}$$ < $${41.03}$$ $${-0}$$ $${9.8}$$ < $${10.62}$$
iot_agent_rpm_amd64 $${0}$$ $${42.92}$$ < $${43.29}$$ $${-0}$$ $${11.25}$$ < $${12.06}$$
iot_agent_suse_amd64 $${0}$$ $${42.92}$$ < $${43.29}$$ $${-0}$$ $${11.25}$$ < $${12.06}$$

@cit-pr-commenter

cit-pr-commenter Bot commented Nov 21, 2025

Copy link
Copy Markdown

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 337d643c-2f3e-4e92-b0a5-b5c95da9ebfd

Baseline: bb99c82
Comparison: 080a59d
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf experiment goal Δ mean % Δ mean % CI trials links
docker_containers_cpu % cpu utilization +0.47 [-2.57, +3.52] 1 Logs

Fine details of change detection per experiment

perf experiment goal Δ mean % Δ mean % CI trials links
quality_gate_metrics_logs memory utilization +1.63 [+1.41, +1.86] 1 Logs bounds checks dashboard
ddot_logs memory utilization +0.81 [+0.73, +0.88] 1 Logs
quality_gate_idle_all_features memory utilization +0.53 [+0.50, +0.57] 1 Logs bounds checks dashboard
docker_containers_cpu % cpu utilization +0.47 [-2.57, +3.52] 1 Logs
uds_dogstatsd_20mb_12k_contexts_20_senders memory utilization +0.33 [+0.27, +0.38] 1 Logs
ddot_metrics_sum_cumulativetodelta_exporter memory utilization +0.06 [-0.17, +0.29] 1 Logs
file_to_blackhole_1000ms_latency egress throughput +0.04 [-0.39, +0.46] 1 Logs
uds_dogstatsd_to_api_v3 ingress throughput +0.01 [-0.11, +0.13] 1 Logs
file_to_blackhole_0ms_latency egress throughput +0.00 [-0.41, +0.41] 1 Logs
tcp_dd_logs_filter_exclude ingress throughput -0.00 [-0.08, +0.07] 1 Logs
uds_dogstatsd_to_api ingress throughput -0.00 [-0.13, +0.12] 1 Logs
file_to_blackhole_100ms_latency egress throughput -0.02 [-0.07, +0.03] 1 Logs
file_to_blackhole_500ms_latency egress throughput -0.05 [-0.43, +0.32] 1 Logs
ddot_metrics_sum_delta memory utilization -0.06 [-0.26, +0.15] 1 Logs
quality_gate_idle memory utilization -0.10 [-0.14, -0.05] 1 Logs bounds checks dashboard
ddot_metrics memory utilization -0.12 [-0.35, +0.11] 1 Logs
file_tree memory utilization -0.24 [-0.31, -0.17] 1 Logs
otlp_ingest_logs memory utilization -0.27 [-0.37, -0.17] 1 Logs
otlp_ingest_metrics memory utilization -0.32 [-0.47, -0.16] 1 Logs
docker_containers_memory memory utilization -0.42 [-0.50, -0.35] 1 Logs
quality_gate_logs % cpu utilization -0.46 [-1.94, +1.02] 1 Logs bounds checks dashboard
tcp_syslog_to_blackhole ingress throughput -0.69 [-0.77, -0.61] 1 Logs
ddot_metrics_sum_cumulative memory utilization -0.85 [-1.01, -0.69] 1 Logs

Bounds Checks: ✅ Passed

perf experiment bounds_check_name replicates_passed links
docker_containers_cpu simple_check_run 10/10
docker_containers_memory memory_usage 10/10
docker_containers_memory simple_check_run 10/10
file_to_blackhole_0ms_latency lost_bytes 10/10
file_to_blackhole_0ms_latency memory_usage 10/10
file_to_blackhole_1000ms_latency lost_bytes 10/10
file_to_blackhole_1000ms_latency memory_usage 10/10
file_to_blackhole_100ms_latency lost_bytes 10/10
file_to_blackhole_100ms_latency memory_usage 10/10
file_to_blackhole_500ms_latency lost_bytes 10/10
file_to_blackhole_500ms_latency memory_usage 10/10
quality_gate_idle intake_connections 10/10 bounds checks dashboard
quality_gate_idle memory_usage 10/10 bounds checks dashboard
quality_gate_idle_all_features intake_connections 10/10 bounds checks dashboard
quality_gate_idle_all_features memory_usage 10/10 bounds checks dashboard
quality_gate_logs intake_connections 10/10 bounds checks dashboard
quality_gate_logs lost_bytes 10/10 bounds checks dashboard
quality_gate_logs memory_usage 10/10 bounds checks dashboard
quality_gate_metrics_logs cpu_usage 10/10 bounds checks dashboard
quality_gate_metrics_logs intake_connections 10/10 bounds checks dashboard
quality_gate_metrics_logs lost_bytes 10/10 bounds checks dashboard
quality_gate_metrics_logs memory_usage 10/10 bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

  • ✅ = significantly better comparison variant performance
  • ❌ = significantly worse comparison variant performance
  • ➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

  1. Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.

  2. Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.

  3. Its configuration does not mark it "erratic".

CI Pass/Fail Decision

Passed. All Quality Gates passed.

  • quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
  • quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
  • quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.

@Lewis-E
Lewis-E force-pushed the lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix branch 4 times, most recently from 54ab478 to 3a58bff Compare November 27, 2025 15:31
@Lewis-E
Lewis-E marked this pull request as ready for review December 4, 2025 21:22
@Lewis-E
Lewis-E requested review from a team as code owners December 4, 2025 21:22
Comment thread releasenotes/notes/compute-stats-metrics-only-aae48020db10708b.yaml
Comment thread cmd/serverless-init/tag/tag.go Outdated
Comment thread releasenotes/notes/compute-stats-metrics-only-aae48020db10708b.yaml Outdated
dd-mergequeue Bot pushed a commit that referenced this pull request Dec 5, 2025
…43837)

### What does this PR do?

This is a temporary fix to remove high cardinality tags from Cloud Run Jobs enhanced metrics.
A more thorough refactor will be done in a future PR (#43370)

High cardinality removed:
<img width="400" height="393" alt="Screenshot 2025-12-04 at 2 11 05 PM" src="https://github.com/user-attachments/assets/0d002d7f-5d58-448d-ae24-8a9c62bf5849" />

Low cardinality still exists:
<img width="400" height="382" alt="Screenshot 2025-12-04 at 2 11 39 PM" src="https://github.com/user-attachments/assets/076e73c6-769b-4f11-b023-b2579905a9c7" />

### Motivation

Cost savings

### Describe how you validated your changes

Manually with a real Cloud Run Job

### Additional Notes

No changelog because Cloud Run Jobs support is not officially released yet.


Co-authored-by: nicholas.hulston <[email protected]>
@iadjivon

iadjivon commented Dec 5, 2025

Copy link
Copy Markdown

Hi there, thanks for this PR. I've added a wip tag since it seems this is still being drafted. Please let us know once this is ready for review and we'll be happy to review! Thanks so much and let us know if you have any questions.

@Lewis-E
Lewis-E force-pushed the lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix branch 3 times, most recently from 1c262e0 to ae56b67 Compare December 5, 2025 18:09
@Lewis-E Lewis-E added the changelog/no-changelog No changelog entry needed label Dec 5, 2025
@github-actions github-actions Bot added long review PR is complex, plan time to review it and removed medium review PR review might take time labels Dec 5, 2025
@Lewis-E
Lewis-E marked this pull request as draft December 5, 2025 18:34

@apiarian-datadog apiarian-datadog left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how are we testing this? should we have some kind of self-monitoring alert for the resulting tagged things?

Comment thread cmd/serverless-init/main.go Outdated
Comment thread cmd/serverless-init/tag/tag.go Outdated
Comment thread cmd/serverless-init/main.go
@Lewis-E
Lewis-E force-pushed the lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix branch from ca24a7d to c7cf356 Compare December 9, 2025 17:19
@Lewis-E

Lewis-E commented Dec 9, 2025

Copy link
Copy Markdown
Contributor Author

how are we testing this? should we have some kind of self-monitoring alert for the resulting tagged things?

@apiarian-datadog There's a checklist for manual tests in the PR description under Describe how you validated your changes. Unchecked them all when going back into draft, and will re-test with the new code. Are there existing practices around monitors + combinations of env vars in self monitoring, given env vars can change per-deploy? I see there's some compute stats related code in the e2e tests. I have that repo on my to-onboard list, so adding tests there seems like a good next step to me.

@@ -123,10 +123,11 @@ func setup(secretComp secrets.Component, _ mode.Conf, tagger tagger.Component, c

log.Debugf("Detected cloud service: %s", cloudService.GetOrigin())

configuredTags := configUtils.GetConfiguredTags(pkgconfigsetup.Datadog(), false)
tags := serverlessInitTag.GetBaseTagsMapWithMetadata(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: while we're in here, might be worth renaming "tags" to "base tags" or something like that, so it's clear that we probably shouldn't use it directly without serious thought.

@apiarian-datadog

Copy link
Copy Markdown
Contributor

how are we testing this? should we have some kind of self-monitoring alert for the resulting tagged things?

@apiarian-datadog There's a checklist for manual tests in the PR description under Describe how you validated your changes. Unchecked them all when going back into draft, and will re-test with the new code. Are there existing practices around monitors + combinations of env vars in self monitoring, given env vars can change per-deploy? I see there's some compute stats related code in the e2e tests. I have that repo on my to-onboard list, so adding tests there seems like a good next step to me.

i think the existing trace stats tests should cover your trace stats needs on this one.

i suppose it might be worth adding a small test that creates an app with some tags and confirms that those tags (and the additional automatic ones) end up on the metrics, traces, and logs that we ingest. or perhaps this is an element of self-monitoring.

@Lewis-E
Lewis-E force-pushed the lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix branch from 0cc5213 to 13a353a Compare December 11, 2025 21:30
@Lewis-E
Lewis-E requested a review from a team as a code owner December 15, 2025 14:10
@agent-platform-auto-pr

agent-platform-auto-pr Bot commented Dec 15, 2025

Copy link
Copy Markdown
Contributor

Go Package Import Differences

Baseline: 2441ad4
Comparison: 080a59d

binaryosarchchange
trace-agentlinuxamd64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags
trace-agentlinuxarm64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags
trace-agentwindowsamd64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags
trace-agentdarwinamd64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags
trace-agentdarwinarm64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags
heroku-trace-agentlinuxamd64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags
otel-agentlinuxamd64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags
otel-agentlinuxarm64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags
full-host-profilerlinuxamd64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags
full-host-profilerlinuxarm64
+0, -1
-github.com/DataDog/datadog-agent/pkg/serverless/tags

@Lewis-E
Lewis-E force-pushed the lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix branch from d0e1091 to ff7d6c5 Compare December 16, 2025 15:08
@Lewis-E Lewis-E added the ask-review Ask in slack required teams to review this PR label Dec 16, 2025
@Lewis-E
Lewis-E force-pushed the lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix branch from ff7d6c5 to 38fe7b9 Compare December 16, 2025 17:21
@Lewis-E
Lewis-E force-pushed the lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix branch 4 times, most recently from 3af4456 to 702b72e Compare December 17, 2025 21:34
@Lewis-E
Lewis-E force-pushed the lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix branch from 702b72e to 080a59d Compare December 18, 2025 14:21
@Lewis-E

Lewis-E commented Dec 18, 2025

Copy link
Copy Markdown
Contributor Author

/merge

@dd-devflow-routing-codex

dd-devflow-routing-codex Bot commented Dec 18, 2025

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2025-12-18 16:19:31 UTC ℹ️ Start processing command /merge


2025-12-18 16:19:37 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 1h (p90).


2025-12-18 17:16:35 UTC ℹ️ MergeQueue: This merge request was merged

@dd-mergequeue
dd-mergequeue Bot merged commit ca93105 into main Dec 18, 2025
330 checks passed
@dd-mergequeue
dd-mergequeue Bot deleted the lewis/SVLS-4573-serverless-init-No-separation-of-tags-between-metrics-logs-and-traces-fix branch December 18, 2025 17:16
@github-actions github-actions Bot added this to the 7.75.0 milestone Dec 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ask-review Ask in slack required teams to review this PR changelog/no-changelog No changelog entry needed long review PR is complex, plan time to review it qa/done QA done before merge and regressions are covered by tests team/serverless-azure-gcp

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants