feat: [SVLS-8237] fix cloud run job trace parenting by apiarian-datadog · Pull Request #44727 · DataDog/datadog-agent

apiarian-datadog · 2026-01-02T19:13:06Z

What does this PR do?

Fixes our cloud run job span reparenting.

Motivation

The original code destroyed trace hierarchies and resulted in broken logs/trace correlation.

Describe how you validated your changes

Unit tests and tests against an actual cloud run job.

apiarian-datadog · 2026-01-02T19:18:47Z

a test trace: https://ddserverless.datadoghq.com/apm/trace/9128031905749743469?config_trace_show_hidden_metadata=true&graphType=flamegraph&shouldShowLegend=true&spanID=3280614947310590697&spanViewType=logs&timeHint=1767380458818.3357&trace=91280319057497434693280614947310590697&traceQuery= (and the same in json format: https://ddserverless.datadoghq.com/api/ui/trace/9128031905749743469 )
the logs: https://ddserverless.datadoghq.com/logs?query=%40raw_dd_trace_id%3A695815ef000000007ead48a7e8a5376d&agg_m=count&agg_m_source=base&agg_t=count&clustering_pattern_field_path=message&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=paused&storage=flex_tier&stream_sort=time%2Casc&viz=stream&from_ts=1767380462071&to_ts=1767380467419&live=false

unfortunately there's something happening with 128bit logs/trace correlation either in the ui or in our agent/tracer. seems, perhaps, that the trace ui doesn't correctly correlate a trace with a 128bit id to the logs? maybe we configured something wrong in the app https://github.com/DataDog/serverless-examples/pull/147

agent-platform-auto-pr · 2026-01-02T19:52:00Z

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor ba0d1dc
📊 Static Quality Gates Dashboard

31 successful checks with minimal change (< 2 KiB)

	Quality gate	Current Size
✅	agent_deb_amd64	705.336 MiB
✅	agent_deb_amd64_fips	700.618 MiB
✅	agent_heroku_amd64	326.935 MiB
✅	agent_msi	571.471 MiB
✅	agent_rpm_amd64	705.323 MiB
✅	agent_rpm_amd64_fips	700.604 MiB
✅	agent_rpm_arm64	686.906 MiB
✅	agent_rpm_arm64_fips	682.989 MiB
✅	agent_suse_amd64	705.323 MiB
✅	agent_suse_amd64_fips	700.604 MiB
✅	agent_suse_arm64	686.906 MiB
✅	agent_suse_arm64_fips	682.989 MiB
✅	docker_agent_amd64	767.551 MiB
✅	docker_agent_arm64	773.694 MiB
✅	docker_agent_jmx_amd64	958.430 MiB
✅	docker_agent_jmx_arm64	953.292 MiB
✅	docker_cluster_agent_amd64	180.754 MiB
✅	docker_cluster_agent_arm64	196.620 MiB
✅	docker_cws_instrumentation_amd64	7.135 MiB
✅	docker_cws_instrumentation_arm64	6.689 MiB
✅	docker_dogstatsd_amd64	38.808 MiB
✅	docker_dogstatsd_arm64	37.128 MiB
✅	dogstatsd_deb_amd64	30.031 MiB
✅	dogstatsd_deb_arm64	28.176 MiB
✅	dogstatsd_rpm_amd64	30.031 MiB
✅	dogstatsd_suse_amd64	30.031 MiB
✅	iot_agent_deb_amd64	43.037 MiB
✅	iot_agent_deb_arm64	40.155 MiB
✅	iot_agent_deb_armhf	40.736 MiB
✅	iot_agent_rpm_amd64	43.038 MiB
✅	iot_agent_suse_amd64	43.038 MiB

On-wire sizes (compressed)

	Quality gate	Change	Size (prev → curr → max)
✅	agent_deb_amd64	-20.95 KiB (0.01% reduction)	173.389 → 173.368 → 174.490
✅	agent_deb_amd64_fips	-19.23 KiB (0.01% reduction)	172.318 → 172.300 → 173.750
✅	agent_heroku_amd64	neutral	87.130 MiB
✅	agent_msi	-40.0 KiB (0.03% reduction)	142.934 → 142.895 → 143.020
✅	agent_rpm_amd64	-39.74 KiB (0.02% reduction)	176.094 → 176.056 → 177.660
✅	agent_rpm_amd64_fips	neutral	175.022 MiB
✅	agent_rpm_arm64	-8.56 KiB (0.01% reduction)	159.379 → 159.370 → 161.260
✅	agent_rpm_arm64_fips	-4.63 KiB (0.00% reduction)	158.766 → 158.761 → 160.550
✅	agent_suse_amd64	-39.74 KiB (0.02% reduction)	176.094 → 176.056 → 177.660
✅	agent_suse_amd64_fips	neutral	175.022 MiB
✅	agent_suse_arm64	-8.56 KiB (0.01% reduction)	159.379 → 159.370 → 161.260
✅	agent_suse_arm64_fips	-4.63 KiB (0.00% reduction)	158.766 → 158.761 → 160.550
✅	docker_agent_amd64	neutral	261.086 MiB
✅	docker_agent_arm64	-4.63 KiB (0.00% reduction)	250.106 → 250.102 → 252.630
✅	docker_agent_jmx_amd64	neutral	329.723 MiB
✅	docker_agent_jmx_arm64	neutral	314.726 MiB
✅	docker_cluster_agent_amd64	neutral	63.860 MiB
✅	docker_cluster_agent_arm64	neutral	60.141 MiB
✅	docker_cws_instrumentation_amd64	neutral	2.994 MiB
✅	docker_cws_instrumentation_arm64	neutral	2.726 MiB
✅	docker_dogstatsd_amd64	neutral	15.026 MiB
✅	docker_dogstatsd_arm64	neutral	14.351 MiB
✅	dogstatsd_deb_amd64	neutral	7.946 MiB
✅	dogstatsd_deb_arm64	neutral	6.823 MiB
✅	dogstatsd_rpm_amd64	neutral	7.956 MiB
✅	dogstatsd_suse_amd64	neutral	7.956 MiB
✅	iot_agent_deb_amd64	neutral	11.273 MiB
✅	iot_agent_deb_arm64	-2.68 KiB (0.03% reduction)	9.640 → 9.637 → 10.450
✅	iot_agent_deb_armhf	neutral	9.834 MiB
✅	iot_agent_rpm_amd64	neutral	11.290 MiB
✅	iot_agent_suse_amd64	neutral	11.290 MiB

cit-pr-commenter · 2026-01-02T20:08:28Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: 79b112b3-d2aa-469f-a730-a4f3226ab788

Baseline: b33f703
Comparison: 1a97ca2
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+3.84	[+0.77, +6.90]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+3.84	[+0.77, +6.90]	1	Logs
➖	ddot_logs	memory utilization	+0.60	[+0.53, +0.66]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	+0.57	[+0.51, +0.63]	1	Logs
➖	otlp_ingest_metrics	memory utilization	+0.39	[+0.23, +0.54]	1	Logs
➖	otlp_ingest_logs	memory utilization	+0.30	[+0.20, +0.41]	1	Logs
➖	ddot_metrics_sum_delta	memory utilization	+0.24	[+0.04, +0.45]	1	Logs
➖	docker_containers_memory	memory utilization	+0.21	[+0.14, +0.29]	1	Logs
➖	quality_gate_idle	memory utilization	+0.19	[+0.15, +0.24]	1	Logs bounds checks dashboard
➖	quality_gate_logs	% cpu utilization	+0.14	[-1.34, +1.62]	1	Logs bounds checks dashboard
➖	file_to_blackhole_500ms_latency	egress throughput	+0.09	[-0.29, +0.46]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.01	[-0.07, +0.09]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.01	[-0.12, +0.13]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.00	[-0.41, +0.42]	1	Logs
➖	ddot_metrics_sum_cumulative	memory utilization	-0.00	[-0.16, +0.16]	1	Logs
➖	uds_dogstatsd_to_api_v3	ingress throughput	-0.00	[-0.13, +0.13]	1	Logs
➖	ddot_metrics_sum_cumulativetodelta_exporter	memory utilization	-0.01	[-0.24, +0.22]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.03	[-0.07, +0.02]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-0.04	[-0.11, +0.03]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	-0.08	[-0.49, +0.32]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	-0.12	[-0.16, -0.09]	1	Logs bounds checks dashboard
➖	quality_gate_metrics_logs	memory utilization	-0.27	[-0.46, -0.07]	1	Logs bounds checks dashboard
➖	file_tree	memory utilization	-0.47	[-0.53, -0.42]	1	Logs
➖	ddot_metrics	memory utilization	-1.16	[-1.37, -0.95]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	docker_containers_cpu	simple_check_run	10/10
✅	docker_containers_memory	memory_usage	10/10
✅	docker_containers_memory	simple_check_run	10/10
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	lost_bytes	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.

apiarian-datadog · 2026-01-12T15:25:51Z

with our 128bit trace id handling fixed, here's a fully functioning trace:

apiarian-datadog · 2026-01-13T15:26:31Z

and now with some more reparenting:

https://ddserverless.datadoghq.com/apm/trace/16679114230451531130?graphType=flamegraph&shouldShowLegend=true&spanID=4127598440122206283&timeHint=1768316577744.114&trace=166791142304515311304127598440122206283&traceQuery=
https://ddserverless.datadoghq.com/apm/trace/17618648402267774522?graphType=flamegraph&shouldShowLegend=true&spanID=9379642606563809657&timeHint=1768316876467.196&trace=176186484022677745229379642606563809657&traceQuery=
that second one has a manually propagated trace. it's not very pretty, since the cloud run span doesn't get inserted into the trace hierarchy as expected. but this should be enough to get us started.

…an.go

but still only reparent the trace if we see a zero parent id. trace propagation will look a little strange for now

apiarian-datadog · 2026-01-13T19:04:00Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-01-13T19:04:05Z

View all feedbacks in Devflow UI.

2026-01-13 19:04:04 UTC ℹ️ Start processing command /merge

2026-01-13 19:04:11 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 60m (p90).

2026-01-13 19:28:07 UTC ℹ️ MergeQueue: This merge request was merged

### What does this PR do? Use the helpers added in #44727 in more places. ### Motivation Set up good patterns for the future. ### Describe how you validated your changes Automated tests. Co-authored-by: aleksandr.pasechnik <[email protected]>

apiarian-datadog commented Jan 2, 2026

View reviewed changes

Comment thread cmd/serverless-init/trace/span_modifier.go Outdated

github-actions Bot added the medium review PR review might take time label Jan 2, 2026

apiarian-datadog force-pushed the aleksandr.pasechnik/svls-8237-fix-cloud-run-jobs-trace-reparenting branch from 85d10d2 to 66a2103 Compare January 12, 2026 20:08

apiarian-datadog marked this pull request as ready for review January 13, 2026 15:30

apiarian-datadog requested review from a team as code owners January 13, 2026 15:30

apiarian-datadog added 3 commits January 13, 2026 10:33

feat: [SVLS-8237] fix cloud run job trace parenting

7c7877b

feat: [SVLS-8237] move CopyTraceID function to pkg/trace/traceutil/sp…

9dac161

…an.go

feat: [SVLS-8237] universal trace adoption

1a97ca2

but still only reparent the trace if we see a zero parent id. trace propagation will look a little strange for now

apiarian-datadog force-pushed the aleksandr.pasechnik/svls-8237-fix-cloud-run-jobs-trace-reparenting branch from 66a2103 to 1a97ca2 Compare January 13, 2026 15:33

apiarian-datadog added changelog/no-changelog No changelog entry needed qa/done QA done before merge and regressions are covered by tests team/serverless-azure-gcp labels Jan 13, 2026

apiarian-datadog requested a review from Lewis-E January 13, 2026 16:44

purple4reina reviewed Jan 13, 2026

View reviewed changes

Comment thread cmd/serverless-init/trace/span_modifier.go

purple4reina reviewed Jan 13, 2026

View reviewed changes

Comment thread pkg/trace/traceutil/span.go

purple4reina approved these changes Jan 13, 2026

View reviewed changes

Lewis-E approved these changes Jan 13, 2026

View reviewed changes

ajgajg1134 approved these changes Jan 13, 2026

View reviewed changes

dd-mergequeue Bot merged commit dbab403 into main Jan 13, 2026
370 of 373 checks passed

dd-mergequeue Bot deleted the aleksandr.pasechnik/svls-8237-fix-cloud-run-jobs-trace-reparenting branch January 13, 2026 19:28

github-actions Bot added this to the 7.76.0 milestone Jan 13, 2026

apiarian-datadog mentioned this pull request Jan 13, 2026

chore: use the new CopyTraceID helper in more places #45061

Merged

robbiet480 mentioned this pull request Mar 24, 2026

[BUG] serverless-init: concurrent map panic at process exit on Cloud Run Jobs (Go 1.26) #48308

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: [SVLS-8237] fix cloud run job trace parenting#44727

feat: [SVLS-8237] fix cloud run job trace parenting#44727
dd-mergequeue[bot] merged 3 commits into
mainfrom
aleksandr.pasechnik/svls-8237-fix-cloud-run-jobs-trace-reparenting

apiarian-datadog commented Jan 2, 2026

Uh oh!

Uh oh!

apiarian-datadog commented Jan 2, 2026

Uh oh!

agent-platform-auto-pr Bot commented Jan 2, 2026 •

edited

Loading

Uh oh!

cit-pr-commenter Bot commented Jan 2, 2026 •

edited

Loading

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

Uh oh!

apiarian-datadog commented Jan 12, 2026

Uh oh!

apiarian-datadog commented Jan 13, 2026

Uh oh!

Uh oh!

Uh oh!

apiarian-datadog commented Jan 13, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jan 13, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

apiarian-datadog commented Jan 2, 2026

What does this PR do?

Motivation

Describe how you validated your changes

Uh oh!

Uh oh!

apiarian-datadog commented Jan 2, 2026

Uh oh!

agent-platform-auto-pr Bot commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Static quality checks

Uh oh!

cit-pr-commenter Bot commented Jan 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

apiarian-datadog commented Jan 12, 2026

Uh oh!

apiarian-datadog commented Jan 13, 2026

Uh oh!

Uh oh!

Uh oh!

apiarian-datadog commented Jan 13, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Jan 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

agent-platform-auto-pr Bot commented Jan 2, 2026 •

edited

Loading

cit-pr-commenter Bot commented Jan 2, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Jan 13, 2026 •

edited

Loading