[Serverless-Init] Support Cloud Run Jobs with serverless-init by nhulston · Pull Request #38737 · DataDog/datadog-agent

nhulston · 2025-07-11T20:21:46Z

What does this PR do?

Support Cloud Run Jobs in serverless-init. So we get a variety of features:

stdout & stderr piping for in-process instrumentation
setting logs source to cloudrun
add tags to spans: job name, execution name, task index, task attempt #, task count, and resource name. Looks like we already get container_id (which is some long hex string) out of the box by the agent

Added logic to skip cold start & shutdown metrics, since they're irrelevant for Cloud Run jobs (every execution will have a "cold start", every execution will have a shutdown).

I follow the GCP convention and prefix logs with gcp.run.job instead of gcp.run.

Motivation

Describe how you validated your changes

Tested manually and unit tests

Possible Drawbacks / Trade-offs

Additional Notes

Jobs kind of work with the sidecar approach, but the flushing logic is sketchy. When the main app is done executing, all other containers immediately receive a SIGKILL, so any pending traces or logs will not be sent.

Therefore, we will be recommending customers use the in-process approach in our documentation.

cit-pr-commenter · 2025-07-11T21:56:53Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: ba20afe4-230e-4608-8126-748bce12b6cb

Baseline: f4aa22d
Comparison: b798d5f
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+2.82	[-0.24, +5.88]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
➖	docker_containers_cpu	% cpu utilization	+2.82	[-0.24, +5.88]	1	Logs
➖	quality_gate_metrics_logs	memory utilization	+0.94	[+0.56, +1.33]	1	Logs bounds checks dashboard
➖	tcp_syslog_to_blackhole	ingress throughput	+0.82	[+0.76, +0.89]	1	Logs
➖	docker_containers_memory	memory utilization	+0.40	[+0.32, +0.47]	1	Logs
➖	ddot_metrics	memory utilization	+0.26	[+0.14, +0.38]	1	Logs
➖	quality_gate_idle	memory utilization	+0.20	[+0.17, +0.24]	1	Logs bounds checks dashboard
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	+0.20	[+0.16, +0.24]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	+0.09	[+0.05, +0.13]	1	Logs bounds checks dashboard
➖	file_to_blackhole_0ms_latency	egress throughput	+0.05	[-0.52, +0.61]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	+0.02	[-0.51, +0.55]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	+0.01	[-0.58, +0.60]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.01	[-0.30, +0.31]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.02, +0.02]	1	Logs
➖	file_to_blackhole_1000ms_latency	egress throughput	-0.00	[-0.59, +0.59]	1	Logs
➖	otlp_ingest_metrics	memory utilization	-0.00	[-0.15, +0.15]	1	Logs
➖	otlp_ingest_logs	memory utilization	-0.01	[-0.13, +0.12]	1	Logs
➖	file_tree	memory utilization	-0.06	[-0.10, -0.02]	1	Logs
➖	ddot_logs	memory utilization	-0.06	[-0.15, +0.03]	1	Logs
➖	quality_gate_logs	% cpu utilization	-2.37	[-5.10, +0.37]	1	Logs bounds checks dashboard

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	docker_containers_cpu	simple_check_run	10/10
✅	docker_containers_memory	memory_usage	10/10
✅	docker_containers_memory	simple_check_run	10/10
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.

agent-platform-auto-pr · 2025-07-11T22:29:11Z

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor 57027c9

Successful checks

Info

	Quality gate	Delta	On disk size (MiB)	Delta	On wire size (MiB)
✅	agent_deb_amd64	DataNotFound	$${704.72}$$ < $${704.84}$$	DataNotFound	$${177.88}$$ < $${178.58}$$
✅	agent_deb_amd64_fips	DataNotFound	$${702.97}$$ < $${703.09}$$	DataNotFound	$${177.33}$$ < $${178.12}$$
✅	agent_heroku_amd64	DataNotFound	$${354.61}$$ < $${355.37}$$	DataNotFound	$${94.85}$$ < $${95.72}$$
✅	agent_msi	DataNotFound	$${979.28}$$ < $${979.61}$$	DataNotFound	$${148.63}$$ < $${149.31}$$
✅	agent_rpm_amd64	DataNotFound	$${704.71}$$ < $${704.83}$$	DataNotFound	$${179.58}$$ < $${180.22}$$
✅	agent_rpm_amd64_fips	DataNotFound	$${702.96}$$ < $${703.08}$$	DataNotFound	$${179.21}$$ < $${179.85}$$
✅	agent_rpm_arm64	DataNotFound	$${694.51}$$ < $${694.74}$$	DataNotFound	$${163.25}$$ < $${163.96}$$
✅	agent_rpm_arm64_fips	DataNotFound	$${692.83}$$ < $${693.05}$$	DataNotFound	$${162.22}$$ < $${163.0}$$
✅	agent_suse_amd64	DataNotFound	$${704.71}$$ < $${704.83}$$	DataNotFound	$${179.58}$$ < $${180.22}$$
✅	agent_suse_amd64_fips	DataNotFound	$${702.96}$$ < $${703.08}$$	DataNotFound	$${179.21}$$ < $${179.85}$$
✅	agent_suse_arm64	DataNotFound	$${694.51}$$ < $${694.74}$$	DataNotFound	$${163.25}$$ < $${163.96}$$
✅	agent_suse_arm64_fips	DataNotFound	$${692.83}$$ < $${693.05}$$	DataNotFound	$${162.22}$$ < $${163.0}$$
✅	docker_agent_amd64	DataNotFound	$${788.47}$$ < $${788.65}$$	DataNotFound	$${271.37}$$ < $${272.01}$$
✅	docker_agent_arm64	DataNotFound	$${801.78}$$ < $${802.0}$$	DataNotFound	$${259.0}$$ < $${259.7}$$
✅	docker_agent_jmx_amd64	DataNotFound	$${979.71}$$ < $${979.84}$$	DataNotFound	$${340.38}$$ < $${340.95}$$
✅	docker_agent_jmx_arm64	DataNotFound	$${981.62}$$ < $${981.8}$$	DataNotFound	$${323.99}$$ < $${324.65}$$
✅	docker_cluster_agent_amd64	DataNotFound	$${213.9}$$ < $${214.5}$$	DataNotFound	$${72.71}$$ < $${73.51}$$
✅	docker_cluster_agent_arm64	DataNotFound	$${229.75}$$ < $${230.33}$$	DataNotFound	$${68.97}$$ < $${69.77}$$
✅	docker_cws_instrumentation_amd64	DataNotFound	$${7.07}$$ < $${7.12}$$	DataNotFound	$${2.95}$$ < $${3.29}$$
✅	docker_cws_instrumentation_arm64	DataNotFound	$${6.69}$$ < $${6.92}$$	DataNotFound	$${2.7}$$ < $${3.07}$$
✅	docker_dogstatsd_amd64	DataNotFound	$${38.55}$$ < $${39.57}$$	DataNotFound	$${14.89}$$ < $${15.76}$$
✅	docker_dogstatsd_arm64	DataNotFound	$${37.23}$$ < $${38.2}$$	DataNotFound	$${14.34}$$ < $${14.83}$$
✅	dogstatsd_deb_amd64	DataNotFound	$${29.79}$$ < $${31.4}$$	DataNotFound	$${7.85}$$ < $${8.95}$$
✅	dogstatsd_deb_arm64	DataNotFound	$${28.39}$$ < $${29.97}$$	DataNotFound	$${6.81}$$ < $${7.89}$$
✅	dogstatsd_rpm_amd64	DataNotFound	$${29.79}$$ < $${31.4}$$	DataNotFound	$${7.86}$$ < $${8.96}$$
✅	dogstatsd_suse_amd64	DataNotFound	$${29.79}$$ < $${31.4}$$	DataNotFound	$${7.86}$$ < $${8.96}$$
✅	iot_agent_deb_amd64	DataNotFound	$${54.05}$$ < $${54.55}$$	DataNotFound	$${13.63}$$ < $${14.45}$$
✅	iot_agent_deb_arm64	DataNotFound	$${51.37}$$ < $${51.9}$$	DataNotFound	$${11.79}$$ < $${12.63}$$
✅	iot_agent_deb_armhf	DataNotFound	$${50.88}$$ < $${51.42}$$	DataNotFound	$${11.86}$$ < $${12.74}$$
✅	iot_agent_rpm_amd64	DataNotFound	$${54.05}$$ < $${54.55}$$	DataNotFound	$${13.64}$$ < $${14.47}$$
✅	iot_agent_rpm_arm64	DataNotFound	$${51.37}$$ < $${51.91}$$	DataNotFound	$${11.81}$$ < $${12.65}$$
✅	iot_agent_suse_amd64	DataNotFound	$${54.05}$$ < $${54.55}$$	DataNotFound	$${13.64}$$ < $${14.47}$$

nina9753 · 2025-07-17T16:03:57Z

+// GetTags returns a map of gcp-related tags for Cloud Run Jobs.
+func (c *CloudRunJobs) GetTags() map[string]string {
+	tags := metadataHelperFunc(GetDefaultConfig(), false)
+	tags["origin"] = CloudRunJobsOrigin


I think this can be removed. We wanted to make origin a hidden tag, but in case some customers were using it, we needed to be backward compatible. Since this is a new product, we can exclude it here

good call, thanks for catching this!

apiarian-datadog · 2025-07-17T23:56:14Z


-	metric.AddShutdownMetric(prefix, origin, metricAgent.GetExtraTags(), time.Now(), metricAgent.Demux)
+	// Don't emit shutdown metric for Cloud Run Jobs
+	if _, ok := cloudService.(*cloudservice.CloudRunJobs); !ok {


nit: this sort of thing is usually a smell, adding service-specific logic gated by an if statement checking the service's struct type.

can we move this logic into the cloudService class, with the cloud run job implementation as a noop? or at the very least, can we add a shouldEmitShutdownMetric method to the cloud run services so that we consolidate the service-specific logic in with the cloud service implementation?

(i know we have other things like this already. i think i've seen some azure-specific code sprinkled around, but let's not propagate unfortunate patterns if we don't have to.)

actually, not a nit. let's do this differently. this and the cold start metric as well.

We emit this enhanced metric for Azure and GCP. I agree we can move this into cloud service, but we were also talking about adding cloud run job specific enhanced metrics relating to the jobs and tasks, like AWS step functions. I'll leave it up to Nick if he wants to do this in a separate PR or in this one

Yes, since we're going to emit an enhanced metric, we can rename AddColdStartMetric to AddStartMetric, which will be implemented by each service. Same with shutdown. wdyt?

apiarian-datadog

looks fine to me!

apiarian-datadog · 2025-07-21T18:29:02Z

 		return &CloudRun{spanNamespace: cloudRunService}
 	}

+	if isCloudRunJob() {


probably not in this pr, but this whole if ladder is a bit funky since it leaks the individual service details into this overall interface. might be nice to refactor this. but if you're not going to be making any more changes, maybe add a note for us to do it in the future?

i agree, that's wonky. left a comment

nhulston · 2025-07-23T19:32:34Z

/merge

dd-devflow-routing-codex · 2025-07-23T19:32:39Z

View all feedbacks in Devflow UI.

2025-07-23 19:32:38 UTC ℹ️ Start processing command /merge

2025-07-23 19:32:48 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in main is approximately 1h (p90).

2025-07-23 21:32:06 UTC ℹ️ MergeQueue: This merge request was merged

nhulston added 4 commits July 11, 2025 16:19

detect cloud run jobs

7f8e1b6

implement GetTags()

452eb7c

unit tests

1c23425

don't emit cold start or shutdown metrics for cloud run jobs

4e46dbe

github-actions Bot added the medium review PR review might take time label Jul 11, 2025

nhulston added team/serverless-azure-gcp qa/done QA done before merge and regressions are covered by tests labels Jul 11, 2025

nhulston added 2 commits July 11, 2025 16:37

lint

3a07b2c

lint

c535fd6

nhulston added 2 commits July 14, 2025 10:10

add changelog

93c05ec

Merge branch 'main' into nicholas.hulston/serverless-init-cloud-run-jobs

b5c706b

nhulston changed the title ~~[Serverless-Init] Support Cloud Run Jobs with serverless-init and sidecar~~ [Serverless-Init] Support Cloud Run Jobs with serverless-init Jul 14, 2025

nhulston marked this pull request as ready for review July 14, 2025 18:45

nhulston requested review from a team as code owners July 14, 2025 18:45

brett0000FF approved these changes Jul 14, 2025

View reviewed changes

nina9753 reviewed Jul 17, 2025

View reviewed changes

nhulston added 3 commits July 17, 2025 12:37

remove origin tag but keep it hidden under _dd.origin

fca0399

change span tags prefix from gcrjob to gcrj to follow convention

6922085

Merge branch 'main' into nicholas.hulston/serverless-init-cloud-run-jobs

76a4c2e

nhulston requested a review from nina9753 July 17, 2025 18:03

apiarian-datadog reviewed Jul 17, 2025

View reviewed changes

refactor; add enhanced start and shutdown metric for cloud run jobs

ce588d5

nhulston requested a review from apiarian-datadog July 21, 2025 17:23

nhulston added 3 commits July 21, 2025 13:28

Merge branch 'main' into nicholas.hulston/serverless-init-cloud-run-jobs

1821ef7

Merge branch 'main' into nicholas.hulston/serverless-init-cloud-run-jobs

011ebe1

fix metric_test.go

a6e6f95

apiarian-datadog approved these changes Jul 21, 2025

View reviewed changes

nhulston added 4 commits July 21, 2025 14:39

remove AddStartMetric, which was just wrapping Add

a51ad3f

add todo comment

c924866

Merge branch 'main' into nicholas.hulston/serverless-init-cloud-run-jobs

67802c5

Merge branch 'main' into nicholas.hulston/serverless-init-cloud-run-jobs

b798d5f

dd-mergequeue Bot merged commit c219db9 into main Jul 23, 2025
262 checks passed

dd-mergequeue Bot deleted the nicholas.hulston/serverless-init-cloud-run-jobs branch July 23, 2025 21:32

github-actions Bot added this to the 7.70.0 milestone Jul 23, 2025

Uh oh!

Conversation

nhulston commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Describe how you validated your changes

Possible Drawbacks / Trade-offs

Additional Notes

Uh oh!

cit-pr-commenter Bot commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

agent-platform-auto-pr Bot commented Jul 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Static quality checks

Info

Uh oh!

nina9753 Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

nhulston Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

apiarian-datadog Jul 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

apiarian-datadog Jul 17, 2025

Choose a reason for hiding this comment

Uh oh!

nina9753 Jul 18, 2025

Choose a reason for hiding this comment

Uh oh!

nhulston Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

apiarian-datadog left a comment

Choose a reason for hiding this comment

Uh oh!

apiarian-datadog Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

nhulston Jul 21, 2025

Choose a reason for hiding this comment

Uh oh!

nhulston commented Jul 23, 2025

Uh oh!

dd-devflow-routing-codex Bot commented Jul 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

nhulston commented Jul 11, 2025 •

edited

Loading

cit-pr-commenter Bot commented Jul 11, 2025 •

edited

Loading

agent-platform-auto-pr Bot commented Jul 11, 2025 •

edited

Loading

apiarian-datadog Jul 17, 2025 •

edited

Loading

dd-devflow-routing-codex Bot commented Jul 23, 2025 •

edited

Loading