Extend `bazel` test scope by rdesgroppes · Pull Request #44455 · DataDog/datadog-agent

rdesgroppes · 2025-12-19T11:06:30Z

What does this PR do?

extend bazel test scope in CI from //bazel/tests/... to //...,
fix remaining failure after doing so.

Motivation

the deps/acl/acl.MODULE.bazel file was out of sync with its generated version (using url instead of urls), but this inconsistency was not caught: bazel test //bazel/tests/... was missing consistency tests in //deps/** (addressed by Fix bazel test //deps/acl:module_consistency_test #44750 since),
after discovery: Import sd-agent #44881, was merged, //pkg/discovery/module/rust:dd_discovery_test panicked on assertion failed: result.is_some() (addressed by discovery: Add missing files from sd-agent import #45140 since),
more to come...

Describe how you validated your changes

CI.

Additional Notes

We should make sure bazel test //... always works, whatever the OS our colleagues are on.

agent-platform-auto-pr · 2025-12-19T11:50:23Z

Static quality checks

✅ Please find below the results from static quality gates
Comparison made with ancestor 45cdc09
📊 Static Quality Gates Dashboard

31 successful checks with minimal change (< 2 KiB)

	Quality gate	Current Size
✅	agent_deb_amd64	746.980 MiB
✅	agent_deb_amd64_fips	707.009 MiB
✅	agent_heroku_amd64	324.136 MiB
✅	agent_msi	658.952 MiB
✅	agent_rpm_amd64	746.964 MiB
✅	agent_rpm_amd64_fips	706.992 MiB
✅	agent_rpm_arm64	726.148 MiB
✅	agent_rpm_arm64_fips	688.661 MiB
✅	agent_suse_amd64	746.964 MiB
✅	agent_suse_amd64_fips	706.992 MiB
✅	agent_suse_arm64	726.148 MiB
✅	agent_suse_arm64_fips	688.661 MiB
✅	docker_agent_amd64	809.452 MiB
✅	docker_agent_arm64	813.234 MiB
✅	docker_agent_jmx_amd64	1000.363 MiB
✅	docker_agent_jmx_arm64	992.832 MiB
✅	docker_cluster_agent_amd64	180.867 MiB
✅	docker_cluster_agent_arm64	196.732 MiB
✅	docker_cws_instrumentation_amd64	7.135 MiB
✅	docker_cws_instrumentation_arm64	6.689 MiB
✅	docker_dogstatsd_amd64	38.449 MiB
✅	docker_dogstatsd_arm64	36.812 MiB
✅	dogstatsd_deb_amd64	29.669 MiB
✅	dogstatsd_deb_arm64	27.833 MiB
✅	dogstatsd_rpm_amd64	29.669 MiB
✅	dogstatsd_suse_amd64	29.669 MiB
✅	iot_agent_deb_amd64	42.766 MiB
✅	iot_agent_deb_arm64	39.884 MiB
✅	iot_agent_deb_armhf	40.454 MiB
✅	iot_agent_rpm_amd64	42.767 MiB
✅	iot_agent_suse_amd64	42.767 MiB

On-wire sizes (compressed)

	Quality gate	Change	Size (prev → curr → max)
✅	agent_deb_amd64	-8.7 KiB (0.00% reduction)	182.663 → 182.655 → 184.810
✅	agent_deb_amd64_fips	-5.58 KiB (0.00% reduction)	174.144 → 174.139 → 177.560
✅	agent_heroku_amd64	neutral	87.173 MiB → 88.450
✅	agent_msi	-44.0 KiB (0.03% reduction)	142.492 → 142.449 → 143.300
✅	agent_rpm_amd64	neutral	185.686 MiB → 188.160
✅	agent_rpm_amd64_fips	+14.57 KiB (0.01% increase)	176.461 → 176.475 → 178.900
✅	agent_rpm_arm64	+27.13 KiB (0.02% increase)	168.259 → 168.286 → 169.930
✅	agent_rpm_arm64_fips	+19.25 KiB (0.01% increase)	160.612 → 160.631 → 163.120
✅	agent_suse_amd64	neutral	185.686 MiB → 188.160
✅	agent_suse_amd64_fips	+14.57 KiB (0.01% increase)	176.461 → 176.475 → 178.900
✅	agent_suse_arm64	+27.13 KiB (0.02% increase)	168.259 → 168.286 → 169.930
✅	agent_suse_arm64_fips	+19.25 KiB (0.01% increase)	160.612 → 160.631 → 163.120
✅	docker_agent_amd64	+3.6 KiB (0.00% increase)	274.752 → 274.756 → 277.400
✅	docker_agent_arm64	-3.01 KiB (0.00% reduction)	262.311 → 262.308 → 266.040
✅	docker_agent_jmx_amd64	neutral	343.404 MiB → 346.020
✅	docker_agent_jmx_arm64	+3.09 KiB (0.00% increase)	326.918 → 326.921 → 330.660
✅	docker_cluster_agent_amd64	neutral	63.887 MiB → 64.510
✅	docker_cluster_agent_arm64	neutral	60.151 MiB → 61.170
✅	docker_cws_instrumentation_amd64	neutral	2.994 MiB → 3.330
✅	docker_cws_instrumentation_arm64	neutral	2.726 MiB → 3.090
✅	docker_dogstatsd_amd64	neutral	14.877 MiB → 15.820
✅	docker_dogstatsd_arm64	neutral	14.219 MiB → 14.830
✅	dogstatsd_deb_amd64	neutral	7.839 MiB → 8.790
✅	dogstatsd_deb_arm64	neutral	6.726 MiB → 7.710
✅	dogstatsd_rpm_amd64	neutral	7.850 MiB → 8.800
✅	dogstatsd_suse_amd64	neutral	7.850 MiB → 8.800
✅	iot_agent_deb_amd64	neutral	11.217 MiB → 12.040
✅	iot_agent_deb_arm64	neutral	9.589 MiB → 10.450
✅	iot_agent_deb_armhf	neutral	9.784 MiB → 10.620
✅	iot_agent_rpm_amd64	-2.56 KiB (0.02% reduction)	11.236 → 11.234 → 12.060
✅	iot_agent_suse_amd64	-2.56 KiB (0.02% reduction)	11.236 → 11.234 → 12.060

cit-pr-commenter · 2025-12-19T13:03:34Z

Regression Detector

Regression Detector Results

Metrics dashboard
Target profiles
Run ID: cb1f1b22-aa23-4e86-bd5d-19a753a24058

Baseline: 500dcf3
Comparison: 4bbe0a0
Diff

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Regressions in experiments with settings containing erratic: true are ignored.

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
❌	docker_containers_cpu	% cpu utilization	+5.13	[+2.12, +8.14]	1	Logs

Fine details of change detection per experiment

perf	experiment	goal	Δ mean %	Δ mean % CI	trials	links
❌	docker_containers_cpu	% cpu utilization	+5.13	[+2.12, +8.14]	1	Logs
➖	quality_gate_logs	% cpu utilization	+0.84	[-0.67, +2.34]	1	Logs bounds checks dashboard
➖	quality_gate_metrics_logs	memory utilization	+0.76	[+0.53, +0.99]	1	Logs bounds checks dashboard
➖	otlp_ingest_logs	memory utilization	+0.38	[+0.28, +0.49]	1	Logs
➖	uds_dogstatsd_20mb_12k_contexts_20_senders	memory utilization	+0.31	[+0.25, +0.36]	1	Logs
➖	ddot_logs	memory utilization	+0.09	[+0.03, +0.14]	1	Logs
➖	quality_gate_idle_all_features	memory utilization	+0.06	[+0.02, +0.09]	1	Logs bounds checks dashboard
➖	file_to_blackhole_1000ms_latency	egress throughput	+0.03	[-0.38, +0.43]	1	Logs
➖	uds_dogstatsd_to_api	ingress throughput	+0.01	[-0.13, +0.14]	1	Logs
➖	file_to_blackhole_0ms_latency	egress throughput	+0.01	[-0.50, +0.51]	1	Logs
➖	tcp_dd_logs_filter_exclude	ingress throughput	+0.00	[-0.10, +0.10]	1	Logs
➖	ddot_metrics_sum_delta	memory utilization	-0.00	[-0.21, +0.21]	1	Logs
➖	quality_gate_idle	memory utilization	-0.01	[-0.06, +0.03]	1	Logs bounds checks dashboard
➖	uds_dogstatsd_to_api_v3	ingress throughput	-0.02	[-0.14, +0.11]	1	Logs
➖	file_to_blackhole_500ms_latency	egress throughput	-0.02	[-0.40, +0.37]	1	Logs
➖	file_to_blackhole_100ms_latency	egress throughput	-0.02	[-0.07, +0.02]	1	Logs
➖	ddot_metrics_sum_cumulativetodelta_exporter	memory utilization	-0.02	[-0.26, +0.21]	1	Logs
➖	ddot_metrics	memory utilization	-0.12	[-0.34, +0.11]	1	Logs
➖	ddot_metrics_sum_cumulative	memory utilization	-0.26	[-0.43, -0.10]	1	Logs
➖	file_tree	memory utilization	-0.37	[-0.42, -0.31]	1	Logs
➖	otlp_ingest_metrics	memory utilization	-0.37	[-0.52, -0.21]	1	Logs
➖	docker_containers_memory	memory utilization	-0.37	[-0.45, -0.30]	1	Logs
➖	tcp_syslog_to_blackhole	ingress throughput	-2.18	[-2.25, -2.10]	1	Logs

Bounds Checks: ✅ Passed

perf	experiment	bounds_check_name	replicates_passed	links
✅	docker_containers_cpu	simple_check_run	10/10
✅	docker_containers_memory	memory_usage	10/10
✅	docker_containers_memory	simple_check_run	10/10
✅	file_to_blackhole_0ms_latency	lost_bytes	10/10
✅	file_to_blackhole_0ms_latency	memory_usage	10/10
✅	file_to_blackhole_1000ms_latency	lost_bytes	10/10
✅	file_to_blackhole_1000ms_latency	memory_usage	10/10
✅	file_to_blackhole_100ms_latency	lost_bytes	10/10
✅	file_to_blackhole_100ms_latency	memory_usage	10/10
✅	file_to_blackhole_500ms_latency	lost_bytes	10/10
✅	file_to_blackhole_500ms_latency	memory_usage	10/10
✅	quality_gate_idle	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_idle_all_features	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_logs	memory_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	cpu_usage	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	intake_connections	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	lost_bytes	10/10	bounds checks dashboard
✅	quality_gate_metrics_logs	memory_usage	10/10	bounds checks dashboard

Explanation

Confidence level: 90.00%
Effect size tolerance: |Δ mean %| ≥ 5.00%

Performance changes are noted in the perf column of each table:

✅ = significantly better comparison variant performance
❌ = significantly worse comparison variant performance
➖ = no significant change in performance

A regression test is an A/B test of target performance in a repeatable rig, where "performance" is measured as "comparison variant minus baseline variant" for an optimization goal (e.g., ingress throughput). Due to intrinsic variability in measuring that goal, we can only estimate its mean value for each experiment; we report uncertainty in that value as a 90.00% confidence interval denoted "Δ mean % CI".

For each experiment, we decide whether a change in performance is a "regression" -- a change worth investigating further -- if all of the following criteria are true:

Its estimated |Δ mean %| ≥ 5.00%, indicating the change is big enough to merit a closer look.
Its 90.00% confidence interval "Δ mean % CI" does not contain zero, indicating that if our statistical model is accurate, there is at least a 90.00% chance there is a difference in performance between baseline and comparison variants.
Its configuration does not mark it "erratic".

CI Pass/Fail Decision

✅ Passed. All Quality Gates passed.

quality_gate_metrics_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check cpu_usage: 10/10 replicas passed. Gate passed.
quality_gate_metrics_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle_all_features, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_idle, bounds check memory_usage: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check intake_connections: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check lost_bytes: 10/10 replicas passed. Gate passed.
quality_gate_logs, bounds check memory_usage: 10/10 replicas passed. Gate passed.

aiuto

The module generator is something we should do.
bazel test //... is probably not.

aiuto · 2025-12-20T03:14:39Z

.gitlab/build/bazel/tests.yml

  script:
-    - bazel test //bazel/tests/...
+    - bazel test //...


Let's discuss this as a team first. It is going to get big and complex soon, especially with multiple platforms. It is not always possible to write strict test skipping with target_compatible_with.

OK. I've been extracting "less disputed" bits into small PRs until we're all back.

agent-platform-auto-pr · 2026-01-16T11:12:36Z

Gitlab CI Configuration Changes

Modified Jobs

bazel:test:windows-amd64

  bazel:test:windows-amd64:
    before_script:
    - C:\ci-identities-gitlab-job-client.exe assume-role
    id_tokens:
      CI_IDENTITIES_GITLAB_ID_TOKEN:
        aud: ci-identities
    needs: []
    rules:
-   - if: $SKIP_WINDOWS == "true"
-     when: never
    - when: on_success
    script:
    - '$FailFast = @''
  
      $ErrorActionPreference = ''Stop''
  
      $PSNativeCommandUseErrorActionPreference = $true
  
      Set-StrictMode -Version 3.0
  
      ''@
  
      Invoke-Expression $FailFast
  
      $Utf8Script = [Text.Encoding]::UTF8.GetString([Text.Encoding]::GetEncoding([Console]::OutputEncoding.CodePage).GetBytes($POWERSHELL_SCRIPT))
  
      $EncodedCommand = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes("$FailFast`n$Utf8Script"))'
    - .\tools\ci\docker-run-with-bazel-cache.ps1 --env=CI_IDENTITIES_GITLAB_ID_TOKEN
      --volume="${CI_PROJECT_DIR}:$CI_PROJECT_DIR" --workdir="$CI_PROJECT_DIR" "$WINBUILDIMAGE"
      powershell -NonInteractive -NoLogo -NoProfile -InputFormat Text -OutputFormat
      Text -EncodedCommand $EncodedCommand
    stage: source_test
    tags:
    - windows-v2:2022
    timeout: 60m
    variables:
      ARCH: x64
      AWS_SHARED_CREDENTIALS_FILE: ${CI_PROJECT_DIR}\.aws\credentials-by-job-id\${CI_JOB_ID}
      BAZELISK_HOME: $XDG_CACHE_HOME/bazelisk
      GIT_DEPTH: 2
      OVERRIDE_GIT_STRATEGY: clone
-     POWERSHELL_SCRIPT: bazel test //bazel/tests/... //rtloader/...
+     POWERSHELL_SCRIPT: bazel test //...
      WINBUILDIMAGE: registry.ddbuild.io/ci/datadog-agent-buildimages/windows_ltsc2022_${ARCH}${CI_IMAGE_WIN_LTSC2022_X64_SUFFIX}:${CI_IMAGE_WIN_LTSC2022_X64}
      XDG_CACHE_HOME: c:/bzl

Changes Summary

Removed	Modified	Added	Renamed
0	1	0	0

ℹ️ Diff available in the job log.

### What does this PR do? This upgrade includes a Windows compatibility fix for batch file line endings (bazel-contrib/bazel-lib#1222), which resolves a cmd parsing bug where GOTO/CALL to labels fails when batch files use non CRLF-only line endings and a label crosses a 512-byte boundary: ```diff > bazel test //deps/openssl3:module_consistency_test [...] ==================== Test output for //deps/openssl3:module_consistency_test: -The system cannot find the batch label specified - compare_files +compare_files +FAIL: files "C:\[...]\bazel-out\[...]\deps\openssl3\openssl3.MODULE.bazel.new" and "C:\[workspace]\deps\openssl3\openssl3.MODULE.bazel" differ. + + +@@//deps/openssl3:openssl3.MODULE.bazel is out of date. To update this file, run: + + bazel run //deps/openssl3:module_consistency +To see differences run: + + diff "C:\[...]\bazel-out\[...]\deps\openssl3\openssl3.MODULE.bazel.new" "C:\[workspace]\deps\openssl3\openssl3.MODULE.bazel" + ================================================================================ ``` Because this now surfaces we'd been generating `*.MODULE.bazel` with OS-specific newlines, the present change fixes it by always issuing UNIX-style newlines: ``` > bazel test //deps/openssl3:module_consistency_test [...] //deps/openssl3:module_consistency_test PASSED in 0.4s ``` ### Motivation Keep up to date and better support colleagues working on Windows. ### Describe how you validated your changes Locally for the time being because we still have very limited test coverage due to #44455 being disputed. ### Additional Notes `bazel_lib` 3.1.0/3.1.1 comes with further improvements, like: - bazel-contrib/bazel-lib#1217 - bazel-contrib/bazel-lib#1220 - bazel-contrib/bazel-lib#1232 Co-authored-by: regis.desgroppes <[email protected]>

### What does this PR do? - extend `bazel test` scope in CI from `//bazel/tests/...` to `//...`, - fix remaining failures after doing so. ### Motivation The `deps/acl/acl.MODULE.bazel` file was out of sync with its generated version (using `url` instead of `urls`), but this inconsistency was not caught.

agent-platform-auto-pr · 2026-03-21T03:20:10Z

Files inventory check summary

File checks results against ancestor b1362953:

Results for datadog-agent_7.79.0~devel.git.40.ad896a6.pipeline.103842958-1_amd64.deb:

No change detected

rdesgroppes requested a review from a team as a code owner December 19, 2025 11:06

rdesgroppes added changelog/no-changelog No changelog entry needed qa/no-code-change No code change in Agent code requiring validation labels Dec 19, 2025

github-actions bot added medium review PR review might take time team/agent-build labels Dec 19, 2025

rdesgroppes force-pushed the regis.desgroppes/extend-bazel-test-scope branch 3 times, most recently from 41faa56 to c408cf2 Compare December 19, 2025 11:45

rdesgroppes force-pushed the regis.desgroppes/extend-bazel-test-scope branch from c408cf2 to 3c73c00 Compare December 19, 2025 12:03

rdesgroppes force-pushed the regis.desgroppes/extend-bazel-test-scope branch 2 times, most recently from d968077 to b8e899a Compare December 19, 2025 15:33

github-actions bot added long review PR is complex, plan time to review it and removed medium review PR review might take time labels Dec 19, 2025

aiuto requested changes Dec 20, 2025

View reviewed changes

rdesgroppes force-pushed the regis.desgroppes/extend-bazel-test-scope branch from b8e899a to 6acf7cc Compare January 8, 2026 13:01

github-actions bot added medium review PR review might take time and removed long review PR is complex, plan time to review it labels Jan 8, 2026

rdesgroppes changed the title ~~Extend bazel deps module consistency enforcement~~ Extend bazel test scope Jan 8, 2026

rdesgroppes force-pushed the regis.desgroppes/extend-bazel-test-scope branch 2 times, most recently from e5f9dba to d1864d3 Compare January 16, 2026 11:08

rdesgroppes force-pushed the regis.desgroppes/extend-bazel-test-scope branch from d1864d3 to ed34984 Compare January 19, 2026 12:15

rdesgroppes mentioned this pull request Jan 19, 2026

Bump bazel_lib from 3.0.1 to 3.1.1 #45220

Merged

rdesgroppes force-pushed the regis.desgroppes/extend-bazel-test-scope branch 2 times, most recently from 1de9de4 to 4bbe0a0 Compare January 22, 2026 15:23

rdesgroppes force-pushed the regis.desgroppes/extend-bazel-test-scope branch from 4bbe0a0 to 7e238ba Compare February 3, 2026 08:32

rdesgroppes force-pushed the regis.desgroppes/extend-bazel-test-scope branch from 7e238ba to ad896a6 Compare March 21, 2026 02:59

dd-octo-sts bot added the internal Identify a non-fork PR label Mar 21, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend `bazel` test scope#44455

Extend `bazel` test scope#44455
rdesgroppes wants to merge 1 commit intomainfrom
regis.desgroppes/extend-bazel-test-scope

rdesgroppes commented Dec 19, 2025 •

edited

Loading

Uh oh!

agent-platform-auto-pr bot commented Dec 19, 2025 •

edited

Loading

Uh oh!

cit-pr-commenter bot commented Dec 19, 2025 •

edited by cit-pr-commenter-54b7da bot

Loading

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

Uh oh!

aiuto left a comment

Uh oh!

aiuto Dec 20, 2025

Uh oh!

rdesgroppes Jan 5, 2026

Uh oh!

agent-platform-auto-pr bot commented Jan 16, 2026 •

edited

Loading

Uh oh!

agent-platform-auto-pr bot commented Mar 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

rdesgroppes commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Describe how you validated your changes

Additional Notes

Uh oh!

agent-platform-auto-pr bot commented Dec 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Static quality checks

Uh oh!

cit-pr-commenter bot commented Dec 19, 2025 • edited by cit-pr-commenter-54b7da bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Regression Detector

Regression Detector Results

Optimization Goals: ✅ No significant changes detected

Experiments ignored for regressions

Fine details of change detection per experiment

Bounds Checks: ✅ Passed

Explanation

CI Pass/Fail Decision

Uh oh!

aiuto left a comment

Choose a reason for hiding this comment

Uh oh!

aiuto Dec 20, 2025

Choose a reason for hiding this comment

Uh oh!

rdesgroppes Jan 5, 2026

Choose a reason for hiding this comment

Uh oh!

agent-platform-auto-pr bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Gitlab CI Configuration Changes

Modified Jobs

Changes Summary

Uh oh!

agent-platform-auto-pr bot commented Mar 21, 2026

Files inventory check summary

Results for datadog-agent_7.79.0~devel.git.40.ad896a6.pipeline.103842958-1_amd64.deb:

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

rdesgroppes commented Dec 19, 2025 •

edited

Loading

agent-platform-auto-pr bot commented Dec 19, 2025 •

edited

Loading

cit-pr-commenter bot commented Dec 19, 2025 •

edited by cit-pr-commenter-54b7da bot

Loading

agent-platform-auto-pr bot commented Jan 16, 2026 •

edited

Loading