Harden Linux Docker retries in CI by NachoEchevarria · Pull Request #8358 · DataDog/dd-trace-dotnet

NachoEchevarria · 2026-03-24T14:06:52Z

Summary

Add Linux Docker readiness check to CI, mirroring the existing Windows approach (PR #8245). Targets transient cgroup/systemd/dbus failures — the largest CI flake category (10 commits blocked).

Reason for change

retryCountOnTaskFailure retries on the same agent, so if Docker is broken on that VM, all retries fail. This adds active recovery (service restart) before Docker commands run.

For instance:

2026-03-25T15:05:08.0833114Z docker: Error response from daemon: failed to create task for container: Unavailable: error reading from server: EOF
2026-03-25T15:05:08.0833917Z 
2026-03-25T15:05:08.0834644Z Run 'docker run --help' for more information

Changes

New ensure-docker-ready-linux.sh: waits for Docker daemon, attempts systemctl restart docker if the service is down (mirrors Windows Restart-Service), emits diagnostics on failure
ensure-docker-ready.yml: Linux readiness step added alongside existing Windows step
run-in-docker.yml: embedded ensure-docker-ready.yml call — automatically covers all ~27 Linux jobs that use this template.
ultimate-pipeline.yml: added readiness check to integration_tests_windows_msi job (the only uncovered Windows docker-compose caller)

Co-authored-by: NachoEchevarria <[email protected]>

andrewlock · 2026-03-27T10:31:07Z

+#!/usr/bin/env bash
+# Linux Docker readiness check — mirrors the Windows PowerShell logic in ensure-docker-ready.yml.
+# Waits for the Docker daemon, attempts service restarts if needed, and fails fast
+# so the job can be rescheduled on a different agent.


# so the job can be rescheduled on a different agent.

This doesn't happen automatically though right, you have to manually retry? But it would never work anyway, so that's fine, just checking 🙂

andrewlock · 2026-03-27T10:36:52Z

+# Waits for the Docker daemon, attempts service restarts if needed, and fails fast
+# so the job can be rescheduled on a different agent.
+
+set -u


Should probably exit on failing commands too? Or does that break things? Meh, maybe best to leave it 😅

Suggested change

set -u

set -eu

Yes, we should avoid early exits in some commands that could possibly fail in the script

andrewlock · 2026-03-27T10:42:30Z

+        fi
+
+        # If Docker is not responding, try restarting the service
+        if command -v systemctl >/dev/null 2>&1; then


If this command fails (because systemctl is not available, then we don't actually restart anything, so we may as well just exit early instead I think? We can do that after logging the docker state if we want to. Alternatively, we could just not restart it, but still try to wait?

andrewlock · 2026-03-27T10:44:29Z

+                log "Docker service is ${svc_status}. Attempting restart ${restart_count}/${DOCKER_MAX_RESTARTS}..."
+                try_restart_docker


(suggestion based on previous comment)

Suggested change

log "Docker service is ${svc_status}. Attempting restart ${restart_count}/${DOCKER_MAX_RESTARTS}..."

try_restart_docker

if ! command -v systemctl >/dev/null 2>&1; then

log "Docker service is ${svc_status}, but systmctl is not available, so unable to explicitly restart service."

else

log "Docker service is ${svc_status}. Attempting restart ${restart_count}/${DOCKER_MAX_RESTARTS}..."

try_restart_docker

fi

In this case, we are already checking systemctl is-active docker in line 93, so if we reach this part of the code, the command was successful. I will leave the original code. WDYT?

Yeah, buy the problem is if it wasn't successful, we end up stuck in this loop where we're just sleeping repeatedly, not restarting anything, but still sleeping 🤔 That's the scenario this is trying to highlight 😄

andrewlock · 2026-03-27T10:45:53Z

@@ -0,0 +1,113 @@
+#!/usr/bin/env bash


Generally we try to just use POSIX shell instead of bash - if it's simple to convert to that, then it may save us some pain later

Nice catch! Done!

andrewlock · 2026-03-27T10:52:22Z

      displayName: BuildWindowsIntegrationTests
      retryCountOnTaskFailure: 3

+    - template: steps/ensure-docker-ready.yml


Good catch 👍

Co-authored-by: Andrew Lock <[email protected]>

…m/DataDog/dd-trace-dotnet into dd/ci/linux-docker-cgroup-retries

andrewlock

Great, thanks!

NachoEchevarria · 2026-04-09T14:20:21Z

Thank you for the feedback and review!

## Summary of changes Add a targeted retry wrapper around `docker run` in `run-in-docker.yml` to handle transient Docker daemon errors without blanket-retrying the entire CI step. ## Reason for change After merging #8358 (Docker daemon readiness checks), we still observed transient failures where the daemon was confirmed ready but `docker run` failed with D-Bus/cgroup errors like: > `unable to start unit "docker-....scope": Message recipient disconnected from message bus without replying` The daemon is healthy (passes `docker info`), but systemd's D-Bus has a momentary hiccup during container cgroup creation. This is a transient infrastructure issue, not a test failure. ## Implementation details - Adds a `docker_run_with_retry` shell function that wraps the `docker run` invocation - **Only retries on exit code 125** (Docker daemon error — container creation failure), up to 3 attempts with 10s delay - **Does not retry** on any other exit code (test/build failures inside the container pass through immediately) - **Always logs the exit code** on failure (`##[warning]`), so we can observe actual exit codes for future Docker errors regardless of whether they trigger a retry ## Test coverage CI pipeline change — no unit tests. Validated by reviewing Docker's exit code conventions (125 = daemon error). ## Other details Follow-up to #8358.

NachoEchevarria and others added 30 commits March 19, 2026 13:16

initial implementation

e7ea99d

update snapshots

43c805c

update snapshots

45a3914

fix duplicated

fa1e92c

Avoid two calls to GetServiceName

dfd5800

Add warning comments for enum sync

891cd46

rename param

92b5de7

Remove not needed

9495c55

Remove not needed

1603764

Refactor: Use ServiceNameMetadata to decrease double method calls.

83c03be

Fix unit tests

fb7e24a

remove integration arrays

b487692

Refactor schemas

76423c9

Use opt.service_mapping even if default

2e0044c

Fix dbscopefactory

d76c65e

Final check serviceNameEqualsDefault

6ef7bb5

SpanMessagePackFormatter opt case

18105ba

Add unit tests

42401b0

Minor refactor

564ff19

remove hardcoded

19abb0a

Initial implementation

3df9559

Fix unit. Azure Aws add source.

571a6e8

fix unit tests and integration tests

cbf0b86

Update v1 rules

02ecf55

Update OTEL

1721030

Remove commas

5bc3666

serivoce source stats. Initial implementation.

337c5fe

protect against empty strings

c80fcaf

Merge branch 'master' into nacho/ServiceSourceStats

d6e12c2

Add Linux docker cgroup retry wrapper

7add326

Co-authored-by: NachoEchevarria <[email protected]>

Improve logging

a78fcd8

NachoEchevarria marked this pull request as ready for review March 27, 2026 09:30

NachoEchevarria requested a review from a team as a code owner March 27, 2026 09:30

andrewlock reviewed Mar 27, 2026

View reviewed changes

NachoEchevarria and others added 5 commits March 27, 2026 12:47

Update .azure-pipelines/steps/ensure-docker-ready-linux.sh

134aa23

Co-authored-by: Andrew Lock <[email protected]>

Apply suggestion from @andrewlock

974b4fe

Co-authored-by: Andrew Lock <[email protected]>

Apply suggestion from @andrewlock

bf76fc4

Co-authored-by: Andrew Lock <[email protected]>

Merge branch 'dd/ci/linux-docker-cgroup-retries' of https://github.co…

16f424f

…m/DataDog/dd-trace-dotnet into dd/ci/linux-docker-cgroup-retries

undo

d98d1f7

NachoEchevarria requested a review from a team as a code owner March 27, 2026 12:13

NachoEchevarria and others added 11 commits April 6, 2026 11:51

Merge branch 'master' into dd/ci/linux-docker-cgroup-retries

f4f367f

Add log line

49bb57d

Merge branch 'dd/ci/linux-docker-cgroup-retries' of https://github.co…

9cc81c3

…m/DataDog/dd-trace-dotnet into dd/ci/linux-docker-cgroup-retries

Use POSIX shell

a66e824

Merge branch 'master' into dd/ci/linux-docker-cgroup-retries

47d9cc7

Early return

88d5788

Simplified try_restart_docker

4e1cdcd

Make ensure-docker-ready-linux.sh executable

ee16ba0

adress nits

9388163

Early return

26d3048

Avoid too verbose output

eea7deb

NachoEchevarria requested a review from andrewlock April 9, 2026 12:46

andrewlock approved these changes Apr 9, 2026

View reviewed changes

NachoEchevarria merged commit 549e87e into master Apr 9, 2026
137 checks passed

NachoEchevarria deleted the dd/ci/linux-docker-cgroup-retries branch April 9, 2026 14:20

github-actions Bot added this to the vNext-v3 milestone Apr 9, 2026

NachoEchevarria mentioned this pull request Apr 13, 2026

Docker run retry #8447

Merged

andrewlock added the type:flake-fix label Apr 20, 2026

		log "Docker service is ${svc_status}. Attempting restart ${restart_count}/${DOCKER_MAX_RESTARTS}..."
		try_restart_docker

Uh oh!

Conversation

NachoEchevarria commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Reason for change

Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewlock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NachoEchevarria commented Apr 9, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

NachoEchevarria commented Mar 24, 2026 •

edited

Loading