Skip to content

Prune docker images/volumes in free-disk-space script#19229

Open
zeitlinger wants to merge 2 commits into
open-telemetry:mainfrom
zeitlinger:free-more-disk-space-docker
Open

Prune docker images/volumes in free-disk-space script#19229
zeitlinger wants to merge 2 commits into
open-telemetry:mainfrom
zeitlinger:free-more-disk-space-docker

Conversation

@zeitlinger

@zeitlinger zeitlinger commented Jul 15, 2026

Copy link
Copy Markdown
Member

Summary

  • Test suites pull many testcontainers images over the course of a single CI job (elasticsearch, kafka connect, etc.), which can exhaust runner disk space mid-run — saw this cause a spurious CI failure on Consolidate markdown formatting onto rumdl/flint, drop spotless prettier #19183 (docs-only PR, No space left on device on a test3 matrix job, disk went from 39GB free to 0 during the test run).
  • Adds docker system prune -af --volumes to .github/scripts/gha-free-disk-space.sh, removing unused Docker content already present on the runner before tests begin.
  • GitHub-hosted jobs start on a fresh runner VM, so this is not intended to clean Docker data from a prior workflow run. It targets unused content present on the fresh runner image or created during job setup.

Caveat: this only frees space before tests start, matching the existing script's approach — it does not address images pulled during the job, so it may reduce but not eliminate this failure mode.

Observed evidence

  • The 16 Ubuntu smoke jobs in run 29429470350 and run 29580716222 each reported 1.851 GB reclaimed by Docker pruning.
  • The overall filesystem cleanup increased available space by roughly 25–26 GB in those jobs; that larger number includes the script's non-Docker cleanup and should not be attributed to Docker alone.
  • Both runs completed successfully. This is evidence of additional capacity, not a controlled measurement of failure prevention or total runtime improvement.

Test plan

  • CI passes

Test suites pull many testcontainers images over the course of a single
CI job (elasticsearch, kafka connect, etc.), which can exhaust runner
disk space mid-run. Prune whatever shipped with the runner image first
to make more room, following the same pattern as open-telemetry#15597 and open-telemetry#15851.

Signed-off-by: Gregor Zeitlinger <[email protected]>
@zeitlinger
zeitlinger marked this pull request as ready for review July 15, 2026 15:48
@zeitlinger
zeitlinger requested a review from a team as a code owner July 15, 2026 15:48
Copilot AI review requested due to automatic review settings July 15, 2026 15:48

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Docker cleanup to reduce disk exhaustion in CI jobs.

Changes:

  • Prunes unused Docker images, containers, networks, and volumes before tests.
  • Documents why Testcontainers workloads require additional space.

Comment thread .github/scripts/gha-free-disk-space.sh Outdated
@opentelemetry-pr-dashboard

opentelemetry-pr-dashboard Bot commented Jul 15, 2026

Copy link
Copy Markdown

Pull request dashboard status

Status last refreshed: 2026-07-26 21:34:55 UTC.

  • Waiting on: Reviewers
  • Next step: Review the latest changes.

This automated status or its linked feedback items may be incorrect. If something looks wrong, please report it with the result you expected.

Comment thread .github/scripts/gha-free-disk-space.sh Outdated
Probe Docker before pruning so Windows jobs with a stopped service can reach the existing service-start workaround. Keep actual prune failures fatal once the daemon responds.

Observed in the existing PR run 29429470350 on ubuntu-24.04: Docker prune reported 1.851GB reclaimed from runner-image content. This is image-dependent and not a guaranteed amount; GitHub-hosted jobs use fresh runners, so prior workflow runs are not a source of Docker data.
Signed-off-by: Gregor Zeitlinger <[email protected]>
@zeitlinger

Copy link
Copy Markdown
Member Author

A possible follow-up would be to prune Docker data during a long-running job, but only at safe boundaries between independent test batches or when disk pressure crosses a threshold—not concurrently with active Testcontainers workloads.

That would need separate measurement: reclaimed space, prune duration, image re-pulls, total job time, and whether it prevents No space left on device failures. An unrestricted mid-run docker system prune -af --volumes could remove stopped containers, unused volumes, networks, and images that later need to be downloaded again. For now, this PR intentionally keeps the lower-risk pre-test cleanup.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants