Skip to content

Commit 1ec9ede

Browse files
compheadalamb
andauthored
minor: make MinIO tests more stable (apache#25092)
## Rationale for this change I observed PRs sometimes got kicked out of merge queue because flaky `datafusion-cli` tests, which require full CI again The `datafusion-cli` storage integration tests (`test_cli`, `test_aws_options`, `test_s3_url_fallback`, `test_object_store_profiling`) each start their own MinIO container, so a single CI run pulls the image several times concurrently. Any transient Docker failure during that pull fails the test immediately: ``` thread 'test_aws_options' panicked at datafusion-cli/tests/cli_integration.rs:607:25: called `Result::unwrap()` on an `Err` value: "Failed to start MinIO container. Ensure Docker is running and accessible: failed to pull the image 'minio/minio:RELEASE.2025-02-28T09-55-16Z', error: bytes remaining on stream" ``` These errors are transient and clear on a second attempt, so a flaky pull should not fail the job. ## Rationale for this change The `datafusion-cli` storage integration tests (`test_cli`, `test_aws_options`, `test_s3_url_fallback`, `test_object_store_profiling`) each start their own MinIO container, so a single CI run pulls the image several times concurrently. Any transient Docker failure during that pull fails the test immediately: ``` thread 'test_aws_options' panicked at datafusion-cli/tests/cli_integration.rs:607:25: called `Result::unwrap()` on an `Err` value: "Failed to start MinIO container. Ensure Docker is running and accessible: failed to pull the image 'minio/minio:RELEASE.2025-02-28T09-55-16Z', error: bytes remaining on stream" ``` These errors are transient and clear on a second attempt, so a flaky pull should not fail the job. Co-authored-by: Andrew Lamb <[email protected]>
1 parent 4f0faac commit 1ec9ede

2 files changed

Lines changed: 204 additions & 105 deletions

File tree

‎.github/workflows/rust.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -395,6 +395,21 @@ jobs:
395395
with:
396396
save-if: false # set in linux-test
397397
shared-key: "amd-ci"
398+
# The storage integration tests start MinIO containers. Pulling the image
399+
# once up front keeps the pull off the critical path of the tests, which
400+
# would otherwise pull it several times concurrently and occasionally fail
401+
# with transient Docker transport errors. The tests retry the pull
402+
# themselves, so a failure here is only a warning.
403+
#
404+
# MINIO_IMAGE must match the image used by the `minio` module of the
405+
# `testcontainers-modules` crate. The `minio_image_matches_ci_prepull`
406+
# test in `datafusion-cli/tests/cli_integration.rs` fails if it drifts.
407+
- name: Pre-pull MinIO image
408+
env:
409+
MINIO_IMAGE: minio/minio:RELEASE.2025-02-28T09-55-16Z
410+
run: |
411+
ci/scripts/retry timeout 120 docker pull "$MINIO_IMAGE" \
412+
|| echo "::warning::Could not pre-pull $MINIO_IMAGE, the tests will pull it themselves"
398413
- name: Run tests (excluding doctests)
399414
env:
400415
RUST_BACKTRACE: 1

0 commit comments

Comments
 (0)