Skip to content

Commit e2ca7f3

Browse files
a10ya10y
authored
fix datafusion-cli CI: switch from minio to rustfs (#25706)
## Which issue does this PR close? Closes #25705 ## Rationale for this change This morning it seems like Minio pulled its images from quay.io, which has caused recent CI runs to start failing, e.g. https://github.com/apache/datafusion/actions/runs/36006680489/job/107657362588 RustFS is a pretty comparable (and more permissively licensed) alternative that provides a suitable S3 interface for integration tests. This was substantially authored by Codex, but reviewed by me and tested locally. ## What is the testing strategy for this PR? Existing integration tests ## Are there any user-facing changes? No Signed-off-by: Andrew Duffy <[email protected]>
1 parent 21a3215 commit e2ca7f3

4 files changed

Lines changed: 149 additions & 122 deletions

File tree

‎.github/workflows/rust.yml‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -480,22 +480,21 @@ jobs:
480480
with:
481481
save-if: false # set in linux-test
482482
shared-key: "amd-ci"
483-
# The storage integration tests start MinIO containers. Pulling the image
483+
# The storage integration tests start RustFS containers. Pulling the image
484484
# once up front keeps the pull off the critical path of the tests, which
485485
# would otherwise pull it several times concurrently and occasionally fail
486486
# with transient Docker transport errors. The tests retry the pull
487487
# themselves, so a failure here is only a warning.
488488
#
489-
# MINIO_IMAGE must match what the tests start: the tag comes from the
490-
# `minio` module of `testcontainers-modules`, the registry from
491-
# MINIO_IMAGE_NAME in `datafusion-cli/tests/cli_integration.rs`. The
492-
# `minio_image_matches_ci_prepull` test fails if either drifts.
493-
- name: Pre-pull MinIO image
489+
# RUSTFS_IMAGE must match RUSTFS_IMAGE_NAME and RUSTFS_IMAGE_TAG in
490+
# `datafusion-cli/tests/cli_integration.rs`. The
491+
# `rustfs_image_matches_ci_prepull` test fails if they differ.
492+
- name: Pre-pull RustFS image
494493
env:
495-
MINIO_IMAGE: quay.io/minio/minio:RELEASE.2025-02-28T09-55-16Z
494+
RUSTFS_IMAGE: docker.io/rustfs/rustfs:1.0.0
496495
run: |
497-
ci/scripts/retry timeout 120 docker pull "$MINIO_IMAGE" \
498-
|| echo "::warning::Could not pre-pull $MINIO_IMAGE, the tests will pull it themselves"
496+
ci/scripts/retry timeout 120 docker pull "$RUSTFS_IMAGE" \
497+
|| echo "::warning::Could not pre-pull $RUSTFS_IMAGE, the tests will pull it themselves"
499498
- name: Run tests (excluding doctests)
500499
env:
501500
RUST_BACKTRACE: 1

‎datafusion-cli/CONTRIBUTING.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ cargo test --all-targets
3535

3636
## Running Storage Integration Tests
3737

38-
By default, storage integration tests are not run. These tests use the `testcontainers` crate to start up a local MinIO server using Docker on port 9000.
38+
By default, storage integration tests are not run. These tests use the `testcontainers` crate to start up a local RustFS server using Docker with a dynamically assigned host port.
3939

4040
To run them you will need to set `TEST_STORAGE_INTEGRATION`:
4141

@@ -47,8 +47,8 @@ For some of the tests, [snapshots](https://datafusion.apache.org/contributor-gui
4747

4848
### AWS
4949

50-
S3 integration is tested against [Minio](https://github.com/minio/minio) with [TestContainers](https://github.com/testcontainers/testcontainers-rs)
51-
This requires Docker to be running on your machine and port 9000 to be free.
50+
S3 integration is tested against [RustFS](https://github.com/rustfs/rustfs) with [TestContainers](https://github.com/testcontainers/testcontainers-rs)
51+
This requires Docker to be running on your machine.
5252

5353
If you see an error mentioning "failed to load IMDS session token" such as
5454

‎datafusion-cli/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ ctor = { workspace = true }
7878
insta = { workspace = true }
7979
insta-cmd = "0.7.0"
8080
rstest = { workspace = true }
81-
testcontainers-modules = { workspace = true, features = ["minio"] }
81+
testcontainers-modules = { workspace = true }
8282
# Makes sure `test_display_pg_json` behaves in a consistent way regardless of
8383
# feature unification with dependencies
8484
serde_json = { workspace = true, features = ["preserve_order"] }

0 commit comments

Comments
 (0)