Skip to content

Commit 9f5714b

Browse files
compheada10y
andauthored
[branch-55] fix datafusion-cli CI: switch from minio to rustfs (#25706) (#25759)
## Which issue does this PR close? - Fixes #25705 on `branch-55`. ## Rationale for this change This PR backports #25706 from @a10y to the `branch-55` line. `quay.io/minio/minio` no longer allows anonymous pulls. `branch-55` pulls that image for the `datafusion-cli` storage integration tests (it moved there from Docker Hub in #25485), so the `cargo test datafusion-cli (amd64)` job is expected to hit the same failure that #25705 reported on `main`. The last `branch-55` CI run was on 2026-09-19, before the image became unavailable, so no run on this branch has failed yet. ## What changes are included in this PR? A clean `git cherry-pick -x` of e2ca7f3 with no conflicts. The patch is identical to the original (same `git patch-id`). - `datafusion-cli/tests/cli_integration.rs`: run the S3 tests against a RustFS container (`docker.io/rustfs/rustfs:1.0.0`) instead of MinIO. - `.github/workflows/rust.yml`: pre-pull the RustFS image instead of the MinIO one. - `datafusion-cli/Cargo.toml`: drop the `minio` feature of `testcontainers-modules`. - `datafusion-cli/CONTRIBUTING.md`: update the storage test docs. ## What is the testing strategy for this PR? The existing `datafusion-cli` storage integration tests cover this change, and CI runs them in the `cargo test datafusion-cli (amd64)` job. On this branch, `datafusion-cli/tests/` and its snapshots match `main` apart from one unrelated Spark test. `testcontainers` and `testcontainers-modules` also resolve to the same versions as on `main`. Locally, `cargo fmt --all -- --check`, `cargo clippy -p datafusion-cli --all-targets --all-features -- -D warnings`, `taplo format --check` and the prettier docs check pass. The Docker-based storage tests were not run locally. CI runs them. ## Are there any user-facing changes? No. The change only affects tests, CI and contributor docs. Signed-off-by: Andrew Duffy <[email protected]> Co-authored-by: Andrew Duffy <[email protected]>
1 parent 5c4db5a commit 9f5714b

4 files changed

Lines changed: 149 additions & 122 deletions

File tree

‎.github/workflows/rust.yml‎

Lines changed: 8 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -370,22 +370,21 @@ jobs:
370370
with:
371371
save-if: false # set in linux-test
372372
shared-key: "amd-ci"
373-
# The storage integration tests start MinIO containers. Pulling the image
373+
# The storage integration tests start RustFS containers. Pulling the image
374374
# once up front keeps the pull off the critical path of the tests, which
375375
# would otherwise pull it several times concurrently and occasionally fail
376376
# with transient Docker transport errors. The tests retry the pull
377377
# themselves, so a failure here is only a warning.
378378
#
379-
# MINIO_IMAGE must match what the tests start: the tag comes from the
380-
# `minio` module of `testcontainers-modules`, the registry from
381-
# MINIO_IMAGE_NAME in `datafusion-cli/tests/cli_integration.rs`. The
382-
# `minio_image_matches_ci_prepull` test fails if either drifts.
383-
- name: Pre-pull MinIO image
379+
# RUSTFS_IMAGE must match RUSTFS_IMAGE_NAME and RUSTFS_IMAGE_TAG in
380+
# `datafusion-cli/tests/cli_integration.rs`. The
381+
# `rustfs_image_matches_ci_prepull` test fails if they differ.
382+
- name: Pre-pull RustFS image
384383
env:
385-
MINIO_IMAGE: quay.io/minio/minio:RELEASE.2025-02-28T09-55-16Z
384+
RUSTFS_IMAGE: docker.io/rustfs/rustfs:1.0.0
386385
run: |
387-
ci/scripts/retry timeout 120 docker pull "$MINIO_IMAGE" \
388-
|| echo "::warning::Could not pre-pull $MINIO_IMAGE, the tests will pull it themselves"
386+
ci/scripts/retry timeout 120 docker pull "$RUSTFS_IMAGE" \
387+
|| echo "::warning::Could not pre-pull $RUSTFS_IMAGE, the tests will pull it themselves"
389388
- name: Run tests (excluding doctests)
390389
env:
391390
RUST_BACKTRACE: 1

‎datafusion-cli/CONTRIBUTING.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ cargo test --all-targets
3535

3636
## Running Storage Integration Tests
3737

38-
By default, storage integration tests are not run. These tests use the `testcontainers` crate to start up a local MinIO server using Docker on port 9000.
38+
By default, storage integration tests are not run. These tests use the `testcontainers` crate to start up a local RustFS server using Docker with a dynamically assigned host port.
3939

4040
To run them you will need to set `TEST_STORAGE_INTEGRATION`:
4141

@@ -47,8 +47,8 @@ For some of the tests, [snapshots](https://datafusion.apache.org/contributor-gui
4747

4848
### AWS
4949

50-
S3 integration is tested against [Minio](https://github.com/minio/minio) with [TestContainers](https://github.com/testcontainers/testcontainers-rs)
51-
This requires Docker to be running on your machine and port 9000 to be free.
50+
S3 integration is tested against [RustFS](https://github.com/rustfs/rustfs) with [TestContainers](https://github.com/testcontainers/testcontainers-rs)
51+
This requires Docker to be running on your machine.
5252

5353
If you see an error mentioning "failed to load IMDS session token" such as
5454

‎datafusion-cli/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ ctor = { workspace = true }
7777
insta = { workspace = true }
7878
insta-cmd = "0.7.0"
7979
rstest = { workspace = true }
80-
testcontainers-modules = { workspace = true, features = ["minio"] }
80+
testcontainers-modules = { workspace = true }
8181
# Makes sure `test_display_pg_json` behaves in a consistent way regardless of
8282
# feature unification with dependencies
8383
serde_json = { workspace = true, features = ["preserve_order"] }

0 commit comments

Comments
 (0)