Repository navigation
[branch-55] fix datafusion-cli CI: switch from minio to rustfs (#25706) - #25759
Merged
Merged
Conversation
## Which issue does this PR close? Closes apache#25705 ## Rationale for this change This morning it seems like Minio pulled its images from quay.io, which has caused recent CI runs to start failing, e.g. https://github.com/apache/datafusion/actions/runs/36006680489/job/107657362588 RustFS is a pretty comparable (and more permissively licensed) alternative that provides a suitable S3 interface for integration tests. This was substantially authored by Codex, but reviewed by me and tested locally. ## What is the testing strategy for this PR? Existing integration tests ## Are there any user-facing changes? No Signed-off-by: Andrew Duffy <[email protected]> (cherry picked from commit e2ca7f3)
github-actions
Bot
added
the
development-process
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## branch-55 #25759 +/- ##
=============================================
- Coverage 81.21% 81.21% -0.01%
=============================================
Files 1110 1110
Lines 388499 388499
Branches 388499 388499
=============================================
- Hits 315524 315507 -17
- Misses 54424 54440 +16
- Partials 18551 18552 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
6 of 21 tasks
alamb
approved these changes
Sep 25, 2026
Contributor
|
Thanks @comphead |
MassivePizza
added a commit
to massive-com/arrow-datafusion
that referenced
this pull request
Sep 30, 2026
Brings in the upstream 55.1.0 backports. Content matches rebasing the fork patches onto upstream/branch-55, with these fork commits superseded upstream: - #75 fix: preserve fetch across distribution reoptimization -> apache#24809 (backport apache#25821) - #79 fix(ci): pull the MinIO test image from quay.io -> apache#25092, apache#25216 (backport apache#25485), later replaced by the switch to rustfs in apache#25706 (backport apache#25759) Co-Authored-By: Claude Opus 5.5 <[email protected]>
bvolpato-dd
added a commit
to DataDog/datafusion
that referenced
this pull request
Nov 4, 2026
Docker Hub no longer serves the MinIO image used by the four CLI storage tests. Adapt the RustFS 1.0.0 fixture from apache#25759 and keep the existing S3 queries and snapshots.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
branch-55.Rationale for this change
This PR backports #25706 from @a10y to the
branch-55line.quay.io/minio/miniono longer allows anonymous pulls.branch-55pulls that image for thedatafusion-clistorage integration tests (it moved there from Docker Hub in #25485), so thecargo test datafusion-cli (amd64)job is expected to hit the same failure that #25705 reported onmain. The lastbranch-55CI run was on 2026-09-19, before the image became unavailable, so no run on this branch has failed yet.What changes are included in this PR?
A clean
git cherry-pick -xof e2ca7f3 with no conflicts. The patch is identical to the original (samegit patch-id).datafusion-cli/tests/cli_integration.rs: run the S3 tests against a RustFS container (docker.io/rustfs/rustfs:1.0.0) instead of MinIO..github/workflows/rust.yml: pre-pull the RustFS image instead of the MinIO one.datafusion-cli/Cargo.toml: drop theminiofeature oftestcontainers-modules.datafusion-cli/CONTRIBUTING.md: update the storage test docs.What is the testing strategy for this PR?
The existing
datafusion-clistorage integration tests cover this change, and CI runs them in thecargo test datafusion-cli (amd64)job. On this branch,datafusion-cli/tests/and its snapshots matchmainapart from one unrelated Spark test.testcontainersandtestcontainers-modulesalso resolve to the same versions as onmain.Locally,
cargo fmt --all -- --check,cargo clippy -p datafusion-cli --all-targets --all-features -- -D warnings,taplo format --checkand the prettier docs check pass. The Docker-based storage tests were not run locally. CI runs them.Are there any user-facing changes?
No. The change only affects tests, CI and contributor docs.