Skip to content

Commit 105819d

Browse files
osipovartema10y
andauthored
Backport RustFS for DataFusion CLI S3 tests (#166)
* fix datafusion-cli CI: switch from minio to rustfs (apache#25706) Closes apache#25705 This morning it seems like Minio pulled its images from quay.io, which has caused recent CI runs to start failing, e.g. https://github.com/apache/datafusion/actions/runs/36006680489/job/107657362588 RustFS is a pretty comparable (and more permissively licensed) alternative that provides a suitable S3 interface for integration tests. This was substantially authored by Codex, but reviewed by me and tested locally. Existing integration tests No Signed-off-by: Andrew Duffy <[email protected]> (cherry picked from commit e2ca7f3) * Repair DataFusion catalog tests after identifier option update (cherry picked from commit 0bc89d6) * Format datasource manifest for Taplo CI (cherry picked from commit 5ddc41bb19e776f2490acae6b95c12ebd3881eac) * Refresh parser option docs and clippy test (cherry picked from commit 52580648fa40ce24a28b466044872f59615d6a8c) * Compare ranged CSV diagnostics without backtraces (cherry picked from commit 69bb1e18cdb5358b3dd776de89c34ec8de8619c5) * Refresh generated array_reduce function docs --------- Signed-off-by: Andrew Duffy <[email protected]> Co-authored-by: Andrew Duffy <[email protected]>
1 parent 432ee5f commit 105819d

11 files changed

Lines changed: 315 additions & 138 deletions

File tree

‎.github/workflows/rust.yml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -371,6 +371,21 @@ jobs:
371371
with:
372372
save-if: false # set in linux-test
373373
shared-key: "amd-ci"
374+
# The storage integration tests start RustFS containers. Pulling the image
375+
# once up front keeps the pull off the critical path of the tests, which
376+
# would otherwise pull it several times concurrently and occasionally fail
377+
# with transient Docker transport errors. The tests retry the pull
378+
# themselves, so a failure here is only a warning.
379+
#
380+
# RUSTFS_IMAGE must match RUSTFS_IMAGE_NAME and RUSTFS_IMAGE_TAG in
381+
# `datafusion-cli/tests/cli_integration.rs`. The
382+
# `rustfs_image_matches_ci_prepull` test fails if they differ.
383+
- name: Pre-pull RustFS image
384+
env:
385+
RUSTFS_IMAGE: docker.io/rustfs/rustfs:1.0.0
386+
run: |
387+
ci/scripts/retry timeout 120 docker pull "$RUSTFS_IMAGE" \
388+
|| echo "::warning::Could not pre-pull $RUSTFS_IMAGE, the tests will pull it themselves"
374389
- name: Run tests (excluding doctests)
375390
env:
376391
RUST_BACKTRACE: 1

‎datafusion-cli/CONTRIBUTING.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ cargo test --all-targets
3535

3636
## Running Storage Integration Tests
3737

38-
By default, storage integration tests are not run. These tests use the `testcontainers` crate to start up a local MinIO server using Docker on port 9000.
38+
By default, storage integration tests are not run. These tests use the `testcontainers` crate to start up a local RustFS server using Docker with a dynamically assigned host port.
3939

4040
To run them you will need to set `TEST_STORAGE_INTEGRATION`:
4141

@@ -47,8 +47,8 @@ For some of the tests, [snapshots](https://datafusion.apache.org/contributor-gui
4747

4848
### AWS
4949

50-
S3 integration is tested against [Minio](https://github.com/minio/minio) with [TestContainers](https://github.com/testcontainers/testcontainers-rs)
51-
This requires Docker to be running on your machine and port 9000 to be free.
50+
S3 integration is tested against [RustFS](https://github.com/rustfs/rustfs) with [TestContainers](https://github.com/testcontainers/testcontainers-rs)
51+
This requires Docker to be running on your machine.
5252

5353
If you see an error mentioning "failed to load IMDS session token" such as
5454

‎datafusion-cli/Cargo.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -77,7 +77,7 @@ ctor = { workspace = true }
7777
insta = { workspace = true }
7878
insta-cmd = "0.7.0"
7979
rstest = { workspace = true }
80-
testcontainers-modules = { workspace = true, features = ["minio"] }
80+
testcontainers-modules = { workspace = true }
8181
# Makes sure `test_display_pg_json` behaves in a consistent way regardless of
8282
# feature unification with dependencies
8383
serde_json = { workspace = true, features = ["preserve_order"] }

0 commit comments

Comments
 (0)