Repository navigation
fix: revert Apache datafusion-iceberg adoption (#742) - #749
Merged
alexanderbianchi merged 1 commit intoSep 24, 2026
Merged
Conversation
…n-contrib#742)" This reverts commit 2e6e1e3.
gabotechs
approved these changes
Sep 24, 2026
alexanderbianchi
added a commit
that referenced
this pull request
Oct 13, 2026
## Summary Reland [#742](#742) as a narrow dependency/predicate-conversion refactor. Add Apache `iceberg-datafusion` pinned to `20327db`, use its public `read::convert_filters_to_predicate`, and remove DFD's 931-line copied converter and tests. DFD's table providers, data source, work-unit feed, codec, worker reader, statistics, and task assignment are unchanged. This does not depend on the open Apache eager-planning PR. [#749](#749) reverted #742 because its Apache pin pulled iceberg-rust `665c64e`, which lacked working `FileScanTask` serialization: `executes_with_estimated_scan_tasks` panicked and hung. The pinned Apache revision (`20327db`, current Apache `main` when tested) includes #21 and uses iceberg-rust `5b82ab9`, which retains `FileScanTask` serialization. Both DFD's direct `iceberg` and `iceberg-storage-opendal` dependencies now use that same revision. The existing distributed regression test, including worker task transport, passes. No DFD protocol or work-unit encoding code changes; mixed-version rolling upgrades were not tested. The iceberg-rust pin changes types exposed through DFD's public Iceberg APIs; `docs/upgrade/5.0.0.md` tells users with direct iceberg-rust dependencies to align their revision or use DFD's `iceberg` re-export. ## Validation - `cargo check -p datafusion-distributed-iceberg` — pass. - `cargo test --locked -p datafusion-distributed-iceberg --features integration --test desired_task_count tests::executes_with_estimated_scan_tasks -- --exact` — pass (the #749 reproducer). - `cargo test --locked -p datafusion-distributed-iceberg --features integration` — pass, including predicate pushdown, codecs, statistics, and distributed execution. - `cargo test --locked -p datafusion-distributed --lib` — pass (319 passed, 1 ignored). - `cargo clippy --locked -p datafusion-distributed-iceberg --features integration --all-targets -- -D warnings` — pass. - `cargo fmt --all -- --check` and `git diff --check` — pass. ## AI disclosure Developed with AI assistance; changes were reviewed and validated locally before opening this draft.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Revert #742 until Apache
datafusion-iceberguses aniceberg-rustrevision containing FileScanTask serialization support.#742 aligned our Iceberg dependencies with upstream's older
665c64erevision, dropping that support. Distributed scans panic inFileScanTaskWorkUnit::serialized()withSerialization not implemented for this field, andexecutes_with_estimated_scan_tasksthen hangs. Reproduced on the merge commit.This is an exact revert: restore the previous
4d83bc7dependency pins and local predicate conversion implementation, without changing task-count or transport behavior.Fixes #748.
Validation
cargo test --locked -p datafusion-distributed-iceberg --features integration— all 84 unit/integration tests and 1 doctest pass, including the previously hanging test.cargo fmt --all -- --checkgit diff --check HEAD^ HEAD