Skip to content

fix: revert Apache datafusion-iceberg adoption (#742) - #749

Merged
alexanderbianchi merged 1 commit into
datafusion-contrib:mainfrom
alexanderbianchi:fix/revert-742
Sep 24, 2026
Merged

alexanderbianchi merged 1 commit into
datafusion-contrib:mainfrom
alexanderbianchi:fix/revert-742

Conversation

@alexanderbianchi

Copy link
Copy Markdown
Collaborator

Summary

Revert #742 until Apache datafusion-iceberg uses an iceberg-rust revision containing FileScanTask serialization support.

#742 aligned our Iceberg dependencies with upstream's older 665c64e revision, dropping that support. Distributed scans panic in FileScanTaskWorkUnit::serialized() with Serialization not implemented for this field, and executes_with_estimated_scan_tasks then hangs. Reproduced on the merge commit.

This is an exact revert: restore the previous 4d83bc7 dependency pins and local predicate conversion implementation, without changing task-count or transport behavior.

Fixes #748.

Validation

  • cargo test --locked -p datafusion-distributed-iceberg --features integration — all 84 unit/integration tests and 1 doctest pass, including the previously hanging test.
  • cargo fmt --all -- --check
  • git diff --check HEAD^ HEAD

@alexanderbianchi
alexanderbianchi merged commit 588a07f into datafusion-contrib:main Sep 24, 2026
35 checks passed
alexanderbianchi added a commit that referenced this pull request Oct 13, 2026
## Summary

Reland
[#742](#742)
as a narrow dependency/predicate-conversion refactor. Add Apache
`iceberg-datafusion` pinned to `20327db`, use its public
`read::convert_filters_to_predicate`, and remove DFD's 931-line copied
converter and tests. DFD's table providers, data source, work-unit feed,
codec, worker reader, statistics, and task assignment are unchanged.
This does not depend on the open Apache eager-planning PR.


[#749](#749)
reverted #742 because its Apache pin pulled iceberg-rust `665c64e`,
which lacked working `FileScanTask` serialization:
`executes_with_estimated_scan_tasks` panicked and hung. The pinned
Apache revision (`20327db`, current Apache `main` when tested) includes
#21 and uses iceberg-rust `5b82ab9`, which retains `FileScanTask`
serialization. Both DFD's direct `iceberg` and `iceberg-storage-opendal`
dependencies now use that same revision. The existing distributed
regression test, including worker task transport, passes. No DFD
protocol or work-unit encoding code changes; mixed-version rolling
upgrades were not tested.

The iceberg-rust pin changes types exposed through DFD's public Iceberg
APIs; `docs/upgrade/5.0.0.md` tells users with direct iceberg-rust
dependencies to align their revision or use DFD's `iceberg` re-export.

## Validation

- `cargo check -p datafusion-distributed-iceberg` — pass.
- `cargo test --locked -p datafusion-distributed-iceberg --features
integration --test desired_task_count
tests::executes_with_estimated_scan_tasks -- --exact` — pass (the #749
reproducer).
- `cargo test --locked -p datafusion-distributed-iceberg --features
integration` — pass, including predicate pushdown, codecs, statistics,
and distributed execution.
- `cargo test --locked -p datafusion-distributed --lib` — pass (319
passed, 1 ignored).
- `cargo clippy --locked -p datafusion-distributed-iceberg --features
integration --all-targets -- -D warnings` — pass.
- `cargo fmt --all -- --check` and `git diff --check` — pass.

## AI disclosure

Developed with AI assistance; changes were reviewed and validated
locally before opening this draft.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deadlock in Iceberg integration test

2 participants