refactor(iceberg): adopt Apache datafusion-iceberg - #742

## Summary

Revert #742 until Apache `datafusion-iceberg` uses an `iceberg-rust`
revision containing [FileScanTask serialization
support](apache/iceberg-rust#3091).

#742 aligned our Iceberg dependencies with upstream's older `665c64e`
revision, dropping that support. Distributed scans panic in
`FileScanTaskWorkUnit::serialized()` with `Serialization not implemented
for this field`, and `executes_with_estimated_scan_tasks` then hangs.
Reproduced on the merge commit.

This is an exact revert: restore the previous `4d83bc7` dependency pins
and local predicate conversion implementation, without changing
task-count or transport behavior.

Fixes #748.

## Validation

- `cargo test --locked -p datafusion-distributed-iceberg --features
integration` — all 84 unit/integration tests and 1 doctest pass,
including the previously hanging test.
- `cargo fmt --all -- --check`
- `git diff --check HEAD^ HEAD`

## What
Bump the datafusion-distributed ref so we stay current. It picks up
distributed dynamic filters
(datafusion-contrib/datafusion-distributed#634,
datafusion-contrib/datafusion-distributed#635),
reduced producer fan-out in NetworkShuffleExec
(datafusion-contrib/datafusion-distributed#717),
the AQE worker-assignment retry
(datafusion-contrib/datafusion-distributed#723),
fractional desired task counts
(datafusion-contrib/datafusion-distributed#714),
and the datafusion-iceberg refactor
(datafusion-contrib/datafusion-distributed#742).

The introduction of distributed dynamic filters introduces a large hash
join regression when used over our transport layer, so this PR disables
their use until we get out a fix.

## Tests
CI's green. Once benchmarks run we should be able to verify this _does
not_ introduce the aforementioned perf regression.

---------

Co-authored-by: paradedb-github-bot[bot] <282009505+paradedb-github-bot[bot]@users.noreply.github.com>

## Summary

Reland
[#742](#742)
as a narrow dependency/predicate-conversion refactor. Add Apache
`iceberg-datafusion` pinned to `20327db`, use its public
`read::convert_filters_to_predicate`, and remove DFD's 931-line copied
converter and tests. DFD's table providers, data source, work-unit feed,
codec, worker reader, statistics, and task assignment are unchanged.
This does not depend on the open Apache eager-planning PR.


[#749](#749)
reverted #742 because its Apache pin pulled iceberg-rust `665c64e`,
which lacked working `FileScanTask` serialization:
`executes_with_estimated_scan_tasks` panicked and hung. The pinned
Apache revision (`20327db`, current Apache `main` when tested) includes
#21 and uses iceberg-rust `5b82ab9`, which retains `FileScanTask`
serialization. Both DFD's direct `iceberg` and `iceberg-storage-opendal`
dependencies now use that same revision. The existing distributed
regression test, including worker task transport, passes. No DFD
protocol or work-unit encoding code changes; mixed-version rolling
upgrades were not tested.

The iceberg-rust pin changes types exposed through DFD's public Iceberg
APIs; `docs/upgrade/5.0.0.md` tells users with direct iceberg-rust
dependencies to align their revision or use DFD's `iceberg` re-export.

## Validation

- `cargo check -p datafusion-distributed-iceberg` — pass.
- `cargo test --locked -p datafusion-distributed-iceberg --features
integration --test desired_task_count
tests::executes_with_estimated_scan_tasks -- --exact` — pass (the #749
reproducer).
- `cargo test --locked -p datafusion-distributed-iceberg --features
integration` — pass, including predicate pushdown, codecs, statistics,
and distributed execution.
- `cargo test --locked -p datafusion-distributed --lib` — pass (319
passed, 1 ignored).
- `cargo clippy --locked -p datafusion-distributed-iceberg --features
integration --all-targets -- -D warnings` — pass.
- `cargo fmt --all -- --check` and `git diff --check` — pass.

## AI disclosure

Developed with AI assistance; changes were reviewed and validated
locally before opening this draft.