[EPIC] Bounded-memory execution operations for pyiceberg-core #2716

What's the feature you're trying to implement?

Expose bounded-memory DataFusion execution operations through pyiceberg-core's Python bindings, enabling PyIceberg to perform compute-heavy table operations (compaction, delete resolution, CoW rewrites) entirely in Rust with spill-to-disk guarantees.

Motivation

PyIceberg (iceberg-python#3554) needs to perform operations that are infeasible under bounded memory with PyArrow alone. pyiceberg-core already provides Rust-backed transforms and a read-only DataFusion TableProvider. This epic extends it with execution operations — each exposed as a single Python function that runs an entire DataFusion plan in Rust (GIL released, memory bounded by FairSpillPool, spill-to-disk via DiskManager).

Design principle: operation-level FFI. Python passes file paths + config, Rust does all I/O + compute internally, returns only metadata (DataFile JSON). Data never crosses the FFI boundary.

Part 1: Primitives (iceberg-rust core + iceberg-datafusion)

These are foundational pieces needed before the execution operations can be built. Some already have issues/PRs in progress by other contributors.

Transaction actions (others' work — we depend on these)

DataFusion integration primitives (our work — no blockers)

Part 2: Python execution module (bindings/python) — #2718

The pyiceberg_core.execution module — new submodule in bindings/python/src/execution.rs exposing operations to Python via PyO3.

Infrastructure (no blockers)

• Module structure + register_module in lib.rs
• Result types (CowRewriteResult, CompactionResult)
• Function signatures with todo!() bodies

Read-path operations (needs: bounded session helper only)

• execute_equality_resolution — anti-join of data × equality delete files
• execute_antijoin_paths — path anti-join for orphan file detection

Write-path operations (needs: bounded session + OverwriteAction/RewriteFiles)

• execute_cow_rewrite — streaming filter → write for CoW deletes/overwrites
• execute_compaction — sort + rewrite with target file size splitting

Future operations

• execute_upsert — hash join for matched rows
• execute_eq_to_positional — convert equality deletes to positional
• execute_position_delete_compaction — rewrite data files with pos deletes applied

Related

• iceberg-python#3554 — PyIceberg DataFusion integration epic (consumer of this work)
• [EPIC] Implement Missing Write Actions #2269 — [EPIC] Missing Write Actions
• Copy-on-Write and Merge-on-Read support #2186 — CoW and MoR support epic
• [Discussion] Reduce the need for iceberg-rust forks / how can we grow the community? #1797 — Community discussion on making iceberg-rust more usable

Willingness to contribute

I would be willing to contribute to this feature with guidance from the Iceberg Rust community