Repository navigation
PyIceberg-Core: Push down Parquet reading to Iceberg-Rust #1144
Description
Is your feature request related to a problem or challenge?
As a next step in integrating PyIceberg and Iceberg-Rust, it would be great to push down the Parquet reading (including all the schema evolution) to Iceberg-Rust. Today, in PyIceberg, we go over each of the record batches, which causes a lot of pressure on the GIL. This logic should all happen in the Parquet reader (schema evolution, projecting missing columns, renames, re-ordering, etc), but from PyArrow we don't have the flexibility to project on ID, so this is what we ended up with.
The most logical separation would be to pass the FileScanTask into Iceberg-Rust.
We can break it down into building blocks:
- Ability to leverage the Iceberg-Rust FileIO in PyIceberg to open up streams
- Ability to pass down a PyIceberg schema into Iceberg-Rust.
- Can we serialize it into JSON? But that seems to be costly. Ideally, we want to reuse objects and not have to copy them from one to the other.
- Pass down expressions.
From the callgraph:
Describe the solution you'd like
No response
Willingness to contribute
None
Reacted by Kevin Liu, Renjie Liu and Abdul Rauf
Reacted by Hendrik Makait, Xuanwo, Jonathan Chen, Renjie Liu, Christian and Kevin Liu
Reacted by Kevin Liu and aschreiber1
Activity
[-]PyIceberg core: Push down Parquet reading to Iceberg-Rust[/-][+]PyIceberg-Core: Push down Parquet reading to Iceberg-Rust[/+]ion-elgreco commentedon Apr 15, 2025
This can be reasonably straight forward, depending on how you want to handle predicates. SQL strings or expressions.
A lazy record batch iterator can be done with this: https://github.com/delta-io/delta-rs/blob/main/python%2Fsrc%2Freader.rs
Then you would construct the TableProvider, scan, project it and then return that lazy iterator.
kaushiksrini commentedon Oct 14, 2025
Hey @Fokko, interested in picking this up!
github-actions commentedon Apr 13, 2026
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
github-actions commentedon Oct 11, 2026
This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.
github-actions commentedon Oct 26, 2026
This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'