Repository navigation
File Format API for PyIceberg #3100
Description
Feature Request / Improvement
Problem
The write path in pyiceberg/io/pyarrow.py is hardcoded to Parquet. The write.format.default table property exists but is never read. Adding a new format (ORC, Vortex, Lance) requires modifying the monolithic write_file() function. The read path already dispatches multiple formats; the write path should too.
Proposal
Introduce a File Format API aligned with Java Iceberg's File Format API (design doc).
New module pyiceberg/io/fileformat.py:
FileFormatWriter(ABC)FileFormatModel(ABC)FormatRegistryDataFileStatistics(it's inpyarrow.pycurrently but I think this might be good to consolidate for metrics)
Changes to pyiceberg/io/pyarrow.py:
ParquetFormatWriter/ParquetFormatModelusing thewrite_parquet()(insidewrite_file()write_file()refactored to readwrite.format.default, look up the format model, and dispatch.
TCK tests/io/test_file_format_tck.py:
- pytest-parameterized round-trip, statistics, type coverage, and null handling tests for every registered format.
Phased rollout:
- ABCs and registry first, then Parquet extraction with TCK tests, then
write_file()dispatch
Java ↔ Python Mapping
| Java | Python |
|---|---|
FormatModel<D, S> |
FileFormatModel (ABC, no type params) |
FileAppender<D> / ModelWriteBuilder |
FileFormatWriter (ABC) |
FormatModelRegistry |
FormatRegistry (keyed by FileFormat only) |
Metrics |
DataFileStatistics (existing) |
| TCK | test_file_format_tck.py |
Scope
This proposal covers the abstraction layer and the Parquet extraction only. No new format writers are included; ORC write support (#20) and any future formats (Avro, etc.) would be follow-ups once this lands.
References
- Java File Format API: apache/iceberg#12774
- Design doc: Google Doc
- Format impls: Parquet #15253, ORC #15255, Avro #15254
- TCK: apache/iceberg#15415
- Prior pyiceberg ORC work: #20, #790, #2236
Reacted by jj.lee
Metadata
Metadata
Assignees
Labels
No labels
Activity
rambleraptor commentedon Feb 25, 2026
This is going to be great, especially for new file formats!
I'm helping out with the TCK work over on Java and the shape of it might change a bit as we figure out exactly what we want the tests to look like. I don't think it would hurt for us to figure out our own testing story and then we can look into TCK when it's merged in.
nssalian commentedon Feb 26, 2026
Sounds good @rambleraptor. The TCK can come in when ready. The first few pieces are critical.
CC: @kevinjqliu @Fokko @geruh to see if the proposal makes sense
kevinjqliu commentedon Feb 26, 2026
Thanks for writing this up! This is super exciting. I think we can focus on both the read and write path for parquet. I'm interested to see what the API looks like.
We dont have to mirror the java apis, but perhaps we can reuse similar ideas
kevinjqliu commentedon Feb 26, 2026
btw @pvary, you started a revolution 🥳
nssalian commentedon Feb 26, 2026
Thanks @kevinjqliu . Could you assign this issue to me and I'll start pushing out the fix in phases.
mrutunjay-kinagi commentedon Mar 7, 2026
@kevinjqliu @nssalian This is exciting. Is there anything I can contribute to this?
Fokko commentedon Apr 16, 2026
Thanks @nssalian for bringing this up, and it is very exciting indeed 🚀
I would like to echo @kevinjqliu comment there. If we follow Java then we end up with many ABC's which are very common in Java, but are not considered very Pythonic.
nssalian commentedon Apr 16, 2026
Thanks @Fokko . I know you already looked at #3119 but feel free to comment if you think it could use a different direction or something needs to change. I'd like to set things right directionally before adding on more plumbing so I don't mind waiting to get the first foundational setup correct prior to proceeding.
Initial work for file format writer API (#3119)
Implement `ParquetFormatModel` and update `write_file` to use the for…
Convert `FileFormatModel` from `ABC` to `typing.Protocol` (#3692)