Repository navigation
feat(python): Add DataFrame.write_iceberg - #15018
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #15018 +/- ##
==========================================
- Coverage 79.81% 79.80% -0.02%
==========================================
Files 1592 1592
Lines 229268 229496 +228
Branches 2623 2625 +2
==========================================
+ Hits 182987 183142 +155
- Misses 45676 45749 +73
Partials 605 605 ☔ View full report in Codecov by Sentry. |
|
waiting for pyiceberg 0.6.1 release, should be released soon, which will fix the schema issue described above |
9bff9b8 to
0801012
Compare
stinodego
left a comment
There was a problem hiding this comment.
Thanks for the PR. A few comments.
There was a problem hiding this comment.
Why does this method take a Table and not a Pathlike, like other write functions?
There was a problem hiding this comment.
Writing to an iceberg table requires creating the pyiceberg.table.Table object, which also requires an iceberg catalog.
I think it's best to delegate the creation of the catalog and table to be outside of this function and just pass in the table object.
The original implementation takes a str/Pathlike representing the "warehouse location" for the iceberg table. In Iceberg world, writing to a table usually requires an update to the catalog.
There was a problem hiding this comment.
This logic is so simple that this does not warrant its own method, in my opinion.
There was a problem hiding this comment.
The main goal is to implement the top-level write_iceberg function for the dataframe API. The actual implementation code can be moved somewhere else
There was a problem hiding this comment.
The point is that, if users already have an Iceberg table object, they can just write table.append(df.to_arrow). It's even shorter than df.write_iceberg(table, mode="append"). So there is not much added value to a write_iceberg method.
If there is some complex logic required to set up the iceberg table, or if to_arrow is not sufficient to handle all data types correctly, we can consider adding a write_iceberg to save all users some implementation hassle. But right now it doesn't seem warranted.
There was a problem hiding this comment.
That is true. Pyiceberg is well integrated with arrow. With an arrow dataframe and a PyIceberg table object, one can just invoke the pyiceberg write functions .append/.overwrite.
Given the above, is there still value in implementing a simple write_iceberg method?
Looking at the write_delta method, aside from the merge functionality, the function just pass data as arrow into the write_deltalake function.
There was a problem hiding this comment.
If there is some complex logic required to set up the iceberg table
There's a scenario where a user might want to write an Iceberg table to a location in blob store. In such case, the write_iceberg function can take care of creating an in-memory catalog and iceberg table object before writing.
That was the initial version of this PR
0801012
There was a problem hiding this comment.
Seems like a write_iceberg would be worth it just by virtue of having full support from polars / parity with delta.
DataFrame.write_iceberg
There was a problem hiding this comment.
This needs much more extensive testing. Try using the df fixture (it has a mix of many data types), or define your own dataframe with lots of different data types.
There was a problem hiding this comment.
Thanks! I'll take a look at this fixture
stinodego
left a comment
There was a problem hiding this comment.
See comments. Needs testing on more different data types before this can be merged.
|
This is great. Excited to try this out |
|
Excited to try this one out! |
1fef815 to
62a923a
Compare
|
hey @stinodego thanks for the previous review, could you take another look at this PR? I've changed the test to use the |
|
Hey @kevinjqliu , pyiceberg now supports partition and hiddent partitions The overwrite method supports a overwrite filter parameter, similar to delete_where on delta writer |
|
@mariotaddeucci thanks for the pointer. I want to get this simple implementation in ASAP and add the partitioned write later |
|
Hey, really looking forward to this feature. Any reason progress has stopped / merging is blocked? @stinodego |
|
Commenting for the same reason as @andreasbolstad |
|
This kind of goes to @stinodego's earlier point about taking a table instead of a path. I'd take that a step further and say if users still have to manually import pyiceberg and do other prep before they can use That said, you can still make it work by my own personal (Disclaimer: not at all binding on the polars dev team) threshold. I'd recommend having it accept either a Table, str, or tuple. If it's not a Table then it would load the catalog using defaults and then the str | tuple would be the table name. What about something more like this? def write_iceberg(
self,
table: pyiceberg.table.Table | str | tuple[str,str],
mode: Literal["append", "overwrite"] = "append",
) -> None:
from pyiceberg.catalog import load_catalog
if isinstance(table, str):
catalog = load_catalog()
table = catalog.load_table(table)
if mode == "append":
table.append(self.to_arrow())
elif mode == "overwrite":
table.overwrite(self.to_arrow())and then its usable without any lines of prep code and would be just as seamless as Another feature would be to create table if it doesn't already exist. That should probably have a parameter like from pyiceberg.types import (
BooleanType, IntegerType, LongType, FloatType, DoubleType, DecimalType,
StringType, BinaryType, DateType, TimeType, TimestampType, DurationType,
ListType, StructType, NullType
)
iceberg_type_mapping = {
pl.Boolean: BooleanType,
pl.Int8: IntegerType,
pl.Int16: IntegerType,
pl.Int32: IntegerType,
pl.Int64: LongType,
pl.UInt8: IntegerType,
pl.UInt16: IntegerType,
pl.UInt32: IntegerType,
pl.UInt64: LongType,
pl.Float32: FloatType,
pl.Float64: DoubleType,
pl.Decimal: DecimalType,
pl.Utf8: StringType,
pl.Binary: BinaryType,
pl.Date: DateType,
pl.Time: TimeType,
pl.Datetime: TimestampType,
pl.Duration: DurationType,
pl.List: ListType,
pl.Categorical: StringType,
pl.Struct: StructType,
pl.Object: None, # Not directly supported
pl.Null: NullType
} |
|
Thanks @kevinjqliu |
|
Thank you @ritchie46 |
Co-authored-by: ritchie <[email protected]>
Co-authored-by: ritchie <[email protected]>
Resolves #14610
This PR introduces the
write_icebergfunction to the Dataframe API.write_icebergrequires apyiceberg.table.Tableobject and amode(eitherappendoroverwrite).Note, partitioned write is currently not supported (blocked on iceberg-python #208)
Constructing the Iceberg catalog and Table object is left outside of this function. The test (
test_write_iceberg) provides an example of creating an in-memory catalog and a Iceberg Table object.Example
Files created