Skip to content

feat(python): Add DataFrame.write_iceberg - #15018

Merged
ritchie46 merged 11 commits into
pola-rs:mainfrom
kevinjqliu:kevinjqliu/iceberg-write
Feb 28, 2025
Merged

ritchie46 merged 11 commits into
pola-rs:mainfrom
kevinjqliu:kevinjqliu/iceberg-write

Conversation

@kevinjqliu

@kevinjqliu kevinjqliu commented Mar 12, 2024 •

Copy link
Copy Markdown
Contributor

Resolves #14610

This PR introduces the write_iceberg function to the Dataframe API. write_iceberg requires a pyiceberg.table.Table object and a mode (either append or overwrite).

Note, partitioned write is currently not supported (blocked on iceberg-python #208)

Constructing the Iceberg catalog and Table object is left outside of this function. The test (test_write_iceberg) provides an example of creating an in-memory catalog and a Iceberg Table object.

Example

import polars as pl
df = pl.DataFrame(
    {
        "foo": [1, 2, 3, 4, 5],
        "bar": [6, 7, 8, 9, 10],
        "ham": ["a", "b", "c", "d", "e"],
    }
)

table_path = "/tmp/path/to/iceberg-table/"

# create catalog and table
from pyiceberg.catalog.sql import SqlCatalog
catalog = SqlCatalog(
    "default", uri="sqlite:///:memory:", warehouse=f"file://{table_path}"
)
catalog.create_namespace("foo")
table = catalog.create_table(
    "foo.bar",
    schema=df.to_arrow().schema,
)

df.write_iceberg(table, mode="overwrite")
print(table.location())

# read back the iceberg table 
pl.scan_iceberg(table).collect()

Files created

(.venv) ➜  py-polars git:(kevinjqliu/iceberg-write) tree /tmp/path/to/iceberg-table/                  
/tmp/path/to/iceberg-table/
└── default.db
    └── table
        ├── data
        │   └── 00000-0-3a2c5080-0e03-4dc5-a21a-73b8423ae181.parquet
        └── metadata
            ├── 00000-4ed15e3e-cafb-436c-b0ec-3bd08e6079ab.metadata.json
            ├── 00001-47503a3a-c2d9-4024-8605-28e92b6448c0.metadata.json
            ├── 3cd50fc2-83c6-4e5d-a68a-c34b56418631-m0.avro
            └── snap-3013071076850462756-0-3cd50fc2-83c6-4e5d-a68a-c34b56418631.avro

4 directories, 5 files

@codecov

codecov Bot commented Mar 12, 2024 •

Copy link
Copy Markdown

Codecov Report

Attention: Patch coverage is 72.72727% with 3 lines in your changes missing coverage. Please review.

Project coverage is 79.80%. Comparing base (69612d4) to head (afac997).
Report is 9 commits behind head on main.

Files with missing lines Patch % Lines
py-polars/polars/dataframe/frame.py 72.72% 2 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main   #15018      +/-   ##
==========================================
- Coverage   79.81%   79.80%   -0.02%     
==========================================
  Files        1592     1592              
  Lines      229268   229496     +228     
  Branches     2623     2625       +2     
==========================================
+ Hits       182987   183142     +155     
- Misses      45676    45749      +73     
  Partials      605      605              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@kevinliu-stripe

Copy link
Copy Markdown

waiting for pyiceberg 0.6.1 release, should be released soon, which will fix the schema issue described above

@mkarbo

mkarbo commented May 3, 2024

Copy link
Copy Markdown

@kevinliu-stripe fyi 0.6.1 has been released now :)

https://github.com/apache/iceberg-python/releases/tag/pyiceberg-0.6.1

@kevinjqliu
kevinjqliu force-pushed the kevinjqliu/iceberg-write branch from 9bff9b8 to 0801012 Compare July 2, 2024 17:51
@kevinjqliu
kevinjqliu marked this pull request as ready for review July 2, 2024 19:04
@kevinjqliu kevinjqliu changed the title [WIP] Write support for Iceberg Write support for Iceberg Jul 2, 2024

@stinodego stinodego left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. A few comments.

Comment thread py-polars/polars/dataframe/frame.py Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this method take a Table and not a Pathlike, like other write functions?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Writing to an iceberg table requires creating the pyiceberg.table.Table object, which also requires an iceberg catalog.
I think it's best to delegate the creation of the catalog and table to be outside of this function and just pass in the table object.

The original implementation takes a str/Pathlike representing the "warehouse location" for the iceberg table. In Iceberg world, writing to a table usually requires an update to the catalog.

Comment thread py-polars/polars/dataframe/frame.py Outdated
Comment on lines 3857 to 4080

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is so simple that this does not warrant its own method, in my opinion.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The main goal is to implement the top-level write_iceberg function for the dataframe API. The actual implementation code can be moved somewhere else

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The point is that, if users already have an Iceberg table object, they can just write table.append(df.to_arrow). It's even shorter than df.write_iceberg(table, mode="append"). So there is not much added value to a write_iceberg method.

If there is some complex logic required to set up the iceberg table, or if to_arrow is not sufficient to handle all data types correctly, we can consider adding a write_iceberg to save all users some implementation hassle. But right now it doesn't seem warranted.

@kevinjqliu kevinjqliu Jul 4, 2024 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is true. Pyiceberg is well integrated with arrow. With an arrow dataframe and a PyIceberg table object, one can just invoke the pyiceberg write functions .append/.overwrite.

Given the above, is there still value in implementing a simple write_iceberg method?

Looking at the write_delta method, aside from the merge functionality, the function just pass data as arrow into the write_deltalake function.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is some complex logic required to set up the iceberg table

There's a scenario where a user might want to write an Iceberg table to a location in blob store. In such case, the write_iceberg function can take care of creating an in-memory catalog and iceberg table object before writing.

That was the initial version of this PR
0801012

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a write_iceberg would be worth it just by virtue of having full support from polars / parity with delta.

@stinodego stinodego changed the title Write support for Iceberg feat(python): Add DataFrame.write_iceberg Jul 4, 2024
@github-actions github-actions Bot added enhancement New feature or an improvement of an existing feature python Related to Python Polars labels Jul 4, 2024
Comment thread py-polars/tests/unit/io/test_iceberg.py Outdated
Comment on lines 174 to 180

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs much more extensive testing. Try using the df fixture (it has a mix of many data types), or define your own dataframe with lots of different data types.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! I'll take a look at this fixture

@stinodego stinodego left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See comments. Needs testing on more different data types before this can be merged.

@glesperance

Copy link
Copy Markdown

This is great. Excited to try this out

@mkarbo

mkarbo commented Aug 20, 2024

Copy link
Copy Markdown

Excited to try this one out!

@kevinjqliu
kevinjqliu force-pushed the kevinjqliu/iceberg-write branch from 1fef815 to 62a923a Compare September 12, 2024 20:24
@kevinjqliu

Copy link
Copy Markdown
Contributor Author

hey @stinodego thanks for the previous review, could you take another look at this PR?

I've changed the test to use the df test fixture.
Currently, pyiceberg does not support time64[ns] and returning the dtype as Enum or Categorical. Hopefully, we can add those incrementally.

@mariotaddeucci

Copy link
Copy Markdown

Hey @kevinjqliu , pyiceberg now supports partition and hiddent partitions

The overwrite method supports a overwrite filter parameter, similar to delete_where on delta writer

@kevinjqliu

Copy link
Copy Markdown
Contributor Author

@mariotaddeucci thanks for the pointer. I want to get this simple implementation in ASAP and add the partitioned write later

@andreasbolstad

Copy link
Copy Markdown

Hey, really looking forward to this feature. Any reason progress has stopped / merging is blocked? @stinodego
Not to rush anyone, just bumping this PR as I don't want it to go stale.

@ryanovas

Copy link
Copy Markdown

Commenting for the same reason as @andreasbolstad

@deanm0000

deanm0000 commented Jan 24, 2025 •

Copy link
Copy Markdown
Collaborator

This kind of goes to @stinodego's earlier point about taking a table instead of a path. I'd take that a step further and say if users still have to manually import pyiceberg and do other prep before they can use write_iceberg then they might as well keep using the native library for the last step too.

That said, you can still make it work by my own personal (Disclaimer: not at all binding on the polars dev team) threshold. I'd recommend having it accept either a Table, str, or tuple. If it's not a Table then it would load the catalog using defaults and then the str | tuple would be the table name.

What about something more like this?

def write_iceberg(
    self,
    table: pyiceberg.table.Table | str | tuple[str,str],
    mode: Literal["append", "overwrite"] = "append",
) -> None:
    from pyiceberg.catalog import load_catalog
    if isinstance(table, str):
        catalog = load_catalog()
        table = catalog.load_table(table)
    if mode == "append":
        table.append(self.to_arrow())
    elif mode == "overwrite":
        table.overwrite(self.to_arrow())

and then its usable without any lines of prep code and would be just as seamless as write_parquet. If loading the catalog is expensive (I don't actually have experience with iceberg so don't know) then maybe it could be stored in globals() for repeated use.

Another feature would be to create table if it doesn't already exist. That should probably have a parameter like create_on_not_exist set to False by default so people don't create tables based on typos or maybe just have it be a third mode. Looks like you'd have to map all the polars types to the iceberg types for the schema. Copilot came up with this if it helps

from pyiceberg.types import (
    BooleanType, IntegerType, LongType, FloatType, DoubleType, DecimalType,
    StringType, BinaryType, DateType, TimeType, TimestampType, DurationType,
    ListType, StructType, NullType
)

iceberg_type_mapping = {
    pl.Boolean: BooleanType,
    pl.Int8: IntegerType,
    pl.Int16: IntegerType,
    pl.Int32: IntegerType,
    pl.Int64: LongType,
    pl.UInt8: IntegerType,
    pl.UInt16: IntegerType,
    pl.UInt32: IntegerType,
    pl.UInt64: LongType,
    pl.Float32: FloatType,
    pl.Float64: DoubleType,
    pl.Decimal: DecimalType,
    pl.Utf8: StringType,
    pl.Binary: BinaryType,
    pl.Date: DateType,
    pl.Time: TimeType,
    pl.Datetime: TimestampType,
    pl.Duration: DurationType,
    pl.List: ListType,
    pl.Categorical: StringType,
    pl.Struct: StructType,
    pl.Object: None,  # Not directly supported
    pl.Null: NullType
}

@ritchie46

Copy link
Copy Markdown
Member

Thanks @kevinjqliu

@ritchie46
ritchie46 merged commit a1ee3b5 into pola-rs:main Feb 28, 2025
@kevinjqliu
kevinjqliu deleted the kevinjqliu/iceberg-write branch February 28, 2025 16:48
@kevinjqliu

Copy link
Copy Markdown
Contributor Author

Thank you @ritchie46
Just a heads up, pyiceberg is in the process of releasing 0.9.0 and partition writes is now supported too

anath2 pushed a commit to anath2/polars that referenced this pull request Mar 5, 2025
jsjasonseba pushed a commit to jsjasonseba/polars that referenced this pull request Mar 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or an improvement of an existing feature python Related to Python Polars

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Write support for Apache Iceberg

10 participants