Repository navigation
Polars Support #735
Description
It would be great to offer Polars support, it is currently half as popular as Pandas, and generally work better for large datasets. Polars is bound to replace most data-scientist day to day operations within the next five years.
Thanks for developing bigframes, it very useful.
Reacted by rpanai-tc
Activity
TrevorBergeron commentedon May 30, 2024
What kind of polars support would you find useful? Would you want BigQuery DataFrames to have an polars-like DataFrame API (as an alternative to the current pandas-like one) or simply interop with polars objects more easily?
lmmx commentedon Jun 13, 2024
I would like automatic schema supply, this is currently the limiting step in automatically uploading Polars DataFrames:
write_ndjsonseems to be the only way I can upload list dtypes (Parquet seems to not be viable, see this issue), but NDJSON requires the schema to be passed. I'm really looking for something that will just let me put my Polars DataFrame in a BQ table without fiddling with schemas: there should be enough info already here to do that for me.tswast commentedon Jun 26, 2024
For going from BigQuery DataFrames to polars, I'm adding a
to_arrowmethod in #807 as well as an example for how to create a polars DataFrame from the results.tswast commentedon Jan 3, 2025
For uploading to BigQuery, I have updated the polars docs to indicate how to get BigQuery to correctly handle list types pola-rs/polars#20292
I think that
read_polarsandto_polarsmethods would be reasonable requests for bigframes. I have done some refactoring recently to our I/O that might make it a bit easier, but would probably require a little more refactoring to have pyarrow tables/recordbatches as the intermediate format instead of pandas dataframes. The other thing to be careful about is that polars would be an optional "extra" dependency insetup.pyto avoid a hard dependency on the polars package.Edit: Or at the very least, a
read_arrow(...)method to correspond to theto_arrow()I implemented in #807. There are fewer concerns with depending on pyarrow in bigframes because we already have that as a required dependency.lmmx commentedon Feb 13, 2025
Amazing! Should this issue be closed now?
tswast commentedon Jun 26, 2025
I just mailed #1855 with
bpd.read_arrow(pyarrow.Table)to round out the other side of this conversion.Technically I think this was possible before by going through the DataFrame constructor, but that ended up translating to pandas as an in-between layer. Now we can just go from polars -> Arrow -> BigFrames without pandas in the middle.