Repository navigation
Support inserting data into BigQuery directly from a Polars DataFrame #15710
Description
Is your feature request related to a problem? Please describe.
The Polars DataFrame library has been gaining a lot of traction and many are writing new pipelines in Polars and/or moving from pandas to Polars. It would be great to add native support between the BigQuery client library and Polars.
Describe the solution you'd like
This request is to allow inserting data directly from a Polars DataFrame into a BigQuery table.
An additional bonus would be not requiring PyArrow to be installed.
I would be open to expanding client.load_table_from_dataframe to also accept Polars DataFrames, or new dedicated method(s) being created.
Describe alternatives you've considered
-
Convert to pandas at the end of the pipeline and use
client.load_table_from_dataframeto insert the data. Not ideal to require an additional dependency just to insert data. Furthermore, I don't believe that pandas supports complex types available in both BigQuery and Polars, such as structs and arrays. -
Write the DataFrame to a bytes stream as a parquet file and insert the data with
client.load_table_from_file. This intent of this code is a lot less obvious, and it would be much nicer to have more native support. Note that this is also the suggested approach in the Polars user guide (rightfully so IMO as it does not require any additional dependencies). -
Do not support Polars directly, but instead support inserting data from a PyArrow table. This is not currently feasible, but would be an alternative feature request. This is not preferable as the option above already allows inserting data without a PyArrow dependency. From looking at the docs (haven't checked the source), this potentially looks to have some overlap with what
client.load_table_from_dataframealready does.
Activity
tswast commentedon Aug 16, 2024
I have some thoughts on this: I think a separate package for BigQuery + Polars (similar to pandas-gbq) would be most appropriate here, especially given the desire to avoid pyarrow as a dependency.
henryharbeck commentedon Aug 31, 2024
Thanks for the response @tswast
It would be great if the BigQuery client accepted Polars DataFrames in addition to pandas DataFrames in
client.load_table_from_dataframe. It would currently be possible without a pyarrow dependency as outlined in point 2 in the issue description. I would be happy to provide an implementation if accepted. I do acknowledge that this would be an additional optional dependency.I can also appreciate a separate package would provide a consolidated reading and writing interface, which I am definitely in support of. Do you have in mind that this package would be developed / owned by Google? If yes, would be it be possible / in-scope to support reading from BigQuery into Polars without a pyarrow dependency? At the moment it seems that
polars.from_arrow(client.query(...).to_arrow())is (from my understanding) the most performant option.As a heads up, I think I will also request that Polars provide some support for the BigQuery client, and implement the approaches mentioned in their user guide.
EDIT: here is that request - pola-rs/polars#18547
tswast commentedon Sep 6, 2024
Yes, that's my thought, though if the community were to create one before I make it through all the red tape needed to make such a thing happen, I'd gladly contribute there, instead. ;-)
That's my thought with regards to a separate package. No pyarrow necessary if the focus is just polars. The test suite for google-cloud-bigquery is complicated enough as it is without adding a test environment where polars is installed but pyarrow is not.
tswast commentedon Sep 6, 2024
I know you asked for writes, but I figured I'd try the read path today, and I was able to get BigQuery table -> polars DataFrame without pyarrow. Checkout this gist: https://gist.github.com/tswast/99b017b20386e324f5c7d2bd49f21b5f#file-bigquery-to-polars-no-pyarrow-ipynb
Obviously, it's single threaded, missing a lot of boilerplate, and doesn't support query inputs, but as a proof of concept, I was happy to see it's possible.
tswast commentedon Dec 13, 2024
I've confirmed it is possible to write to BigQuery from polars without pyarrow in this gist: https://gist.github.com/tswast/4e2fb2cca1c1fecf8fb697e94102358f
I've mailed pola-rs/polars#20292 to update the polars docs to set this option.
henryharbeck commentedon Dec 18, 2024
Thank you for your Polars PR as well.
Is there any movement on creating a separate Polars / BQ package within Google?
tswast commentedon Jan 3, 2025
Since I see there is some appetite to include such functionality in polars itself (pola-rs/polars#17326) or possibly in the a polars plugin package now that I/O plugins seem to be supported (https://github.com/pola-rs/pyo3-polars/tree/main/example/io_plugin, pola-rs/pyo3-polars#94), my approach has been to use some 20% time to try and contribute first a read plugin and then a write plugin. I don't have anything working yet, but https://github.com/tswast/polars/blob/issue17326-scan-bigquery/py-polars/polars/io/bigquery.py is the start of the branch I have for that.
The other approach we could take is to integrate this functionality into BigQuery DataFrames (google/bigframes#735), which I would definitely like to support and would be a bit simpler to implement, since bigframes already depends on pyarrow.
henryharbeck commentedon Jan 5, 2025
Thank you @tswast, that is awesome! I very much appreciate your work on this.
In the Polars repo or a Polars I/O plugin sounds great. A scan function supporting column/predicate pushdown would be amazing.
BigFrames support also seems in reasonable demand, but as you can probably tell I am personally trying to avoid the pyarrow dependency, haha
CesarArroyo09 commentedon Jan 15, 2025
Hey @tswast, I am very interested in participating in this development. I have around 6 years working with Python and very noob on rust but eager to learn.
If you have specific tasks that need to be executed and you can point to me, I can dedicate ~3 hours a week to this.
tswast commentedon Jan 21, 2025
Hi @CesarArroyo09 , thanks for reaching out!
I think the most promising way to get this done would be to contribute to the polars project directly. I'm making some good progress on a
scan_bigqueryfunction (pola-rs/polars#17326 (comment)) a few hours a week as well.Edit: Mailed pola-rs/polars#22976 with a Python-only implementation.
I could definitely use some help. Besides a "scan" method that I'm working on, there's also
read_databaseandwrite_databasein the polars library that could use some BigQuery support.read_database: @alexander-beedie proposes allowingread_databaseagoogle.cloud.bigquery.Clientobject as the "connection" argument in Use BigQuery Dataframes as Read-Connector to BigQuery pola-rs/polars#17326 (comment) -- There is a sample for reading from BigQuery at https://docs.pola.rs/user-guide/io/bigquery/ which could be used as the basis for this. A few thoughts:query_and_wait()instead ofquery(), but this comes with the tradeoff that it might be harder to avoid pyarrow and usescan_bigquerywhen it's available.write_database: I have contributed an update to the documentation at https://docs.pola.rs/user-guide/io/bigquery/#write which shows how to create a BigQuery-compatible parquet file without the need for a pyarrow dependency. Similar toread_database, allowing agoogle.cloud.bigquery.Clientobject to write in this manner.I'm eager to learn Rust, too, but official Rust clients for Google Cloud are still very much in their infancy (https://github.com/googleapis/google-cloud-rust). I suspect it'll probably be another year or so before we can really take advantage of them. In the meantime, I think the Polars + BigQuery connectors can be written in pure Python, even if it won't be as efficient as a Rust integration.
tswast commentedon Nov 6, 2025
Good news, folks, I'm working with the Polars team to create an official connector at: https://github.com/pola-rs/polars-bigquery-client
I'm hoping to provide the following methods:
polars_bigquery.read_bigquery-- read from BigQuery table (or query) to Polars DataFramepolars_bigquery.scan_bigquery-- read from BigQuery table (or query) to Polars LazyFrame, with push-down support for row/column filters and possibly more in the future.polars_bigquery.write_bigquery-- write to a BigQuery table using the BigQuery Storage Write API Arrow support.So far, I've got a working prototype of
read_bigquery, which is several times faster compared to the status quo of our Python client by using Rust for the connection to BigQuery.parthea commentedon Jan 12, 2026
Transferring to google-cloud-python as the code for python-bigquery will be migrated there in 1-3 weeks.
parthea commentedon Jan 12, 2026
Transferring this issue back to python-bigquery.
chalmerlowe commentedon Mar 2, 2026
This issue was moved from the following repo as part of the monorepo migration: googleapis/python-bigquery
tswast commentedon Aug 12, 2026
Closing in favor of this tracking bug: pola-rs/polars-bigquery-client#18