support adbc APIs in DB-API package #2015

Activity

1. product-auto-label

   added

   api: bigqueryIssues related to the googleapis/python-bigquery API.Issues related to the googleapis/python-bigquery API.

   on Sep 6, 2024
2. blunderbuss-gcf

   assigned

   suzmue

   on Sep 6, 2024
3. tswast

   added

   type: feature request‘Nice-to-have’ improvement, new feature or different behavior or design.‘Nice-to-have’ improvement, new feature or different behavior or design.

   on Sep 6, 2024
4. henryharbeck commented on Dec 18, 2024

   

   henryharbeck

   on Dec 18, 2024

   More actions

   A (better?) alternative would be for BigQuery to provide a real C/C++ ADBC driver, which could then be automatically usable from the Python adbc_driver_manager package.

   I'm unsure if you're aware, but it looks like BigQuery does have ADBC drivers implemented in C# and Go. Python wheels were included for the first time in the recent release.

   There is still a fair bit to be desired (including a page in the docs), with a to do list outlined here.

   Also, according to the writing new drivers page:

   Currently, new drivers can be written in C#, C/C++, Go, and Java. A driver written in C/C++ or Go can be used from either of those languages, as well as C#, Python, R, and Ruby. (C# can experimentally export drivers to the same set of languges as well.)

   I haven't installed or tested anything - just observations. Just figured I would give a heads up on it.

   Cheers

   🎉React with 🎉1Reacted by Tim Sweña (Swast)

   👀React with 👀1Reacted by Tim Sweña (Swast)
5. tswast commented on Jan 3, 2025

   

   tswast

   on Jan 3, 2025

   ContributorAuthor

   More actions

   I was not aware. That's awesome news.
6. henryharbeck commented on Jan 5, 2025

   

   henryharbeck

   on Jan 5, 2025

   Last edited by henryharbeck

   More actions

   Here is the PyPi package - https://pypi.org/project/adbc-driver-bigquery/

   After some tinkering with the low-level API I've also figured out how to read data directly into Polars without PyArrow or the BigQuery client libraries.
   This is thanks to the Arrow PyCapsule Interface, implemented by Polars and the ADBC driver manger (specifically on the ArrowArrayStreamHandle returned by stmt.execute_query()

   import adbc_driver_bigquery
   import adbc_driver_manager
   import polars as pl
   
   PROJECT = "my-project"
   db_kwargs = {adbc_driver_bigquery.DatabaseOptions.PROJECT_ID.value: PROJECT}
   query = f"SELECT * FROM `{PROJECT}.testing.test` LIMIT 10"
   
   with (
       adbc_driver_bigquery.connect(db_kwargs) as db,
       adbc_driver_manager.AdbcConnection(db) as conn,
       adbc_driver_manager.AdbcStatement(conn) as stmt,
   ):
       stmt.set_sql_query(query)
       stream_handle, rows = stmt.execute_query()
   
       df = pl.DataFrame(stream_handle)
       print(df)
       print(f"{rows = }")
   
   # shape: (1, 1)
   # ┌─────┐
   # │ foo │
   # │ --- │
   # │ i64 │
   # ╞═════╡
   # │ 1   │
   # └─────┘
   # rows = 1

   $ pip list
   Package              Version
   -------------------- -------
   adbc-driver-bigquery 1.3.0
   adbc-driver-manager  1.3.0
   importlib_resources  6.5.2
   pip                  24.3.1
   polars               1.19.0
   setuptools           65.5.0
   typing_extensions    4.12.2
   

   The DBAPI (which is where adbc_ingest also is) currently requires PyArrow. I think I'll ask the ADBC people if this can be relaxed/removed at all. adbc_ingest also implements the Arrow PyCapsule Interface, so it could also be possible to write to BQ without PyArrow (with ADBC)
   EDIT: Here is the feature request apache/arrow-adbc#2413
   EDIT EDIT: It has been completed apache/arrow-adbc#2609 and released in version 1.6.0

   Perhaps the BigQuery client / BigQuery storage client could also implement Arrow PyCapsule Interface 😉, there is a growing list
   Could this be possible for methods that already return results in the Arrow memory format? (like google.cloud.bigquery.table.RowIterator.to_arrow)
7. tswast commented on Jan 21, 2025

   

   tswast

   on Jan 21, 2025

   ContributorAuthor

   More actions

   Closing since apache/arrow-adbc#168 has already completed a adbc-driver-bigquery. Thanks @henryharbeck for the pointer and code sample!

   Opening googleapis/google-cloud-python#15699 to track the Arrow PyCapsule item.

   👍React with 👍1Reacted by Henry Harbeck
8. tswast

   closed this as completedon Jan 21, 2025