feat(python/adbc_driver_manager): enable DB-API without PyArrow #2609

lidavidm · 2025-03-12T07:34:18Z

Enable limited use of the DB-API interface without PyArrow for people using other Arrow-based libraries, like polars (which is used here to test the new functionality).

This doesn't enable everything (e.g. get_objects), but it does enable (parameterized) queries and ingestion, which I would guess are the most important things.

Fixes #2413.

paleolimbot · 2025-03-12T16:03:21Z

python/adbc_driver_manager/adbc_driver_manager/dbapi.py

        elif hasattr(parameters, "__arrow_c_stream__"):
            self._stmt.bind_stream(parameters)
-        elif isinstance(parameters, pyarrow.RecordBatch):
+        # TODO: PyCapsule


nanoarrow.c_array_stream() has this path if you'd like to copy it/copy the tests:

https://github.com/apache/arrow-nanoarrow/blob/63fca8bafafbe574929e2fe662f82dcd60fde0d0/python/src/nanoarrow/c_array_stream.py#L24-L97

Thanks, I think I'm going to steal obj_is_capsule

Fixes apache#2413.

zeroshade · 2025-03-14T15:02:48Z

python/adbc_driver_manager/adbc_driver_manager/dbapi.py

 if typing.TYPE_CHECKING:
    import pandas
+    import polars
+    import pyarrow


don't we need another try/except here too?

This is under TYPE_CHECKING so it's only run under pyright, and pyright would complain anyways if you're missing these

zeroshade · 2025-03-14T15:04:10Z

python/adbc_driver_manager/adbc_driver_manager/dbapi.py

+        data: Union[
+            "pyarrow.RecordBatch", "pyarrow.Table", "pyarrow.RecordBatchReader"


should this include pycapsule or something too now?

zeroshade · 2025-03-14T15:05:03Z

python/adbc_driver_manager/adbc_driver_manager/dbapi.py

+        elif _has_pyarrow and isinstance(data, pyarrow.RecordBatch):
            array = _lib.ArrowArrayHandle()
            schema = _lib.ArrowSchemaHandle()
            data._export_to_c(array.address, schema.address)
            self._stmt.bind(array, schema)


should we use __arrow_c_array__ and just universally use pycapsules?

We could probably skip some of the type/instance checking by just checking for the pycapsule methods? Or do we still need to keep them to support older pyarrow versions?

I bumped up the minimum PyArrow earlier and it seems that always supports this now

I'll remove this

zeroshade · 2025-03-14T15:07:10Z

python/adbc_driver_manager/adbc_driver_manager/dbapi.py

-        return self._results._reader._reader
+        return self._results.reader._reader
+
+    def fetch_pycapsule(self) -> _lib.ArrowArrayStreamHandle:


wouldn't this return a pycapsule and not a _lib.ArrowArrayStreamHandle?

That type is the PyCapsule

You can't explicitly "name" a PyCapsule (without typing_extensions or 3.13 I guess)

Changed the docstring to clarify that it returns something implementing the PyCapsule interface (which we want anyways since it's slightly more flexible than directly returning the capsule object itself)

My personal vote on the name would be fetch_arrow(), although I am not sure anybody else is inventing another type of relevant PyCapsule related to databases anytime soon, nor do I think anybody would confuse this with a DLPack capsule. (FWIW GeoPandas went with to_arrow() for this concept).

Or maybe to_arrow()? (Most implementations of this don't actually do any fetching until you start consuming the result?)

(Feel free to ignore any of this 🙂 )

Hmm, fetch_arrow or to_arrow works.

Maybe we should have a little discussion about trying to standardize names like this? (Ditto perhaps for these DB-API compatible methods since as we can see DuckDB and turboarrow already diverged 😅)

I also sort of wonder if I shouldn't have the cursor object itself implement the PyCapsule protocol now that I think about it.

A GeoDataFrame should probably just implement the PyCapsule protocol too (but I think they were unsure about defaults for exactly what type to generate at the time they added to_arrow()). I haven't spent that much time with Python DBAPI to know what a cursor implies but implementing PyCapsule on it seems right?

A cursor does have methods like fetchone implying that it itself is the result set, but it also kind of bundles up both of what would normally be separate statement and result set interfaces in other APIs, so I could go either way. I think I'll leave it out for now and just have fetch_arrow

lidavidm · 2025-03-20T00:34:30Z

Any more comments here?

paleolimbot

Great as is but I left a few things that might be worth considering 🙂

paleolimbot · 2025-03-20T03:19:28Z

python/adbc_driver_manager/adbc_driver_manager/dbapi.py

+        elif _lib.is_pycapsule(parameters, b"arrow_array"):
+            self._stmt.bind(parameters)


I'm not sure this one will work (how would it know the schema?)

Ah, right. The __arrow_c_array__ returns both, but obviously once you have a PyCapsule it makes no sense.

paleolimbot · 2025-03-20T03:20:50Z

python/adbc_driver_manager/adbc_driver_manager/dbapi.py

+        elif _lib.is_pycapsule(data, b"arrow_array"):
+            self._stmt.bind(data)


Again, will this work?

paleolimbot · 2025-03-20T03:30:37Z

python/adbc_driver_manager/tests/test_dbapi_nopyarrow.py

+            polars.DataFrame({"theresult": [1]}).__arrow_c_stream__(),
+            id="PyCapsule_Stream",
+        ),
+        # TODO: It seems we can't get an array capsule from a series


I think a Series is more like a ChunkedArray and so I'm not sure that will ever happen?

paleolimbot · 2025-03-20T03:37:57Z

python/adbc_driver_manager/tests/test_dbapi_nopyarrow.py

+        cursor.execute("SELECT 1 + ? AS theresult", parameters=parameters)
+
+        cursor.execute("SELECT 1 AS theresult")


Is this intentional? (I don't see the result of the first one being tested?)

D'oh, copy-paste error

paleolimbot · 2025-03-20T03:42:18Z

python/adbc_driver_manager/tests/test_dbapi_nopyarrow.py

+
+        capsule = cursor.fetch_arrow()
+
+        # TODO: test new method with pyarrow installed too


Did you want to track this with an issue?

…he#2609) Enable limited use of the DB-API interface without PyArrow for people using other Arrow-based libraries, like polars (which is used here to test the new functionality). This doesn't enable everything (e.g. get_objects), but it does enable (parameterized) queries and ingestion, which I would guess are the most important things. Fixes apache#2413.

lidavidm force-pushed the gh-2413 branch 2 times, most recently from 6968e27 to f7973d3 Compare March 12, 2025 08:11

paleolimbot reviewed Mar 12, 2025

View reviewed changes

lidavidm force-pushed the gh-2413 branch 5 times, most recently from 79e42ef to 9bf2b8b Compare March 13, 2025 06:30

feat(python/adbc_driver_manager): enable DB-API without PyArrow

2bb8836

Fixes apache#2413.

lidavidm force-pushed the gh-2413 branch from 9bf2b8b to 2bb8836 Compare March 14, 2025 01:43

lidavidm marked this pull request as ready for review March 14, 2025 03:25

lidavidm requested a review from kou as a code owner March 14, 2025 03:25

lidavidm requested a review from zeroshade March 14, 2025 03:25

github-actions bot added this to the ADBC Libraries 18 milestone Mar 14, 2025

zeroshade reviewed Mar 14, 2025

View reviewed changes

lidavidm added 4 commits March 17, 2025 09:14

simplify

e2a9b4e

docstring

a2930e9

docs build

3ef81ad

fetch_pycapsule -> fetch_arrow

75da699

paleolimbot approved these changes Mar 20, 2025

View reviewed changes

lidavidm added 2 commits March 21, 2025 14:33

Add more tests/remove useless branch

c951334

Fix test, doh

4a41ab9

lidavidm merged commit 687da37 into apache:main Mar 21, 2025
61 of 62 checks passed

lidavidm deleted the gh-2413 branch March 21, 2025 08:35

This was referenced May 14, 2025

Drop PyArrow requirement for database IO functions with ADBC driver pola-rs/polars#22745

Open

support adbc APIs in DB-API package googleapis/python-bigquery#2015

Closed

henryharbeck mentioned this pull request May 14, 2025

Feature Request: Don't require PyArrow for parameterized queries that use a Python sequence #2827

Closed

		data: Union[
		"pyarrow.RecordBatch", "pyarrow.Table", "pyarrow.RecordBatchReader"

		elif _lib.is_pycapsule(parameters, b"arrow_array"):
		self._stmt.bind(parameters)

		elif _lib.is_pycapsule(data, b"arrow_array"):
		self._stmt.bind(data)

		cursor.execute("SELECT 1 + ? AS theresult", parameters=parameters)

		cursor.execute("SELECT 1 AS theresult")


		capsule = cursor.fetch_arrow()

		# TODO: test new method with pyarrow installed too

feat(python/adbc_driver_manager): enable DB-API without PyArrow #2609

feat(python/adbc_driver_manager): enable DB-API without PyArrow #2609

Uh oh!

Conversation

lidavidm commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zeroshade Mar 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

paleolimbot Mar 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lidavidm commented Mar 20, 2025

Uh oh!

paleolimbot left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

lidavidm commented Mar 12, 2025 •

edited

Loading

zeroshade Mar 14, 2025 •

edited

Loading

paleolimbot Mar 17, 2025 •

edited

Loading