Repository navigation
Consider moving pyarrow's pandas compatibility and conversion code to the pandas project? #59780
Description
This issue is to discuss the idea of moving a significant part of the pandas conversion and compatibility code that currently lives in pyarrow to the pandas project itself. Of course we would keep all low-level conversions (e.g. everything that lives in pyarrow C++) at the array-level in pyarrow itself (i.e. what pandas would use), but I think that a large part of what lives in pyarrow/pandas_compat.py could live in pandas.
Some reasons to do this:
- It's a lot of pandas specific code that might "fit" better in pandas itself
- It would allow pandas to control the conversion more tightly
- Example: now with upcoming pandas 3.0 and the new string dtype, pandas could ensure to use that new dtype in any conversion, while now with older versions of pyarrow
to_pandas()will still give object dtype ([Python] Support pandas future default string dtype apache/arrow#43683)
- Example: now with upcoming pandas 3.0 and the new string dtype, pandas could ensure to use that new dtype in any conversion, while now with older versions of pyarrow
- The required low-level functionality in pyarrow should now also be stable enough to allow having this code live in pandas itself (which might not have been the case at the inception of pyarrow)
A potential downside is that it makes the dependency structure even more complex (pyarrow's to_pandas() relying on pandas relying on pyarrow), although pyarrow already has infrastructure set up to lazily import pandas today.
The idea is not that we would change any public pyarrow API that supports pandas (ingesting pandas in various pyarrow constructors, to_pandas() methods on objects, etc), but that at least for the DataFrame and Series level, pyarrow would under the hood rely on a method from pandas to do that conversion.
For example, I think that most of the handling of the "pandas metadata" (to guarantee a better pandas <-> arrow roundtrip) could live in pandas itself, or the code to convert column labels to strings and reconstruct an Index in the other direction, determining which columns should be converted as an extension dtype vs numpy dtype, etc
There are of course a lot of details to figure out, but wanted to already open the issue to get a general idea of what people think about this, and if we want to maintain this in pandas.
Equivalent issue on the pyarrow side: apache/arrow#44068
Activity
jorisvandenbossche commentedon Sep 11, 2024
Two additional notes:
rok commentedon Jun 26, 2026
I suppose we could start work on Pandas side by defining a private Pandas API that PyArrow would call, so something like
pandas.core.interchange.from_arrow._table_to_dataframe(...). Once that is in place we can proceed with the PyArrow side.Would it be ok for @aboderinsamuel and I to prepare a draft? @jorisvandenbossche
rok commentedon Jul 27, 2026
ping @jorisvandenbossche
jorisvandenbossche commentedon Aug 6, 2026
Yes, I think that would be a good first start to get the ball rolling. I think we can have a private function or method for both the conversion to/from pyarrow.
We can discuss the exact place where to put this in the PR. The
interchangemodule is mostly about the deprecated dataframe protocol, although that code will be removed in the future, so maybe we can indeed reuse that module for the arrow conversion.For the API to expose to pyarrow, it might also make sense to wire this through a (static) method on the DataFrame/Series, which would avoid compatibility issues (eg when pandas would rename
pandas.coretopandas._corein the future)jorisvandenbossche commentedon Aug 6, 2026
Thinking a bit more about this: while implementing things on our side, we also need to test that it behaves the same as the current conversion. So it could also make to start with building a test suite (based on the one that lives in pyarrow, but it would also be a good occasion to clean it up / start somewhat from scratch).
And for the actual implementation of the conversion, we also need to decide how to approach this: implement this bit-by-bit (like #66636) and directly have a polished implementation per bit, or first essentially copy over the full pandas_compat code from pyarrow and just make it work, and then only afterwards polish it bit-by-bit inside the pandas code base.
[Data] Keep pandas metadata on Arrow blocks parsable by pandas (#67576)