Skip to content

Adopt Python dataframe interchange protocol #13427

Closed

Description

@nstarman

Description

A protocol for zero-copy data interchange between Python dataframe libraries.

https://data-apis.org/dataframe-protocol/latest/

Activity

  1. pllim commented on Jul 4, 2022

    @pllim
    Member

    It looks like they still trying to finalize things?

  2. nstarman commented on Jul 4, 2022

    @nstarman
    MemberAuthor

    Yeah, this is more of a long term thing to keep on our radar.

  3. mhvk commented on Jul 5, 2022

    @mhvk
    Contributor

    What are concrete examples of this? Would it be between a pandas column and a table one?

  4. nstarman commented on Jul 5, 2022

    @nstarman
    MemberAuthor

    Yeah, with this protocol, we could have

    from astropy import table
    
    def somefunc(df, ...):
        """`df` can be any dataframe supporting the protocol."""
        astropy_df = table.from_dataframe(df)  # converts without copying
        # From now on, use Table internally
  5. MridulS commented on Jan 19, 2024

    @MridulS
    Contributor

    I was poking around the Table code (due to #15905 (comment)) and I was wondering if there is an appetite to get this moving. This should (in theory) enable to use the plotting code already written of libraries like pandas with Table.

    I think it is still moving slowly https://github.com/data-apis/dataframe-api but maybe we can poke around?

  6. ManonMarchand commented on Jul 31, 2024

    @ManonMarchand
    Member
  7. nstarman commented on Jul 31, 2024

    @nstarman
    MemberAuthor

    Thanks for pointing this out @ManonMarchand. Do you have any interest in contributing the necessary methods for astropy.table.Table to support the protocol?

  8. ManonMarchand commented on Aug 2, 2024

    @ManonMarchand
    Member

    I don't understand all the points that have to be implemented in buffer.py and from_dataframe.py. Do astropy Tables have chunks?

  9. nstarman commented on Aug 12, 2024

    @nstarman
    MemberAuthor

    @mhvk, perhaps you could lend some insight?
    But @ManonMarchand, I don't believe Tables support chunking.

  10. mhvk commented on Aug 12, 2024

    @mhvk
    Contributor

    Not sure I have much insight, but maybe helps to keep in mind that a QTable is really just a collection of array-likes that have the same length. If all those array likes support chunking, then the table holding them should too, but for a generic table that has some Quantity, Time, or SkyCoord that would not be the case. My guess is that we'd need Quantity 2.0 to make real progress here -- though really the first thing may be to set out some specific use cases...

  11. nstarman commented on Nov 25, 2024

    @nstarman
    MemberAuthor

    Bumping this. I was just looking at using Altair for some plotting, but it requires pandas dataframes or a table that implements the dataframe interchange protocol. Astropy could become compatible with Altair for nice interactive plots!

  12. taldcroft commented on Dec 7, 2024

    @taldcroft
    Member

    I took a quick look and this seems like a very good thing and something we should pursue. To a large extent the base Table API is close to numpy arrays (column) and structured array (table). Is there a relevant example for making those objects compliant with the DataFrame protocol? I.e. has someone done some or most of our work already?

  13. pllim commented on May 8, 2026

    @pllim
    Member

    Perhaps fixed in #18435

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions