Repository navigation
DOC: Why use astropy Table/QTable over pandas DataFrame #17146
Description
From #15905 (comment)
@bsipocz why Table is "better" than pandas (the question comes up regularly)
@taldcroft We should have a FAQ, but for the record (and for your tutorial), Table provides these capabilities that are not in pandas DataFrame:
- Robust support for table and column metadata. This is needed for representing data in file formats like FITS and VOTable.
- Support for N-dimensional column data, for example a table of thumbnail images in a source catalog.
- Support for columns of astropy Time, Quantity, and SkyCoord / Frame objects.
- General support for external "mixin" columns, for instance dask arrays.
Reacted by Marten van Kerkwijk
Activity
mhvk commentedon Oct 9, 2024
If this becomes a kind of comparison page, it would also be good to highlight how to import pandas dataframes into tables, and possibly have a list of things one does on dataframes and their equivalent operations on tables. (If that's relevant -- I don't use pandas so am of little use here!).
pllim commentedon Oct 9, 2024
I do use pandas when I want some quick stats for reports but that need is not often.
bsipocz commentedon Oct 9, 2024
Oh, I know the list above, but the list above gets weaker if working in a notebook we need to switch to a third party tool as conveniences from Table are gutted out (again, with good reason, from the developer pov, I understand why the deprecation happened, but I try to look at it from the users pov now)
pllim commentedon Dec 2, 2024
This came up again at Astropy ALOHA 2024 workshop.
One new point brought up for
pandaswas SQL support .FYI.
taldcroft commentedon Dec 3, 2024
That FAQ still isn't done so I'll try to take a crack at it.
I'm not sure what more to say about this except that tutorials and documentation need to emphasize that the Astropy ecosystem built up since 2011 depends on the unique features of
TableandQTablethat are not available inDataFrame(pandas or polars).We can certainly add convenience wrappers in
Tablefor things like SQL support.mhvk commentedon Dec 3, 2024
I think you are right that the main advantage of
QTableis the flexibility in what it can store. Perhaps the FAQ could focus on how to interoperate? This of course immediately brings up an idea for a PR: could one make the serialization a public method? That way, the user could also serialize to/from pandas (if they keep track of the metadata, not sure if pandas allows storing any table metadata).taldcroft commentedon Dec 3, 2024
@mhvk - Yes, I am thinking about a section near the end of the Table front page devoted to the pandas question (with a link to it near the top). This will go into a little detail on each of the bullet points listed in this issue, and also highlight interoperability.'
Pandas has "experimental" attrs: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.attrs.html. It's been there as experimental for at least 4 years, so I'm not sure about the status, but I assume it is not getting a lot of attention. Series also has
attrs, but it seems support is uneven. E.g. if you slice a DataFrame then a Series in that frame withattrswill lose theattrs.But yes, finding ways to improve interoperability and leverage DataFrame methods would be a good thing. And of course pandas is no longer the only DataFrame in town, and maybe it won't be long before polars is the gold standard.
bsipocz commentedon Dec 3, 2024
this has definitely started :)
taldcroft commentedon Dec 5, 2024
I was just checking on pandas support for leap seconds in its datetime handling (for developing a response to this issue). Google's GenAI response was particularly satisfying, read to the end.
neutrinoceros commentedon Dec 6, 2024
We might want to adopt
narwhalsas an abstraction layer to get portability across dataframe implementations.Although I haven't studied this deeply (yet ?), I recognize there are two aspects to this idea:
Tablecompatible with the DataFrame Interchange Protocol (this is Adopt Python dataframe interchange protocol #13427)pandaswithnarwhalsto improve portability (this would probably imply that we'd makenarwhalsa hard dependency to astropy, though it's worth noting thatnarwhalsas zero hard dependency of its own)