Repository navigation
index column not preserved on table write/read #16836
Description
Description
When writing and reading a table with an index column, I would expect that index to be saved, as it's an important part of efficient table lookups. We use the table index for fast cross-matching in LSST code with pandas, and we're trying to transition away from using DataFrame (pandas.DataFrame saved as parquet files) datasets to ArrowAstropy (astropy.table.Table saved as parquet), due to some of the known problems with pandas (e.g. the lack of a fixed schema).
It doesn't look like this is dependent on the file format, as I see the same behavior with parquet, ecsv, and FITS output.
Expected behavior
Round tripping a table with an index should preserve the index that was set.
How to Reproduce
import astropy.table
data=astropy.table.Table(data=[[1, 5.0], [2, 6.0]], names=["id", "thing"], dtype=[int, float])
data.add_index("id")
print(data.indices)
# prints the table index information
data.write("test.parquet")
print(astropy.table.Table.read("test.parquet").indices)
# prints an empty list
data.write("test.ecsv")
print(astropy.table.Table.read("test.ecsv").indices)
# prints an empty listVersions
import astropy
try:
astropy.system_info()
except AttributeError:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import astropy; print("astropy", astropy.__version__)
import numpy; print("Numpy", numpy.__version__)
import erfa; print("pyerfa", erfa.__version__)
try:
import scipy
print("Scipy", scipy.__version__)
except ImportError:
print("Scipy not installed")
try:
import matplotlib
print("Matplotlib", matplotlib.__version__)
except ImportError:
print("Matplotlib not installed")macOS-14.5-arm64-arm-64bit
Python 3.11.7 | packaged by conda-forge | (main, Dec 23 2023, 14:38:07) [Clang 16.0.6 ]
astropy 6.0.0
Numpy 1.24.4
pyerfa 2.0.1.1
Scipy 1.11.4
Matplotlib 3.8.2
Activity
pllim commentedon Aug 15, 2024
Kinda unrelated but is it just me or the string output of indices seems weird in v7.0.dev?
pllim commentedon Aug 15, 2024
I can confirm that v7.0.dev also have indices roundtripping issue, i.e., I get back empty list as reported when read it back in from file.
taldcroft commentedon Aug 16, 2024
This has been in my head for a long time as a lingering gap in Table serialization. It is a little tricky because the underlying index machinery is pretty flexible (e.g. the different engines). But I agree this is a high priority and something I would like to address.
One question is whether an immediate workaround would be of use for LSST? Is this intended for internal data tables or user-facing ones?
erykoff commentedon Aug 19, 2024
As it happens, we do not need this functionality in LSST right now. So it's not urgent from our perspective.
saimn commentedon Aug 27, 2024
There is an old issue (#6925) but this one has more information and an example so maybe we can close the old one?
neutrinoceros commentedon Sep 2, 2024
(it's not just you, but it's not astropy 7.0.dev either: it looks the same in 6.0.0)
pllim commentedon Sep 3, 2024
Re: #16836 (comment)
Should it (#16836 (comment)) be considered a bug or a feature I do not understand? I can open a new issue about it if it is a bug. Hope someone can advise. Thanks!
taldcroft commentedon Sep 3, 2024
@pllim - I believe the output referenced there for the str/repr of table
indicesis expected and looks normal to me. I did a PR at some point to make the indices repr more complete. It is still slightly awkward sinceindicesis a list of index objects.taldcroft commentedon Oct 9, 2025
@parejkoj - I have an implementation #18703 that I'm hopeful will get into 7.2. Feel free to comment!