Repository navigation
Slicing FITS_rec objects produces unexpected dtypes (VLA columns) #16990
Description
Original conversation from #5490 (comment) onwards.
@peppedilillo , can you please give us a minimally reproducible code snippet that you encountered? What numpy version? Python version, OS, distro, etc?
from astropy.io import fits
if __name__ == "__main__":
hdul = fits.open("mre.fits")
fields = ["CHANNEL", "PHA", "CID", "SDDID"]
for field in fields:
print(f"slice, then access {field} field: ", hdul[2].data[:1][field])
for field in fields:
print(f"access field, then access {field} slice: ", hdul[2].data[field][:1])
hdul.close()This outputs:
slice, then access CHANNEL field: [array([ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 2,
0, 0, 3, 228, 0, 0, 0, 0, 0, 76, 106, 162, 65,
140, 52, 98, 12, 1, 165, 55, 0, 64, 215, 77, 89, 166,
110, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 2,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 2, 0, 0, 0, 0, 0, 12, 61, 246, 0, 0,
0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 36, 185,
226, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0,
0, 48, 247, 216, 1, 1, 68, 37, 228, 119, 0, 0, 0,
203], dtype=uint8) ]
slice, then access PHA field: [array([], dtype=int16)]
slice, then access CID field: [array([], dtype=uint8)]
slice, then access SDDID field: [array([], dtype=uint8)]
access CHANNEL field, then slice: [array([13, 19], dtype=uint8)]
access PHA field, then slice: [array([21786, 22649], dtype=int16)]
access CID field, then slice: [array([7, 7], dtype=uint8)]
access SDDID field, then slice: [array([1, 0], dtype=uint8)]The first four prints are garbage. The last four are correct, and take a long time in each statement. You can download the input file and the script here. Note that the problem occurs also if fields is set to [PHA], which is signed. Some of these columns are variable length.
This occurred to me because I am working on a FITS viewer application for the terminal. The application show FITS tables in short "pages". Each page consists of a slice of FITS_rec. I could not get the right output for certain input files because of this problem. I'm trying to work on slices because, if some scaling or other slow astropy operation are carried out over the whole table, that would result in annoying delays on the readout.
Just tested on Debian 12 bookworm, but observed the same behaviour also on Windows 11 and MacOS Ventura.
Yes i'm on numpy 2 and Python 3.12.5. The whole environment is the following, I'm on anaconda presently:
# packages in environment at /home/deppep/miniforge3/envs/misfits:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
astropy 6.1.3 pypi_0 pypi
astropy-iers-data 0.2024.9.2.0.33.23 pypi_0 pypi
bzip2 1.0.8 h4bc722e_7 conda-forge
ca-certificates 2024.8.30 hbcca054_0 conda-forge
click 8.1.7 pypi_0 pypi
ld_impl_linux-64 2.40 hf3520f5_7 conda-forge
libexpat 2.6.2 h59595ed_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc 14.1.0 h77fa898_1 conda-forge
libgcc-ng 14.1.0 h69a702a_1 conda-forge
libgomp 14.1.0 h77fa898_1 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libsqlite 3.46.0 hde9e2c9_0 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.3.1 h4ab18f5_1 conda-forge
linkify-it-py 2.0.3 pypi_0 pypi
markdown-it-py 3.0.0 pypi_0 pypi
mdit-py-plugins 0.4.1 pypi_0 pypi
mdurl 0.1.2 pypi_0 pypi
misfits 0.0.1 pypi_0 pypi
ncurses 6.5 he02047a_1 conda-forge
numpy 2.1.0 pypi_0 pypi
openssl 3.3.1 hb9d3cd8_3 conda-forge
packaging 24.1 pypi_0 pypi
pandas 2.2.2 pypi_0 pypi
pip 24.2 pyh8b19718_1 conda-forge
platformdirs 4.2.2 pypi_0 pypi
pyerfa 2.0.1.4 pypi_0 pypi
pygments 2.18.0 pypi_0 pypi
python 3.12.5 h2ad013b_0_cpython conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.2 pypi_0 pypi
readline 8.2 h8228510_1 conda-forge
rich 13.8.0 pypi_0 pypi
setuptools 72.2.0 pyhd8ed1ab_0 conda-forge
six 1.16.0 pypi_0 pypi
textual 0.79.1 pypi_0 pypi
tk 8.6.13 noxft_h4845f30_101 conda-forge
typing-extensions 4.12.2 pypi_0 pypi
@saimn : @peppedilillo - your issue is a different one, it has to do with VLA columns. Interestingly if you first access the full columns and then slice it, it works.
Activity
peppedilillo commentedon Sep 10, 2024
Thank you for having opened this issue! Is anyone aware of any workaround for getting correct values over a slice without accessing the whole column? This would reduce the processing time for my usecase.
saimn commentedon Sep 10, 2024
Can be reproduced with out test file: