Repository navigation
astropy.io.fits cannot open any FITS-IDI file #9709
Description
When I try to open any FITS-IDI file using astropy 3.2 or newer, the following error occurs:
In [1]: from astropy.io import fits
In [2]: hdu_list = fits.open('test.fits')
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-2-29c6c8a8af01> in <module>()
----> 1 hdu_list = fits.open('test.fits')
/usr/lib64/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py in fitsopen(name, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs)
149
150 return HDUList.fromfile(name, mode, memmap, save_backup, cache,
--> 151 lazy_load_hdus, **kwargs)
152
153
/usr/lib64/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py in fromfile(cls, fileobj, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs)
388 return cls._readfrom(fileobj=fileobj, mode=mode, memmap=memmap,
389 save_backup=save_backup, cache=cache,
--> 390 lazy_load_hdus=lazy_load_hdus, **kwargs)
391
392 @classmethod
/usr/lib64/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py in _readfrom(cls, fileobj, data, mode, memmap, save_backup, cache, lazy_load_hdus, **kwargs)
1062
1063 # Make sure at least the PRIMARY HDU can be read
-> 1064 read_one = hdulist._read_next_hdu()
1065
1066 # If we're trying to read only and no header units were found,
/usr/lib64/python3.6/site-packages/astropy/io/fits/hdu/hdulist.py in _read_next_hdu(self)
1137 fileobj.seek(offset, os.SEEK_SET)
1138
-> 1139 hdu = _BaseHDU.readfrom(fileobj, **kwargs)
1140 except EOFError:
1141 self._read_all = True
/usr/lib64/python3.6/site-packages/astropy/io/fits/hdu/base.py in readfrom(cls, fileobj, checksum, ignore_missing_end, **kwargs)
326 hdu = cls._readfrom_internal(fileobj, checksum=checksum,
327 ignore_missing_end=ignore_missing_end,
--> 328 **kwargs)
329
330 # If the checksum had to be checked the data may have already been read
/usr/lib64/python3.6/site-packages/astropy/io/fits/hdu/base.py in _readfrom_internal(cls, data, header, checksum, ignore_missing_end, **kwargs)
462 del new_kwargs[key]
463
--> 464 hdu = cls(data=DELAYED, header=header, **new_kwargs)
465
466 # One of these may be None, depending on whether the data came from a
/usr/lib64/python3.6/site-packages/astropy/io/fits/hdu/groups.py in __init__(self, data, header)
275 if len(self._axes) <= 0:
276 self._axes = [0]
--> 277 self._header['NAXIS'] = 1
278 self._header.set('NAXIS1', 0, after='NAXIS')
279
TypeError: '_BasicHeader' object does not support item assignment
At the same time, astropy 3.1.2 or older opens such files without any issues.
Test data: ftp://odin.asc.rssi.ru/outgoing/voitsik/tmp/test.fits
System Details
Linux-4.19.82-gentoo-x86_64-Intel-R-_Xeon-R-CPU_E5-1650_0@_3.20GHz-with-gentoo-2.6
Python 3.6.9 (default, Dec 2 2019, 12:56:17)
[GCC 9.2.0]
Numpy 1.17.4
Scipy 1.1.0
astropy 3.2.3
Activity
MSeifert04 commentedon Dec 2, 2019
Thank you for the report. I can verify that the exact same problem happens for astropy v4.0rc1 and master.
MSeifert04 commentedon Dec 2, 2019
cc @saimn Given the error message it could be that e74726b is involved in the regression.
pllim commentedon Dec 2, 2019
Whoever fixes this also need to add a regression test for this file format. FYI.
saimn commentedon Dec 3, 2019
Ok, so this is indeed related to the "fast Header" feature ( #8502), and to the fact that this missing some keyword (NAXIS1) that then needs to be set on the header.
@voitsik - Would it be possible to use your file in a test in the Astropy repository ? I actually extracted only the primary HDU to make the file smaller, would be great of I can use it directly instead of having to create a fake one.
Create Header instance when _BasicHeader fails
saimn commentedon Dec 3, 2019
@voitsik - I found a mean to generate a test file. See #9711 for a fix (though even with this some operations will crash because of the incorrect values for
NAXIS*).voitsik commentedon Dec 3, 2019
@saimn - Thank you, #9711 fixed the issue, at least for my read-only purposes.
I should note that this "invalid header" is part of FITS-IDI format. See section 3.1 in this document: http://fits.gsfc.nasa.gov/registry/fitsidi/AIPSMEM114.PDF.
Here is the "official" sample file:
http://fits.gsfc.nasa.gov/registry/fitsidi/BL146_1.fits.Create Header instance when _BasicHeader fails
Create Header instance when _BasicHeader fails
Create Header instance when _BasicHeader fails