Repository navigation
Issue with error about missing END card #4351
Description
The following example raises an error:
In [1]: from astropy.io import fits
In [2]: fits.open('https://dataverse.harvard.edu/api/access/datafile/2439198')
Downloading https://dataverse.harvard.edu/api/access/datafile/2439198
|==================================================================| 511M/511M (100.00%) 14m38s
---------------------------------------------------------------------------
OSError Traceback (most recent call last)
<ipython-input-2-95ceb74571c6> in <module>()
----> 1 fits.open('https://dataverse.harvard.edu/api/access/datafile/2439198')
/Volumes/Raptor/miniconda3/envs/production/lib/python3.4/site-packages/astropy-1.1.dev14143-py3.4-macosx-10.5-x86_64.egg/astropy/io/fits/hdu/hdulist.py in fitsopen(name, mode, memmap, save_backup, cache, **kwargs)
136 raise ValueError('Empty filename: %s' % repr(name))
137
--> 138 return HDUList.fromfile(name, mode, memmap, save_backup, cache, **kwargs)
139
140
/Volumes/Raptor/miniconda3/envs/production/lib/python3.4/site-packages/astropy-1.1.dev14143-py3.4-macosx-10.5-x86_64.egg/astropy/io/fits/hdu/hdulist.py in fromfile(cls, fileobj, mode, memmap, save_backup, cache, **kwargs)
278
279 return cls._readfrom(fileobj=fileobj, mode=mode, memmap=memmap,
--> 280 save_backup=save_backup, cache=cache, **kwargs)
281
282 @classmethod
/Volumes/Raptor/miniconda3/envs/production/lib/python3.4/site-packages/astropy-1.1.dev14143-py3.4-macosx-10.5-x86_64.egg/astropy/io/fits/hdu/hdulist.py in _readfrom(cls, fileobj, data, mode, memmap, save_backup, cache, **kwargs)
836
837 try:
--> 838 hdu = _BaseHDU.readfrom(ffo, **kwargs)
839 except EOFError:
840 break
/Volumes/Raptor/miniconda3/envs/production/lib/python3.4/site-packages/astropy-1.1.dev14143-py3.4-macosx-10.5-x86_64.egg/astropy/io/fits/hdu/base.py in readfrom(cls, fileobj, checksum, ignore_missing_end, **kwargs)
420 hdu = cls._readfrom_internal(fileobj, checksum=checksum,
421 ignore_missing_end=ignore_missing_end,
--> 422 **kwargs)
423
424 # If the checksum had to be checked the data may have already been read
/Volumes/Raptor/miniconda3/envs/production/lib/python3.4/site-packages/astropy-1.1.dev14143-py3.4-macosx-10.5-x86_64.egg/astropy/io/fits/hdu/base.py in _readfrom_internal(cls, data, header, checksum, ignore_missing_end, **kwargs)
480 if header is None:
481 header_offset = data.tell()
--> 482 header = Header.fromfile(data, endcard=not ignore_missing_end)
483 hdu_fileobj = data
484 data_offset = data.tell() # *after* reading the header
/Volumes/Raptor/miniconda3/envs/production/lib/python3.4/site-packages/astropy-1.1.dev14143-py3.4-macosx-10.5-x86_64.egg/astropy/io/fits/header.py in fromfile(cls, fileobj, sep, endcard, padding)
458
459 return cls._from_blocks(block_iter, is_binary, sep, endcard,
--> 460 padding)[1]
461 finally:
462 if close_file:
/Volumes/Raptor/miniconda3/envs/production/lib/python3.4/site-packages/astropy-1.1.dev14143-py3.4-macosx-10.5-x86_64.egg/astropy/io/fits/header.py in _from_blocks(cls, block_iter, is_binary, sep, endcard, padding)
527 # TODO: Pass this error to validation framework as an ERROR,
528 # rather than raising an exception
--> 529 raise IOError('Header missing END card.')
530
531 header_str = ''.join(read_blocks)
OSError: Header missing END card.(the data is a gzipped 500MB cube)
However, at first glance the FITS file is valid and the header has an END card. What could be going on here? It looks like this fails while trying to read a second HDU (based on preliminary debugging on my side), but there is only one HDU. Could it be that there are more data bytes in the file than the header would suggest and this is causing issues?
cc @embray
Activity
embray commentedon Nov 25, 2015
Probably unexpected trailing bytes #3941
embray commentedon Nov 25, 2015
Yep, that's all it is.
I want to say this is a known issue (it's very old), but I can't find a specific issue to point to.
embray commentedon Nov 25, 2015
Try
fits.open('https://dataverse.harvard.edu/api/access/datafile/2439198', ignore_missing_end=True)astrofrog commentedon Nov 25, 2015
Yes, that works, thanks! And I then see a warning about the padding bytes being present.
embray commentedon Nov 25, 2015
I think, if nothing else, if this is encounted at the end of the file and "extra data" is nothing but zeros it should definitely recognize this as padding and not try to parse it as an extra header. However, if it contains anything but zeros it's harder to just automatically dismiss it as garbage...
astrofrog commentedon Nov 25, 2015
I think in the present case, it's just zeros (well, null bytes)
embray commentedon Nov 25, 2015
Yep
SaOgaz commentedon Jun 27, 2016
astropy/astropy/io/fits/header.py
Lines 554 to 562 in 4910fa0
I don't know if I'm linking this correctly..
@embray would this be the right method to edit to fix this bug?
SaOgaz commentedon Jun 27, 2016
And maybe work somehow with this line?
astropy/astropy/io/fits/header.py
Lines 31 to 32 in 4910fa0
goes and studies the
remoduleembray commentedon Jun 29, 2016
@SaraOgaz I don't think the problem is really with the regular expression for the END card, or finding the END card that's the problem.
The problem is more that the way HDUs are read is kind of dumb. When reading in a FITS file it goes into this loop (albeit significantly modified in #5065, though the end result is still the same) where as long as there are bytes left to be read in the file, it will try to read those bytes as an HDU. And the first thing that happens when trying to read an HDU is that it tries to read the header.
The dumb thing is that it just assumes that if there are some remaining bytes in the file, it had better be a valid FITS header. But if there's just trailing padding, or garbage, any attempt to process it as a header will fail.
I can't tell you off the top of my head (would have to think about it a bit more) exactly where this should be fixed. But I think a special case could be made for a failed attempt to read part of the file as a header if the end of the file is reached.
E.g. say all the valid HDUs have already been read and processed, but there's something like 20 bytes of invalid padding at the end. Trying to read that as a header will fail with some error or another (the most typical being that it's read in those 20 bytes and failed to find an END card). I think if it's reached the EOF, but failed to find an END card, and, to be extra sure, it does not even look like a valid FITS header (i.e. does not begin with
SIMPLE = T, etc.) then ignore any errors about invalid headers and at the most produce a warning that there's trailing garbage at the end of the file.There's a special case where this already happens, in fact, and there's a warning message there you can reuse:
astropy/astropy/io/fits/header.py
Line 524 in 4910fa0
pllim commentedon Apr 13, 2017
Running into the same problem but was using
FITSDiff. Any chance this can be a keyword forFITSDiffwhen user passes in filenames?