Repository navigation
Different behavior of io.fits in Python 2.7 and 3.5 #5093
Description
I came across some FITS files created by a third party using their proprietary software. It is clear to me that files are non-standard. However, I have noticed that while I cannot even open most of the files in Python 3.5, they work just file in Python 2.7.
I put the offending files in the following directory https://stsci.box.com/s/y5op9sxhd2vgoyatfc0krap2arovmpuw (Link will expire on Oct. 1st, 2016)
Errors in Python 3.5.1 but not in Python 2.7.5 when opening files:
PIX_002.fits:
WARNING: VerifyWarning: Error validating header for HDU #0 (note: Astropy uses zero-based indexing).
'ascii' codec can't decode byte 0xaa in position 2081: ordinal not in range(128)
There may be extra bytes after the last HDU or the file is corrupted. [astropy.io.fits.hdu.hdulist]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../lib/python3.5/site-packages/astropy/io/fits/hdu/hdulist.py", line 139, in fitsopen
return HDUList.fromfile(name, mode, memmap, save_backup, cache, **kwargs)
File ".../lib/python3.5/site-packages/astropy/io/fits/hdu/hdulist.py", line 281, in fromfile
save_backup=save_backup, cache=cache, **kwargs)
File ".../lib/python3.5/site-packages/astropy/io/fits/hdu/hdulist.py", line 871, in _readfrom
raise IOError('Empty or corrupt FITS file')
OSError: Empty or corrupt FITS filePIX_006.fits
WARNING: VerifyWarning: Error validating header for HDU #0 (note: Astropy uses zero-based indexing).
'ascii' codec can't decode byte 0x82 in position 2081: ordinal not in range(128)
There may be extra bytes after the last HDU or the file is corrupted. [astropy.io.fits.hdu.hdulist]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../lib/python3.5/site-packages/astropy/io/fits/hdu/hdulist.py", line 139, in fitsopen
return HDUList.fromfile(name, mode, memmap, save_backup, cache, **kwargs)
File ".../lib/python3.5/site-packages/astropy/io/fits/hdu/hdulist.py", line 281, in fromfile
save_backup=save_backup, cache=cache, **kwargs)
File ".../lib/python3.5/site-packages/astropy/io/fits/hdu/hdulist.py", line 871, in _readfrom
raise IOError('Empty or corrupt FITS file')
OSError: Empty or corrupt FITS filePIX_008.fits
WARNING: VerifyWarning: Error validating header for HDU #0 (note: Astropy uses zero-based indexing).
'ascii' codec can't decode byte 0x80 in position 2082: ordinal not in range(128)
There may be extra bytes after the last HDU or the file is corrupted. [astropy.io.fits.hdu.hdulist]
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File ".../lib/python3.5/site-packages/astropy/io/fits/hdu/hdulist.py", line 139, in fitsopen
return HDUList.fromfile(name, mode, memmap, save_backup, cache, **kwargs)
File ".../lib/python3.5/site-packages/astropy/io/fits/hdu/hdulist.py", line 281, in fromfile
save_backup=save_backup, cache=cache, **kwargs)
File ".../lib/python3.5/site-packages/astropy/io/fits/hdu/hdulist.py", line 871, in _readfrom
raise IOError('Empty or corrupt FITS file')
OSError: Empty or corrupt FITS fileAll these files can be opened in Python 2.7.5 (and loaded in DS9) without any warning messages. The main concern in this "Issue" is not that files are corrupt (though it would be interesting to know what is the problem) but rather the different behavior of astropy.io.fits in different Python versions regarding file errors.
Activity
pllim commentedon Jun 14, 2016
The key here is:
I have seen similar error in #4757. Might need some decoding somwhere in
io.fits?mcara commentedon Jun 14, 2016
Additional info: Opening the files in Python 2.7.5 and then writing them out (with a different name) creates file that can be opened in Python 3.5.1.
astrofrog commentedon Jun 14, 2016
@mcara - not sure if this will help, but could you possibly diff the FITS files before and after to see if it's likely to be a header or data issue?
mcara commentedon Jun 15, 2016
@astrofrog Screenshot below shows the difference between the user supplied file,
PIX_002.fitson the left, and re-written file (usingfits.writeto()in Python 2.7) -pix002.fitson the right. The difference seems to indicate that user's image has garbage in the padding between the END card and the beginning of the image block while re-saved image (pix002.fits) has 0x20 filling the end of the block (I think this is how it should have been according to specifications). For unknown reasonsfits.open()works fine with garbage in the padding while the same function fails in Python 3.5.mwcraig commentedon Jun 15, 2016
Not saying I know which line of code needs to be changed here, but the underlying cause is clear: handling of strings/unicode/byte streams is different in python 3 than in 2, and in this fits file there are characters outside the range of ascii characters, so python 3 fails when it tries to read it using that codec (which is what it will default to if no other codec is supplied).
Won't realistically have time to dive deeper until next week, but something along the lines of what @pllim suggested will be the fix.
pllim commentedon Jun 15, 2016
But philosophically, should it even be allowed to succeed? If it has garbage, then failure should not be surprising.
mwcraig commentedon Jun 15, 2016
I think it should, though perhaps that could be controlled by a flag, or emit a warning.
dhomeier commentedon Aug 17, 2016
The breach of the standard here actually seems to the header not being a multiple of 2880 bytes long, but instead it is truncated at 2080 (the 80 character 'line' including the
ENDkeyword).I'd agree that the Python2 version should at the very least give a warning as well, in particular since it looks as if it is now simply reading the following 800 bytes into the
headerunit, thus truncating thedatafield by that amount (at least when manually padding theheaderblock, thedataimage looks notably different from the original file loaded under Python2.7)It looks like setting the
padding=Falsekeyword should change the wayheaderunits of nonstandard length are dealt with, but it does not appear to make any difference here...mcara commentedon Aug 19, 2016
@dhomeier I am not sure I agree with your conclusions. Specifically, I will refer to
PIX_002.fitsfile which is compared to the version re-saved in Python 2.7 in #5093 (comment). For this file, if I open both versions (original and re-saved one) in Python 2.7 and compare data I get that the data are identical.Screenshot in #5093 (comment) shows that the header block has correct size and that it is not truncated. I think it does not matter where the
ENDcard is as long as it is within a block. That is, theENDcard indicates the end of the header but not of the header block. I believe a fits reader should read 2880 bytes regardless of whether or not theENDcard was in the block. The problem withPIX_002.fitsis that the header contains non ASCII characters (in fact non-space) in the padding between the end card and byte 2880 and Python 3.5 version ofastropy.io.fitschokes on it because of this.dhomeier commentedon Aug 20, 2016
@mcara, yes, you are probably right. I had been looking into the issue of latin character treatment in Python 3, and since it was not immediately clear where the real data block begins, a truncated header unit seemed a possibility. But looking at your comparison picture the byte pattern looks obviously regular, so it's more likely as you say that the writer just used something else than blanks for the padding.
In that case, since there is an unambiguous and non-damaging way to deal with the nonstandard file, I agree that Astropy should handle it better under Python 3 as well.
saimn commentedon May 19, 2017
I think this issue is also fixed with #6084, as I can now print the headers of the 3 files with both Python 2 and 3. The bytes after the END card are ignored so it does not matter.