Repository navigation
More descriptive/accurate error when opening non-FITS files #3941
Description
I tried to open a FITS file that happened to be bzip2 compressed (e.g. frame-g-001000-1-0027.fits.bz2). This is the error I got:
Traceback (most recent call last):
File "./populate_frame_polygons.py", line 102, in <module>
points = get_polygon_points(filepath=filepath, hdu=0)
File "./populate_frame_polygons.py", line 44, in get_polygon_points
image_hdu = fits.open(filepath)[hdu]
File "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/astropy-1.0.2-py2.7-linux-x86_64.egg/astropy/io/fits/hdu/hdulist.py", line 129, in fitsopen
return HDUList.fromfile(name, mode, memmap, save_backup, cache, **kwargs)
File "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/astropy-1.0.2-py2.7-linux-x86_64.egg/astropy/io/fits/hdu/hdulist.py", line 271, in fromfile
save_backup=save_backup, cache=cache, **kwargs)
File "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/astropy-1.0.2-py2.7-linux-x86_64.egg/astropy/io/fits/hdu/hdulist.py", line 829, in _readfrom
hdu = _BaseHDU.readfrom(ffo, **kwargs)
File "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/astropy-1.0.2-py2.7-linux-x86_64.egg/astropy/io/fits/hdu/base.py", line 378, in readfrom
**kwargs)
File "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/astropy-1.0.2-py2.7-linux-x86_64.egg/astropy/io/fits/hdu/base.py", line 438, in _readfrom_internal
header = Header.fromfile(data, endcard=not ignore_missing_end)
File "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/astropy-1.0.2-py2.7-linux-x86_64.egg/astropy/io/fits/header.py", line 460, in fromfile
padding)[1]
File "/uufs/chpc.utah.edu/sys/pkg/python/2.7.3_rhel6/lib/python2.7/site-packages/astropy-1.0.2-py2.7-linux-x86_64.egg/astropy/io/fits/header.py", line 529, in _from_blocks
raise IOError('Header missing END card.')
IOError: Header missing END card.
The error incorrectly points out that the header is missing an END card; rather, the problem is that the file being opened was compressed. It should be easy to check for obvious problems like this.
Alternatively (preferably), it would be great if io.fits could just go ahead and open the file, e.g.:
import bz2
if filename.endswith(".bz2"):
decompressed_file = bz2.BZ2File(filename)
# continue with opening that file...
Activity
embray commentedon Jul 13, 2015
For the latter, #3789.
Pyfits makes a lot of effort to read totally invalid FITS files (but that might still be valid if, say, accidental padding is ignored and things like that. Usually it can just tell you it's not a valid FITS file, but some odd cases slip through the cracks and instead you get the less obvious "missing END card" when there was nothing resembling a FITS header in the first place.
That could definitely be improved by checking for that exception and then looking back to see if anything we did find makes any sense (and if not raise a better worded exception), but it might not be as easy as you think while keeping the flexibility.
[-]More descriptive/accurate error when opening FITS files, or, open bz2 files.[/-][+]More descriptive/accurate error when opening non-FITS files[/+]demitri commentedon Jul 14, 2015
I ran into similar issues with my FITS viewer. I asked Bill Pence for a suggestion on the easiest (i.e. minimal resource) way to check if a file is at least pretending to be a FITS file, and he said to check for the word "SIMPLE" at the beginning of the file. I do this now:
That string must be the first 30 bytes/characters of any valid FITS file. (Yes, the above is in C, but you can easily adopt.) I almost suggest doing this test as the very first thing before anything else.
As a lesser suggestion, I'd recommend not having
IOError: Header missing END card.effectively be thefinallyin the code - it is strongly suggesting that a header (and thus somewhat valid FITS file) was found. A "final" message should be more like "The file {0} could not be interpreted as a FITS file."embray commentedon Jul 14, 2015
I've seen cases where the SIMPLE = T is missing and yet the rest of the file is still readable, so if you think it's a reliable check for your files (and it darn well should be) then by all means. Unfortunately there are plenty of crap hand-written FITS files out there by software that just thought that was optional.
demitri commentedon Jul 14, 2015
I'm taking a much harder line than that. There's no point in having an actual standard if people don't adhere to it. Opinions of FITS aside, having standardized on a single format is a huge advantage in the astronomical community. The more forgiving the software is, the less people will fix their files. I would still suggest to fail in the absence of "SIMPLE", and when people complain, tell people to fix their @&#^. A very valid, correct, and appropriate response is, “that is not a FITS file.” For my program, I am currently making no effort to fix files. It’s beyond my scope. I don't know if io.fits does this now, but I'd go as far to print a stream of warnings if you're "fixing" FITS files behind the user's back. /2¢
embray commentedon Jul 14, 2015
I agree strongly. I'll let you have a look at the pile of issues I have reporting "I have this FITS file that doesn't conform to the standard in this one trivial way that the software couldn't read, but if you look at it by hand it's 'obvious' what the intent was. Can you at least let me read the file so I can fix the error?" :)
demitri commentedon Jul 14, 2015
Certainly, one should provide tools to repair files, and Astropy is the right place to do it. But for every “I just want to fix it” person, there are forty, “oh, it works, I don’t have time to fix this (and it worked anyway)”. So you’re in a more difficult position. Perhaps a new flag, e.g.
that is required to force open bad files? It's a constant reminder, and hopefully someone smart won't allow such a flag to be used in production code. In addition, a warning should still be output. Yes, provide means for people to fix files, but a stick is still necessary to get people to do that right thing.
For anyone who says "it’s obvious what the intent was”, point them to the FITS spec and say that there’s nothing in there about “obvious”.
This is definitely one place where I deviate from my philosophy of “if the intent is unambiguous, the software should handle it.” I know for my program I’m not going to spend time patching people’s bad files instead of creating new features.
embray commentedon Jul 14, 2015
Yeah, I've had plans to rework FITS validation from the grounds up for a long time, but it's pretty low priority.
bsipocz commentedon Jul 14, 2015
I would think forcing the crowd to follow the standard is a valid approach, but the main source of the problem is usually doesn't lay on the small individual user but on the big players (just look at an ESO header from the archive, let alone their file naming convention horror).
embray commentedon Jul 14, 2015
Right--most of the time those reports are coming from Jane Astronomer who just wants to read the friggin' (15 year old) data they got off of .
demitri commentedon Jul 14, 2015
I know, and I definitely appreciate the problem. But if there is no "stick" to stop people from writing bad files it will continue, and then software writers will have to support all the bad stuff. The line has to be drawn somewhere, otherwise there will be no price to not adhering to the standard (which is where we are now). As for 15 year old files, I would argue the archives need to fix them, not push the work onto users or people who are not getting paid for it. Fix it at the source. If it takes Jane Astronomer to start complaining to the archives, I think that's a good start.
pllim commentedon Jul 14, 2015
@embray , or custom FITS writer written in Fortran. Anyways, @mcara has a fix that might be useful here for the affected user:
demitri commentedon Jul 14, 2015
I would be (unsurprisingly) opposed to that. It just enables (and tacitly approves) bad files. If it doesn’t have an
ENDcard, it’s NOT a FITS file.Unless the code above inserts the card, in which case never mind. Well, actually, I'd still oppose
ignore_missing_end; it’s something people would use in their code but not necessarily fix. Hence writing a warning would still be good anytime it's used.embray commentedon Jul 14, 2015
That's usually how it goes, and they do fix them. Usually fairly responsively. But in the meantime that mistake shouldn't impede science. For ASDF I think we are planning to take a much stricter stance on allowing invalid files to be read. But we're also taking a stricter stance on allowing invalid files to be written, and trying to find the right places to apply pressure to inhibit it (though no effort will be 100% effective).
demitri commentedon Jul 14, 2015
Agreed; hence the warnings and the clear "opt-in to open a bad file" suggestions.
parejkoj commentedon Oct 14, 2015
Has this been dealt with at the same time as #3881? I'm dealing with exactly the same type of file as Demitri (the SDSS Frame files), and the astropy message (I'm on 1.0.3) wasn't exactly helpful when diagnosing the problem.
embray commentedon Oct 15, 2015
Astropy v1.1 will be released probably in a couple weeks (I'm releasing a beta testing version later today).