replace non-ASCII characters in header - #2887

@pkubanek - technically speaking, I think the FITS standard requires ASCII headers (see http://www.aanda.org/articles/aa/pdf/2010/16/aa15362-10.pdf, Section 3.2).But @embray should comment on this.

I know FITS standard would be more strict, and I will urge my friends at Astrometry.net to drop local day names from HISTORY keyword. But so far this is the first case when we hit problems with our FITS - other tools and libraries, including IRAF, cftsio and ds9, works with them fine. I think this change will not cause any other problem, and will align WCS package with what is de-facto accepted. I can provide test FITS frame to support my claims.

FWIW, currently PyFITS/astropy.io.fits will choke on non-ASCII characters in the header (last I checked). I've been meaning to fix that for a while--the header parsing already has code to handle invalid characters, but the loop that actually reads through FITS files and determines the beginnings and ends of headers is broken in that respect. So maybe that should be handled at the io.fits level rather than anywhere in the WCS module.

That said, Astrometry.net should definitely be given a talking to about this.

@pkubanek Also, you wrote

Such FITS headers open fine in Python and are probably valid

I'm surprised they open, given my comments that non-ASCII headers don't generally work. Maybe I fixed that after all and forgot that I did? :) In any case they're definitely not valid regardless of whether they happen to work. Could you send me a link to download a sample file that exhibits this problem?

Proof file: http://lascaux.asu.cas.cz/~petr/broken_wcs.fits

Code:

import astropy.wcs
import astropy.io.fits

f = astropy.io.fits.open('broken_wcs.fits')

print f[0].header['HISTORY']

w = astropy.wcs.WCS('broken_wcs.fits')
ra,dec = w.all_pix2world(100,200,0)
print ra,dec

Ah yes, now I remember--this does read on Python 2 but not on Python 3 due to currently somewhat unsophisticated handling of encodings. That should be fixed, but in the fits module, not the wcs module.

Although, on the other hand, as PyFITS does try to display non-ASCII characters (so that they can be corrected, if nothing else), maybe this patch is still a good idea just in case, if the wcslib code will blow up otherwise.

Maybe we do what this patch does, but additionally display a warning if the header contains non-ascii characters?

I am digging through old FITS PRs. Is this still necessary?

I think it's still relevant but the issue needs to be fixed on the fits.Header end of things.

I have done horrible work arounds for some solar data where there are unicode chars in the comments of a header (I know I know), so I would like a nicer way of dealing with this situation.

It's probably a relatively easy fix. If nothing else the header parser needs to catch non-ASCII bytes and either replace them with a replacement character (e.g. '?' and/or raise an error--perhaps depending on a setting).

Opened #6074 to keep track of this issue, and closing as the fix has to be done in io.fits.