Skip to content
This repository was archived by the owner on Dec 19, 2017. It is now read-only.
This repository was archived by the owner on Dec 19, 2017. It is now read-only.

Poor handling of illegal characters in Header #37

Closed

Description

@embray

Per an e-mail to the Astropy mailing list:

Hi,

I want to use the "ñ" character in a fit header. The verify in astropy is giving me an error.

WARNING: VerifyWarning:         Card 'OBSERVER' is not FITS standard (invalid value string: 'Maria Rosa Mu�oz ').  Fixed 'OBSERVER' card to meet the FITS standard. [astropy.io.fits.verify]
ERROR: UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 1946: ordinal not in range(128) [astropy.io.fits.header]
Traceback (most recent call last):
  File "./fixfitheader.py", line 30, in <module>
    hdulist.writeto(sys.argv[2]+'/'+filename, output_verify='fix')
  File "/usr/local/lib/python2.6/dist-packages/astropy/io/fits/hdu/hdulist.py", line 675, in writeto
    hdu._writeto(hdulist.__file)
  File "/usr/local/lib/python2.6/dist-packages/astropy/io/fits/hdu/base.py", line 497, in _writeto
    self._writeheader(fileobj)
  File "/usr/local/lib/python2.6/dist-packages/astropy/io/fits/hdu/base.py", line 428, in _writeheader
    self._header.tofile(fileobj)
  File "/usr/local/lib/python2.6/dist-packages/astropy/io/fits/header.py", line 625, in tofile
    fileobj.write(blocks.encode('ascii'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xf1 in position 1946: ordinal not in range(128)

Is there a solution for this issue?

Obviously PyFITS is correct to raise an exception here, but the specific exception given is not terribly useful. First of all, the VerificationWarning states: "Fixed 'OBSERVER' card to meet the FITS standard." which it clearly did not, because there's still a crash later one when trying to write out the header.

Need to decide what the correct behavior should be, but it should either raise an UnfixableError exception immediately upon validation, or it should have a default fix of removing (or possibly replacing) the offending character. There are a number of possible approaches for replacement--there do exist transliteration tables. But it's probably not worth the effort in an initial fix.

Activity

  1. embray commented on Feb 1, 2014

    @embray
    CollaboratorAuthor

    I was wondering how the user even got and ñ into the header in the first place, because normally the Card class prevents assignment of values that are not ASCII-encodable. But it turns out that on Python 2 if you pass in a str containing non-ASCII characters of questionable encoding it will allow that. So that is a secondary bug that should be fixed first.

    Once that is fixed we still need to hack in a way to force non-ASCII characters into a header so that we can ensure that that case is handled better in case it somehow happens by accident as it did here.

  2. embray commented on Feb 1, 2014

    @embray
    CollaboratorAuthor

    I've fixed the issue that motivated this, that Python 2 was allowing non-ASCII values to slip by.

    The bigger issue is still how to handle existing headers that contain non-ASCII characters. Fortunately this is generally only going to be an issue for files coming from the wild, and even there the issue has come up thankfully infrequently. Even when it does, this will generally result in a unicode decode error when trying to parse the file.

    This does need better handling at the validation level, but that will need to be part of the broader validation overhaul that needs to be performed so I am going to put that off for the time being.

  3. added a commit that references this issue on Feb 6, 2014
  4. modified the milestones: v3.4.0, v3.3.0, v3.3.1 on Jul 17, 2014
  5. modified the milestone: v3.3.1 on Jan 28, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions