Workaround to deal with invalid headers?

CASA 4.7 was just released and it appears to produce invalid FITS headers, at least sometimes:

ValueError: FITS header values must contain standard printable ASCII characters; "'CASA 4.7.0-REL (r38335)\n'" contains characters not representable in ASCII or non-printable characters.

There is already a bug report upstream to fix this, but is there a workaround for the files that are created by this rogue version? These files will fail whenever the header is accessed or written.

Note that fh.verify('fix') results in the same traceback.

(edit: tested on astropy 1.1.2)

It would be interesting to know which unprintable characters. Could you share one "corrupted" file?

@MSeifert04

@MSeifert04 it's a newline at the end of the string: the \n in the above example

This problem has gotten a bit worse for me. I tried the obvious workaround of editing the HDU inplace:

    fh = fits.open(fn)
    fh[0].header['ORIGIN'] = 'CASA 4.7.0-REL (r38335)'
    print('Header origin: "{0}"'.format(fh[0].header['ORIGIN'])) # confirms no newline
    wcs.WCS(fh[0].header) #crashes!!

but it results in the same traceback:

Header origin: "CASA 4.7.0-REL (r38335)"
Traceback (most recent call last):
  File "<ipython-input-72-2c3f12ce28f0>", line 16, in <module>
    wcs.WCS(fh[0].header)
  File "/users/aginsbur/anaconda/lib/python3.5/site-packages/astropy/wcs/wcs.py", line 391, in __init__
    try:
  File "/users/aginsbur/anaconda/lib/python3.5/site-packages/astropy/io/fits/header.py", line 639, in tostring
    s += sep + _pad('END')
  File "/users/aginsbur/anaconda/lib/python3.5/site-packages/astropy/io/fits/card.py", line 461, in __str__
    return self.image
  File "/users/aginsbur/anaconda/lib/python3.5/site-packages/astropy/io/fits/card.py", line 765, in image
    self.verify('fix')
  File "/users/aginsbur/anaconda/lib/python3.5/site-packages/astropy/io/fits/verify.py", line 74, in verify
    errs = self._verify(opt)
  File "/users/aginsbur/anaconda/lib/python3.5/site-packages/astropy/io/fits/card.py", line 1392, in _verify
    fix=self._fix_value))
  File "/users/aginsbur/anaconda/lib/python3.5/site-packages/astropy/io/fits/verify.py", line 48, in run_option
    fix()
  File "/users/aginsbur/anaconda/lib/python3.5/site-packages/astropy/io/fits/card.py", line 1131, in _fix_value
    self.value = valuecomment
  File "/users/aginsbur/anaconda/lib/python3.5/site-packages/astropy/io/fits/card.py", line 582, in value
    'ASCII or non-printable characters.' % value)
ValueError: FITS header values must contain standard printable ASCII characters; "'CASA 4.7.0-REL (r38335)\n'" contains characters not representable in ASCII or non-printable characters.

How is WCS accessing some older version of the header? It's as if the header isn't being updated.

Try mode=update or... make a copy of that header, modify the copy, and pass in the modified copy to WCS. Does either of those work?

@pllim

@pllim Thanks for the ideas. Copy fails (same). It actually shows that WCS is not accessing the same header... which I don't understand at all.

    fh = fits.open(fn)
    fh[0].header['ORIGIN'] = 'CASA 4.7.0-REL (r38335)'
    print('Header origin: "{0}"'.format(fh[0].header['ORIGIN']))
    header = copy.copy(fh[0].header)
    header['ORIGIN'] = 'BLANK'

ValueError: FITS header values must contain standard printable ASCII characters; "'CASA 4.7.0-REL (r38335)\n'" contains characters not representable in ASCII or non-printable characters.

same problem with fh=fits.open(fn, mode='update')

EDIT: this works:

    fh = fits.open(fn)
    fh[0].header['ORIGIN'] = 'CASA 4.7.0-REL (r38335)'
    print('Header origin: "{0}"'.format(fh[0].header['ORIGIN']))
    header = copy.copy(fh[0].header)
    header['ORIGIN'] = 'BLANK'
    header.update()
    try:
        print(wcs.WCS(header))
    except ValueError:
        print(wcs.WCS(header))

Is the WCS call triggering some sort of flush action?

This has some heisenbug elements to it. If I run this as a block of code (i.e., with %cpaste), I get the error. If I run each command in turn (just paste the whole thing on the command line), it doesn't happen. Maybe it's something peculiar ipython is doing. The overall issue remains important, though.

@keflavich

@keflavich Is there a way to get your test image?

@keflavich

@keflavich I have tried to reproduce your example from #5396 (comment) (using an STScI image modified to contain \n in the ORIGIN string value) and I cannot reproduce your crash: WCS can be successfully constructed from a "corrected" header. Are you sure you are not attempting to create the WCS by providing the file name instead of the header to the WCS constructor? It looks to me that you are reloading the old file...

In [56]: h=fits.open('test1n.fits')

In [57]: w=wcs.WCS(h[0].header)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/Users/.../lib/python3.4/site-packages/astropy/io/fits/card.py in _fix_value(self)
   1128                 self.value = value.strip()
-> 1129                 self.comment = comment.strip()
   1130             except (ValueError, IndexError):

/Users/.../lib/python3.4/site-packages/astropy/io/fits/card.py in comment(self, comment)
    703                     'representable in ASCII or non-printable characters.' %
--> 704                     comment)
    705 

ValueError: FITS header comments must contain standard printable ASCII characters; "CFITSIO March 2010\n'" contains characters not representable in ASCII or non-printable characters.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-57-01ff27fbd8dc> in <module>()
----> 1 w=wcs.WCS(h[0].header)

/Users/.../lib/python3.4/site-packages/astropy/wcs/wcs.py in __init__(self, header, fobj, key, minerr, relax, naxis, keysel, colsel, fix, translate_units, _do_set)
    401 
    402             if isinstance(header, fits.Header):
--> 403                 header_string = header.tostring().rstrip()
    404             else:
    405                 header_string = header

/Users/.../lib/python3.4/site-packages/astropy/io/fits/header.py in tostring(self, sep, endcard, padding)
    640         lines = []
    641         for card in self._cards:
--> 642             s = str(card)
    643             # Cards with CONTINUE cards may be longer than 80 chars; so break
    644             # them into multiple lines

/Users/.../lib/python3.4/site-packages/astropy/io/fits/card.py in __str__(self)
    459 
    460     def __str__(self):
--> 461         return self.image
    462 
    463     def __len__(self):

/Users/.../lib/python3.4/site-packages/astropy/io/fits/card.py in image(self)
    763 
    764         if self._image and not self._verified:
--> 765             self.verify('fix')
    766         if self._image is None or self._modified:
    767             self._image = self._format_image()

/Users/.../lib/python3.4/site-packages/astropy/io/fits/verify.py in verify(self, option)
     72             return
     73 
---> 74         errs = self._verify(opt)
     75 
     76         # Break the verify option into separate options related to reporting of

/Users/.../lib/python3.4/site-packages/astropy/io/fits/card.py in _verify(self, option)
   1390                              'string: %r).' % (self.keyword, valuecomment),
   1391                     fix_text=fix_text,
-> 1392                     fix=self._fix_value))
   1393 
   1394         # verify the comment (string), it is never fixable

/Users/.../lib/python3.4/site-packages/astropy/io/fits/verify.py in run_option(self, option, err_text, fix_text, fix, fixable)
     46         else:
     47             if fix:
---> 48                 fix()
     49             text += '  ' + fix_text
     50 

/Users/.../lib/python3.4/site-packages/astropy/io/fits/card.py in _fix_value(self)
   1129                 self.comment = comment.strip()
   1130             except (ValueError, IndexError):
-> 1131                 self.value = valuecomment
   1132             self._valuestring = self._value
   1133             return

/Users/.../lib/python3.4/site-packages/astropy/io/fits/card.py in value(self, value)
    580                     'FITS header values must contain standard printable ASCII '
    581                     'characters; %r contains characters not representable in '
--> 582                     'ASCII or non-printable characters.' % value)
    583         elif isinstance(value, binary_type):
    584             # Allow str, but only if they can be decoded to ASCII text; note

ValueError: FITS header values must contain standard printable ASCII characters; "'HSTIO/CFITSIO March 2010\n'" contains characters not representable in ASCII or non-printable characters.

In [58]: h[0].header['origin']='goodval'

In [59]: w=wcs.WCS(h[0].header)

In [60]: h.verify()

In [61]:

@mcara

@mcara - in your example, you get a traceback the first time you access the header, but not the second. I get the same, even if I don't correct the header. i.e.:

fh = fits.open(fn)
fh[0].header.tostring() # produces exception
fh[0].header.tostring() # does not produce exception

Can you reproduce that behavior?

@keflavich

@keflavich Yes, I can reproduce this behavior.

If in my previous example I do not attempt to create a WCS immediately after opening FITS file, then attempting to create a WCS after correcting the header still crashes (as you have reported):

>>> h = fits.open('test1n.fits')
>>> h[0].header['origin'] = 'goodval'
>>> w = wcs.WCS(h[0].header)  # <- crashes

This is exciting! 😄

... On the other hand, if I use tostring() instead of attempting to create a WCS, I get the following:

>>> from astropy.io import fits
>>> from astropy import wcs
>>> h = fits.open('test1n.fits')
>>> h[0].header['origin'] = 'goodval'
>>> h[0].header.tostring()
WARNING: VerifyWarning: Verification reported errors: [astropy.io.fits.verify]
WARNING: VerifyWarning: Card 'ORIGIN' is not FITS standard (invalid value string: "'HSTIO/CFITSIO March 2010\n'").  Fixed 'ORIGIN' card to meet the FITS standard. [astropy.io.fits.verify]
WARNING: VerifyWarning: Note: astropy.io.fits uses zero-based indexing.
 [astropy.io.fits.verify]
"SIMPLE  =                    T / conforms to FITS standard                      BITPIX  =                  -32 / array data type                                NAXIS   =                    2 / number of array dimensions                     NAXIS1  =                 2061                                                  NAXIS2  =                 1921                                                  COMMENT   FITS (Flexible Image Transport System) format is defined in 'AstronomyCOMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H ORIGIN  = '''HSTIO '           / CFITSIO March 2010\n'                           DATE    = '2014-09

(scroll to the right to see 'ORIGIN'). So, it seems that h[0].header['origin'] = 'goodval' had no effect (but it does work if run twice)!

Interesting! If I do the same, I get a crash on tostring the first time I call it, but the second time it shows that the keyword was successfully changed:

>>> fh = fits.open(fn)
>>> fh[0].header['ORIGIN']='goodval'
# exception:
# ValueError: FITS header values must contain standard printable ASCII characters; "'CASA 4.7.0-REL (r38335)\n'" contains characters not representable in ASCII or non-printable characters.
>>> fh[0].header.tostring()
# ... snip ... ORIGIN  = \'goodval \' ... snip ...

Well, I think this has to be a bug in fits.io but it would probably be better to open a new issue as this is independent of CASA's bug.

Hmm, right. There are two independent issues here. This issue is about how best to use files that have illegal header keywords.

The upstream fix has been implemented, so most likely we will not hear much about this specific error - we caught it early enough that probably no data have been distributed with the bad headers. However, a preferred workaround would still be helpful to have here.

I am still worried that one cannot change header values right away - it seems that header is re-loading old values from somewhere...

@mcara

@mcara: agreed. I opened the new issue here now: #5408

This seems to be fixed by #6084 based on #6084 (comment).

Please feel free to reopen if I misunderstood and it didn't get fixed.

Navigation Menu

Repository navigation

Workaround to deal with invalid headers? #5396

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Workaround to deal with invalid headers? #5396

Description

Activity

MSeifert04 commented on Oct 6, 2016

keflavich commented on Oct 6, 2016

keflavich commented on Oct 7, 2016

pllim commented on Oct 7, 2016

keflavich commented on Oct 7, 2016

keflavich commented on Oct 7, 2016

mcara commented on Oct 8, 2016

mcara commented on Oct 8, 2016

keflavich commented on Oct 10, 2016

mcara commented on Oct 10, 2016

mcara commented on Oct 10, 2016

mcara commented on Oct 10, 2016

keflavich commented on Oct 10, 2016

mcara commented on Oct 10, 2016

keflavich commented on Oct 10, 2016

keflavich commented on Oct 14, 2016

mcara commented on Oct 14, 2016

keflavich commented on Oct 14, 2016

bsipocz commented on Jun 19, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions