Issue with null values in IPAC tables

@taldcroft

According to the IPAC table definition (http://irsa.ipac.caltech.edu/applications/DDGEN/Doc/ipac_tbl.html) any column can have string null values. However, the following script fails:

from astropy.io.ascii import read, Ipac

table = """
|   a    |
| double |
|  mJy   |
|  null  |
    1
   null
"""

t = read(table, Reader=Ipac, guess=False)

and the exception is:

Traceback (most recent call last):
  File "test_ipac.py", line 12, in <module>
    t = read(table, Reader=Ipac, guess=False)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1572-py2.7-macosx-10.6-x86_64.egg/astropy/io/ascii/ui.py", line 133, in read
    dat = reader.read(table)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1572-py2.7-macosx-10.6-x86_64.egg/astropy/io/ascii/core.py", line 875, in read
    self.table = self.outputter(cols)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1572-py2.7-macosx-10.6-x86_64.egg/astropy/io/ascii/core.py", line 786, in __call__
    self._convert_vals(cols)
  File "/Users/tom/Library/Python/2.7/lib/python/site-packages/astropy-0.2.dev1572-py2.7-macosx-10.6-x86_64.egg/astropy/io/ascii/core.py", line 721, in _convert_vals
    raise ValueError('Column %s failed to convert' % col.name)
ValueError: Column a failed to convert

If I print out the original exception that was caught in core.py, I get:

ValueError: could not convert string to float: null

I don't know the astropy.io.ascii codebase to know how to deal with this - @taldcroft, do you have any suggestions?

It looks like I implemented the IPAC spec incorrectly and did not allow for the string "null" as a possible null value. It considers a value of "null" in the 4th row to mean there is no defined null value for that column:

            if len(header_vals) > 3:
                null = header_vals[3][i].strip()
                if null.lower() != 'null':
                    col.null = null  # Can't strip dashes here
                    if issubclass(col.type, core.FloatType):
                        fillval = 'nan'
                    else:
                        fillval = '-999'
                    self.data.fill_values.append((col.null, fillval, col.name))
            start = col.end + 1
            cols.append(col)

I think if the if null.lower() != 'null': statement is removed this might work per the spec. There is something about not stripping dashes here that needs to be checked (since the header rows can use dashes instead of spaces).

With the current code the user can set fill_values explicitly as a workaround:

>>> t = read(table, Reader=Ipac, guess=False, fill_values=('null', -999))
>>> t
masked_rec.array(data = [(1.0,) (--,)],
                 mask = [(False,) (True,)],
           fill_value = (1e+20,),
                dtype = [('a', '<f8')])

This can be done on a per-column basis if needed. Of course this results in a masked array, not a Table object.

Ok, I will see if that fixes my issue and will submit a pull request if so. By the way, this also relates to #149 - I added some comments there recently with ideas on implementing masking in Table. I'm happy to do the work to support masking, but I'd appreciate any feedback you have about that in that issue.

(closed by mistake)

I'll attach code to this issue, but I'm first waiting to hear back from IRSA. I'm not sure I understand how to interpret:

|-a----|
|-int--|
|-unit-|
|--99--|
  -99
   99

is the null 99 or -99? Is that why you weren't sure about stripping dashes?

I emailed IRSA, and their conclusion after a couple of exchanges was "I agree that dashes should be disallowed from the NULL line. People like to use negatives for nulls, and I can't see how to do it if dashes are allowed.". So it seems we can just assume that dashes never appear in the null line (except for minus signs).

Merge pull request #483 from taldcroft/ascii-table

Correctly support IPAC table null values

Merge pull request astropy#483 from taldcroft/ascii-table

Temporarily allow to fail setuptools 27+ builds, revert when astropy#361

Merge pull request astropy#361 from astropy/extension-helpers

Navigation Menu

Repository navigation

Issue with null values in IPAC tables #361

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue with null values in IPAC tables #361

Description

Activity

taldcroft commented on Aug 27, 2012

astrofrog commented on Aug 28, 2012

astrofrog commented on Aug 28, 2012

astrofrog commented on Aug 29, 2012

astrofrog commented on Aug 31, 2012

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions