-
-
Notifications
You must be signed in to change notification settings - Fork 2k
Description
When the fill_value of a masked Table Column is equal to an unmasked value, the round-trip to a FITS format file is broken [EDIT taldcroft example changed]. Note that #7481 in astropy 3.1 provides a work-around by specifying a serialize_method, but does not fix the default behavior.
In [23]: from astropy.table.table_helpers import simple_table
In [24]: t = simple_table(masked=True)
In [25]: t['a'].fill_value = 2
In [26]: t.write('junk.fits', overwrite=True)
In [27]: t2 = Table.read('junk.fits')
In [28]: t
Out[28]:
<Table masked=True length=3>
a b c
int64 float64 str1
----- ------- ----
-- 1.0 c
2 2.0 --
3 -- e
In [29]: t2
Out[29]:
<Table masked=True length=3>
a b c
int64 float64 bytes1
----- ------- ------
-- 1.0 c
-- 2.0 N
3 nan e
This is because the Table is storing the fill_value in the TNULLn keywords to flag mask values, resulting in masking any row that has that value, even if it wasn't originally masked. [EDIT taldcroft] In addition, handling of floating point masking via NaN is broken as well as string-type masking.
To fix this, it seems that the Table needs to separate out the concept of what values are masked (using TNULLn following the FITS standard, vs. what the fill_value should be for those masked elements. AFAIK, the FITS standard doesn't define how to store a fill_value, but TFILLn would seem reasonable.
The current behavior is especially problematic when coupled with issue #4707 where filtering masked tables resets the fill_value to 1 -- a common number that can result in any other 1 getting masked after round tripping to a FITS file.