Consider using IntegerArrays in to_pandas() for masked int columns

Currently, to_pandas converts any masked columns automatically to float (even if they are integers). This is actually a pretty big problem for integer identifiers (for example, GAIA source ids). If the integer is too large, the double-precision float actually gets approximated in the last few digits, completely changing the identifier. As this can happen with something as common as GAIA data, it's a big problem. What I assume most people would do is convert to_pandas, and then convert the data-type back to int, but this will now produce a different number.

More recent versions of pandas now support nullable integers (https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html), using the new IntegerArrays (with data type pd.Int64Dtype()). With that now supported, it would make sense for the astropy.table.to_pandas to turn masked int columns into this new data type, rather than float. pandas IntegerArrays are apparently still "experimental" but at the same time I think it's worth doing anyway since turning integers to floats can have pretty severe side-effects in astronomical data, which can take a long time to debug.

At the very least, if the above isn't implemented yet, I think it would make sense for astropy to produce a warning when it's converting integers to floats in this way, especially if the integers are large enough that the floats will only approximate them.

Indeed this has been a known limitation of pandas. The issue you raised is explicitly documented and hence not "hidden", see the caveats here:
https://docs.astropy.org/en/stable/table/pandas.html?highlight=pandas

That's great news about the new IntegerArrays class. I'm editing the issue title to be a bit more concise and actionable.

@mar-ses

@mar-ses - in your work did you not see the warning that is emitted?

In [9]: c = MaskedColumn([1, 2], mask=[False, True])

In [10]: t = Table([c])

In [11]: df = t.to_pandas()
/Users/aldcroft/miniconda3/envs/astropy/lib/python3.6/site-packages/IPython/core/interactiveshell.py:2963: TableReplaceWarning: converted column 'col0' from integer to float
  exec(code_obj, self.user_global_ns, self.user_ns)

No I didn't see the warning as far as I can tell.

Maybe you are using an older version? This was put into astropy in Aug. 2018 in 032fba7.

I don't know, I updated since then anyway so I don't know which version it was. In any case, I made a workaround fix to convert to strings first, then to_pandas, then .astype('float128'), and then .astype('Int64'). The issue is pandas also doesn't fully support Int64 yet, you can't directly change a Series of str into Int64, you need to convert to float128 first in order not to lose precision, then from float128 to Int64.

So the warning part is done, but I still think it would also be good to allow to_pandas to support conversion to Int64 when dealing with masked arrays of integers.

I've never contributed to these big projects, so I don't know how it works. If I wanted to contribute this myself (unless someone else here is interested and knows how), could you recommend how to start? I guess I would need to know the inner workings of these astropy Tables first.

Okay I was looking into it a bit, it seems the issue is here: out[name] = column.astype(float).filled(np.nan) at:

astropy/astropy/table/table.py

Line 3286 in ad4e0f2

out[name] = column.astype(float).filled(np.nan)

Right now, if it detects a MaskedColumn, even if the column.dtype.kind in ('i', 'u') is True, it still converts to float.

Would it be as simple as something like changing the above to: out[name] = pd.Series(column, dtype='Int64')? I tested it on my own and that seems to work, so should I do a pull/push request? Haven't really done this before.

Follow the instructions here, and if you run into any trouble I can come by and help on Wednesday. Thanks for working on the fix!

EDIT: Updated link to point to latest instead of stable.

Adding pandas Int64 support [fix astropy#9280]

Merge pull request #9541 from bmorris3/fix-issue-9280

Navigation Menu

Repository navigation

Consider using IntegerArrays in to_pandas() for masked int columns #9280

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Consider using IntegerArrays in to_pandas() for masked int columns #9280

Description

Activity

taldcroft commented on Sep 24, 2019

taldcroft commented on Sep 24, 2019

mar-ses commented on Sep 30, 2019

taldcroft commented on Sep 30, 2019

mar-ses commented on Nov 4, 2019

mar-ses commented on Nov 4, 2019

bmorris3 commented on Nov 4, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions