Astropy tables / pandas / hdf5 / Endian issue #11286

Activity

1. tomr-stargazer commented on Jan 30, 2021

   

   tomr-stargazer

   on Jan 30, 2021

   Author

   More actions

   Also h5py version, if it's meaningful:

   >>> import h5py
   >>> h5py.__version__
   '2.10.0'
   
2. taldcroft commented on Jan 30, 2021

   

   taldcroft

   on Jan 30, 2021

   Member

   More actions

   @tomr-stargazer - astropy is supposed to be converting the data to native byte order before passing to pandas. This is a known issue that has been discussed and indeed fixed some time ago. See #1156 and #3729.

   Perhaps the discussion in those two issues will help you find a workaround. We would probably need a data file and/or minimum working example showing the problem to get much further in fixing this if indeed there is a real bug in astropy.
3. tomr-stargazer commented on Jan 30, 2021

   

   tomr-stargazer

   on Jan 30, 2021

   Last edited by tomr-stargazer

   Author

   More actions

   @taldcroft, thank you!
   On first pass, it's not clear to me how I would make a workaround - does astropy.table.Table itself have a method to swap to a new byte ordering, or (equally fine by me) is there a hackish way to modify the byte ordering of the internal data in-place?

   I've tried the following, in an attempt to in-place modify the byte order of the table's data:

       dat = Table.read(filename)
       data = np.array(dat, copy=False)
       data = data.byteswap().newbyteorder()
       df = dat.to_pandas()
   
       ds = spreadsheet_maker(df) # this is a custom function I wrote which mostly consists of a groupby / apply step

   but it didn't seem to have an affect (same error occurs), and I'm trying to figure out how else I might modify the table's byte-order in-place.

   In the meantime, I wondered if part of the issue involved h5py, so I re-generated my data files as .fits instead of .hdf5, and then changed my data handling chain from:

   (original)

   1. astropy.table -> .hdf5 file
   2. .hdf5 file -> astropy.table
   3. astropy.table -> pandas DataFrame
   4. pandas DataFrame -> Groupy & Apply # this is where the endian error occured

   (new)

   1. astropy.table -> .fits file
   2. .fits file -> astropy.table
   3. astropy.table -> pandas DataFrame
   4. pandas DataFrame -> Groupy & Apply # no error!

   For my purposes I seem to have found a workaround: avoid saving my intermediate data as .h5 and use .fits instead. (This points to a possible bug in how astropy and/or h5py are handling data, perhaps?)

   My data file is 291 MB, and I still have the .hdf5 version in case anyone would like to see it.
4. tomr-stargazer

   changed the title [-]Astropy tables / pandas / Endian issue[/-] [+]Astropy tables / pandas / hdf5 / Endian issue[/+] on Jan 30, 2021
5. taldcroft commented on Jan 30, 2021

   

   taldcroft

   on Jan 30, 2021

   Member

   More actions

   @tomr-stargazer - For the table that fails going to pandas DataFrame, can you print the dtype with tbl.dtype and report that? Also, perhaps you can generate a small version of your HDF5 file with just a few rows and see if that replicates the issue. That will help debugging (probably don't need all 291 Mb).

   As an aside, astropy Table natively supports group_by and aggregation operations, so maybe you don't need Pandas? Using pandas is great, I'm just always wondering about any missing functionality in Table.
6. tomr-stargazer commented on Jan 30, 2021

   

   tomr-stargazer

   on Jan 30, 2021

   Author

   More actions

   @taldcroft - here's a 5 KB version of the hdf5 file (hosted on Google Drive) that causes the same endian error on my computer.

   Here's the dtype:

   tab = astropy.table.Table.read("/Users/tsrice/Documents/Variability_Project_2020/wuvars/Data/reduction_artifacts/wserv11/WSERV11_graded_clipped0.95_scrubbed0.1_dusted0.5_new_error_corrected_debug_small.h5")
   
   tab.dtype
   
   dtype([('SOURCEID', '<i8'), ('MEANMJDOBS', '<f8'), ('RA', '<f8'), ('DEC', '<f8'), ('JMHPNT', '<f4'), ('JMHPNTERR', '<f4'), ('HMKPNT', '<f4'), ('HMKPNTERR', '<f4'), ('JAPERMAG3', '<f4'), ('JAPERMAG3ERR', '<f4'), ('HAPERMAG3', '<f4'), ('HAPERMAG3ERR', '<f4'), ('KAPERMAG3', '<f4'), ('KAPERMAG3ERR', '<f4'), ('JPPERRBITS', '<i4'), ('HPPERRBITS', '<i4'), ('KPPERRBITS', '<i4'), ('MERGEDCLASS', '<i2'), ('PSTAR', '<f4'), ('JGRADE', '<f4'), ('HGRADE', '<f4'), ('KGRADE', '<f4')])

   Here's the associated error traceback.

   also, wow! I had no idea that groupby and aggregation existed in Astropy Tables. I certainly switched to pandas specifically in search of this kind of functionality. I may be a little too deep in pandas now to switch back, but in case you're curious, I have a fairly convoluted groupby/apply function that is vital to my multi-band time-series photometry analysis. The code appears here:
   https://github.com/tomr-stargazer/Variability_Project_2020/blob/master/wuvars/analysis/variability_selection.py#L269
7. dhomeier commented on Jan 30, 2021

   

   dhomeier

   on Jan 30, 2021

   Contributor

   More actions

   Looked quite mysterious how any big-endian values would come to appear there in the first place, since you read in an all-little endian dtype, and I guess the original table written to HDF5 was native byteorder as well (or have some columns been read from FITS)?
   But I could in fact reproduce this directly with

   >>> df = tab.to_pandas()
   >>> df.dtypes
   SOURCEID        int64
   MEANMJDOBS        >f8
   RA                >f8
   DEC               >f8
   JMHPNT            >f4
   JMHPNTERR         >f4
   HMKPNT            >f4
   HMKPNTERR         >f4
   JAPERMAG3         >f4
   JAPERMAG3ERR      >f4
   HAPERMAG3         >f4
   HAPERMAG3ERR      >f4
   KAPERMAG3         >f4
   KAPERMAG3ERR      >f4
   JPPERRBITS      int32
   HPPERRBITS      int32
   KPPERRBITS      int32
   MERGEDCLASS     int16
   PSTAR             >f4
   JGRADE            >f4
   HGRADE            >f4
   KGRADE            >f4
   dtype: object

   and any operation such as trying to print the entire data frame (not individual columns though) raises the same
   ValueError: Big-endian buffer not supported on little-endian compiler you triggered (in my case on a float64 column, probably because it is the first float encountered).

   Whereas directly reading the file in with pandas (which might be a shorter workaround for the time being ;-)

   >>> df = pd.read_hdf('Downloads/WSERV11_graded_clipped0.95_scrubbed0.1_dusted0.5_new_error_corrected_debug_small.h5', '__astropy_table__')
   >>> df.dtypes
   SOURCEID          int64
   MEANMJDOBS      float64
   RA              float64
   DEC             float64
   JMHPNT          float32
   JMHPNTERR       float32
   HMKPNT          float32
   HMKPNTERR       float32
   JAPERMAG3       float32
   JAPERMAG3ERR    float32
   HAPERMAG3       float32
   HAPERMAG3ERR    float32
   KAPERMAG3       float32
   KAPERMAG3ERR    float32
   JPPERRBITS        int32
   HPPERRBITS        int32
   KPPERRBITS        int32
   MERGEDCLASS       int16
   PSTAR           float32
   JGRADE          float32
   HGRADE          float32
   KGRADE          float32
   dtype: object

   returned an apparently functional data frame.

   So it looks indeed like the bug is somewhere in the tab.to_pandas() method somehow triggering the byteswap, which would be unwarranted in the first place.
   In contrast, both tab.as_array(), which is supposed to behave in the same way, and tab.as_array(keep_byteorder=True) preserve the original all-little endian tab.dtype.
8. dhomeier commented on Jan 30, 2021

   

   dhomeier

   on Jan 30, 2021

   Last edited by dhomeier

   Contributor

   More actions

   OK, that was rather easy. The table read by h5py has explicit < byteorders, but
   

   astropy/astropy/table/table.py

   Lines 3679 to 3681 in 4d31e6c

   	if (hasattr(out[name].dtype, 'byteorder')
   	and out[name].dtype.byteorder not in ('=', '|')):
   	out[name] = out[name].byteswap().newbyteorder()

   
   only makes a string comparison to native or string/object-type byteorders.

   At the core changing the conversion to

                    out[name] = out[name].byteswap().newbyteorder('=')

   would already take care of this. Alternatively or additionally (to avoid unnecessary byteswap() entirely), we could test against a native_order tuple like @embray did in #4080.
9. dhomeier commented on Jan 30, 2021

   

   dhomeier

   on Jan 30, 2021

   Contributor

   More actions

   For a temporary workaround (other than the intermediate FITS file or directly using pd.read_hdf) you'd probably need to byteswap explicitly to native as well, and feed the ndarray back to a table:

   dat = Table.read(filename)
   df = Table(dat.as_array().byteswap().newbyteorder('=')).to_pandas()
10. tomr-stargazer commented on Jan 31, 2021

    

    tomr-stargazer

    on Jan 31, 2021

    Author

    More actions

    @dhomeier this is all awesome!
    You wrote:

    (or have some columns been read from FITS)?

    and indeed, the .hdf5 file that I'm creating contains data that were originally read from a .fits file, before being manipulated as an astropy Table object and then re-saved in .hdf5 format.

    Thank you for the notes on all of this, especially with the multiple workaround suggestions. If you'd like me to test anything else out let me know! Also, if it would be appropriate for me to be the one to try and do a bugfix pull request on astropy/astropy/table/table.py, do let me know, though I feel I'm a little less than qualified to submit a patch to astropy.
11. mhvk commented on Jan 31, 2021

    

    mhvk

    on Jan 31, 2021

    Contributor

    More actions

    @dhomeier - wow, nice catch!

    I think one would want to change the test to explicitly check if the byte-order is swapped, for instance by (following https://numpy.org/devdocs/reference/generated/numpy.dtype.byteorder.html):

    if getattr(out[name].dtype, 'byteorder', '=') == ('>' if sys.byteorder == 'little' else '<'):
    
12. dhomeier commented on Jan 31, 2021

    

    dhomeier

    on Jan 31, 2021

    Contributor

    More actions

    if getattr(out[name].dtype, 'byteorder', '=') == ('>' if sys.byteorder == 'little' else '<'):

    Now I am not sure if this is fancier or the indexing trick...
    For the patch I pushed now I took the lazy way out by just using dtype.isnative, but I think that should catch all cases in our supported numpy versions. Became a bit clearer from the numpy docs why dtype only identifies as '<' on little endian if explicitly created with .newbyteorder('<')); that may also have caused at least one of the tests to check the wrong thing, when it correctly, but unintendedly, asserted that for t[column].dtype.byteorder == '<' the column was swapped.
13. dhomeier commented on Jan 31, 2021

    

    dhomeier

    on Jan 31, 2021

    Contributor

    More actions

    @tomr-stargazer thanks; I have already put my ideas for a fix into a PR, but your input and testing on #11288 will be highly appreciated!
14. pllim

    added

    table

    Bug

    on Feb 1, 2021
15. mhvk

    closed this as completedin #11288on Feb 2, 2021