Repository navigation
fits: Inconsistent type conversion for FitsRecord #4034
Description
I have this fitsfile with a FitsRecord that has the following dtype:
dtype([('SC_CLK_COARSE', '>i4'), ('SC_CLK_FINE', '>i2'), ('FUV_CHIP_TEMP_C', '>f8'),
('FUV_P3_3_V_ADC_V', '>f4'),...]and while dtypes like >i2 and >i4 are being converted to numpy.float64 when I read the arrays with the standard key access of the FitsRecord, dtypes like >f4 or >f8 keep stuck in BigEndian mode:
print(hk.AnalogConv['SC_CLK_COARSE'].dtype)
print(hk.AnalogConv['SC_CLK_FINE'].dtype)
print(hk.AnalogConv['FUV_CHIP_TEMP_C'].dtype)
print(hk.AnalogConv['FUV_P3_3_V_ADC_V'].dtype)
float64
float64
>f8
>f4This fact throws off working with Pandas arrays, b/c for some reason if some of the columns in the dataframe are float64, i.e. when I include the SC_CLK_xx columns, then the whole dataframe is being converted to float64, but if I exclude the integer columns that are being auto-converted to float64 and only read >f4/8 types, then the resulting pandas array (that uses numpy under the hood) is remaining in BigEndian mode and math operations like df.describe() are failing:
KeyError: ValueError('Big-endian buffer not supported on little-endian compiler',)My question: Is this behavior of the FitsRecord reading as designed and I have to write my own conversion to make sure things work out, or is this a bug somewhere?
Activity
embray commentedon Aug 4, 2015
I'm not totally sure I understand the issue--I feel like I'm missing some context.
If you have a look at this comment it might help explain what's going on though: #3073 (comment)
The lack of big-endian support in Pandas has been a problem before with working with FITS files. I would recommend just looping over each column and ensuring that it's converted to native byte-order before passing it to Pandas.
michaelaye commentedon Aug 4, 2015
Well, forget the Pandas part for a moment that's just an effect on what happens on numpy level. Why are the types converted to little endian for '>ix' and not for '>fx' ? Isn't that inconsistent?
Edited: correcting DamnYouAutoCorrect.
embray commentedon Aug 4, 2015
Heh.
My point was that some of those columns might be scaled, and the rescaled array is just in native byte order, since there's no reason to add the overhead of converting it back to big-endian until and unless it's written back out to a a FITS file. The columns that are still in big-endian are just accessing the raw data directly out of the FITS file since they aren't scaled.
It's true that the dtype is deceptive. I think I have a fix for that in a branch.
michaelaye commentedon Aug 4, 2015
Hm, indeed the integer dtypes are the only ones that have scaling and offset information:
and from the header:
I still think that this is quite a bad interface for the user. I'm speaking as the naive user here (easy for me ;) and if I get a mix of big endian and little endian back into numpy, that's just very confusing. As the scaling and offset operations seem to convert to little-endian without doing any harm, is there a reason not to simply enforce conversion to little-endian whenever accessing it via the FitsRecord[key] interface?
embray commentedon Aug 4, 2015
Yes--that requires (at least naively, without using on-disk caching) copying the entire array into memory, rather than allowing it to be read from disk as needed. Make sure also that you've read this if you haven't.
In most cases the endianness isn't relevant since Numpy handles that pretty transparently. It's only when interfacing with other software that doesn't handle it that this is a problem, such as Pandas and some SciPy routines. This has come up before. But I agree it's a bad interface. There's no silver bullet though when it comes to trying to perfectly hide away FITS-isms that make working with FITS files hazardous, though I've outlined plans elsewhere for improvement (that said, consistent byte orders between FITS columns is not a to-do item since in most cases byte order should be irrelevant to the user and hidden away entirely).
Oh, also, it looks like the columns you have there are getting rescaled are pseudo-unsigned integers, so really you want those being converted to unsigned ints instead of floats. To get that, use
uint=Truewhen opening the file. In Astropy v1.1 this will be the default.embray commentedon Aug 4, 2015
Though this makes me think, I could add a bullet point to my plans for the
FITSDataclass (this is my planned container for FITS arrays and FITS table columns) to include a convenience method for returning the logical data in native byteorder.michaelaye commentedon Aug 4, 2015
Thanks a lot! I was not imaging that Numpy would be handling this transparently, I was worried that I end up with wrong numbers if I don't have little-endians upfront.
embray commentedon Aug 4, 2015
No, it's not a problem for Numpy--if a Numpy array is specified to contain big-endian values it will automatically swap them to little-endian before displaying them or doing any other calculations with them.
It's just a bit of a pain if you want to convert multiple columns to native byte order. Since you can't be sure up front what the array is stored in you have to do something like:
michaelaye commentedon Nov 2, 2015
i don't understand the
needs_clarificationlabel? What is missing?embray commentedon Nov 16, 2015
Well originally it was because I didn't understand the question. I'll have to go back and re-read this now.
saimn commentedon Jun 4, 2026
Not sure if something needs fixing here but given the last comments I will close :)