Repository navigation
astropy.io.fits: Unexpected behavior of scaled data in random groups and binary table #2751
Description
When you create a GroupData object and set PZEROn keywords to non-zero values, once you save out the GroupsHDU object to a file and reread it, the PZEROn values replace all the values inside parameter arrays instead of properly scaling the array. Data type of the arrays will also not be preserved and change to float64.
>>> from astropy.io import fits
>>> import numpy as np
>>> imdata = np.zeros(20, dtype=np.float32)
>>> imdata.shape = (5, 1, 1, 2, 2)
>>> pdata = np.arange(5, dtype=np.float32) * 10
>>> x = fits.GroupData(imdata, parnames=['PAR1',], pardata=[pdata,],
... parbzeros=[3.14,],
... parbscales=[1.,], bitpix=-32)
>>> hdu = fits.GroupsHDU(x)
>>> hdu.writeto('test_grouptb_scaled.fits', clobber=True)
>>> t = fits.open('test_grouptb_scaled.fits')
>>> print hdu.data['par1']
[ 0. 10. 20. 30. 40.]
>>> print hdu.data['par1'].dtype
float32
>>> print t[0].data['par1']
[ 3.14000000e+00 3.14000000e+00 3.14000000e+00 -1.58456325e+29
3.14000000e+00]
>>> print t[0].data['par1'].dtype
float64
>>> hdu.header['pzero1']
3.14
>>> t[0].header['pzero1']
3.14Scaling at read-in for a random group data seems to work properly if the file was written with other software packages (for example VLA data converted to AIPS UVFITS), but the data type of the array after scaling will always be float64 - not preserved - and you still run into the above issue if you save out the HDU.
The binary table has somewhat similar problems. When you create a binary table hdu, array values in a column with non-default TZEROn keyword will be shown as if they have been replaced by negative of the TZEROn value. But when you save out the HDU and reread it, the array values seem to not have been replaced although the data type is, again, not preserved, and there are some accuracy problems.
>>> names = np.array(['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j'])
>>> values = np.arange(10, dtype=np.float32)
>>> c1 = fits.Column(name='names', format='1A', array=names)
>>> c2 = fits.Column(name='values', format='E', bzero=24251.3141, array=values)
>>> tbhdu = fits.BinTableHDU.from_columns([c1, c2])
>>> print repr(tbhdu.header)
XTENSION= 'BINTABLE' / binary table extension
BITPIX = 8 / array data type
NAXIS = 2 / number of array dimensions
NAXIS1 = 5 / length of dimension 1
NAXIS2 = 10 / length of dimension 2
PCOUNT = 0 / number of group parameters
GCOUNT = 1 / number of groups
TFIELDS = 2 / number of table fields
TTYPE1 = 'names '
TFORM1 = '1A '
TTYPE2 = 'values '
TFORM2 = 'E '
TZERO2 = 24251.3141
>>> tbhdu.writeto('test_bintb.fits', clobber=True)
>>> test_tb = fits.open('test_bintb.fits')
>>> values
array([ 0., 1., 2., 3., 4., 5., 6., 7., 8., 9.], dtype=float32)
>>> tbhdu.data['values']
array([-24251.31445312, -24250.31445312, -24249.31445312, -24248.31445312,
-24247.31445312, -24246.31445312, -24245.31445312, -24244.31445312,
-24243.31445312, -24242.31445312], dtype=float32)
>>> test_tb[1].data['values']
array([ -3.53125000e-04, 9.99646875e-01, 1.99964687e+00,
2.99964687e+00, 3.99964687e+00, 4.99964687e+00,
5.99964687e+00, 6.99964687e+00, 7.99964687e+00,
8.99964687e+00])
>>> test_tb[1].data['values'].dtype
dtype('float64')Both of the above issues seem be related to how astropy handle scaled data in a binary table, and my suggestion for a solution is to not scale a binary table data at read-in, or at least provide an option for that like in the IDL Astro library.
Activity
embray commentedon Jul 15, 2014
Have you read
http://astropy.readthedocs.org/en/latest/io/fits/appendix/faq.html#why-is-an-image-containing-integer-data-being-converted-unexpectedly-to-floats
and did none of the options discussed there work for you?
piyanatk commentedon Jul 15, 2014
Okay. So the conversion from 32-bit float to 64-bit float is expected. I have tried
do_not_scale_image_data=True, but it does not work with table data and random group parameter arrays. I will trymode='update'andscale_back=Trueand get back to you.However, the behavior here is not just about conversion from 32-bit float to 64-bit float. The scaling just does not work as expected from the formula:
piyanatk commentedon Jul 16, 2014
Update: open a random group structure with
mode='update' andscale_back=True`.piyanatk commentedon Jul 16, 2014
@embray I think you are right that scaling at read-in is not an issue here. I did more test with the random groups structure, and I think the problem in the first post happens when you write out a group hdu with non-default scaling parameters to a fits file. Please the code snippet below, and sorry for a long post here.
embray commentedon Jul 16, 2014
Yeah, I wouldn't be surprised if this doesn't work quite right for random groups HDUs. It's not a use case cross section that I think is tested much. Reworking the UI for this is high on the todo list (though automatic rescaling will remain the default).
Thanks for investigating this though. I'll see what I can do about this when I get a chance.
cmarmo commentedon Mar 29, 2026
Dear maintainers,
I have rerun the code above with astropy
'8.0.0.dev546+gd8705bd60', after adapting some parameters (clobber->overwrite).The issue is still there.