BinTableHDU.data.columns doesn't apply scaling #6837

Activity

1. astrofrog

   added

   io.fits

   on Nov 12, 2017
2. pllim commented on Nov 13, 2017

   

   pllim

   on Nov 13, 2017

   Member

   More actions

   This looks very familiar; Are you sure this this not a known bug? (I found a few in a quick search but can't be sure if they are indeed the same or related.)
3. saimn commented on Nov 15, 2017

   

   saimn

   on Nov 15, 2017

   Contributor

   More actions

   hmm all this is really weird:

   In [1]: from astropy.io import fits
   
   In [2]: hdu = fits.open('astropy/io/fits/tests/data/tb.fits')[1]
   
   In [3]: hdu.columns[2].array[0]
   Out[3]: 3.7000000715255736
   
   In [4]: hdu.columns[2].array
   Out[4]: array([ 1.10000002,  2.0999999 ], dtype=float32)
   
   In [5]: hdu.columns[2].array[0]
   Out[5]: 1.1
   

   #5490 is probably related.
4. saimn commented on Nov 15, 2017

   

   saimn

   on Nov 15, 2017

   Contributor

   More actions

   And it has to do with the delayed data loading. At first hdu.columns[2].array is a Delayed object, but as soon as the data is loaded to a np.array the scaling is no more applied;

   In [13]: hdu = fits.open('astropy/io/fits/tests/data/tb.fits')[1]
   
   In [14]: type(hdu.columns[2].array)
   Out[14]: astropy.io.fits.column.Delayed
   
   In [15]: hdu.columns[2].array[0]
   Out[15]: 3.7000000715255736
   
   In [16]: type(hdu.columns[2].array)
   Out[16]: numpy.ndarray
   
   In [17]: hdu.columns[2].array[0]
   Out[17]: 1.1
   
5. saimn commented on Dec 6, 2017

   

   saimn

   on Dec 6, 2017

   Contributor

   More actions

   After investigating #6887 I understand better now. The issue is that at the beginning columns are Delayed objects, and if you use exactly hdu.data.columns[2].array[0] it goes to the Delayed.__getitem__ method which then use hdu.data[key] which scales the data:
   https://github.com/astropy/astropy/blob/master/astropy/io/fits/column.py#L143

   But this can happen only once, when the data is not yet loaded. As soon as the data is loaded, any access to hdu.data.columns[2].array or hdu.columns[2].array will return the unscaled data, because the scaled data is stored in hdu.data._converted and can be accessed only with hdu.data['c3']!

   The root of the issue (and many issues with this table hdu code!) is that the data can be access both with the FITS_rec, hdu.data, and with individual columns, hdu.columns or hdu.data.columns...