Repository navigation
astropy.io.fits: repr() error in GroupData object with more than 1000 groups #2748
Description
When creating a random access group data with more than 1000 groups, viewing the string representation of the data object, via print object or repr(object) or just typing it on python terminal, will throw an error that traces back to fitsrec.py.
>>> from astropy.io import fits
>>> import numpy
>>> def make_group_data(ngroups):
... imdata = numpy.arange(ngroups * 2 * 5)
... imdata.shape = (ngroups, 1, 1, 2, 5)
... pdata1 = numpy.arange(ngroups) + 0.1
... pdata2 = 42
... x = fits.GroupData(imdata, parnames=['abc', 'xyz'],
... pardata=[pdata1, pdata2], bitpix=-32)
... return x
...
>>> a = make_group_data(1000)
>>> b = make_group_data(1001)
>>> # a # This will print the whole record array
...
>>> b # This won't work.
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py", line 463, in __repr__
return np.recarray.__repr__(self)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/records.py", line 465, in __repr__
ret = ndarray.__repr__(self)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/numeric.py", line 1553, in array_repr
', ', "array(")
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/arrayprint.py", line 454, in array2string
separator, prefix, formatter=formatter)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/arrayprint.py", line 250, in _array2string
data = _leading_trailing(a)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/arrayprint.py", line 208, in _leading_trailing
b = _nc.concatenate((a[:_summaryEdgeItems],
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py", line 553, in __getslice__
return self[slice(start, end)]
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/hdu/groups.py", line 200, in __getitem__
out = super(GroupData, self).__getitem__(key)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py", line 498, in __getitem__
arrays.append(self._coldefs._arrays[idx][key])
TypeError: 'NoneType' object has no attribute '__getitem__'```
>>> b[0] # But this work
(0.1, 42.0, array([[[[ 0., 1., 2., 3., 4.],
[ 5., 6., 7., 8., 9.]]]], dtype=float32))The error disappear once you put the object into a GroupsHDU object, write it out, and reread it back in.
>>> hdu = fits.GroupsHDU(b)
>>> hdu.writeto('test.fits', clobber=True)
>>> c = fits.open('test.fits')
>>> c[0].data
GroupData([ (0.1, 42.0, array([[[[ 0., 1., 2., 3., 4.],
[ 5., 6., 7., 8., 9.]]]], dtype=float32)),
(1.1, 42.0, array([[[[ 10., 11., 12., 13., 14.],
[ 15., 16., 17., 18., 19.]]]], dtype=float32)),
(2.0999999, 42.0, array([[[[ 20., 21., 22., 23., 24.],
[ 25., 26., 27., 28., 29.]]]], dtype=float32)),
...,
(998.09998, 42.0, array([[[[ 9980., 9981., 9982., 9983., 9984.],
[ 9985., 9986., 9987., 9988., 9989.]]]], dtype=float32)),
(999.09998, 42.0, array([[[[ 9990., 9991., 9992., 9993., 9994.],
[ 9995., 9996., 9997., 9998., 9999.]]]], dtype=float32)),
(1000.1, 42.0, array([[[[ 10000., 10001., 10002., 10003., 10004.],
[ 10005., 10006., 10007., 10008., 10009.]]]], dtype=float32))],
dtype=[('abc', '>f4'), ('xyz', '>f4'), ('DATA', '>f4', (1, 1, 2, 5))])Also, I found that calling view() method from a GroupData object will throw a similar error that also traces back to fitsrec.py. This does not matter how many groups are in the GroupData object. FIT_rec object inherited this function from numpy, so I am not sure if this is related to the above issue.
>>> a.view()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py", line 463, in __repr__
return np.recarray.__repr__(self)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/records.py", line 465, in __repr__
ret = ndarray.__repr__(self)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/numeric.py", line 1553, in array_repr
', ', "array(")
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/arrayprint.py", line 454, in array2string
separator, prefix, formatter=formatter)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/arrayprint.py", line 250, in _array2string
data = _leading_trailing(a)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/numpy/core/arrayprint.py", line 208, in _leading_trailing
b = _nc.concatenate((a[:_summaryEdgeItems],
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py", line 553, in __getslice__
return self[slice(start, end)]
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/hdu/groups.py", line 200, in __getitem__
out = super(GroupData, self).__getitem__(key)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py", line 490, in __getitem__
out = self.view(np.recarray).__getitem__(key).view(subtype)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/hdu/groups.py", line 193, in __array_finalize__
super(GroupData, self).__array_finalize__(obj)
File "/Users/piyanat/src/anaconda/envs/conda-sim/lib/python2.7/site-packages/astropy/io/fits/fitsrec.py", line 251, in __array_finalize__
self._nfields = len(obj.dtype.names)
TypeError: object of type 'NoneType' has no len()I am using astropypy 0.4rc1 and numpy 1.8.1.
Activity
embray commentedon Jul 15, 2014
Thanks for the detailed report--these should both be fixed. The random groups structure doesn't come up very often in use so that particular feature isn't as well tested as others. This will have to be fixed in the next patch release.
piyanatk commentedon Jul 15, 2014
@embray Please also take a look at another issue that I just submit.
I am involve in an epoch of reionization experiment with a low frequency radio interferometer, the MWA to be specific, and for some unknown reason, UVFITS, which is a random groups structure, is a data format of choice for these kind of experiments. LOFAR and PAPER also use it as far as I know. I am developing a python tools for manipulating UVFITS produced from the telescope pipeline and simulation, and that how I ran into these issues.
embray commentedon Jul 15, 2014
Right, to my knowledge UVFITS files are the only place where this feature is still used, so whenever someone is starting a new project involving those files they often find new issues with it, so thanks :) I should be able to have fixes for this fairly soon.