Repository navigation
Mixed Multidimensional Arrays #101
Description
Hello PyFits Developers,
I am having an issue with writing out FITS table files containing a mix of multidimensional arrays of different types.
I have written the following 'helpful' wrapper function for PyFits to allow me to quickly write out table data:
def WriteTable(filename,data,cols,clobber=True,Header=None):
'''
Write FITS table data.
Arguments:
filename -- Filename to write FITS table data to.
data_list -- Contains data to write to table.
E.g. [dataset1,dataset2,...]
cols -- Column names for the input data.
Keyword Arguments:
clobber -- Overwrite fits files (Default: True)
Header -- Dictionary containing header names and values;
E.g. Header={'CRVAL1': 88.52, 'CRVAL2': 1.44, ...}
NOTES: Format of input data:
- data[0].shape == (1,dataLength,nColumns)
Even if you have only 1 data column, the format of the array must
be as shown above, e.g. (1,dataLength,1) .
'''
#Setup a couple of containers for format codes
f = np.empty(len(data),dtype='object')
dim = np.empty(len(data),dtype='object')
#Calculate format and dim of each data column
for i in range(len(cols)):
print data[i].shape
print data[i].shape[2],data[i].shape[1]
if isinstance(data[i][0,0,0],type('')):
#Calculate format and dim for string data *BROKEN*
#-------------
# (2) HOWEVER I CANNOT WRITE A MULTI-DIM STRING ARRAY IN THE SAME FORMAT
#-------------
length = (len(max(data[i][0,:,0],key = len)))
formatcode = 'A'
dim[i] = '( '+str(data[i].shape[2])+', '+str(data[i].shape[1])+')'
f[i] = str(length)+formatcode
else:
#Calculate format and dim for numerical data
#-------------
# (1) THIS ALL WORKS FINE! I CAN WRITE AS MANY MULTI-DIM ARRAYS AS I LIKE
#-------------
length = data[i].shape[1]*data[i].shape[2]
code = 'D'
dim[i] = '( '+str(data[i].shape[2])+', '+str(data[i].shape[1])+')'
formatcode = str(length) + code
f[i] = formatcode
#Generate the table HDU
tablelist = [pyfits.Column(name=c,format=f[i],array=data[i],dim=dim[i]) for i,c in enumerate(cols)]
coldefs = pyfits.ColDefs(tablelist)
tbhdu = pyfits.BinTableHDU.from_columns(coldefs)
hdu = pyfits.PrimaryHDU()
#Read in any header info
if Header:
for key,val in Header.iteritems():
hdu.header.set(key,val)
thdulist = pyfits.HDUList([hdu,tbhdu])
thdulist.verify('fix') #Check the header info order is correct
print 'WRITING FILE AS: ', filename
thdulist.writeto(filename,clobber=clobber)
thdulist.close()
Okay so in the code above I pass the function 'data' that is a list of numpy arrays of the format (1,dataLength,dataColumns) and 'cols' that is a list of column names. If 'data' is just a list of numpy arrays containing just numerical data the above code will work fine. Numerical data is processed by the code after the comment labelled "(1)" . However, if one of the arrays in the list 'data' contains a numpy array of strings the code will fail. String arrays are processed by the code after the comment labelled "(2)". Note the strings passed to the function could be of any length, but in my case are typically around 30 characters long.
The error I receive upon crashing is:
"The repeat count of the colum format 'STRINGS' for column '29A' is fewer than the number of elements per TDIM argument '(1,307)'. The invalid TDIMn value will be ignored for the purpose of formatting this column."
Now I'm fairly certain the issue is related to me misunderstanding how the TDIM keyword works for strings but after trying numerous combinations of TDIM values I am at a bit of a loss.
Thank you for any help.
Activity
embray commentedon May 18, 2015
Hi @SharperJBCA,
I'm going to take a closer look at your code in a bit, and see if I can give you some more concrete advice. But to start with, I think that most of the work you're doing to manually determine the format code and dim keywords for the columns is unnecessary. (I think this is a common confusion that should be addressed in the docs.)
In most cases you can just pass in an array in the Numpy recarray format, and PyFITS will automatically determine the correct formatting for the FITS columns. To give a simplified example:
The docs are somewhat outdated to the extent to which they show you how you can work directly with the FITS format. And while that's fine for some cases, I find in most cases it's easier to bypass all that and let PyFITS figure out how to format the columns from the data, especially when it comes to things like string arrays.
embray commentedon May 18, 2015
By the way, if while following this example you get a warning like "UserWarning: TDIM2 value (3,2) does not fit with the size of the array items (3). TDIM2 will be ignored.", that's a known issue and can be ignored. The warning is not actually correct, and nothing will be ignored.