Skip to content
This repository was archived by the owner on Dec 19, 2017. It is now read-only.
This repository was archived by the owner on Dec 19, 2017. It is now read-only.

Mixed Multidimensional Arrays #101

Closed

Description

@SharperJBCA

Hello PyFits Developers,

I am having an issue with writing out FITS table files containing a mix of multidimensional arrays of different types.

I have written the following 'helpful' wrapper function for PyFits to allow me to quickly write out table data:

def WriteTable(filename,data,cols,clobber=True,Header=None):
    '''
    Write FITS table data.

    Arguments:
    filename  -- Filename to write FITS table data to.
    data_list -- Contains data to write to table.
                  E.g. [dataset1,dataset2,...]
    cols -- Column names for the input data.

    Keyword Arguments:
    clobber -- Overwrite fits files (Default: True)
    Header -- Dictionary containing header names and values;
               E.g. Header={'CRVAL1': 88.52, 'CRVAL2': 1.44, ...}


    NOTES: Format of input data:
    - data[0].shape == (1,dataLength,nColumns)

    Even if you have only 1 data column, the format of the array must
    be as shown above, e.g. (1,dataLength,1) .

    '''

    #Setup a couple of containers for format codes
    f = np.empty(len(data),dtype='object')
    dim = np.empty(len(data),dtype='object')

    #Calculate format and dim of each data column
    for i in range(len(cols)):
        print data[i].shape
        print data[i].shape[2],data[i].shape[1]
        if isinstance(data[i][0,0,0],type('')):
            #Calculate format and dim for string data *BROKEN*

            #-------------
            # (2) HOWEVER I CANNOT WRITE A MULTI-DIM STRING ARRAY IN THE SAME FORMAT
            #-------------

            length = (len(max(data[i][0,:,0],key = len)))
            formatcode = 'A'
            dim[i]  = '( '+str(data[i].shape[2])+', '+str(data[i].shape[1])+')'            
            f[i] =  str(length)+formatcode
        else:
            #Calculate format and dim for numerical data

            #-------------
            # (1) THIS ALL WORKS FINE! I CAN WRITE  AS MANY MULTI-DIM ARRAYS AS I LIKE
            #-------------

            length = data[i].shape[1]*data[i].shape[2]
            code = 'D'
            dim[i]  = '( '+str(data[i].shape[2])+', '+str(data[i].shape[1])+')'
            formatcode = str(length) + code
            f[i] = formatcode

    #Generate the table HDU
    tablelist = [pyfits.Column(name=c,format=f[i],array=data[i],dim=dim[i]) for i,c in enumerate(cols)]
    coldefs = pyfits.ColDefs(tablelist)
    tbhdu = pyfits.BinTableHDU.from_columns(coldefs)

    hdu = pyfits.PrimaryHDU()

    #Read in any header info 
    if Header:
        for key,val in Header.iteritems():
            hdu.header.set(key,val)

    thdulist = pyfits.HDUList([hdu,tbhdu])
    thdulist.verify('fix') #Check the header info order is correct

    print 'WRITING FILE AS: ', filename
    thdulist.writeto(filename,clobber=clobber)
    thdulist.close()

Okay so in the code above I pass the function 'data' that is a list of numpy arrays of the format (1,dataLength,dataColumns) and 'cols' that is a list of column names. If 'data' is just a list of numpy arrays containing just numerical data the above code will work fine. Numerical data is processed by the code after the comment labelled "(1)" . However, if one of the arrays in the list 'data' contains a numpy array of strings the code will fail. String arrays are processed by the code after the comment labelled "(2)". Note the strings passed to the function could be of any length, but in my case are typically around 30 characters long.

The error I receive upon crashing is:

"The repeat count of the colum format 'STRINGS' for column '29A' is fewer than the number of elements per TDIM argument '(1,307)'. The invalid TDIMn value will be ignored for the purpose of formatting this column."

Now I'm fairly certain the issue is related to me misunderstanding how the TDIM keyword works for strings but after trying numerous combinations of TDIM values I am at a bit of a loss.

Thank you for any help.

Activity

  1. embray commented on May 18, 2015

    @embray
    Collaborator

    Hi @SharperJBCA,

    I'm going to take a closer look at your code in a bit, and see if I can give you some more concrete advice. But to start with, I think that most of the work you're doing to manually determine the format code and dim keywords for the columns is unnecessary. (I think this is a common confusion that should be addressed in the docs.)

    In most cases you can just pass in an array in the Numpy recarray format, and PyFITS will automatically determine the correct formatting for the FITS columns. To give a simplified example:

    >>> import numpy as np
    >>> import pyfits
    >>> arr = np.rec.array([(1.0, ['abc', 'def']), (2.0, ['ghi', 'jkl'])], names=['c1', 'c2'], formats=['f8', '2a3'])
    >>> hdu = pyfits.BinTableHDU(data=arr)
    >>> hdu.writeto('test.fits')
    >>> hdul = pyfits.open('test.fits')
    >>> hdul[1].header
    XTENSION= 'BINTABLE'           / binary table extension                         
    BITPIX  =                    8 / array data type                                
    NAXIS   =                    2 / number of array dimensions                     
    NAXIS1  =                   14 / length of dimension 1                          
    NAXIS2  =                    2 / length of dimension 2                          
    PCOUNT  =                    0 / number of group parameters                     
    GCOUNT  =                    1 / number of groups                               
    TFIELDS =                    2 / number of table fields                         
    TTYPE1  = 'c1      '                                                            
    TFORM1  = 'D       '                                                            
    TTYPE2  = 'c2      '                                                            
    TFORM2  = '6A      '                                                            
    TDIM2   = '(3,2)   '                                                            
    >>> hdul[1].data
    FITS_rec([(1.0, chararray(['abc', 'def'], 
          dtype='|S3')),
           (2.0, chararray(['ghi', 'jkl'], 
          dtype='|S3'))], 
          dtype=[('c1', '>f8'), ('c2', 'S6')])

    The docs are somewhat outdated to the extent to which they show you how you can work directly with the FITS format. And while that's fine for some cases, I find in most cases it's easier to bypass all that and let PyFITS figure out how to format the columns from the data, especially when it comes to things like string arrays.

  2. embray commented on May 18, 2015

    @embray
    Collaborator

    By the way, if while following this example you get a warning like "UserWarning: TDIM2 value (3,2) does not fit with the size of the array items (3). TDIM2 will be ignored.", that's a known issue and can be ignored. The warning is not actually correct, and nothing will be ignored.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions