astropy.table.Table - truncated strings

The behavior of Table regarding strings is really not what one expects, e.g.

from astropy.table import Table

x=['a','b','c']
y=[1,2,3]
z=Table([x,y],names=['A','B'])
print(z)
z['A'][1]='XXX'
print(z)
z['A'].format='20s'
z['A'][2]='YYY'
print(z)

In both cases, the longer string (XXX or YYY) is truncated in the table. I consider this a very nasty problem since it generates no errors or warnings, and since I've seen it in other guises as well.

This is effectively an upstream issue related to the way numpy works:

In [1]: x = np.array(['x', 'y'])

In [2]: x[0] = 'ABCDE'

In [3]: x
Out[3]: 
array(['A', 'y'], 
      dtype='|S1')

The tradeoff here is for memory efficiency by pre-declaring a fixed string width, vs. using variable width strings stored as Python objects. A solution is therefore:

In [9]: z=Table([x,y],names=['A','B'], dtype=['object', 'int'])

In [10]: z['A'][1] = 'XXX'

In [11]: z
Out[11]: 
<Table length=3>
  A      B  
object int64
------ -----
     a     1
   XXX     2
     c     3

BTW, here is a way to fix a table so the string columns are all objects:

In [17]: for col in z.itercols():
   ....:     if col.dtype.kind in 'SU':
   ....:         z.replace_column(col.name, col.astype('object'))

Thanks, this solved my immediate problem, although this behavior certainly limits the utility of astropy. ascii tables for me, since this information is lost in most formats when you write and read the file.

For example, even if you write out the table above in fixed_width_two_line format. When you read it back, it truncates the description to the length of the longest string. So if you have a table you want to update, you end up with the truncation problem all over again, even if you have set the format the first time. (I know there are one or two formats where the format is written out, but these are less readable.)

Possibly, for ascii formats, one should make this conversion whenever files are read or manipulated. Certainly, if the situation is going to remain the same, then this "feature" needs to be well-documented.

You can make a custom reader or Table class that always does the type conversion shown above and use that instead of the standard ascii.read or Table.read.

class KSL_Table(Table):
    def read(*args, **kwargs):
        out = super(KSL_Table, self).read(*args, **kwargs)
        for col in out.itercols():
            if col.dtype.kind in 'SU':
                out.replace_column(col.name, col.astype('object'))
        return out

Otherwise, once you are aware that this is an unavoidable behavior of numpy and it can happen in certain situations, then coding around it is typically not that problematic.

Thanks

@kslong

@kslong - can we close this? Unfortunately there is not an easy way to natively fix this given that Column is based on ndarray. I think that injecting custom behavior into __setitem__ would be tricky.

Yes, I agree it can be closed. But I think before it is done, that it should be documented somehow, in part so it is not completely forgotten. The problem can appear whenever you add rows to a table, e.g with vstack, or change string value, so this problem is going to crop up for many who manipulate tables containing strings. And the solution, converting strings to objects seems fairly robust to me, not one that necessarily should be part of the code, but one that should be made easy to find when the problem crops up.

@taldcroft

@taldcroft - just out of curiosity, what about making a mix-in string column type that can automatically adjust its width, or emit a warning if a situation like this occurs?

@taldcroft

@taldcroft - just out of curiosity, what about making a mix-in string column type that can automatically adjust its width, or emit a warning if a situation like this occurs?

Could be possible, but I'm not sure of the advantage over a regular np array of object type. This new column type would not be the default for strings so there would always be the friction at input/output of making sure it gets used. Also, I think the answer would be a new subclass of Column instead of a mix-in.

Thinking this over, while I am merely a consumer of astropy tables, I believe this problem should be dealt with properly, which means at a minimum issuing a warning whenever a string truncation occurs, and at a maximum solving the problem, e.g treating strings as objects in tables, or adding code to routines like add_row below, which have the ability to reformat the column.

Right, now if you create a table, and try to add a row to it, it truncates the string, e.g.

>>> x=Table([['a','b','c']])
>>> x
<Table length=3>
col0
str1
----
   a
   b
   c
>>> x.add_row([['hello']])
>>> x
<Table length=4>
col0
str1
----
   a
   b
   c
   h

The format is set at the time the table is formatted and, at present, there is no way to fix it without going through the procedure described of creating a new column with a different format.
Simply trying to set the format does not work, as far as I can tell.

I recognize this may be challenging (requiring possibly forcing strings into objects), since its not obvious how one would be able to issue a warning for this line.

x['col0'][1]='whatever'

but it would make astropy.tables much more bullet proof for the user.

See partial fix to at least emit warnings in #5624.

@astrofrog

@astrofrog - about the mixin column idea. An issue is that AFAIK you cannot do an in-place change of the dtype of a numpy array (to expand the string length). Therefore you must do one of the two:

Implement the expando-string column as a numpy container class, which I think will be messy to do right, OR
Implement the expando-string column as a Column subclass (or within Column itself) with a built-in limitation that it only does its job within a Table where there is the opportunity to replace the parent table column with a new expanded object.

The code for the latter is straightforward, but it introduces a new vector for bugs in creating a dropped reference. This is even more subtle than the table column replacement issue because it would happen for updating a single item.

>>> t = Table([['a', 'b']], names=['a long column name'])
>>> ta = t['a long column name']
>>> ta[0] = 'bbb'
>>> list(ta)
['a', 'b']  # damn, no update??
>>> list(t['a long column name'])
['bbb', 'b']

So we could play the same game of emitting warnings when this happens (see #5586). My gut feeling is that it is going to be better in the end to provide bona-fide variable-length strings via the object dtype, instead of the expando-string idea. BUT this is not totally clear to me so arguments are welcome. "Support" means good documentation and providing bits of glue here and there to make things easy. E.g. have io.fits serialize an object type using its __str__. And make it easier to specify column dtypes in io.ascii, the existing converters API is really crufty.

Was just bitten by this. +1 for visible warning. (Actually, an exception would be preferable to discarding data, imho.)

@taldcroft

I've been bitten by this numerous times when trying to put a Numpy string array into a Table column. Wanted to give this issue a bump. I definitely do NOT like @taldcroft's second suggestion. That's one nasty reference problem.

@SaraOgaz - is the warning not enough?

In [2]: c = Column(['a', 'b', 'c', 'd'])

In [3]: c[1:3] = np.array(['abc', 'def'])
/proj/sot/ska3/flight/arch/x86_64-linux_CentOS-6/bin/ipython:1: StringTruncateWarning: truncated right side string(s) longer than 1 character(s) during assignment
  #!/proj/sot/ska3/flight/arch/x86_64-linux_CentOS-5/bin/python

Hmm, when did that go in? I think I'm running into cases where the warning is never thrown, but I'll have to get back to you on that...

#5819 in 2.0

And #5624.

Ahh ok, I think the examples I've seen were still in 1.3. Will report back...

@taldcroft

Hi @taldcroft, at least where I'm concerned I think you can close this.

Navigation Menu

Repository navigation

astropy.table.Table - truncated strings #5551

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

astropy.table.Table - truncated strings #5551

Description

Activity

taldcroft commented on Dec 4, 2016

taldcroft commented on Dec 4, 2016

kslong commented on Dec 4, 2016

taldcroft commented on Dec 4, 2016

kslong commented on Dec 14, 2016

taldcroft commented on Dec 15, 2016

kslong commented on Dec 15, 2016

astrofrog commented on Dec 16, 2016

taldcroft commented on Dec 17, 2016

kslong commented on Dec 17, 2016

taldcroft commented on Dec 17, 2016

taldcroft commented on Dec 17, 2016

josePhoenix commented on May 23, 2017

SaOgaz commented on Sep 29, 2017

taldcroft commented on Sep 29, 2017

SaOgaz commented on Sep 29, 2017

taldcroft commented on Sep 29, 2017

taldcroft commented on Sep 29, 2017

SaOgaz commented on Sep 29, 2017

SaOgaz commented on Feb 6, 2018

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions