Add funcs to convert table cols to/from unicode/bytestring - #1974

In general, I agree with the Unicode sandwich concept, however it ignores the reality of really large data structures that need to be shared with C in memory or memmapped on disk.

I wonder how this impacts things like FITS, where the dtype of a string column must be bytes in order to have mem-mapping from disk work, etc.

I wonder if instead a sort of proxy that converts a single column to unicode when accessing it, and back to bytes when setting it, as I think @embray has threatened to do, wouldn't work better.

I also thought about a proxy but fairly quickly gave up on the idea, guessing that it would be a tricky thing to do in complete and robust way. That would certainly be better, but this is meant as a stopgap solution that I think would work in 90% of real world cases. This is purely an "opt-in if it works for you" utility, but of course if consensus is that it's not the right way forward then it shouldn't be in astropy.

I think the best would be a numpy 'ascii' dtype, like 'U' but with ascii encoding. It seems like that would allow a pretty easy virtual remapping from e.g. 'S10' to 'A10' with no need to copy memory. It seems like this should not be technically that difficult since the 'U" machinery is already there for UCS-4, but I don't know.

Indeed, the unicode sandwich you describe here is exactly what PyFITS/astropy.io.fits does for string columns in FITS tables. This actually causes a huge hassle particularly for large tables with either many thousands or millions of rows, or a large number of string columns. Right now the entire column is converted, in memory, to unicode as soon as any row in the column. So even if most of the columns can be read from a mmap you get stuck with this huge column of strings in memory using 4 bytes per character. See this thread for an example of where this can get really ugly: https://groups.google.com/d/msg/astropy-dev/zRIZ6rr4JPg/Qs9IHNTqKPAJ

This could be alleviated partially with more sophisticated code that can handle on-demand conversion of values in a column, possibly caching intermediate results. This is something I've been thinking about the best ways to do for a long time, as there are other parts of the FITS format (besides strings) that would benefit from an efficient way to do this generically.

But in the meantime, for the string case in particular, I think the "proxy" @mdboom mentioned is the way to go, and so far my experiments suggest that it would work. It would require all the strings in a column to have the same encoding, which I think is a reasonable assumption in the vast majority of cases (in particular for most science data that is going to include mostly ASCII strings).

The proxy would have a particular encoding associated with it--the strings would all still be stored in their "native" format of encoded bytes. Whenever displaying a value in the column to the user (like with repr(), etc.) simply decode the string on the fly with the assigned encoding. Likewise when retrieving a single value from the array just immediately decode it to text. For now this wouldn't involve any caching since string decoding is generally fast. Likewise storing values would automatically encode in the assigned encoding (if possible--otherwise an exception is rightfully raised). This is a little tricky on the user's end if using 'utf-8' and wanting to store unicode strings, because the dtype of the array only tells you how many bytes you can store in a single value, not how many characters. So the user might think they can fit a 10 character string in an S10 array, but if any of the characters in their string are encoded to more than 1 byte they'll get a slightly surprising exception. But I think this is still a reasonable middle ground.

That's another option that's sort of in-between. Originally in Python 3 unicode strings were always stored internally as UCS-4. But that was obviously terrible for most strings, so they did a lot of rework and changed things so that internal storage format for unicode text was determined by the character with the highest codepoint in the string. In other words, if all characters in the string are ASCII then they are stored internally as ASCII. It was easy and fast enough when joining strings to upconvert from ASCII to UCS-2 or UCS-4 as needed. It's fast enough since the string object itself has a flag in its type struct specifying what "kind" of string it is. Numpy really ought to do the same for the 'U' dtype. But until and unless it gets fixed we need something else.

I suppose we could write our own Numpy dtype that has this sort of behavior.

I should add: One reason that this sort of thing is easier to do for string arrays than for general arrays is that most if not all of the ufuncs in Numpy don't work on strings :)

I started thinking about this harder (and playing a bit in ipython) and came to the same realization that lack of support for all (I think) ufuncs and even operators like + bring this into the realm of the possible. Interestingly even ufuncs that ought to work, don't. For a as a bytestring array and b as a unicode array:

In [21]: np.greater(a, a)
Out[21]: NotImplemented

In [22]: a > b
Out[22]: 
array([[False, False],
       [False, False]], dtype=bool)

But this does bring up the issue of comparisons. (Note that assignment of unicode to a bytestring array and vice versa already seems to work fine).

In [52]: a
Out[52]: 
array([b'a'], 
      dtype='|S1')

In [53]: a > 'b'
ERROR: TypeError: unorderable types: numpy.ndarray() > str() [IPython.core.interactiveshell]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-53-d258357f34e7> in <module>()
----> 1 a > 'b'

TypeError: unorderable types: numpy.ndarray() > str()

In [54]: a > b'b'
Out[54]: array([False], dtype=bool)

In [55]: a == 'b'
Out[55]: False   # WRONG, should be bool array

In [56]: a == b'b'
Out[56]: array([False], dtype=bool)

So you need to overload comparisons and have the machinery to convert the other side to the appropriate bytestring array. Maybe that's as simple as other = np.array(other, dtype=np.bytes_)?

Indeed. I still need to poke at comparisons a bit, but I already have a prototype of this under way that I will show in a bit. The other thing I'm working out is how to handle things like .view(), but I have some ideas for that which are bearing fruit.

Another question that comes up is how to handle __setitem__ and the like. If given a unicode string or iterable thereof it's easy enough to know what to do--just encode the string in the target encoding and assign. But what if one assigns a bytes object? Should that even be allowed? I'd inclined to say, "no". If you want to directly assign bytes then view the array as a normal ndarray or go through the array's .base.

If we do allow bytes the other question is do we just allow arbitrary bytes, or do we check that they are valid for the array's encoding (by attempting to decode them first)?

If you're interested, here's a rough implementation of this idea that I've worked on incorporating into PyFITS: https://github.com/embray/PyFITS/blob/stringy/lib/pyfits/fitsrec.py#L999

The incorporation is still somewhat partial, but I've gotten all the tests passing at least (in all supported Python 2.x and 3.x versions). I still haven't done any performance testing but I suspect this will be much better (especially for tables containing large string columns). This implementation also has a few FITS-isms built into it, such as the fact that it subclasses np.chararray. This is necessary for backwards compatibility since PyFITS has historically used the chararray class for string arrays. But I hope to get away from that in the future since chararray is basically deprecated.

I should also explain one of the "trickier" parts of this implementation: Given an existing Numpy string array (specifically an 'S' array) like:

a = np.array([b'abc', b'def'])

there are two ways to make an encoded_text_array, which must have an encoding specified. The first, albeit a bit clunkier, is to use its constructor directly like:

a2 = encoded_text_array(a.shape, itemsize=a.itemsize, encoding='ascii', buffer=a)

In general it's much easier to use a.view(), but the API for ndarray.view does not allow passing through custom arguments to the array finalizer, making it not clear how to specify an encoding. For that purpose use encoded_text_array.with_encoding:

a3 = a.view(encoded_text_array.with_encoding('ascii'))

This actually creates a new subclass of encoded_text_array on the fly with the correct encoding "hard-coded" as a class attribute. These are also cached to reduce class creation overhead.

I'm also not 100% sure about comparison. Equal and no-equal are straightforward enough I think. It's greater-than/less-than I'm not sure about. When comparing arrays of unicode strings does it compare their code points? Whereas with encoded strings I assume it's just doing a byte-by-byte comparison. Depending on the encoding this could give very different results for non-ASCII characters. But for PyFITS' case I'm not worried about that difference. Just something to be aware of...

In case people haven't been paying attention to numpy-discussion, it looks like there is movement toward getting a 1-byte unicode type (latin-1 or ascii) implemented in numpy. I have a particular itch to scratch and have been making a case.

http://mail.scipy.org/pipermail/numpy-discussion/2014-January/068622.html

Great--I don't follow the Numpy mailing lists, so I'm glad you're following that discussion. I'll have to read up and see if there's anywhere I can jump in too. Because I think Numpy really needs this so that applications using it can get the Unicode sandwiching correct. In the meantime my little hack works pretty well for many uses cases (it mostly falls over on mixed-width encodings a la utf8. It still 'works', but it's not very user-friendly since it makes it difficult to work with such an array on a per-character basis.)

Though I don't like the implicit "latin1" idea at all. That's sloppy, and falls over if you try to mix data of different encodings. A much better approach would be to do what Python itself does, as I described earlier. It still uses fixed-width encodings so strides will still work, but it will use either 1-byte ASCII, UCS-2, or UCS-4 internally depending on the highest codepoint in the string (that's an oversimplification, but the basic idea). I would repurpose the existing 'U' dtype for this, but with some indicator of which internal representation it's using for a given array.

Maybe I'll jump on that thread after all...

I think the point of the h5py dev that using 'ascii' encoding ensures byte compatibility for conversion to 'utf-8' is a good one. And with ascii encoding there is no sloppiness.

Yeah, I'm fine with ascii for that purpose, but not latin1 :)

@embray @mdboom - although there are better solutions on the horizon, would you be OK with going forward on this workaround for now? When the better solutions are available then we could deprecate and then remove these helper functions.

@taldcroft - this looks good, but I'm wondering if we can also expose the general change_dtype method? Sometimes I want to stack tables with columns that are S5 and S10 and it would be nice to easily be able to say convert from S5 to S10. What do you think?

@astrofrog - the existing code won't handle your use case even if exposed since it just works for a particular kind, and in fact the internal method is only sure to work for S and U. So a general change_dtype might be a fine idea, but it's out of scope here and would be for the next release.

Also, I'm a little confused about your question because the astropy vstack will automatically upgrade the S5 to S10 for stacking.

@taldcroft - ok, ignore my request for now, I may have been basing it on old behavior.

I'll let @embray or @mdboom comment on this, then we can merge it if there are no objections.

I'm just worried about the potential for whining that will ensue from users saying "I opened my 1GB table and it used up for 4GB of memory!!!" I get this enough with pyfits as it is.

Seems fine to me. A long term solution might be a custom dtype that implicitly converts between unicode and ascii, but in the meantime, I think this is fine as a stopgap.

I am (or was, until getting distracted by release activities) working on exactly this. It's been slow going learning all the ins and outs of adding a new dtype to Numpy. Parts of it are straightforward--other parts a bit confusing. But it should work in principle.

My encoded_text_array solution works okay too, but I think ultimately a new dtype will be more robust.

I think the desired long term solution is a built-in numpy bytestring type that plays well with Python 3. It's on the numpy dev radar but we probably need to keep pushing it. In the meantime I'm going to merge this. I'm hoping these functions will be deprecated soon enough, but the really good solutions tend to take a little while.

Well if I end up working something like this and getting it working I'll definitely submit it to Numpy if nobody else does it first. I need this pretty badly for FITS tables.