Add an ASCII String Type for Python 3 Compatibility

The transition towards split string/bytes types in Python 3 has the unfortunate side effect of breaking the following snippet:

np.array("Hello", dtype="|S").item() == "Hello"

which returns True on Python 2 and False on Python 3. To aid the porting of Python 2 applications it would be nice if there was an ASCII string data type ("|A", perhaps?) which will always give a string rather than a bytes array, with the restriction that only strings which can be encoded as ASCII can be assigned.

np.array("Hello", dtype="|S").item().decode() == "Hello" works in both Python 2.x and 3.x.

The string situation is already complicated enough, introducing an extra string type is not a good idea imho.

This kind of suggestion is probably better made on the mailing list by the way.

Yes, this is best on the mailing list (and it is there now), but since it's here:

It looks like the 'S' type is still there, and it's a one-byte per charater type, and it gets printed as text. So it alredy IS a type suitable for ascii text. I think it should be documented as such.

so the remaining question is:

What python type should it be converted to by default when indexing an item. It appears to create a bytes type now, but as bytes are not text in Py3, that may not be right.

I suggest that it should create a unicode object, decoded as ASCII (or maybe latin-1). It could also use the default_encoding, but I think that's a little too ripe for the mess that currently exists in py2 now...

NOTE: I say latin-1 because it is one byte per char, is ASCII compatible, and while you may get garbage for higher value bytes, it won't barf, and it does preserve the values through encoding/decoding.

I personally have code that uses S to mean "arbitrary binary"and I'm sure
I'm not the only one. (Very useful in decoding struct-based binary files.)
Declaring these strings to be ascii is right out.

Latin1 would be a pretty ugly hack, and anyway I bet there's also plenty of
code at this point that would break if we just started returning str
instead of bytes.

S is pretty quirky anyway (note that pandas just ignores it entirely and
uses dtype=object for string arrays!).

I think the only practical solution is to clean up dtypes so that defining
a better string dtype is possible, and then do that. Let me know if you
want to help...
On 15 Jan 2014 21:45, "Chris Barker" [email protected] wrote:

Yes, this is best on the mailing list (and it is there now), but since
it's here:

It looks like the 'S' type is still there, and it's a one-byte per
charater type, and it gets printed as text. So it alredy IS a type suitable
for ascii text. I think it should be documented as such.

so the remaining question is:

What python type should it be converted to by default when indexing an
item. It appears to create a bytes type now, but as bytes are not text in
Py3, that may not be right.

I suggest that it should create a unicode object, decoded as ASCII (or
maybe latin-1). It could also use the default_encoding, but I think that's
a little too ripe for the mess that currently exists in py2 now...

NOTE: I say latin-1 because it is one byte per char, is ASCII compatible,
and while you may get garbage for higher value bytes, it won't barf, and it
does preserve the values through encoding/decoding.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4162#issuecomment-32418605
.

On 15/01/14 22:36, njsmith wrote:

I personally have code that uses S to mean "arbitrary binary"and I'm sure
I'm not the only one. (Very useful in decoding struct-based binary files.)
Declaring these strings to be ascii is right out.

To play Devil's advocate here for a second. I have code which assumes S
to be a string-like type, as in something which can be compared with a
Python string. Byte arrays in Python 3 do not fulfil this criterion and
as such have code which breaks as a consequence.

I note that the documentation defines 'S', along with 'a' (ASCII?), to
be a string type. Isn't 'V' ("raw data (void)") a more natural data
type when one does not want the bytes to be subject to interpretation?

However, the fact that many people use 'S' for storing byte-type data is
precisely why I believe there is a need for a new dtype: one which is
clearly defined to be a simple (latin1 or ASCII) string.

Regards, Freddie.

Yes, but my code works on the version of numpy we're shipping now.
Backwards compatibility is important.

In any case we come to basically the same conclusion. The practical problem
is that deficiencies in numpy's dtype system mean that any patch to add a
new string-like dtype now would be too ugly and unmaintainable to merge; we
need to fix the dtype system first for this to be feasible.
On 15 Jan 2014 22:51, "Freddie Witherden" [email protected] wrote:

On 15/01/14 22:36, njsmith wrote:

I personally have code that uses S to mean "arbitrary binary"and I'm sure
I'm not the only one. (Very useful in decoding struct-based binary
files.)
Declaring these strings to be ascii is right out.

To play Devil's advocate here for a second. I have code which assumes S
to be a string-like type, as in something which can be compared with a
Python string. Byte arrays in Python 3 do not fulfil this criterion and
as such have code which breaks as a consequence.

I note that the documentation defines 'S', along with 'a' (ASCII?), to
be a string type. Isn't 'V' ("raw data (void)") a more natural data
type when one does not want the bytes to be subject to interpretation?

However, the fact that many people use 'S' for storing byte-type data is
precisely why I believe there is a need for a new dtype: one which is
clearly defined to be a simple (latin1 or ASCII) string.

Regards, Freddie.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4162#issuecomment-32424343
.

On Jan 15, 2014, at 5:29 PM, njsmith [email protected] wrote:

Yes, but my code works on the version of numpy we're shipping now.
Backwards compatibility is important.

Indeed, so that cat may be out of the bag. I guess I should have been
paying attention earlier in the py3 development cycle ;-)

Though devil's advocate: if we changed this now, folks with working
code on py3 would have things break. But arguably, fewer folks would
see breakage moving from 2 to 3 -- some breakage is inevitable.

Better to break on a numpy version or a python version?

In general better to break on the switch to py3: it is officially
supposed to break things.

But in this case I'm not so sure, given the level of py3 adoption.

any patch to add a
new string-like dtype now would be too ugly and unmaintainable to merge; we
need to fix the dtype system first for this to be feasible.

I was arguing that we may not want two dtypes that are that similar,
even if it was manageable to add another.

-Chris

Deprecating 'S' was already discussed when adding python 3 support: http://article.gmane.org/gmane.comp.python.numeric.general/34586. Apparently it just wasn't done.

'S' was not deprecated because there is no better letter available. 'B' is already reserved for uint8.
'S' is equivalent to bytes and 'U' to unicode both on Py2 and Py3.
Py3 does not do any automatic decoding/encoding between the two, and Numpy currently follows suit.

On 17 Jan 2014 12:31, "Pauli Virtanen" [email protected] wrote:

Py3 does not do any automatic decoding/encoding between the two, and
Numpy currently follows suit.

Actually, as shown in the original report's example, that's not quite
true... np.array("hello", dtype="|S") silently coerces "hello" to b"hello"
(using the ASCII codec). It should probably be an error instead...

Ah, that's true. I wonder how much code relying on this there is...

" 'S' is equivalent to bytes and 'U' to unicode both on Py2 and Py3."

The trick is that bytes is not the same thing on Py2 and Py3 -- on Py2, bytes is the same as str:
In [4]: b = bytes('this')

In [5]: b
Out[5]: 'this'

In [6]: type(b)
Out[6]: str

Whereas in py3, some of the text processing functionality has been removed (though not all -- bytes having .upper(), etc methods is just plain weird...

The Python devs are very reluctant to add anything back, and are quite clear that bytes is NOT to be considered as a reasonable way to process text. And IMHO, all the auto-encoding stuff was a mistake in the first place...

However, in scientific use, there is an even bigger use case for one-byte-per-character-text -- I see it as the equivalent to numpy providing lower-precision numeric types. So it would be nice if numpy had such a type, and 'S' is really really close, but it doesn't do the expected thing under py3 anymore in that context.

As above, people use 's' for arbitrary bytes, though I'm still confused as to why -- but breaking code is a bad thing.

It seems odd that the limitation here is that we don't have a letter to use to name it, but what about lower case s?: It's apparently not in use:

In [13]: np.zeros((3,), dtype='s')

TypeError Traceback (most recent call last)
in ()
----> 1 np.zeros((3,), dtype='s')

TypeError: data type "s" not understood

Then we could remove the auto encoding from 'S', too.....

This is EXACTLY the stuff that is supposed to break in the py2 to py3 conversion, so maybe not so bad.

-Chris

As far as the binary data itself is concerned, bytes is the same thing on Py2 and Py3, whereas str is not.
This was largely the reason for why things work as they do in Numpy currently on Py3.

It's also worth noting that the Py3 folk are very aware of the need for
better handling of legacy one-byte-per-character text -- I guess in this
respect scientific programmers have a lot in common with people working on
protocols like HTTP. E.g. Nick Coghlan calls this out as the primary
remaining deficiency in Py3, from the POV of the core developers:

http://python-notes.curiousefficiency.org/en/latest/python3/binary_protocols.html

http://python-notes.curiousefficiency.org/en/latest/python3/questions_and_answers.html#room-for-improvement
https://github.com/jeamland/asciicompat

If you're going after this seriously you might want to join forces with
them.

On Fri, Jan 17, 2014 at 5:29 PM, Chris Barker [email protected]:

" 'S' is equivalent to bytes and 'U' to unicode both on Py2 and Py3."

The trick is that bytes is not the same thing on Py2 and Py3 -- on Py2,
bytes is the same as str:
In [4]: b = bytes('this')

In [5]: b
Out[5]: 'this'

In [6]: type(b)
Out[6]: str

Whereas in py3, some of the text processing functionality has been removed
(though not all -- bytes having .upper(), etc methods is just plain
weird...

The Python devs are very reluctant to add anything back, and are quite
clear that bytes is NOT to be considered as a reasonable way to process
text. And IMHO, all the auto-encoding stuff was a mistake in the first
place...

However, in scientific use, there is an even bigger use case for
one-byte-per-character-text -- I see it as the equivalent to numpy
providing lower-precision numeric types. So it would be nice if numpy had
such a type, and 'S' is really really close, but it doesn't do the expected
thing under py3 anymore in that context.

As above, people use 's' for arbitrary bytes, though I'm still confused as
to why -- but breaking code is a bad thing.

It seems odd that the limitation here is that we don't have a letter to
use to name it, but what about lower case s?: It's apparently not in use:
In [13]: np.zeros((3,), dtype='s')

TypeError Traceback (most recent call last)
in ()
----> 1 np.zeros((3,), dtype='s')

TypeError: data type "s" not understood

Then we could remove the auto encoding from 'S', too.....

This is EXACTLY the stuff that is supposed to break in the py2 to py3
conversion, so maybe not so bad.

-Chris

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/4162#issuecomment-32626229
.

Nathaniel J. Smith
Postdoctoral researcher - Informatics - University of Edinburgh
http://vorpus.org

Navigation Menu

Repository navigation

Add an ASCII String Type for Python 3 Compatibility #4162

In [13]: np.zeros((3,), dtype='s')

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Add an ASCII String Type for Python 3 Compatibility #4162

Description

Activity

rgommers commented on Jan 1, 2014

rgommers commented on Jan 1, 2014

ChrisBarker-NOAA commented on Jan 15, 2014

njsmith commented on Jan 15, 2014

FreddieWitherden commented on Jan 15, 2014

njsmith commented on Jan 16, 2014

ChrisBarker-NOAA commented on Jan 16, 2014

rgommers commented on Jan 16, 2014

pv commented on Jan 17, 2014

njsmith commented on Jan 17, 2014

pv commented on Jan 17, 2014

ChrisBarker-NOAA commented on Jan 17, 2014

In [13]: np.zeros((3,), dtype='s')

pv commented on Jan 17, 2014

njsmith commented on Jan 17, 2014

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions