Use errors='replace' in decoding bytes data in Py3 - #6145

Canceling the travis build to rebase this on the very latest master.

cc: @mhvk

Sorry for the extra work. I'm OK with this approach -- in the end, the choice is to treat S dtype as a utf-8 string, so it seems OK to replace stuff to ensure one always gets the same result. It does imply that equality tests will now pass for more than one corrupted value, but that seems a reasonable price to pay.

Does this affect dev or release?

I guess this is getting to be merged asap, so I've cancelled the master build of #6142 to clear up a tiny bit of the backlog.

@mhvk - true...

In [8]: t2 = Table([[b'Z\xf1']])

In [9]: t1 = Table([[b'Z\xf0']])

In [10]: t1['col0'] == t2['col0']  # Bytes get compared
Out[10]: array([False], dtype=bool)

In [11]: t1['col0'][0] == t2['col0'][0]  # Unicode gets compared
Out[11]: True

Using an encoding which is subject to decode errors is basis of the problem. Hence all the discussion on numpy-dev related to latin-1, and my original implementation using latin-1. Yes, it is arbitrary and not actually the correct encoding most of the time, BUT it would give False in Out[11].

I fear there is no really good solution to this -- maybe I need to get to work to implement setting the decoding after all... Anyway, for now, I think we'll have to live with this annoyance.

One issue I think is that the data can be modified silently (which is different from #6117 where the data was modified but only for printing if I'm correct ?). What would you think about doing the replace in a try/except which would allow to issue a warning ?

@saimn - maybe I'm not understanding, but I don't see how the decode operation can in any way leak back to modify table data. This is the internal version of __getitem__ and the decode makes a completely new reference that gets returned.

@taldcroft - Ah sorry, I was thinking about the case of a new table created via slicing, but I missed the fact that the decode is done only for scalar values (not value.shape). (And only chararray has a .decode method anyway, I got confused with the io.fits code).