Repository navigation
Cannot cast one-element object array of empty string to string dtype on 1.21.0rc1 #19085
Description
There's an admittedly edge case that changed behaviour, probably due to the type promotion changes:
>>> import numpy as np
>>> np.array([''], dtype=object).astype('U')On 1.20.3 this returns
array([''], dtype='<U1')whereas on the 1.21.0rc1 release candidate this is an error:
>>> np.array([''], dtype=object).astype('U')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Cannot cast array data from dtype('O') to dtype('<U') according to the rule 'unsafe'I note that casting scalar strings or non-empty arrays still works:
>>> np.array('', dtype=object).astype('U')
array('', dtype='<U1')
>>> np.array(['a'], dtype=object).astype('U')
array(['a'], dtype='<U1')Is this new error expected?
Activity
seberg commentedon May 24, 2021
I am not immediately sure why this is happening, but it must indeed be an error in the promotion changes (this is very recent). Astype should see the
Uand then infer the correct string size from the object array as if you diddtype="U"directly.EDIT: OK, I tracked down the issue, but still have to figur out the correct solution... There are some corner cases, that casting to
"U"has to figure out the size (but cannot normally, althoughastypecan). Sonp.can_cast("O", "U", casting="unsafe")is maybe OK to fail, but.astype("U")is not. The fix depends a bit on whether the first should stay failing, or that should also be taped over.shoyer commentedon May 26, 2021
We hit this in Xarray's test suite as well.
seberg commentedon May 26, 2021
I brought it up briefly on the meeting today. My preference is to not worry about
np.can_cast("O", "U", casting="unsafe")(in theory, this is different fromnp.can_cast("O", "U0", casting="unsafe"), in practice, we are not there yet.There may be another solution available, though: I do have a "default" casting level stored (that could be checked without figuring out the actual string length). I can possibly assume that if the
"U"(without a length) is given, it is sufficient to compare the default fornp.can_cast, because the resolver has the freedom to pick a string that matches (normally, in the case of object to string it will bail, but not forarr.astype("U")!).(I have to think this through, in theory its simple, but in generaly it probably does not work, because the first argument may also be the parametric one. E.g. casting an arbitrary precision floating point dtype to a double must enter the "resolution" step to check the length of the arbitrary precision float (if it is high precision, its "safe", if its short it is "same kind"). We could return a "minimum" casting safety, but that may just shift the problem and the "default" was also useful for some other legacy work-arounds I think – and that could collide.)
Problem recap:
np.can_cast("O", "U", casting="unsafe")should maybe returnTrue, since no matter what string length you would use, it is considered "unsafe". But, NumPy has no way to pick a string length (it could guess something, it used to guess 64-bytes sometimes). Since we would have to guess the correct string length Not being able to pick a string length, an error is raised (internally), which then is suppressed and the cast is considered as impossible.This is problematic, because we don't (yet) distinguish cleanly between
"U0"(zero length) and"U"(undefined length).mattip commentedon May 27, 2021
What about putting a shim into
PyArray_CanCastArrayTobefore this line to explicitly handle this case, witha test and/orperhaps a deprecation warning?seberg commentedon Jun 5, 2021
Well, hopefully I got a complete solution mostly, but not quite yet :/.
My first try, was creating a new set of function
PyArray_CanCastArrayToDescrwhich doesn't do the special casing forS0to meanS. That works forarr.astype(), but I thought I can be more minimal. (Although we should change everything to such new functions anyway.)A second try, trying to be minimal was to rearrange
arr.astype("S")a bit, so that it replaces theS0withS1before checking the casting safety...Both of these fail to solve
np.can_cast(arr, "S", casting="unsafe"). My previous thought was thatcan_castis maybe not important, but I was thinking of thenp.can_cast("O", "S", casting="unsafe")signature. Thenp.can_cast(arr, "S", casting='unsafe')has the same issue and seems more annoying to keep broken and unaligned witharr.astype()?My current try, is the idea of using a "minimal" casting that can be checked before the actual resolution (which fails here). That is, defining
castingimpl->castingto be set to a minimal casting level (i.e. no matter what is passed as thetodescriptor). That should work more or less:str(object)never fails. (For arbitrary user dtypes it would not necessarily alignnp.can_cast("O", DType)withobject_arr.astype(DType), I think)That is more extensive than I expected, but it is probably the best solution. This seems to work fine, but I have some incorrect sanity checks in my code to track down and one test is still failing...
@charris, hopefully, I will just have a good solution tomorrow. Unless we go the "minimal" route, maybe it would make sense to publish the bi-weekly wheels for downstream testing before the rc?
charris commentedon Jun 5, 2021
Maybe. Things still feel a bit up in the air.
charris commentedon Jun 7, 2021
Let's wait to close this until we get some feedback from downstream.
adeak commentedon Jun 7, 2021
Sorry, I was only following emails so I didn't realize that you've pushed a fix.
I can confirm that current numpy main doesn't raise this error in our test suite. Thank you for the fix :)