NEP 35: Finalize like= argument behaviour before 1.21 release

Before we release 1.20, we should finalize the behaviour. Or possibly hide/replace the API. Unless NEP 35 was officially accepted, there are still alternative suggestions such as np.get_array_module which may win the race.

There are also a few issues which require attention if we decide to keep the API largely untouched:

(This is currently NOT valid anymore, so it is the strict version) l = [1, 2]; np.asarray(l, like=l) is currently valid. However, modern API may want to only accept array-object as input and not objects such as lists/tuples, which the user should convert ahead of time. The above spelling could raise an error here, requiring the use of a different function (e.g. np.asduckarray()) to achieve the no-error version, which most existing API currently uses:
```
def strict_library(x):
     x = np.asarray(x, like=x)  # raise error if x is a list

def non_strict_library(x):
     x = np.asduckarray(x)
```
the alternative might be to force the use of a different API:
```
def strict_library(x):
     x = np.asarray(x, like=x, strict=True)
```
(Note that there might be a use for supporting an obj.__asduckarray__() method to mirror obj.__array__(), which would require np.asduckarray())
(This seems like an extension, since 1. is strict it is still possible) There may be a use for finding a common array-like for multiple input functions:
```
def function(arr1, arr2):
    tmp = np.arange(1000, like=np.common_arraylike(arr1, arr2))
```
it was previously discussed that this could be achieved by making like work on tuples: np.arange(1000, like=(arr1, arr2).
Ensure that the name (currently like=) is thought out and has been discussed sufficiently.
(seberg) Are we sure that some of these function should never dispatch based on the existing (usually scalar) input? (I guess so, but I thought I saw np.linspace in there, which made me wonder, but I suppose its not there). In any case, we may want to review each function more individually. (Linspace is not include, so this seems fine)
(We currently dispatch aliases) Should we dispatch the aliases np.asarray(), etc. or rather only use the full np.array() call. The reason for inclusion is currently that CuPy would have issues with the last version, because it doesn't support the subok argument (there are no cupy subclasses):

A few comments on the 4 points from the description:

This currently follows the same pattern of __array_function__, where dispatching on a list would return a np.ndarray, for example:
```
>>> import numpy as np
>>> np.ones_like([1, 2, 3])
array([1, 1, 1])
>>> np.concatenate(([1, 2], (3, 4)))
array([1, 2, 3, 4])
```
I think changing that would go against the behavior in the rest of __array_function__.
I'm still not certain if this is a job for NumPy, maybe it would make more sense that the application handles that? For example, Dask may want to implement a da.common_arraylike and decide what to do if the user passes (cupy_array, sparse_array), or any other special cases it wants to support, maybe?
The naming like= was mainly chosen and somewhat loosely agreed upon given its interchangeability with the name of empty_like and derived functions, where like= behaves similarly to the intent that _like as a suffix have in those functions.
I originally mentioned np.linspace in NEP-35 because I failed to notice that it already dispatches on __array_function__, that mention has been removed in NEP: Adjust NEP-35 to make it more user-accessible #17093 .

I'm still not certain if this is a job for NumPy, maybe it would make more sense that the application handles that?

I agree, a lot of effort was spent into making this work with weird corner cases, but that it was a good idea to do wasn't spelled out clearly anywhere. There may be a use case where one would like to mix something like a Pint array with a numpy array (not sure if that's a subclass), but things like mixing cupy and sparse arrays should just error out.

The naming like= was mainly chosen and somewhat loosely agreed upon given its interchangeability with the name of empty_like and derived functions, where like= behaves similarly to the intent that _like as a suffix have in those functions.

That's a little deceiving rather than helpful maybe. One takes over shape, dtype attributes, the other one only the array type.

I think changing that would go against the behavior in the rest of __array_function__.

It's very different though, like= is a keyword in functions that's clearly user-visible, and therefore should have unambiguous behaviour.

The __array_function__ link is of secondary importance, that's not what a user sees - it's an implementation detail. Saying like=list and then not getting a list back is odd.

I'm still not certain if this is a job for NumPy, maybe it would make more sense that the application handles that?

I agree, a lot of effort was spent into making this work with weird corner cases, but that it was a good idea to do wasn't spelled out clearly anywhere. There may be a use case where one would like to mix something like a Pint array with a numpy array (not sure if that's a subclass), but things like mixing cupy and sparse arrays should just error out.

I'm not sure if you're saying common_arraylike should or should NOT be part of NumPy. I agree some specific mixing isn't always desired, but it's still unclear whether this is NumPy's job to decide or not.

The naming like= was mainly chosen and somewhat loosely agreed upon given its interchangeability with the name of empty_like and derived functions, where like= behaves similarly to the intent that _like as a suffix have in those functions.

That's a little deceiving rather than helpful maybe. One takes over shape, dtype attributes, the other one only the array type.

Yes, but taking shape and dtype is part of empty_like's implementation, a downstream library could still choose not to implement or ignore those attributes.

I think changing that would go against the behavior in the rest of __array_function__.

It's very different though, like= is a keyword in functions that's clearly user-visible, and therefore should have unambiguous behaviour.

The __array_function__ link is of secondary importance, that's not what a user sees - it's an implementation detail. Saying like=list and then not getting a list back is odd.

But saying np.array(..., like=list) would also be deceiving then, is np.array supposed to give you an array or not?

Note that if we error when like=list, it would render like virtually useless in trying to make code more agnostic, consider the following example:

def my_pad(arr, padding):
    padding = np.asarray(padding, like=arr)
    return np.concatenate([padding, arr, padding])

my_pad([1, 2, 3], [0, 0])  # Errors because `arr` is a `list`

@seberg

But saying np.array(..., like=list) would also be deceiving then, is np.array supposed to give you an array or not?

np.array is supposed to give you an array.
When you use like= I expect "an array of type given by the like= parameter".
Returning "an array of type list" makes no sense. That's exactly why it seems logical to raise a TypeError instead.

Note that if we error when like=list, it would render like virtually useless in trying to make code more agnostic, consider the following example:
...

That's exactly why @seberg talks about __duckarray__.

Returning "an array of type list" makes no sense. That's exactly why it seems logical to raise a TypeError instead.

For more context: for creating new libraries, it would be great if there was a clean way of accepting multiple array types, but not accept any kind of sequence or generator. So separating the ultra-flexible (and error-prone) array_like stuff from the support for multiple array types seems like a good idea.

I'm not sure if you're saying common_arraylike should or should NOT be part of NumPy. I agree some specific mixing isn't always desired, but it's still unclear whether this is NumPy's job to decide or not.

I'm saying "I agree with you, it's unclear". Would be nice to go back and retro-actively do for NEP 18 what you just did for your NEP - add motivation, scope, use cases. That would sort this out.

I think the like argument discussion is a bit tricky, partially because we either need extremely complete use cases (which are difficult), or we need to sort out the basic concepts and ~~boil~~ actually map out the differences a bit theoretically (which is also difficult). Ideally, we probably should have both!

The difference in the like argument is seems a lot like the difference between implicit and explicit that NEP 37 details. The current (no error) version extends the implicit dispatching. The version which errors for non-array-object inputs fits much better with an explicit concept where the user picks explicitly what they want.

Now, you may be arguing: That doesn't make sense, both versions pick the output explicitly here! That is true, because there is only a single argument, making implicit and explicit pretty much the same thing.
Thus, my opinion on the implementation PR that we probably don't need to worry too much about it.

But, realizing that differences should show up for multiple arguments, we can remember np.linspace (which was even included in a first NEP draft!):

np.linspace(1, 2, num=10000, like=cupy_array)

very much looks like it should work, but doesn't, because we force you to spell it:

np.linspace(cupy.array(1), 2, num=10000)

Adding a like= argument to linspace is interesting, because the desired behaviour becomes muddled:

np.linspace(cupy.array(1), 2, num=10000, like=numpy_array)

looks a lot like "Try to create a NumPy array, or fail" (although I am sure you can give meaning to returning a cupy array here). But:

np.linspace(cupy.array(1), 2, num=10000, like=[1, 2, 3])

is probably something we can agree on being unclear as to what should be expected.

But, that in large parts circles back to the NEP 37 "We need something explicit" argument. So I don't know if it helps with the specific issue...

With respect to NEP 37, I think a problem is that we do not know how much we need explicit dispatching. E.g. we could also allow to write the above as:

np.call_dipatched(np.linspace, cupy_array)(1, 2, num=1000)

instead of adding like=. Which looks like a kludge, but maybe it could be propped into many of the remaining holes where we want explicit dispatching but __array_function__ doesn't offer it. (Although explicit vs. implicit was just one consideration where NEP 37 differs.)

I guess the main volatility may have been around whether to relax the like= to allow non-array-likes. But, that is the one thing we can also just do without much concern I think.

I guess the main volatility may have been around whether to relax the like= to allow non-array-likes. But, that is the one thing we can also just do without much concern I think.

Apologies for the late reply. I agree we could adjust later, but so far even that I'm not sure would be necessary. Therefore, I'm +1 on just accepting NEP-35 as is at this time, and we can revisit that later if a real use case shows up in the future.

NEP: Accept NEP 35 as final

Navigation Menu

Repository navigation

NEP 35: Finalize `like=` argument behaviour before 1.21 release #17075

30 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

NEP 35: Finalize like= argument behaviour before 1.21 release #17075

Description

Activity

pentschev commented on Aug 19, 2020

rgommers commented on Aug 21, 2020

rgommers commented on Aug 21, 2020

rgommers commented on Aug 21, 2020

pentschev commented on Aug 21, 2020

pentschev commented on Aug 21, 2020

pentschev commented on Aug 21, 2020

rgommers commented on Aug 21, 2020

rgommers commented on Aug 21, 2020

rgommers commented on Aug 21, 2020

seberg commented on Aug 21, 2020

seberg commented on Aug 21, 2020

30 remaining items

seberg commented on May 6, 2021

pentschev commented on May 10, 2021

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

NEP 35: Finalize `like=` argument behaviour before 1.21 release #17075