Skip to content

Using __array_function__ beyond Quantity? #8610

Open

Description

@mhvk

This is a bit of a tracking issue: numpy is introducing __array_function__, with which one can override functionality of numpy functions for one's classes. To implement for:

Activity

  1. eteq commented on Apr 22, 2019

    @eteq
    Member

    Related to ShapedLikeNDArray here and #8593: is it desirable for numpy functions to apply to more complex objects like SkyCoord or even Time? In my mind, numpy functions are meant to return other ndarray's (so e.g. Quantity is fine), but the more complex objects that are really containers for multiple arrays start muddying the waters. So do we end up creating more confusion for the user and/or developers by doing this?

    (Note I can be convinced either way on this... But I think we should at least have a discussion about it before applying the __array_function__ machinery this broadly...)

  2. mhvk commented on Apr 25, 2019

    @mhvk
    ContributorAuthor

    @eteq - I think not that many functions will end up being meaningful for Time and SkyCoord, but don't really see problems with shape-related ones (np.reshape, np.atleast_1d, etc.), especially since many of these work already (as they just look for a corresponding method). np.concatenate would then be quite nice too...

    Similarly, having arithmetic things like np.max and np.mean work on Time seems OK (and, again, some of those work already as they just call the method); given that, it doesn't seem such a big jump to also add np.linspace (which just is syntactic sugar for start + (stop-start) * np.arange(n) / (n-1), something already works well for Time.

    For SkyCoord, much less would be relevant...

  3. changed the title [-]Start using __array_function__[/-] [+]Using __array_function__ beyond Quantity?[/+] on Jun 19, 2019
  4. adrn commented on Jun 19, 2019

    @adrn
    Member

    It could at least help clean up the custom concatenate code in coordinates...

  5. mhvk commented on Jun 19, 2019

    @mhvk
    ContributorAuthor

    @adrn - that code has a bug - see #8877 - the current PR doesn't help that much (OOPS, thinking this is a different issue...), but it is indeed a nice example of something that could become possible if representations also gained an __array_function__.

  6. pllim commented on Jun 20, 2019

    @pllim
    Member

    Just to be clear, the failures in numpy-dev job (e.g., https://travis-ci.org/astropy/astropy/jobs/547998722) will be resolved by resolving this issue?

  7. bsipocz commented on Jun 20, 2019

    @bsipocz
    Member

    Weird, it looks like #8841 came back? never mind, I looked at old logs somehow.

  8. mhvk commented on Jun 20, 2019

    @mhvk
    ContributorAuthor

    @pllim - thanks for noticing the numpy-dev failures - one is due to numpy/numpy#13757 and I guess part of what should be done for #8825 (I'm surprised there is a failure, though), the other is due to numpy/numpy#13634 and also Quantity specific. Anyway, will try to get fixes up...

  9. pllim commented on Jun 20, 2019

    @pllim
    Member

    @mhvk, maybe Numpy should have a job to test against astropy-dev before merging. 😏

  10. mhvk commented on Jun 20, 2019

    @mhvk
    ContributorAuthor

    Actually, it turns out one of them brought an error in my tests to light, so it was OK. The other, with block is less clear, but we had discussion. Anyway, fix is on the way...

  11. jehturner commented on Jun 21, 2019

    @jehturner
    Member

    Sounds like good work on all this. So (at the risk of being contentious) does this mean that NumPy arithmetic functions can/should be made to work on NDData (which already has some equivalent methods)?? Certain things like log10 or power might be useful, eg. for duck-type compatibility? Then there are possibilities like median, which I think would raise the issue of supporting scalar NDData to hold an error & combined mask bits... Just wondering what people think (I expect the answer is let's finish start the long-planned work on refactoring NDData before worrying about new stuff).

  12. mhvk commented on Jun 21, 2019

    @mhvk
    ContributorAuthor

    @jehturner - yes, NDData is, to me, a prime candidate, should have thought about it! (Now added it on top.) The arithmetic functions are mostly ufuncs, and would go via __array_ufunc__ - those would give the option of ensuring that the uncertainty is properly propagated as well. (See #3715 - that one even tracks covariance, so that after b = a + a; c = b - a, c has the same uncertainties as a (rather than sqrt(3) times larger ones).

  13. jehturner commented on Jun 21, 2019

    @jehturner
    Member

    Great. Covariance would be a nice option (though I'm more worried about it in the context of interpolation, where you'd end up with large, sparse, higher-dimensional arrays). We accidentally ran into the problem you describe in our code the other day (not sure how you would go about tracking it between different arrays in general).

    You clearly have a master plan here (I spotted some discussion about masked arrays as well)...

  14. ludwigschwardt commented on Dec 1, 2020

    @ludwigschwardt
    Contributor

    I'm adding a few more to the mix as encouragement for the impending Astropy-SKA hackathon...

    Extend ShapedLikeNDArray to support the following functions:

    • np.shape
    • np.size
    • np.ndim

    These map to properties instead of underlying methods and so probably need another whitelist beyond _APPLICABLE_FUNCTIONS. Or one could simply check whether the underlying attribute is callable, like SkyCoord does in #5751.

    Currently we have

    t = Time.now()
    t.shape   # ()
    np.shape(t)  # TypeError: 'tuple' object is not callable

    These seem kind of pointless if you have working properties anyway, until you realise that they are used under the hood in various interesting places to hide the difference between scalars and arrays:

    np.r_[t, t]  # TypeError: 'int' object is not callable

    Oh wait... These used to work in v4.1. Could we restore this functionality?

  15. added a commit that references this issue on Dec 1, 2020
  16. mhvk commented on Dec 4, 2020

    @mhvk
    ContributorAuthor

    Yikes, that is clearly a regression!! I'll try to make a fix...

  17. ludwigschwardt commented on Dec 4, 2020

    @ludwigschwardt
    Contributor

    Don't try too hard... I was hoping to offer this as a task for the SKA-Astropy Hackathon on Monday 😂

  18. mhvk commented on Dec 4, 2020

    @mhvk
    ContributorAuthor

    Oops, sorry, got the fix already. Well, not for np.r_, but that is a trickier one anyway.... I've added it on top...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions