You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is a bit of a tracking issue: numpy is introducing __array_function__, with which one can override functionality of numpy functions for one's classes. To implement for:
Related to ShapedLikeNDArray here and #8593: is it desirable for numpy functions to apply to more complex objects like SkyCoord or even Time? In my mind, numpy functions are meant to return other ndarray's (so e.g. Quantity is fine), but the more complex objects that are really containers for multiple arrays start muddying the waters. So do we end up creating more confusion for the user and/or developers by doing this?
(Note I can be convinced either way on this... But I think we should at least have a discussion about it before applying the __array_function__ machinery this broadly...)
@eteq - I think not that many functions will end up being meaningful for Time and SkyCoord, but don't really see problems with shape-related ones (np.reshape, np.atleast_1d, etc.), especially since many of these work already (as they just look for a corresponding method). np.concatenate would then be quite nice too...
Similarly, having arithmetic things like np.max and np.mean work on Time seems OK (and, again, some of those work already as they just call the method); given that, it doesn't seem such a big jump to also add np.linspace (which just is syntactic sugar for start + (stop-start) * np.arange(n) / (n-1), something already works well for Time.
@adrn - that code has a bug - see #8877 - the current PR doesn't help that much (OOPS, thinking this is a different issue...), but it is indeed a nice example of something that could become possible if representations also gained an __array_function__.
@pllim - thanks for noticing the numpy-dev failures - one is due to numpy/numpy#13757 and I guess part of what should be done for #8825 (I'm surprised there is a failure, though), the other is due to numpy/numpy#13634 and also Quantity specific. Anyway, will try to get fixes up...
Actually, it turns out one of them brought an error in my tests to light, so it was OK. The other, with block is less clear, but we had discussion. Anyway, fix is on the way...
Sounds like good work on all this. So (at the risk of being contentious) does this mean that NumPy arithmetic functions can/should be made to work on NDData (which already has some equivalent methods)?? Certain things like log10 or power might be useful, eg. for duck-type compatibility? Then there are possibilities like median, which I think would raise the issue of supporting scalar NDData to hold an error & combined mask bits... Just wondering what people think (I expect the answer is let's finish start the long-planned work on refactoring NDData before worrying about new stuff).
@jehturner - yes, NDData is, to me, a prime candidate, should have thought about it! (Now added it on top.) The arithmetic functions are mostly ufuncs, and would go via __array_ufunc__ - those would give the option of ensuring that the uncertainty is properly propagated as well. (See #3715 - that one even tracks covariance, so that after b = a + a; c = b - a, c has the same uncertainties as a (rather than sqrt(3) times larger ones).
Great. Covariance would be a nice option (though I'm more worried about it in the context of interpolation, where you'd end up with large, sparse, higher-dimensional arrays). We accidentally ran into the problem you describe in our code the other day (not sure how you would go about tracking it between different arrays in general).
You clearly have a master plan here (I spotted some discussion about masked arrays as well)...
I'm adding a few more to the mix as encouragement for the impending Astropy-SKA hackathon...
Extend ShapedLikeNDArray to support the following functions:
np.shape
np.size
np.ndim
These map to properties instead of underlying methods and so probably need another whitelist beyond _APPLICABLE_FUNCTIONS. Or one could simply check whether the underlying attribute is callable, like SkyCoord does in #5751.
Currently we have
t=Time.now()
t.shape# ()np.shape(t) # TypeError: 'tuple' object is not callable
These seem kind of pointless if you have working properties anyway, until you realise that they are used under the hood in various interesting places to hide the difference between scalars and arrays:
np.r_[t, t] # TypeError: 'int' object is not callable
Oh wait... These used to work in v4.1. Could we restore this functionality?
Activity
eteq commentedon Apr 22, 2019
Related to
ShapedLikeNDArrayhere and #8593: is it desirable fornumpyfunctions to apply to more complex objects likeSkyCoordor evenTime? In my mind,numpyfunctions are meant to return otherndarray's (so e.g.Quantityis fine), but the more complex objects that are really containers for multiple arrays start muddying the waters. So do we end up creating more confusion for the user and/or developers by doing this?(Note I can be convinced either way on this... But I think we should at least have a discussion about it before applying the
__array_function__machinery this broadly...)mhvk commentedon Apr 25, 2019
@eteq - I think not that many functions will end up being meaningful for
TimeandSkyCoord, but don't really see problems with shape-related ones (np.reshape,np.atleast_1d, etc.), especially since many of these work already (as they just look for a corresponding method).np.concatenatewould then be quite nice too...Similarly, having arithmetic things like
np.maxandnp.meanwork onTimeseems OK (and, again, some of those work already as they just call the method); given that, it doesn't seem such a big jump to also addnp.linspace(which just is syntactic sugar forstart + (stop-start) * np.arange(n) / (n-1), something already works well forTime.For
SkyCoord, much less would be relevant...[-]Start using __array_function__[/-][+]Using __array_function__ beyond Quantity?[/+]adrn commentedon Jun 19, 2019
It could at least help clean up the custom concatenate code in coordinates...
mhvk commentedon Jun 19, 2019
@adrn - that code has a bug - see #8877 -
the current PR doesn't help that much(OOPS, thinking this is a different issue...), but it is indeed a nice example of something that could become possible if representations also gained an__array_function__.pllim commentedon Jun 20, 2019
Just to be clear, the failures in numpy-dev job (e.g., https://travis-ci.org/astropy/astropy/jobs/547998722) will be resolved by resolving this issue?
bsipocz commentedon Jun 20, 2019
Weird, it looks like #8841 came back?never mind, I looked at old logs somehow.mhvk commentedon Jun 20, 2019
@pllim - thanks for noticing the numpy-dev failures - one is due to numpy/numpy#13757 and I guess part of what should be done for #8825 (I'm surprised there is a failure, though), the other is due to numpy/numpy#13634 and also
Quantityspecific. Anyway, will try to get fixes up...pllim commentedon Jun 20, 2019
@mhvk, maybe Numpy should have a job to test against astropy-dev before merging. 😏
mhvk commentedon Jun 20, 2019
Actually, it turns out one of them brought an error in my tests to light, so it was OK. The other, with
blockis less clear, but we had discussion. Anyway, fix is on the way...jehturner commentedon Jun 21, 2019
Sounds like good work on all this. So (at the risk of being contentious) does this mean that NumPy arithmetic functions can/should be made to work on
NDData(which already has some equivalent methods)?? Certain things likelog10orpowermight be useful, eg. for duck-type compatibility? Then there are possibilities likemedian, which I think would raise the issue of supporting scalarNDDatato hold an error & combined mask bits... Just wondering what people think (I expect the answer is let'sfinishstart the long-planned work on refactoringNDDatabefore worrying about new stuff).mhvk commentedon Jun 21, 2019
@jehturner - yes,
NDDatais, to me, a prime candidate, should have thought about it! (Now added it on top.) The arithmetic functions are mostly ufuncs, and would go via__array_ufunc__- those would give the option of ensuring that the uncertainty is properly propagated as well. (See #3715 - that one even tracks covariance, so that afterb = a + a; c = b - a,chas the same uncertainties asa(rather thansqrt(3)times larger ones).jehturner commentedon Jun 21, 2019
Great. Covariance would be a nice option (though I'm more worried about it in the context of interpolation, where you'd end up with large, sparse, higher-dimensional arrays). We accidentally ran into the problem you describe in our code the other day (not sure how you would go about tracking it between different arrays in general).
You clearly have a master plan here (I spotted some discussion about masked arrays as well)...
ludwigschwardt commentedon Dec 1, 2020
I'm adding a few more to the mix as encouragement for the impending Astropy-SKA hackathon...
Extend
ShapedLikeNDArrayto support the following functions:np.shapenp.sizenp.ndimThese map to properties instead of underlying methods and so probably need another whitelist beyond
_APPLICABLE_FUNCTIONS. Or one could simply check whether the underlying attribute is callable, likeSkyCoorddoes in #5751.Currently we have
These seem kind of pointless if you have working properties anyway, until you realise that they are used under the hood in various interesting places to hide the difference between scalars and arrays:
Oh wait... These used to work in v4.1. Could we restore this functionality?
Fix Astropy 4.2 regression
mhvk commentedon Dec 4, 2020
Yikes, that is clearly a regression!! I'll try to make a fix...
ludwigschwardt commentedon Dec 4, 2020
Don't try too hard... I was hoping to offer this as a task for the SKA-Astropy Hackathon on Monday 😂
mhvk commentedon Dec 4, 2020
Oops, sorry, got the fix already. Well, not for
np.r_, but that is a trickier one anyway.... I've added it on top...