You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@mhvk co-authored NEP 42 and it is accepted by Numpy. How will this affect astropy.units and astropy.constants? Do they need a complete refactor? Is NEP 42 sufficient for all the weird unit conversions that astronomers do (looking at you, magnitude systems)?
NEP 42 allows one to place the unit on the dtype, which I think will mean a considerable simplification of the system. It should be quite possible to keep Quantity, though in practice it may become more of a light-weight ndarray subclass (or, perhaps more easily work also with dask arrays, etc.)
I saw @seberg's talk earlier today and it looks super awesome. For what it's worth I would be totally onboard with doing unusual things with the core dependencies and/or deprecation policy for massive wins in units, such as requiring a substantially newer than usual numpy release.
NEP 42 allows one to place the unit on the dtype, which I think will mean a considerable simplification of the system.
Agreed.
It should be quite possible to keep Quantity
I think we will definitely want to!
All the Quantity subclasses that add behavior beyond just unit propagation: angle wrapping, automatic equivalencies
Support for non-numpy data. Some useful libraries, like JAX, might never support units as dtypes. And it would be great to have mpmath + units — but mpmath doesn't use numpy dtypes at all. Upstreaming much of units to NumPy will probably be great, but we should be careful that array calculations are separable from unit propagation. NumPy dtype offers tools, like result_type, and hopefully it will have tools for calculating a dtype without associated numbers. e.g. determine_dtype(np.sin, u.deg) => u.dimensionlesss or determine_dtype(np.multiply, u.deg, u.deg) => u.deg**2. Alternatively, we have similar overrides already in astropy and might continue using them. With this, we could have numpy dtype units and support libraries like JAX, mpmath, etc.
The pangeo/xarray people are also talking about it a bit. They seem to tend towards pint, but also wonder if they can experiment with NEP 42. I am wondering if we should try to at least do one meeting to see what the differences between pint and astropy.units are.
I am more than happy if NEP 42 is not useful for units. One thing I do wonder about is whether we can at least avoid having too many unit implementations even if we need many quantity implementations.
I think it makes sense to adopt NEP 42 -- it won't obviate all of the special functionality in astropy.units or Quantity so I agree that we should preserve our Quantity functionality.
One thing I do wonder about is whether we can at least avoid having too many unit implementations even if we need many quantity implementations.
That's an interesting point and sounds like it might make for an easier to cut shared infrastructure! I would be interested in a meeting to discuss and share ideas between interested parties.
@WilliamJamieson - if the units become part of the dtype, then definitely np.memmap could function as a Quantity. As discussed in #13864, even now it seems one can use an mmap (not memmap) inside Quantity.
I just want to weigh in here as the main unyt maintainer and also as a person who is actively working on NEP-42 as part of my job at Quansight Labs that I'm very interested in this discussion. I think there's definitely a real path forward to substantially simplifying and speeding up both unyt and astropy.units by making both depend on a common unit dtype implementation that does unit computations in a low-level language like C, C++, or rust. As alluded to above, we would still need packages like unyt and astropy.units to provide a nice user-facing API (since dtypes can't inject methods onto ndarray and I doubt they ever will) and to maintain backward compatibility with existing code.
I'd be happy to remotely attend any meetings about this that happen at the next astropy coordination meeting. I'd also be interested in exploring opportunities for funding opportunities this project if that makes sense.
@ngoldbaum - thanks! Your issues over at numpy are clearly helping to improve the actual user type implementation!
Agreed that there is room to speed up the unit conversion machinery -- thought there are different parts to that. E.g., adding arrays with different but compatible units, there is the pure units part of calculating the conversion factor (or function) and then the values/array part on how to implement the conversion. Both can likely be sped up. For instance, for the actual operation, we just multiply the second array with the conversion factor to get it in the unit of the first, then add values, and then attach a unit again. This could all be done in the user dtype. But using fused multiply-add might well give a fair speed-up. For other operations, it might be that doing the multiplication inside the iterator, and thus avoiding the creation of potentially very large arrays, might speed up matters (though some experiments I did with "chained ufuncs" suggested the benefits were smaller than I had hoped).
For the unit calculations proper, @nstarman has suggested that mypyc might help to get a substantial speed-up without having to go to lower-level languages.
Anyway, am definitely interested in pursuing this and happy to meet (though realistically with teaching starting soon not sure how much time I'll have... but I'm on sabbatical next academic year...).
Activity
mhvk commentedon Jul 15, 2022
NEP 42 allows one to place the unit on the dtype, which I think will mean a considerable simplification of the system. It should be quite possible to keep
Quantity, though in practice it may become more of a light-weight ndarray subclass (or, perhaps more easily work also with dask arrays, etc.)mhvk commentedon Jul 15, 2022
Let me add a link to an experiment with units under the dtype put together by @seberg - https://github.com/seberg/unitdtype
I think we should be able to substitute our machinery...
Cadair commentedon Jul 15, 2022
I saw @seberg's talk earlier today and it looks super awesome. For what it's worth I would be totally onboard with doing unusual things with the core dependencies and/or deprecation policy for massive wins in units, such as requiring a substantially newer than usual numpy release.
nstarman commentedon Jul 16, 2022
Agreed.
I think we will definitely want to!
mpmath+ units — but mpmath doesn't use numpy dtypes at all. Upstreaming much of units to NumPy will probably be great, but we should be careful that array calculations are separable from unit propagation. NumPy dtype offers tools, likeresult_type, and hopefully it will have tools for calculating a dtype without associated numbers. e.g.determine_dtype(np.sin, u.deg) => u.dimensionlesssordetermine_dtype(np.multiply, u.deg, u.deg) => u.deg**2. Alternatively, we have similar overrides already in astropy and might continue using them. With this, we could have numpy dtype units and support libraries like JAX, mpmath, etc.seberg commentedon Jul 16, 2022
The pangeo/xarray people are also talking about it a bit. They seem to tend towards pint, but also wonder if they can experiment with NEP 42. I am wondering if we should try to at least do one meeting to see what the differences between pint and astropy.units are.
I am more than happy if NEP 42 is not useful for units. One thing I do wonder about is whether we can at least avoid having too many unit implementations even if we need many quantity implementations.
adrn commentedon Jul 18, 2022
I think it makes sense to adopt NEP 42 -- it won't obviate all of the special functionality in astropy.units or
Quantityso I agree that we should preserve ourQuantityfunctionality.That's an interesting point and sounds like it might make for an easier to cut shared infrastructure! I would be interested in a meeting to discuss and share ideas between interested parties.
pllim commentedon Jul 18, 2022
Units conference? Unitscon? Unicon? 🦄
seberg commentedon Jul 18, 2022
I have created this thread in the hope that we can at least keep everyone in the loop of new developments: https://discuss.scientific-python.org/t/unitsxd-units-across-domains-coordination/349 (or maybe use it to coordinate a meeting if that is to be).
pllim commentedon Jul 18, 2022
Whatever we do, we also want to make sure it won't make fixes for things like these impossible (the list is not exhaustive):
pllim commentedon Oct 10, 2022
Looks like there is another thread in addition to the existing ones. FYI.
https://discuss.scientific-python.org/t/multiple-duck-array-layers-composability-implementations-and-usability-concerns/552
WilliamJamieson commentedon Oct 20, 2022
Would this enable us to
memmap(mmap) the data in aQuantity? See #13864.mhvk commentedon Oct 20, 2022
@WilliamJamieson - if the units become part of the
dtype, then definitelynp.memmapcould function as aQuantity. As discussed in #13864, even now it seems one can use anmmap(notmemmap) insideQuantity.ngoldbaum commentedon Jan 2, 2023
I just want to weigh in here as the main unyt maintainer and also as a person who is actively working on NEP-42 as part of my job at Quansight Labs that I'm very interested in this discussion. I think there's definitely a real path forward to substantially simplifying and speeding up both unyt and astropy.units by making both depend on a common unit dtype implementation that does unit computations in a low-level language like C, C++, or rust. As alluded to above, we would still need packages like unyt and astropy.units to provide a nice user-facing API (since dtypes can't inject methods onto ndarray and I doubt they ever will) and to maintain backward compatibility with existing code.
I'd be happy to remotely attend any meetings about this that happen at the next astropy coordination meeting. I'd also be interested in exploring opportunities for funding opportunities this project if that makes sense.
mhvk commentedon Jan 2, 2023
@ngoldbaum - thanks! Your issues over at numpy are clearly helping to improve the actual user type implementation!
Agreed that there is room to speed up the unit conversion machinery -- thought there are different parts to that. E.g., adding arrays with different but compatible units, there is the pure units part of calculating the conversion factor (or function) and then the values/array part on how to implement the conversion. Both can likely be sped up. For instance, for the actual operation, we just multiply the second array with the conversion factor to get it in the unit of the first, then add values, and then attach a unit again. This could all be done in the user dtype. But using fused multiply-add might well give a fair speed-up. For other operations, it might be that doing the multiplication inside the iterator, and thus avoiding the creation of potentially very large arrays, might speed up matters (though some experiments I did with "chained ufuncs" suggested the benefits were smaller than I had hoped).
For the unit calculations proper, @nstarman has suggested that
mypycmight help to get a substantial speed-up without having to go to lower-level languages.Anyway, am definitely interested in pursuing this and happy to meet (though realistically with teaching starting soon not sure how much time I'll have... but I'm on sabbatical next academic year...).
pllim commentedon Jul 14, 2023
Would this be useful? https://data-apis.org/array-api-tests/
MridulS commentedon Feb 28, 2024
Just wanted to put this here for future reference:
https://numpy.org/devdocs/release/2.0.0-notes.html#new-public-dtype-api
The new DType API is in, so if we want to have another go at astropy.units this seems like a good time :D