Repository navigation
Support for memmap in Quantity #13864
Description
Description
I if possible, would like the Quantity object to support memmap arrays, meaning that given a memmap array, a quantity constructed from that array remains memory mapped.
Additional context
It is possible that Quantity supports this, but it remains unclear. Right now if I run:
import numpy as np
import astropy.units as u
a = np.memmap('test', mode="w+", shape=(3,4))
q1 = u.Quantity(a, u.kg)
q2 = u.Quantity(a, u.kg, copy=False)
print(f'{isinstance(a, np.memmap)=}')
print(f'{isinstance(q1, np.memmap)=}')
print(f'{isinstance(q1.value, np.memmap)=}')
print(f'{isinstance(q2, np.memmap)=}')
print(f'{isinstance(q2.value, np.memmap)=}')It returns
isinstance(a, np.memmap)=True
isinstance(q1, np.memmap)=False
isinstance(q1.value, np.memmap)=False
isinstance(q2, np.memmap)=False
isinstance(q2.value, np.memmap)=False
Which from the memmap documentation indicates to me that the the data is no longer memmap.
Basically, I would like a way to pass a memmap array to a Quantity and that Quantity to use the array data through the memmap (ie the data on disk mapped via the memmap to the array gets updated when I update the Quantity). This includes a way to interrogate the Quantity to confirm it is or is not using a memmap.
Activity
WilliamJamieson commentedon Oct 20, 2022
@mhvk and @nstarman do you have any idea of the state of
memmapsupport forQuantity?nstarman commentedon Oct 20, 2022
That would be very useful. I don't believe it's possible with the current implementation of Quantity, which subclasses
ndarray, notmemmap.WilliamJamieson commentedon Oct 20, 2022
My familiarity with the intricacies of numpy arrays is limited. I did notice this in the
memmapdocumentation:I wonder if this indicates that it might be possible to make
Quantityammapobject? If so, how would I do this? Again not super familiar with the intricacies involved.mhvk commentedon Oct 20, 2022
In principle, it should be possible to make this work, since
np.memmapis anndarraysubclass, but right now our use ofndarrayis perhaps a bit too much built in. The suggestion you got from the documentation does work:WilliamJamieson commentedon Oct 20, 2022
I guess this means there isn't an "easy" test for if
Quantityis doing thismemmapbehavior akin to theisinstance(q, np.memmap)suggested by numpy?mhvk commentedon Oct 20, 2022
Indeed, not immediately. I briefly tried whether it would work to define a new class
QuantityMemmap(u.Quantity, np.memmap)but that in itself did not seem good enough. I have not investigated why, though think in principle it should be possible to get it to work; it might need the kind of automatic class construction that was used forDistributionandMaskedand which @nstarman has been wanting to generalize. With that,Quantitymight also be able to wrap dask arrays, etc.