Repository navigation
Possible fits memmap bug: memmap just doesn't work. #1380
Description
I'm trying to load some gigantic FITS record tables using memmap=True, and I'm getting error: [Errno 12] Cannot allocate memory.
An example session:
filename = '/home/sdfits/AGBT12B_221_01/AGBT12B_221_01.raw.acs.fits'
import astropy.io.fits as fits
filefits = fits.open(filename,memmap=True)
data = filefits[2].data[:50]
The error is at this line:
/users/aginsbur/anaconda/lib/python2.7/site-packages/numpy/core/memmap.py(253)__new__()
--> 253 mm = mmap.mmap(fid.fileno(), bytes, access=acc, offset=start)
ipdb> bytes
23718381056L
ipdb> bytes/1024**2
22619L
ipdb> start
413921280
ipdb> acc
3
I don't really know what's going on, but I suspect memmap is improperly deciding on how much data to read. Any tips on how to further debug? Is this actually a FITS issue, or a numpy issue?
Details:
In [15]: numpy.__version__
Out[15]: '1.7.1'
In [16]: astropy.__version__
Out[16]: '0.2.4'
In [18]: sys.maxint
Out[18]: 9223372036854775807
Activity
embray commentedon Aug 27, 2013
What OS?
What does
ulimit -vreturn?keflavich commentedon Aug 27, 2013
OS is some flavor of linux; don't know off the top of my head or the
easiest command to find out.
Also, was using anaconda install of python/astropy/numpy but upgraded
astropy via pip.
$ ulimit -v
unlimited
On Tue, Aug 27, 2013 at 4:02 PM, Erik Bray [email protected] wrote:
Adam
embray commentedon Aug 28, 2013
What does
cat /proc/meminfoshow?keflavich commentedon Aug 28, 2013
keflavich commentedon Aug 28, 2013
...that seems like a tiny amount of memory; 2 GB? Hrmph.
astrofrog commentedon Nov 30, 2013
@keflavich - are you still seeing this issue?
embray commentedon Dec 2, 2013
Totally forgot about this.
MemTotalis just the total available physical memory. These 2GB is not a lot, sure, but that shouldn't be the issue. You have about 32 TB forVmallocTotalwhich is what should matter here--in principle mmap should be able to use most of that. So there's something fishy going on here.embray commentedon Dec 2, 2013
Ah! I think I see the issue here. By default PyFITS uses the
MAP_PRIVATEwhen opening a file in readonly mode so that users can still modify the data array in place as they would if the entire file were mapped into main memory.The problem is, that means in principle the entire file can be overwritten, so mmap needs to be able to allocate enough memory ahead of time should that occur. That's why this is happening here. PyFITS/Astropy should definitely catch that scenario and provide a more helpful error.
Currently there are two ways around this: You can open the file with
mode='denywrite. I added that a while ago specifically for this case, but it's rarely used. That opens the mmap withMAP_SHARED | PROT_READ--this means the pages are read-only (any attempt to modify the array will result in an exception). But if all you need is to read the data this works fine, and doesn't require allocating any swap space I don't think.Another possibility is to open with
mode='update'. Then any changes to the array can be synced directly back to the file which is fine if you want that, but obviously not so much if you don't.Looking at the man page, it looks like there's also a flag, at least on Linux, called
MAP_NORESERVEwhich will prevent it from pre-allocating space for copy-on-write. So if you don't need to write any changes to the entire array that could work too. But we'd have to be able to catch the SIGSEGV that results if you do end up running out of swap space.embray commentedon Dec 4, 2013
Managed to reproduce this directly--indeed, both of the workarounds I offered (
mode='denywrite'andmode='update'work. Will still try to see what I can do aboutMAP_NORESERVE, and otherwise catching this error and providing a better error message.embray commentedon Dec 4, 2013
Annoyance:
numpy.memmapdoesn't allow tweaking the flags that are passed to themmapcall. Though looking at it, it's not much more than a light subclass ofndarraythat handles the work of creating an mmap with the right flags and then callingndarray.__new__with the mmap as its buffer. It also adds aflushmethod.It should be easy enough to just eschew use of
numpy.memmapat all and handle mmap ourselves. But that's still more than I want to do on this for now. So instead I'll resolve this issue by catching the error and suggesting one of the existing workarounds.saimn commentedon Aug 27, 2018
Note that with #7597 we no more use
np.memmap, so it should easier to use other flags if that is useful.astrofrog commentedon Oct 24, 2018
This can now be closed, as a workaround has been merged in #7926.
MAP_NORESERVEis not available from Python, so that isn't a solution unfortunately.