Repository navigation
Issue with large memory usage when creating a memory mapped array #7732
Description
I am trying to create a memory mapped array. The following example:
import numpy as np
array = np.memmap('test.np', shape=(1000, 1000, 1000), dtype=float, mode='write')
array[:] = 1.2ends up taking 8Gb of memory:
The array[:] = 1.2 assignment starts at t=20 seconds.
If the assignment is done with:
for i in range(1000):
array[i] = 1.2this doesn't actually help:
If I manually flush between each assignment, it takes a lot longer and still doesn't help:
for i in range(1000):
array[i] = 1.2
array.flush()
Is there any way to create this type of memory mapped array while keeping the memory usage low?
Activity
seberg commentedon Jun 12, 2016
Not sure how it works, but might it not be your OS grabbing the memory because it is available, and it would not do it if it was not free?
njsmith commentedon Jun 12, 2016
Yeah, I think that's all working correctly and doing what you want, and your tool is just counting "in memory cache of the mmap'ed file" memory into the same accounting bucket that it uses for regular allocations of non-file-backed memory. Which is actually fairly reasonable; the fundamental problem is that in a virtual memory world there isn't really any meaningful single number answer to "how much memory does this process use". People feel like there should be, so lots of tools will obligingly make something up, but it's very misleading in lots of situations, including this one.
I've been trying to think how to answer more helpfully, but I think really the only solution if you care about these things is to do some study on how virtual memory works, because the conceptual framework is fundamentally different from the simple model we teach beginners.