Large memory overhead change in numpy==1.19.5 vs 1.19.4 on ubuntu 20.04 #18141
Description
Background
We run our compile runs with daemontools to lock virtual memory usage to around 1GB per process (important when running very very parallel builds). During update from 1.19.4 -> 1.19.5 our nightlies trip over code that uses numpy.
Fine with numpy==1.19.4 (1GB):
softlimit -a $(expr 1024 \* 1024 \* 1024) make
Needed to build with numpy==1.19.5 (1.7GB):
softlimit -a $(expr 1740 \* 1024 \* 1024) make
I've whittled it down to the smallest possible reproduction which is simply importing numpy version 1.19.5 with 1GB virtual memory limit. See "Reproducing code example"
To examine further I ratcheted the limit up/down with 1.19.4 and 1.9.5 to find the point at which import succeeds.
| version | approx minimum import limit |
|---|---|
| 1.19.4 | softlimit -a 400000000 python3 |
| 1.19.5 | softlimit -a 1200000000 python3 |
| 1.20.0rc2 | softlimit -a 1200000000 python3 |
| 1.19.3 | softlimit -a 1200000000 python3 |
| 1.19.2 | softlimit -a 400000000 python3 |
Note this is with nothing but import numpy. The overhead difference ~800MB matches the limit changes we needed to get builds running.
Reproducing code example:
You can reproduce this just with loading of the module for version 1.19.5 on ubuntu:20.04 with daemontools. See below for Dockerfile to set up clean 20.04 to reproduce.
import numpy as npInvoked with softlimit -a 1073741824 python3
root@88e4e56b16a1:~# pip3 install numpy==1.19.5
Collecting numpy==1.19.5
Downloading numpy-1.19.5-cp38-cp38-manylinux2010_x86_64.whl (14.9 MB)
|████████████████████████████████| 14.9 MB 10.7 MB/s
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.19.4
Uninstalling numpy-1.19.4:
Successfully uninstalled numpy-1.19.4
Successfully installed numpy-1.19.5
root@88e4e56b16a1:~# softlimit -a 1073741824 python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/dist-packages/numpy/__init__.py", line 143, in <module>
from . import lib
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/__init__.py", line 40, in <module>
from .arraypad import *
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/arraypad.py", line 533, in <module>
def pad(array, pad_width, mode='constant', **kwargs):
File "/usr/local/lib/python3.8/dist-packages/numpy/core/overrides.py", line 180, in decorator
source_object = compile(
MemoryError
>>>
But ok with 1.19.4:
root@88e4e56b16a1:~# pip3 install numpy==1.19.4
Collecting numpy==1.19.4
Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
Installing collected packages: numpy
Attempting uninstall: numpy
Found existing installation: numpy 1.19.5
Uninstalling numpy-1.19.5:
Successfully uninstalled numpy-1.19.5
Successfully installed numpy-1.19.4
root@88e4e56b16a1:~# softlimit -a 1073741824 python3
Python 3.8.5 (default, Jul 28 2020, 12:59:40)
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>
Dockerfile for reproducing - vanilla 20.04 + python3 + pip + softlimit
cat Dockerfile
from ubuntu:20.04
RUN apt-get update && apt-get install -y \
python3 python3-pip \
daemontools \
&& apt-get clean
docker build -t numpylimit .
docker run --rm -it numpylimit /bin/bash
Error message:
>>> import numpy
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib/python3.8/dist-packages/numpy/__init__.py", line 143, in <module>
from . import lib
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/__init__.py", line 40, in <module>
from .arraypad import *
File "/usr/local/lib/python3.8/dist-packages/numpy/lib/arraypad.py", line 533, in <module>
def pad(array, pad_width, mode='constant', **kwargs):
File "/usr/local/lib/python3.8/dist-packages/numpy/core/overrides.py", line 180, in decorator
source_object = compile(
MemoryError
NumPy/Python version information:
1.19.5
Let me know if there is any more information I can give you.