Repository navigation
Memory Leak #7714
Description
The following code has a memory leak with the current numpy version:
import numpy as np
import scipy
import scipy.stats
u = np.random.random(128**2)
for i in range(100000):
scipy.stats.norm.ppf(u)Checking with htop the memory increases as the loops are performed. With older versions of numpy the above code runs with very little memory usage and especially no increase over time.
I tested if the error is due to a bug in scipy but it isn't. Then I checked with which numpy version the error got introduced (since I realized it after an update of all my packages).
According to git bisect the commit that introduced the error is:
9128ed5
OS: Ubuntu 14.04.3 LTS
Python: sys.version_info(major=3, minor=4, micro=3, releaselevel='final', serial=0)
numpy: 1.12.0.dev0+cc51ba8
scipy: 0.18.0.dev0+41e2d00
Activity
charris commentedon Jun 7, 2016
@gfyoung This is yours.
gfyoung commentedon Jun 7, 2016
@aboettcher : Thanks for alerting us about this! Unfortunately, I can't seem to reproduce the problem. I ran your commands and monitored
htopas the loop was running and myMEM%and usage remained constant virtually the entire time. Can you provide further information about your architecture, Python version, andnumpy+scipyversions? Here are mine:OS: Ubuntu 14.04
Python: 2.7.11
numpy: v1.11.0
scipy: 0.18.0.dev0+e2c946c
Also, not really sure at the moment how this memory leak could have been triggered by those changes, as they seem quite distant from what you are running here. Perhaps a more experience C developer could illuminate possible problems with that commit.
aboettcher commentedon Jun 7, 2016
@gfyoung thank you for fast response. Those are my versions:
OS: Ubuntu 14.04.3 LTS
Python: sys.version_info(major=3, minor=4, micro=3, releaselevel='final', serial=0)
numpy: 1.12.0.dev0+cc51ba8
scipy: 0.18.0.dev0+41e2d00
I did not yet test python 2.7.
gfyoung commentedon Jun 7, 2016
@aboettcher : should have mentioned this initially, but could you actually put that system info in your original issue? I think that would be useful for people (maintainers especially) when they read it so they know the entire context, as your wording of "current numpy" threw me off since I thought you were talking about the latest released version.
aboettcher commentedon Jun 7, 2016
@gfyoung I tested your version of numpy (v1.11.0) and the memory leak is indeed not there.
Can you confirm the memory leak for this version?:
9128ed5
gfyoung commentedon Jun 7, 2016
@aboettcher : okay, so at least we know it was introduced after
v1.11.0- however, it did seem strange thatgitwould get it wrong withbisect, so I checked the release and it turns out that PR wasn't included. Sigh...I'll double check now againstmasterto see if I can replicate.charris commentedon Jun 7, 2016
@gfyoung I do see that commit in v1.11.0, but if the bug doesn't show up there it suggests a bisection failure.
aboettcher commentedon Jun 7, 2016
@gfyoung I am sorry for giving the wrong pointer. I trusted bisect at this point.
gfyoung commentedon Jun 7, 2016
Sorry, before we relegate it to a bisect failure, @charris , can you show me where you're seeing that commit show up in
v1.11.0? I couldn't find it anywhere in the maintenance branch nor in the taggedv1.11.0version of the C code that I fixed here.gfyoung commentedon Jun 7, 2016
@aboettcher : I am able to replicate the memory leak using the bleeding versions of
scipy(5a779fd) andnumpy(175476f) on the architecture and Python I gave earlier.charris commentedon Jun 7, 2016
@aboettcher
charris commentedon Jun 7, 2016
Changes in Scipy may have exposed the problem. There is also the possibly related (unlikely?) report #7710.
gfyoung commentedon Jun 7, 2016
@charris : I hate to keep pushing this, but that doesn't convince me. When I do
git checkout v1.11.0and then I rungit log 175476fec7(that's the LRU cache PR that you just merged), I still get output for that commit.aboettcher commentedon Jun 7, 2016
@charris I also cannot find it on github.
At least the changes or not visible here:
https://github.com/numpy/numpy/blob/v1.11.0/numpy/core/src/multiarray/compiled_base.c
charris commentedon Jun 7, 2016
@gfyoung Seems like pilot error on my part, git log also finds it in 1.8.x. Probably need to make git log only search reachable from HEAD.
charris commentedon Jun 7, 2016
OK, this works better
git log --grep 'BUG: Fix string copy'. Looks like searching by hash looks into the whole tree. Learn something everyday ;)gfyoung commentedon Jun 7, 2016
Me too! 😄
In any case, I will run
git bisectbetweenv1.11.0andmasterand see where the bug lies. I'll also take a look to see whether there are sources of improvement upon that PR.aboettcher commentedon Jun 7, 2016
I am currently rerunning bisect with
git bisect good v1.11.1rc1git bisect bad 175476fec763984573e60485a447978cbcb381c0(newest version). Will report the result soon.aboettcher commentedon Jun 7, 2016
The rerun of bisect gave the same result as I reported above (9128ed5)
gfyoung commentedon Jun 7, 2016
Yep, I can confirm as well. It's my PR...now off to figuring out why. 😕