Repository navigation
Severe memory leak in populate_mock() #917
Description
Run the following test case and we observe the RSS size increase continuously. There is no apparent reason for this to happen reading from the face value of the code.
def test_hearin15():
"""
"""
model = PrebuiltHodModelFactory('hearin15')
try:
halocat = CachedHaloCatalog()
except:
halocat = FakeSim()
import resource
for i in range(100):
model.populate_mock(halocat)
maxrss = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss
print(maxrss)
assert maxrss < 1000000
Result
../testenv/lib/python3.6/site-packages/halotools/empirical_models/composite_models/tests/test_preloaded_models.py 169816
185612
199336
216228
230180
230180
230180
241580
257364
276064
286356
291108
302724
315828
330084
345660
359916
...
Activity
[-]Severe memory leak in `populate_mocks()`[/-][+]Severe memory leak in `populate_mock()`[/+]aphearin commentedon Jul 3, 2018
I cannot reproduce your memory leak results using python 2x, @rainwoodman. I have not yet tried python 3.x but will try this soon.
I can say that the
populate_mockfunction was not intended to be used this way. That is a one-time-per-simulation function. After you have called it once, repeated calls on the same simulation should be done usingmodel.mock.populate(). Since I cannot reproduce your memory leak test, could you report the results of modifying your function so thatpopulate_mockis called prior to the loop, and thenmodel.mock.populateis called inside the loop?It is also worth trying to see whether simply forcing garbage collection resolves the issue, which is what @johannesulf and @surhudm found in the closely related #568.
rainwoodman commentedon Jul 3, 2018
Sorry I was testing this with Python 3.6, actually. This is an updated script, which uses
mock.populate()the same behavior unless gc is invoked. This indicates there are circular references in the data model. I think if we get rid of the circular references, this can be fixed without requiring triggering gc. Circular references used to be considered as bugs in the old days.and here is my conda environment list:
aphearin commentedon Jul 3, 2018
@rainwoodman - since there is a perfectly good workaround, can you suggest how to make this explicit call to garbage collect more discoverable in the documentation? Resolving this memory leak could be quite involved, and most of my effort on Halotools now is geared towards Issue #825, so that Halotools scales with the number of available nodes.
rainwoodman commentedon Jul 3, 2018
It's alright. I will give this a go. I may need to introduce some weak references. I'll comment on 825.
rainwoodman commentedon Jul 4, 2018
I can confirm updating astropy to the master branch (included 6277) fixed bulk of memory leak.
#918 further reduced this to a more conservative growth:
johannesulf commentedon Aug 13, 2018
I made a recent test with python3.6, numpy1.14.5, astropy3.0.4 (included 6277), halotools0.6 and the script above. Even without #918, I don't have a memory leak. However, there seems to be a consistent memory leak (even when calling gc.collect and #918) when using numpy 1.15. With numpy1.14.5 there doesn't seem to be a problem. It's probably unrelated to the commit here and maybe not caused by halotools directly. But I just wanted to mention it because it's related.