I've noticed a few issues with Pytorch and garbage collecting. Python tensor objects have GPU memory allocated to them plus CPU memory in the C++ layer, while the garbage collector is only aware (I think) of the CPU memory the interpreter allocates so it doesn't know when these need collecting. The collector in Python tries to run only when it needs to because memory is running out, it has an incorrect read on what this is however. Almost always I would say that invoking collection is bad but this is the only way to do this now.

Pytorch also likes to keep cached buffersaround internally so collecting won't bring down allocated GPU memory as much as you'd expect often. You need torch.cuda.empty_cache() to do that but I would instead leave that up to Pytorch unless you're getting unexpected OOM exceptions.

One suggestion I'd make is to run gc.collect() twice in a row to get a few more objects cleaned up.

thanks @behxyz I'll make sure this is merged for v0.5. (I'm cancelling this CI for now as we'll merge the high priority ones first)

FYI came across this new error (perhaps the first time in 20+ runs)

======================================================================
FAIL: test_content_0 (tests.test_handler_garbage_collector.TestHandlerGarbageCollector)

----------------------------------------------------------------------

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
    return func(*(a + p.args), **p.kwargs)
  File "/__w/MONAI/MONAI/tests/test_handler_garbage_collector.py", line 73, in test_content
    self.assertLess(gb_count[0], first_count)
AssertionError: 44 not less than 38

----------------------------------------------------------------------