Repository navigation
Memory Leak in libllvmlite.so when using Numba #9731
Description
Reporting a bug
Hello,
I am encountering a memory leak issue when using Numba. It appears that the problem originates from the libllvmlite.so library. I used valgrind to identify the memory leaks and received the following output:
grep "definitely lost" -A 20 valgrind_output.txt
==149875== 32 bytes in 2 blocks are definitely lost in loss record 639 of 3,197
==149875== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x1255287B: LLVMCreatePassManager (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0xF3FB3AF: LLVMPY_CreatePassManager (passmanagers.cpp:66)
==149875== by 0xA46CB15: ??? (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA4693EE: ??? (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA46C0BD: ffi_call (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA75D8DB: _call_function_pointer (callproc.c:931)
==149875== by 0xA75E2FE: _ctypes_callproc (callproc.c:1273)
==149875== by 0xA75756D: PyCFuncPtr_call (_ctypes.c:4167)
==149875== by 0x4B894E: _PyObject_Call (call.c:367)
==149875== by 0x4B8A1B: PyObject_Call (call.c:379)
==149875== by 0x5AE25A: _PyEval_EvalFrameDefault (bytecodes.c:3254)
==149875==
==149875== 33 bytes in 1 blocks are still reachable in loss record 640 of 3,197
==149875== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x127509B6: std::pair<llvm::StringMapIterator<llvm::cl::Option*>, bool> llvm::StringMap<llvm::cl::Option*, llvm::MallocAllocator>::try_emplace<llvm::cl::Option*>(llvm::StringRef, llvm::cl::Option*&&) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x12753969: (anonymous namespace)::CommandLineParser::addOption(llvm::cl::Option*, llvm::cl::SubCommand*) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x1275457C: llvm::cl::Option::addArgument() (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0xF328686: _GLOBAL__sub_I_WebAssemblyUtilities.cpp (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x400571E: call_init.part.0 (dl-init.c:74)
--
==149875== 128 bytes in 2 blocks are definitely lost in loss record 2,816 of 3,197
==149875== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x400E150: malloc (rtld-malloc.h:56)
==149875== by 0x400E150: resize_scopes (dl-open.c:295)
==149875== by 0x400E150: dl_open_worker_begin (dl-open.c:728)
==149875== by 0x400151B: _dl_catch_exception (dl-catch.c:237)
==149875== by 0x400CD1F: dl_open_worker (dl-open.c:803)
==149875== by 0x400151B: _dl_catch_exception (dl-catch.c:237)
==149875== by 0x400D163: _dl_open (dl-open.c:905)
==149875== by 0x4A38193: dlopen_doit (dlopen.c:56)
==149875== by 0x400151B: _dl_catch_exception (dl-catch.c:237)
==149875== by 0x4001668: _dl_catch_error (dl-catch.c:256)
==149875== by 0x4A37C72: _dlerror_run (dlerror.c:138)
==149875== by 0x4A3824E: dlopen_implementation (dlopen.c:71)
==149875== by 0x4A3824E: dlopen@@GLIBC_2.34 (dlopen.c:81)
==149875== by 0x62B09A: _PyImport_FindSharedFuncptr (dynload_shlib.c:80)
==149875==
==149875== 136 bytes in 1 blocks are indirectly lost in loss record 2,817 of 3,197
==149875== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x127C73F7: llvm::SmallVectorBase<unsigned int>::grow_pod(void*, unsigned long, unsigned long) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x1262FD92: llvm::SmallVectorImpl<void const*>::operator=(llvm::SmallVectorImpl<void const*> const&) [clone .isra.0] (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
--
==149875== 144 bytes in 2 blocks are definitely lost in loss record 2,834 of 3,197
==149875== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x505D3F: _PyMem_RawMalloc (obmalloc.c:45)
==149875== by 0x506264: _PyMem_DebugRawAlloc (obmalloc.c:2096)
==149875== by 0x5062A0: _PyMem_DebugRawMalloc (obmalloc.c:2129)
==149875== by 0x5070D0: PyMem_RawMalloc (obmalloc.c:662)
==149875== by 0x508DA2: _PyObject_Malloc (obmalloc.c:1569)
==149875== by 0x506264: _PyMem_DebugRawAlloc (obmalloc.c:2096)
==149875== by 0x5062A0: _PyMem_DebugRawMalloc (obmalloc.c:2129)
==149875== by 0x506841: _PyMem_DebugMalloc (obmalloc.c:2281)
==149875== by 0x507B5D: PyObject_Malloc (obmalloc.c:801)
==149875== by 0x4D27B7: PyFloat_FromDouble (floatobject.c:154)
==149875== by 0x69EED2: fill_time (posixmodule.c:2418)
==149875==
==149875== 152 bytes in 1 blocks are possibly lost in loss record 2,835 of 3,197
==149875== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x505D3F: _PyMem_RawMalloc (obmalloc.c:45)
==149875== by 0x506264: _PyMem_DebugRawAlloc (obmalloc.c:2096)
==149875== by 0x5062A0: _PyMem_DebugRawMalloc (obmalloc.c:2129)
==149875== by 0x5070D0: PyMem_RawMalloc (obmalloc.c:662)
==149875== by 0xADF4548: _Numba_hashtable_new_full (_hashtable.cpp:161)
--
==149875== 5,280 (1,112 direct, 4,168 indirect) bytes in 1 blocks are definitely lost in loss record 3,144 of 3,197
==149875== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x108A4E3E: llvm::MCJIT::createJIT(std::unique_ptr<llvm::Module, std::default_delete<llvm::Module> >, std::string*, std::shared_ptr<llvm::MCJITMemoryManager>, std::shared_ptr<llvm::LegacyJITSymbolResolver>, std::unique_ptr<llvm::TargetMachine, std::default_delete<llvm::TargetMachine> >) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x1090D043: llvm::EngineBuilder::create(llvm::TargetMachine*) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0xF3F952C: create_execution_engine (executionengine.cpp:85)
==149875== by 0xF3F95C7: LLVMPY_CreateMCJITCompiler (executionengine.cpp:97)
==149875== by 0xA46CB15: ??? (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA4693EE: ??? (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA46C0BD: ffi_call (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA75D8DB: _call_function_pointer (callproc.c:931)
==149875== by 0xA75E2FE: _ctypes_callproc (callproc.c:1273)
==149875== by 0xA75756D: PyCFuncPtr_call (_ctypes.c:4167)
==149875== by 0x4B894E: _PyObject_Call (call.c:367)
==149875==
==149875== 5,399 bytes in 75 blocks are indirectly lost in loss record 3,145 of 3,197
==149875== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x505D3F: _PyMem_RawMalloc (obmalloc.c:45)
==149875== by 0x506264: _PyMem_DebugRawAlloc (obmalloc.c:2096)
==149875== by 0x5062A0: _PyMem_DebugRawMalloc (obmalloc.c:2129)
==149875== by 0x5070D0: PyMem_RawMalloc (obmalloc.c:662)
==149875== by 0x508DA2: _PyObject_Malloc (obmalloc.c:1569)
--
==149875== 32,280 (40 direct, 32,240 indirect) bytes in 1 blocks are definitely lost in loss record 3,179 of 3,197
==149875== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x120EA14F: llvm::Pass* llvm::callDefaultCtor<llvm::ScalarEvolutionWrapperPass>() (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x1263B97C: llvm::PMTopLevelManager::schedulePass(llvm::Pass*) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0xF3FC802: LLVMPY_LLVMAddLoopRotatePass (passmanagers.cpp:479)
==149875== by 0xA46CB15: ??? (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA4693EE: ??? (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA46C0BD: ffi_call (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA75D8DB: _call_function_pointer (callproc.c:931)
==149875== by 0xA75E2FE: _ctypes_callproc (callproc.c:1273)
==149875== by 0xA75756D: PyCFuncPtr_call (_ctypes.c:4167)
==149875== by 0x4B894E: _PyObject_Call (call.c:367)
==149875== by 0x4B8A1B: PyObject_Call (call.c:379)
==149875==
==149875== 38,612 bytes in 29 blocks are still reachable in loss record 3,180 of 3,197
==149875== at 0x484D953: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x400C72C: calloc (rtld-malloc.h:44)
==149875== by 0x400C72C: _dl_new_object (dl-object.c:92)
==149875== by 0x4007ABE: _dl_map_object_from_fd (dl-load.c:1053)
==149875== by 0x4009528: _dl_map_object (dl-load.c:2268)
==149875== by 0x400D8DB: dl_open_worker_begin (dl-open.c:578)
--
==149875== 47,783 (10,608 direct, 37,175 indirect) bytes in 102 blocks are definitely lost in loss record 3,181 of 3,197
==149875== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x505D3F: _PyMem_RawMalloc (obmalloc.c:45)
==149875== by 0x506264: _PyMem_DebugRawAlloc (obmalloc.c:2096)
==149875== by 0x5062A0: _PyMem_DebugRawMalloc (obmalloc.c:2129)
==149875== by 0x5070D0: PyMem_RawMalloc (obmalloc.c:662)
==149875== by 0x508DA2: _PyObject_Malloc (obmalloc.c:1569)
==149875== by 0x506264: _PyMem_DebugRawAlloc (obmalloc.c:2096)
==149875== by 0x5062A0: _PyMem_DebugRawMalloc (obmalloc.c:2129)
==149875== by 0x506841: _PyMem_DebugMalloc (obmalloc.c:2281)
==149875== by 0x507B5D: PyObject_Malloc (obmalloc.c:801)
==149875== by 0x62F2E9: gc_alloc (gcmodule.c:2307)
==149875== by 0x62F411: _PyObject_GC_NewVar (gcmodule.c:2341)
==149875==
==149875== 49,164 bytes in 1 blocks are still reachable in loss record 3,182 of 3,197
==149875== at 0x484D953: calloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x127D3FEE: llvm::StringMapImpl::RehashTable(unsigned int) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x127509F8: std::pair<llvm::StringMapIterator<llvm::cl::Option*>, bool> llvm::StringMap<llvm::cl::Option*, llvm::MallocAllocator>::try_emplace<llvm::cl::Option*>(llvm::StringRef, llvm::cl::Option*&&) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x12753969: (anonymous namespace)::CommandLineParser::addOption(llvm::cl::Option*, llvm::cl::SubCommand*) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x1275457C: llvm::cl::Option::addArgument() (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0xF3D18EB: _GLOBAL__sub_I_SimplifyCFGPass.cpp (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
--
==149875== 290,911 (56 direct, 290,855 indirect) bytes in 1 blocks are definitely lost in loss record 3,193 of 3,197
==149875== at 0x4846FA3: operator new(unsigned long) (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x11D011DF: llvm::Pass* llvm::callDefaultCtor<(anonymous namespace)::LCSSAWrapperPass>() (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x1263B97C: llvm::PMTopLevelManager::schedulePass(llvm::Pass*) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x10B45F11: llvm::PassManagerBuilder::addFunctionSimplificationPasses(llvm::legacy::PassManagerBase&) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0x10B46B57: llvm::PassManagerBuilder::populateModulePassManager(llvm::legacy::PassManagerBase&) (in /mnt/Apacer/home/m/myenv/lib/python3.12/site-packages/llvmlite/binding/libllvmlite.so)
==149875== by 0xF3FB08B: LLVMPY_PassManagerBuilderPopulateModulePassManager (transforms.cpp:19)
==149875== by 0xA46CB15: ??? (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA4693EE: ??? (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA46C0BD: ffi_call (in /usr/lib/x86_64-linux-gnu/libffi.so.8.1.4)
==149875== by 0xA75D8DB: _call_function_pointer (callproc.c:931)
==149875== by 0xA75E2FE: _ctypes_callproc (callproc.c:1273)
==149875== by 0xA75756D: PyCFuncPtr_call (_ctypes.c:4167)
==149875==
==149875== 325,960 bytes in 1,405 blocks are possibly lost in loss record 3,194 of 3,197
==149875== at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==149875== by 0x505D3F: _PyMem_RawMalloc (obmalloc.c:45)
==149875== by 0x506264: _PyMem_DebugRawAlloc (obmalloc.c:2096)
==149875== by 0x5062A0: _PyMem_DebugRawMalloc (obmalloc.c:2129)
==149875== by 0x5070D0: PyMem_RawMalloc (obmalloc.c:662)
==149875== by 0x508DA2: _PyObject_Malloc (obmalloc.c:1569)
--
==149875== definitely lost: 12,120 bytes in 111 blocks
==149875== indirectly lost: 364,438 bytes in 1,152 blocks
==149875== possibly lost: 3,286,386 bytes in 15,649 blocks
==149875== still reachable: 334,247 bytes in 2,715 blocks
==149875== of which reachable via heuristic:
==149875== stdstring : 2,473 bytes in 44 blocks
==149875== suppressed: 0 bytes in 0 blocks
==149875==
==149875== For lists of detected and suppressed errors, rerun with: -s
==149875== ERROR SUMMARY: 220 errors from 220 contexts (suppressed: 0 from 0)
I am using the following versions:
- Python: 3.12.3
- Numba: 0.60.0
- llvmlite: 0.43.0
- Numpy: 2.0.2
Could you please provide guidance on how to resolve this issue?
Thank you.
Activity
sklam commentedon Sep 30, 2024
They look like harmless un-freed objects in LLVM because these objects are process-wide for Numba. At Python interpreter shutdown, the freeing of these objects are bypassed. These objects are not created repeatedly.
If you are seeing memory-leak for long running process, it is very likely coming from functions that LLVM have JITed. The LLVM JIT that Numba uses do not have a way to remove code loaded into memory.
Extra details:
At python interpreter shutdown, there is no guaranteed sequence of objection deletion. Many globals needed to even access the libLLVM could have be removed before all LLVM objects are freed. See numba/llvmlite@7347d77
github-actions commentedon Oct 31, 2024
This issue is marked as stale as it has had no activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with any updates and confirm that this issue still needs to be addressed.