Repository navigation
Excessive memory retention using either mimalloc v2 or v3 on macOS #1025
Description
Rspack is using mimalloc for speeding up the memory allocation. It's is a tool for transforming and bundling basically JavaScript files. According to our test to resolve this issue, we found out, on macOS, there's some strange memory retention as rspack rebuilds. I tried to create a minimal reproducible demo but unluckily they all failed. So I have to put on my local testing demo for reproduction using Rust:
In this demo, an rspack compiler was created to compile JavaScript in 10000 directory. For each build and rebuild, Rspack would trigger tokio-rs to spawn(if not already spawned) a few green threads to drive asynchronous tasks. Then, Rspack would trigger a series of JavaScript module transformations, then optimizations. Finally, assets generated in each build or rebuild will be emitted to the dist file.
During the compilation, the initial memory on macOS would be around 600 MB, and after a few rebuilds, the memory will skyrocket to 1 GB and more. This does not happen on my ubuntu-22.04 or when I was using macOS's system allocator. This does happen on both mimalloc v2 and v3.
I've added some details to help reproduce the issue in the repo and will try my best to create a minimal reproducible demo. Please bear with me.
Looking forward to hearing from you.
Cheers!
Activity
daanx commentedon Feb 27, 2025
Yikes -- thanks for the report. Strange that it happens with both v2 and v3, and not on ubuntu. Thanks for the repo -- if I find time I will try it out and see. Can you try the following environment settings on the latest
dev3-binbranch:MIMALLOC_ARENA_EAGER_COMMIT=0MIMALLOC_PAGE_COMMIT_ON_DEMAND=1Also, as an experiment, trying,
MIMALLOC_PURGE_DELAY=0would be interesting (this can slow things down though but would perhaps give us a clue).h-a-n-a commentedon Mar 3, 2025
Daan (@daanx) Thanks for the quick reply!
I've tested these options on my macOS. Nothing has been changed in the demo other than changing some value to make it rebuild indefinitely.
Here's some stats I yanked off from
top. The result shows memory consumption on branchdev3-bindo accumulate slower than it was while it was ondev2-bin, but it still accumulates as time passes.With branch
dev3-bin:With branch
dev2-bin:With branch
dev3-binand environment set toMIMALLOC_ARENA_EAGER_COMMIT=0:With branch
dev3-binand environment set toMIMALLOC_PAGE_COMMIT_ON_DEMAND=1:With branch
dev3-binand environment set toMIMALLOC_PURGE_DELAY=0:daanx commentedon Mar 3, 2025
Good to see v3 does much better as v2; I guess because it doesn't occur on Linux it must be something system specific, like an allocation in a thread that is about to be terminated (reinitializing the heap and leave it orphaned). Not sure. I'll try to repro when I find some time.
daanx commentedon Mar 3, 2025
I tried to compile from your repo (awesome that you constructed this! maybe we can use it a standard benchmark in the future).
I first got an error to use nightly, so I use
cargo +nightly build --releasebut now I get:Can you help?
h-a-n-a commentedon Mar 4, 2025
Turned out the rust lang team had changed this API in the latest nightly release. I've pushed a new edit to add a
rust-toolchain.tomlthat locks the rust toolchain to a specific version. Would you please pull and runcargo build --releaseagain?You can also check out the active rust toolchain in the repo directory with command
rustup show.daanx commentedon Mar 5, 2025
Thanks -- I got it running locally; but it doesn't quire reproduce. I set it to 100 iterations using the latest
dev3branch. The memory usage is much lower though than yours (around 280MiB at peak). Is that expected? Secondly, I couldn't quite reproduce the behaviour as it gets stable after about 30 to 50 iterations (from ~180MiB rss to ~280MiB rss). This is on an Apple M1, Sequoia 15.3 withcarge build --releasewith the latestdev3.When I set
MIMALLOC_PURGE_DELAY=0I see that there are about 400 threads, and the memory looks like (usingmi_arenas_printavailable in the latest dev3):Here I see lots of low use pages (the red
P's) which I guess belong to many not often used threads. Over each iteration the heap keeps looking like this with around 9 chunks in use, with sometimes some large singleton objects (like the final greenspage which is about 5 MiB I guess). Maybe the initial growth is due to the threadpool like nature of tokio where per-thread pages get slowly used a bit more depending on the tasks that happen to execute no them -- but in the end it stabilizes? Maybe not, as you remarked you didn't see this on Linux. Is the benchmark reading from disk? Are you writing to a log file in that same directory?Maybe I need the larger workload that you observed of 800MiB+ -- let me know how to do that.
ps. with no options on a release build (with latest
dev3, 100 iterations):chenjiahan commentedon Mar 5, 2025
I can also reproduce the problem by following the steps in this demo.
carge build --releaseResult:
h-a-n-a commentedon Mar 5, 2025
Unfortunately, this is not expected.
I forgot to mention to install node modules in the directory
10000. This might be related to the low memory consumption in your reproduction. I've modified theREADME.md. You might need to installnodeon the machine and callpnpm installto install node modules. My edit toREADME.md: h-a-n-a/rspack-allocation-test@ecf28cb. Then, callingcargo run --releaseshould emit no error.The benchmark contains heavy operation of reading files from disk (in directory
10000) recursively. Doing this would invoke a blocking thread (this type of thread used only for reading files in rspack) of tokio to send filesystem file read command. It will keep alive for the default timeout of 10 seconds (i.e. the thread_keep_alive option)We don't normally write log files. The only file writing operation is called at the end of each rebuild. This is driven by the same blocking thread of tokio.
daanx commentedon Mar 6, 2025
I got it running now -- it still uses much less memory but I can see it grow now, from about 240MiB to 480MiB after 100 iterations. One thing I noticed was that every iteration an extra thread(s) is created, it starts at 339 threads and increases to 526 threads after 100 iterations. I will test on Linux too to see if that happens there too. I guess that extra thread is the issue? edit: on linux the threads stay constant at 199 (but uses more memory and goes much faster?)
Question: how can I configure the project to use the standard macOS allocator?
h-a-n-a commentedon Mar 6, 2025
I did some tests on my local machine and found out the thread count and the memory consumption are indeed related and intertwined. In my opinion, it's strongly linked to these two configurations passed to tokio-rs:
fsoperations)Tuning either of the option will lower the memory usage.
For example, if I set
max_blocking_threadto 8, which is the default blocking thread count in rspack now, it avoids tokio-rs from create as much blocking threads as they might like to. This decreases the memory.Even for mimalloc v2.1.7, which consumes more memory than v3 (dev3-bin), memory usage is way better than it was:
Details
The result for mimalloc-v3 (dev3-bin):
Details
Memory still goes up pretty quick on larger workload though.
To adjust these two options, you may go to
src/main.rsand add the option to the tokio runtime initialization:IIRC, mine test on linux gets stable at around 550~MB. I think this is totally fine if it's not increasing.
Just comment out these two lines at the top of
src/main.rsand recompile.daanx commentedon Mar 6, 2025
Thanks! It is good to see that v3 does much better as it was essentially redesigned to deal with threadpools better.
Just to clarify, the threads on Linux stay constant under 200 (and memory growth stabilizes). Btw. in a threadpool like scenario we expect an initial growth of memory as each thread retains some owned memory but it should level out after a while.
Maybe the creation of the new threads that I see on macOS is the root cause of the memory growth (where there as an increasing amount of threads that are never terminated -- maybe due to mimalloc? maybe not?).
h-a-n-a commentedon Mar 11, 2025
Thanks for the reply!
That's weird. If I set
max_blocking_threadto 8 with mimalloc-v3, the thread count gets stable at33and the memory is still growing over time. I've uploaded a large workload with20000modules, but it still does not quite reproduce the memory growth in a large scale even though it's growing quite slowly. There's a big workload in my company project that I'm unable to share:For each rebuild, it consumed and retained
300MiB memory.Details
So... do you know if there's any recommend way to debug this? For example, checking out the thread of which piece of memory was being hold off from releasing. Or do you know if there's anything should be checked out for me locally?
pitrou commentedon Mar 31, 2025
We might be seeing a related issue on the Apache Arrow CI when trying to bump mimalloc from 2.0.6 (which works fine) to either 2.1.9 or 2.2.3 (both of which incur crashes on some of our ARM64 macOS jobs, when running some large-memory tests). Is there a way to check whether it's the same issue?
Also cc Krisztián Szűcs (@kszucs)
pitrou commentedon Mar 31, 2025
Here's the test log by the way, but it's not very insightful... though the "0x2000000" alignment is weird, is it expected?
pitrou commentedon Mar 31, 2025
Ok, I've tested several versions now and I've determined that the crash appears with 2.1.9. Versions 2.1.7 and older are fine. Should I open a new issue for it?
139 remaining items
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
mimalloc: avoid excessive memory retention
github-actions commentedon Aug 10, 2026
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs in the next 14 days. Thank you for your contributions!
github-actions commentedon Aug 25, 2026
This issue was closed because it has been inactive for 14 days since being marked as stale. Please feel free to reopen if this is still an active issue.