Repository navigation
Make --with-lg-page=16 the default option on ARM64 architecture #2639
Description
ARM64 architecture supports 4K, 16K, and 64K pagesizes. Most ARM64 server hardware is already using 64K pagesize kernels. Many popular modern ARM64 consumer hardware is using 16K pagesize (Raspberry Pi5 and Apple Silicon Macs).
This proliferation has caused all distributions of jemalloc (distros, toolkits, etc) and applications that bundle jemalloc (chromium and all derivatives, geekbench, rust, etc) to set the --with-lg-page=16 to support them. If they do not then the application will crash due to unsupported pagesize for jemalloc.
A good summary of the bug is here -> k3s-io/k3s#7335 (comment) and #467 (comment) . The TLDR of that is not supporting all supported pagesizes of the ARM64 architecture on default configuration options is a bug by definition.
Activity
clementperon commentedon Apr 23, 2024
Related to #467
theofficialgman commentedon Apr 23, 2024
requested by dependent project telegramdesktop/tdesktop#27801 (comment) telegramdesktop/tdesktop#26710 (comment)
theofficialgman commentedon Apr 23, 2024
also just to add to this comment from the other thread #467 (comment)
I do not see why jemalloc could not implement runtime checks for page size to take different codepaths within the binary. You should be able to have your compile time optimizations while also being compatible with the 3 pagesizes supported by the ARM64 architecture (4K/16K/64K) only at the expense of binary size (eg: all three codepaths exists in the final binary but the two non-native pagesize codepaths go unused)
Of course all that effort is unnecessary if using a binary optimized for 64K pagesize (
--with-lg-page=16) on a 4K pagesize does not lead to a considerable (eg: 10%+) regression in performance or memory usage.madscientist commentedon Apr 23, 2024
Unfortunately it IS expensive in some ways for jemalloc to run on systems with larger page sizes. Just as an example, we know that heap profiling is seriously impacted by page size, such that if you have a 64k page you may well have to disable it as it uses too much memory. If you have a 4k page size you don't have this problem and you can leave it on always if you want. There are other issues such as the size of VIRT memory which doesn't matter for performance but makes a very big difference for things like coredumps. There have been some attempts to reduce these impacts but they still exist to some extent.
So, simply always compiling with
--with-lg-page=16by default does have costs. Maybe some projects don't care about these costs, but others do.I think it's totally ridiculous for a project to refuse to add this option to their jemalloc config because it's "not standard". We set this ourselves on ARM since Red Hat EL 8 uses 64k page size. However I think most current Linux distributions (RHEL 9, Debian/Ubuntu, etc.) use 4k pages on ARM.
On the other hand I don't care if jemalloc changes its default setting (for ARM, but NOT for Intel!!) to use 64k pages as long as the docs are clear about this and the
--with-lg-pageoption is still supported to reduce it where that makes sense.I don't really see how it will work for a single libjemalloc.a library to provide support for all page sizes and switch at runtime. I'm not a jemalloc dev but if you read the other issues it's pretty clear that changing this value also changes the size etc. of data structures in jemalloc so it's not just a matter of choosing a different set of algorithms at runtime. You'd basically need to link three complete versions of jemalloc, then select the one you want at runtime, and selecting an allocator at runtime is notoriously tricky because it needs to be in place before anything, including pre-main code, tries to allocate any memory.
theofficialgman commentedon May 8, 2024
@ilya-fedin make your case telegramdesktop/tdesktop#27801 (comment)
Fredvs79 commentedon May 12, 2024
Just want to add a few tidbits to the discussion.
#1, the raspberryPi 5 has optimizations over the rPi4 that include 16K page sizes. I ran into an issue where a package that depends on jemalloc ran fine on the rPi4 but not on the rPi5 with the default kernel. The quick and easy solution was to load a different generic kernel (kernel8) to use a 4K page size.
#2, per the install.MD for jemalloc
Am I incorrect to think that for a 16K page size would yield --with-lg-page=4 (not --with-lg-page=16)
--edit --
I now realize that setting jemalloc to 64K page size would allow for 4K, 16K, and 64K page sizes, and would work with any kernel.
#3 There is some discussion about also optimizing jemalloc to use --with-hugepage=21 as the default build environment variable for arch64. #2624
marcan commentedon Dec 5, 2024
Then those that do, and which are OK with their builds will be non-portable to all ARM64 machines, can make the choice to change to 4K.
What doesn't make any sense whatsoever is for the jemalloc default to simply not work on some machines, because it means that every downstream project has to fix this bug for jemalloc by overriding the build options, which is an endless whack-a-mole of contacting downstream projects (and some, like Telegram, refuse because they think jemalloc should do this). The vast, vast, vast majority of projects' goal with pre-built binaries is that they work at all, not that heap profiling is more efficient. If a project wants to sacrifice portability to have more efficient heap profiling, that should be an explicit opt-in choice (and then perhaps the fact that heap profiling is bad on 64K builds should be considered a jemalloc bug to be fixed later).
ilya-fedin commentedon Dec 9, 2024
Another way for Telegram could be to fix memory usage with glibc (it never releases the RAM allocated for video frames for unknown reason) so jemalloc is not needed at all. Sadly, there's no one to do that work... I believe asahi community has people with low level knowledge of memory allocations, if someone made a call within their community to look at the tdesktop glibc allocations problem and develop a fix, tdesktop worked on asahi and rpi5 since long time ago.
ilya-fedin commentedon Dec 9, 2024
Alternative solutions like finding a replacement for jemalloc that would work with 16K page size out-of-the-box and releasing the RAM properly are welcome, too.
mullcom commentedon Feb 28, 2025
Still bo progress it seams. I just did install docker with six hook and find out.......
This is so bad for open source to not solving this situvation. So is sent easy to come together then blame etch otheras a open source team and not as a company that constantly have this issues because agreed and power. How about get the Pride back by do it instead of blame it.?
ilya-fedin commentedon Feb 28, 2025
The problem is not in blaming but in the lack of someone who will do the actual work (speaking tdesktop side at least - it just lacks a developer with arm64 machine to really support it and fix arm64 specific bugs)
Fredvs79 commentedon Feb 28, 2025
From where I sit, this seems like a chicken or egg kind of puzzle. Do you set 4K as the default and tell everyone who wants 16K to compile their own instance, or do you set 16K as the default and tell everyone who wants 4K page size to compile their own instance?
jemalloc currently uses 4K by default, but can support 16K with build arguments.
Their only option is to custom compile and load jemalloc. That would be fine if there were lots of people who need 4k and very people who need 16k. I don't know what the distribution is, but my hunch is that wherever it is, it is moving in the direction of favor for 16K page size support. So when will jemalloc change to reflect this since there are mounting voices requesting it.
The counter argument presented was there may exist other people who specifically do not want 16k page size as the default. And whether they are the silent majority, or even if they are the minority, obviously they don't want the odious burden of having to custom build and load jemalloc.
Unfortunately jemalloc has to pick a side for what the default should be, and which user base they are going to prioritize. As far as I can tell, for "the work", it basically comes down to inserting build flag arguments in a few locations to make 16k the default.
ilya-fedin commentedon Mar 1, 2025
That might be not the solution maintainers want to see? And the solution they want to see might be way harder to implement.
E.g. tdesktop side, we want page size to not be affected on arches other than arm64 but no one has proposed such a solution. Everyone just suggests to add an argument which changes the size on all arches.
Ideal solution on tdesktop side would be for someone to do debugging to fix leaking with glibc allocator and just stop using jemalloc altogether. But no one has appealed to that work.
sallyhaj commentedon Mar 9, 2025
I use Raspberry Pi 5 with 16K page sizes.
For me it fails to launch telegram-desktop from flathub.
I have already compiled jemcalloc with
--with-lg-page=16but then telegram doesn't work.marcan commentedon Mar 29, 2025
I'm just going to start linking every instance of jemalloc being broken on 16K from end users to this bug.
https://www.reddit.com/r/AsahiLinux/comments/1jm246e/is_electron_for_apps_a_feasible_choice_in_the/
24 remaining items
marcan commentedon Mar 29, 2025
https://meta.discourse.org/t/test-installation-on-aarch64-failed-to-report-error-error-connecting-to-redis-on-localhost-6379/212763/8
marcan commentedon Mar 29, 2025
https://docs.ncsa.illinois.edu/systems/deltaai/en/latest/user-guide/python/tensorflow.html
marcan commentedon Mar 29, 2025
https://forum.qnap.com/viewtopic.php?t=167386
marcan commentedon Mar 29, 2025
falcosecurity/falco#3490
marcan commentedon Mar 29, 2025
https://community.audiorelay.net/t/linux-package-for-arm-support/1265
marcan commentedon Mar 29, 2025
Not even Linux/aarch64 specific: https://mail-index.netbsd.org/port-arm/2020/01/10/msg006424.html
marcan commentedon Mar 29, 2025
https://gitee.com/milvus-io/milvus/blob/master/DEVELOPMENT.md#faqs
marcan commentedon Mar 29, 2025
https://help.nextcloud.com/t/config-php-not-available-after-snap-install-nextcloud-on-pi5-with-bookworm-raspbian-lite-x64/188118
marcan commentedon Mar 29, 2025
Is all this user pain across every downstream project using jemalloc really worth keeping the current broken unportable default?
Remember that a 4K build only works on 4k systems, while a 64K build works on all systems, and the only downsides presented are that it is less optimal (maybe, no real stats have been shared), may use more memory than a pure 4K buid on 4K systems (maybe, sometimes, no real stats have been shared), and some stuff related to heap profiling. None of those hypotheticals sound like good reasons to keep a broken default that does not work at all on a subset of systems.
At the end of the day, every downstream project that intends to be portable, which means any project that intends to package itself for distributions or flatpaks or snaps or AppImages or anything else intended to be used by Linux users in general, has to make this change. So regardless of the downsides, people will be living with them, because the alternative is broken software.
The only people who will willingly keep the 4K default are those who actually know they only run on 4K platforms only and have reason to believe the downsides of a 64K build affect them.
This is completely backwards. The default should cater to portable software intended to run on heterogeneous platforms. Closed, controlled ecosystems where the page size is known are the ones that should be overriding compile time options for micro-optimization.
theofficialgman commentedon Apr 6, 2025
@interwq Could you please outline a plan of action for having this (support for all the valid 4k, 16k, and 64k arm pagesizes in one binary (or set of distributable binaries)) resolved? It seems like all users are in agreement that this should be done (all users want software that works, like myself) but no developer of the jemalloc project has any interest in doing it. I think that outlining the process/plan that you and the other maintainers of jemalloc would be willing to merge once completed would help a 3rd party developer come in to implement the changes in the future.
Additionally, @marcan makes some very valid points here ->
please do share some test cases and statistics that show these statements to be true. It will help everyone understand the gravity (or lack of) a no-code change to
--with-lg-page=16.dneary commentedon May 7, 2025
I submitted this pull request: #2845
I hope it is accepted.
Change the default page size to 64KiB on Aarch64 Linux. Closes jemall…
fix(cubestore): Build with large pagesize support for AMR64, thanks @…
build: pin jemalloc's page size to 4 KiB on aarch64 Linux (#39549)