build: pin jemalloc's page size to 4 KiB on aarch64 Linux - #39549

Summary

Pin jemalloc's allocator page to 4 KiB for aarch64-unknown-linux-gnu builds by setting AARCH64_UNKNOWN_LINUX_GNU_JEMALLOC_SYS_WITH_LG_PAGE = "12" in .cargo/config.toml.

#39339 bumped tikv-jemalloc-sys from 0.6.1 (jemalloc 5.3.0) to 0.7.1 (jemalloc 5.3.1). jemalloc 5.3.1 changed its default allocator page on aarch64 Linux from the build host's page to 64 KiB (facebook/jemalloc#34). Neither #39339 nor the tikv-jemallocator 0.7.0 release notes mention it, and I found no discussion of it in code, issues or Slack.

@antiguru confirmed on this PR that the move to 64 KiB was not a deliberate choice. Every production and staging environmentd and clusterd pod runs on arm64 nodes, so rc.3 runs with a 64 KiB allocator page on 4 KiB-page kernels.

The effect is a step in jemalloc_active minus jemalloc_allocated in every Materialize process. This PR restores the 4 KiB page that v26.44.0 and earlier had on aarch64, and keeps everything else in jemalloc 5.3.1.

What was observed

The v26.45.0-rc.3 sign-off sweeps on staging and the production canaries flagged jemalloc_active rising while jemalloc_allocated stayed flat or fell, from process start, in environmentd and clusterd:

• environmentd on the us-east-1 canaries, v26.44.0 against rc.3 at equal post-restart age: active minus allocated went from 55 MB to 490 MB on one environment and from 84 MB to 558 MB on another.
• clusterd on the largest canary pods: from 0.19 GB to 1.70 GB, scaling with pod size.
• No OOMs and no limit was hit.

How it was bisected

All of this ran locally on an Apple Silicon Mac under OrbStack: linux/arm64 natively, 4 KiB kernel page.

1. Page size per image. Running each image with MALLOC_CONF=stats_print:true <binary> --version prints jemalloc's compiled page size:

kube_pod_container_info shows every rc.3 environmentd and clusterd container on the production canaries (us-east-1, eu-west-1, us-west-2) and on staging (us-east-1, eu-west-1) running the Docker Hub index digests that were tested.

2. Workload bisection on per-commit images. 150 of the 232 main commits between the v26.44.0 and rc.3 branch points have --main.g<sha> images on Docker Hub, including #39339 and its parent. A harness started one fresh materialized container per run on a fixed timeline: 40 s idle, then a fixed SQL workload on quickstart (a 1.5M-row table, an index, an aggregating and a joining materialized view, deletes, some catalog churn), then a 60 s settle and six samples 10 s apart. It recorded jemalloc stats from environmentd's /metrics and from each clusterd's internal HTTP socket, plus VmRSS. Each arm64 configuration ran twice, interleaved.

Active minus allocated, after the workload, in MiB:

The step is in #39339, the only first-parent commit between the two main images. The two new jemalloc 5.3.1 defaults, set back through MALLOC_CONF, do not move it. rc.3 on amd64 runs the same jemalloc 5.3.1 with a 4 KiB page and has v26.44.0's gap.

3. Page size alone. rc.3 was rebuilt twice with cargo build --profile optimized --bin materialized in the ci-builder: once unchanged and once with JEMALLOC_SYS_WITH_LG_PAGE=12. Each binary replaced /usr/local/bin/materialized in the released arm64 rc.3 image. The 4 KiB build keeps experimental_tcache_gc and disable_large_size_classes on.

Values in MiB, two runs each.

4. Where the extra active memory sits. Full malloc_stats_print output was dumped from the live processes after the same workload, for both rebuilt images. The table splits the 64 KiB minus 4 KiB gap difference, in MiB:

• Small-bin slabs are at least one page, so with a 64 KiB page each slab is 320 to 448 KiB instead of 20 to 28 KiB. Each of the 48 arenas holds about one partly used slab per size class, mostly under 25% full.
• With prof:true and lg_prof_sample:19, every sampled small allocation gets its own page-aligned extent. That extent counts as active and not as allocated, so a 64 KiB page makes each sample cost 64 KiB or more. With prof_active:false that term is zero.
• Cutting the arena count from 48 to 16 cut the slab term by 48 to 61%, in single runs. jemalloc creates four arenas per CPU in the process's affinity mask, which would explain why the production gap scales with pod and node size. That link is inferred and not measured in production.

How the cost scales

The two terms in step 4 scale differently, so the effect on small and large pods differs.

Partly filled slabs: roughly a fixed cost per process shape.

• Each arena keeps about one partly used slab per size class in use, so this term follows arenas times size classes in use, not heap size.

• jemalloc creates four arenas per CPU in the process's affinity mask. Swap-enabled replica sizes do not get exclusive CPUs, so a clusterd pod's arena count follows its node's vCPU count. That last step is inferred from the replica-size config and was not measured on a pod.

• Locally, cutting the arena count from 48 to 16 cut this term by 48 to 61%.

• On the production canaries, the clusterd gap as a share of mz_memory_limiter_memory_limit_bytes falls as pods grow. This is a snapshot at 2026-10-04 05:40 UTC across different workloads, not a controlled scaling test:

  Replica sizes	Limiter limit	Gap / limit
  mz_probe	1.9 GiB	3.9 to 6.0%
  25cc	9.5 GiB	0.8 to 2.4%
  50cc	19.0 GiB	0.5%
  M.1-nano, M.1-micro	26.5 to 53.1 GiB	0.2 to 0.7%
  100cc to 400cc, M.1-xsmall	37.9 to 151.6 GiB	0.2 to 0.5%
  M.1-small, M.1-medium, 600cc	212 to 318 GiB	0.3 to 1.4%
  M.1-1.5xlarge, M.1-3xlarge	637 to 1274 GiB	0.1%

  The largest pod's gap, 1.58 GiB, is about 22 times the smallest's, for about 670 times the limit.

• Not measured: frees in a different order from allocation also leave partly empty slabs behind, and a 64 KiB-page slab holds 16 times as many regions. A heap with heavy churn could therefore grow this part of the term with its size.

Heap-profiling samples: linear in live small-object bytes.

• With lg_prof_sample:19, about one small allocation is sampled per 512 KiB allocated. The live sampled allocations therefore number about live small-object bytes divided by 512 KiB, and each takes at least one page.
• That puts the term at roughly 12 to 25% of live small-object bytes with a 64 KiB page, against about 1% with a 4 KiB page. This is an estimate from the sampling rate. The measured term is 7 to 22% of jemalloc_allocated on the three local processes.
• Large allocations are unaffected, so pods whose heap is mostly large allocations see a smaller share.

Resident memory, which the clusterd memory limiter counts, rises less than either term.

• The kernel page stays 4 KiB, so a sparsely used 64 KiB slab, or a 64 KiB extent holding one small sampled object, becomes resident only in the 4 KiB pages actually touched.
• Locally, VmRSS was 32% higher on environmentd and 9 to 18% higher on the clusterd processes with the 64 KiB page, while the gap was about ten times higher.
• That explanation fits the numbers but was not measured per term.

Why the target-prefixed variable

tikv-jemalloc-sys reads <TARGET>_JEMALLOC_SYS_WITH_LG_PAGE before JEMALLOC_SYS_WITH_LG_PAGE (tikv/jemallocator#91, used this way in tikv/jemallocator#122). The unprefixed variable would also apply to aarch64 macOS builds, where the kernel page is 16 KiB. There jemalloc aborts at startup with "Unsupported system page size" (tikv/jemallocator#80). The prefixed form changes only the aarch64 Linux build. x86_64 already uses a 4 KiB page.

Upstream moved to 64 KiB so one aarch64 binary runs on 4, 16 and 64 KiB kernels. The thread it closed, jemalloc/jemalloc#2639, notes the costs, including heap profiling. With this change, a Materialize aarch64 binary again fails to start on a kernel with pages larger than 4 KiB. v26.44.0 and earlier had the same constraint. Their 4 KiB-page jemalloc runs on our arm64 nodes, so those kernels use 4 KiB pages.

Verification

• cargo build -p tikv-jemalloc-sys -vv in the ci-builder on this branch passes --with-lg-page=12 to jemalloc's configure, which reports LG_PAGE : 12 for aarch64-unknown-linux-gnu.
• The rc.3 build with the equivalent unprefixed variable reports Page size: 4096 and gives the measurements above.

No tests were added. Whether the published arm64 images report Page size: 4096 can be checked by running the next main image with MALLOC_CONF=stats_print:true clusterd --version.

🤖 Generated with Claude Code