Repository navigation
allow per-target env detection - #91
Merged
Merged
Conversation
Torrekie
force-pushed
the
patch-per-target-env
branch
from
June 17, 2024 10:00
b316bf7 to
c2234c2
Compare
BusyJay
reviewed
Jun 17, 2024
Signed-off-by: Torrekie <[email protected]>
Signed-off-by: Torrekie <[email protected]>
Torrekie
force-pushed
the
patch-per-target-env
branch
from
June 17, 2024 10:08
c2234c2 to
88b54a4
Compare
BusyJay
reviewed
Jun 17, 2024
Signed-off-by: Torrekie <[email protected]>
Member
|
You can ignore the aarch64 failure. The clippy failure is not related to your change, but if you can fix it in this PR we can merge it all together. |
bosconi
added a commit
to MaterializeInc/materialize
that referenced
this pull request
Oct 5, 2026
## Summary Pin jemalloc's allocator page to 4 KiB for `aarch64-unknown-linux-gnu` builds by setting `AARCH64_UNKNOWN_LINUX_GNU_JEMALLOC_SYS_WITH_LG_PAGE = "12"` in `.cargo/config.toml`. [#39339](#39339) bumped tikv-jemalloc-sys from 0.6.1 (jemalloc 5.3.0) to 0.7.1 (jemalloc 5.3.1). jemalloc 5.3.1 changed its default allocator page on aarch64 Linux from the build host's page to 64 KiB ([meta/jemalloc#34](meta/jemalloc#34)). Neither #39339 nor the tikv-jemallocator [0.7.0 release notes](https://github.com/tikv/jemallocator/releases/tag/0.7.0) mention it, and I found no discussion of it in code, issues or Slack. @antiguru confirmed on this PR that the move to 64 KiB was [not a deliberate choice](#39549 (comment)). Every production and staging environmentd and clusterd pod runs on arm64 nodes, so rc.3 runs with a 64 KiB allocator page on 4 KiB-page kernels. The effect is a step in `jemalloc_active` minus `jemalloc_allocated` in every Materialize process. This PR restores the 4 KiB page that v26.44.0 and earlier had on aarch64, and keeps everything else in jemalloc 5.3.1. ## What was observed The v26.45.0-rc.3 sign-off sweeps on staging and the production canaries flagged `jemalloc_active` rising while `jemalloc_allocated` stayed flat or fell, from process start, in environmentd and clusterd: - environmentd on the us-east-1 canaries, v26.44.0 against rc.3 at equal post-restart age: active minus allocated went from 55 MB to 490 MB on one environment and from 84 MB to 558 MB on another. - clusterd on the largest canary pods: from 0.19 GB to 1.70 GB, scaling with pod size. - No OOMs and no limit was hit. ## How it was bisected All of this ran locally on an Apple Silicon Mac under OrbStack: linux/arm64 natively, 4 KiB kernel page. **1. Page size per image.** Running each image with `MALLOC_CONF=stats_print:true <binary> --version` prints jemalloc's compiled page size: | Image | Arch | jemalloc | Page size | |---|---|---|---| | `environmentd`, `clusterd`, `materialized` `v26.44.0` | arm64 | 5.3.0 | 4096 | | `environmentd`, `clusterd`, `materialized` `v26.44.0` | amd64 | 5.3.0 | 4096 | | `environmentd`, `clusterd`, `materialized` `v26.45.0-rc.3` | arm64 | 5.3.1 | 65536 | | `environmentd`, `clusterd` `v26.45.0-rc.3` | amd64 | 5.3.1 | 4096 | | `materialized:v26.45.0-dev.0--main.geb032636b571a61c1016d2a13f7398b5edceb8ff` (parent of #39339) | arm64 | 5.3.0 | 4096 | | `materialized:v26.45.0-dev.0--main.g0d478e2dc6ba5361b948bc222c61968878508625` (#39339) | arm64 | 5.3.1 | 65536 | `kube_pod_container_info` shows every rc.3 environmentd and clusterd container on the production canaries (us-east-1, eu-west-1, us-west-2) and on staging (us-east-1, eu-west-1) running the Docker Hub index digests that were tested. **2. Workload bisection on per-commit images.** 150 of the 232 main commits between the v26.44.0 and rc.3 branch points have `--main.g<sha>` images on Docker Hub, including #39339 and its parent. A harness started one fresh `materialized` container per run on a fixed timeline: 40 s idle, then a fixed SQL workload on `quickstart` (a 1.5M-row table, an index, an aggregating and a joining materialized view, deletes, some catalog churn), then a 60 s settle and six samples 10 s apart. It recorded jemalloc stats from environmentd's `/metrics` and from each clusterd's internal HTTP socket, plus VmRSS. Each arm64 configuration ran twice, interleaved. Active minus allocated, after the workload, in MiB: | Configuration | environmentd | `mz_catalog_server` | `quickstart` | |---|---:|---:|---:| | `v26.44.0` | 40 | 23 | 36 | | main, parent of #39339 | 42 | 22 | 36 | | main, #39339 | 366 | 233 | 390 | | `v26.45.0-rc.3` | 363 | 222 | 373 | | rc.3, `experimental_tcache_gc:false` | 393 | 235 | 372 | | rc.3, `disable_large_size_classes:false` | 371 | 234 | 380 | | rc.3, both of the above | 382 | 244 | 368 | | `v26.44.0`, amd64 | 43 | 22 | 35 | | rc.3, amd64 | 41 | 24 | 38 | The step is in #39339, the only first-parent commit between the two main images. The two new jemalloc 5.3.1 defaults, set back through `MALLOC_CONF`, do not move it. rc.3 on amd64 runs the same jemalloc 5.3.1 with a 4 KiB page and has v26.44.0's gap. **3. Page size alone.** rc.3 was rebuilt twice with `cargo build --profile optimized --bin materialized` in the ci-builder: once unchanged and once with `JEMALLOC_SYS_WITH_LG_PAGE=12`. Each binary replaced `/usr/local/bin/materialized` in the released arm64 rc.3 image. The 4 KiB build keeps `experimental_tcache_gc` and `disable_large_size_classes` on. | Build | environmentd gap | environmentd VmRSS | `mz_catalog_server` gap | `mz_catalog_server` VmRSS | `quickstart` gap | `quickstart` VmRSS | |---|---:|---:|---:|---:|---:|---:| | rc.3, 64 KiB page | 362 | 434 | 246 | 432 | 382 | 869 | | rc.3, 4 KiB page | 43 | 328 | 24 | 395 | 36 | 739 | Values in MiB, two runs each. **4. Where the extra active memory sits.** Full `malloc_stats_print` output was dumped from the live processes after the same workload, for both rebuilt images. The table splits the 64 KiB minus 4 KiB gap difference, in MiB: | Process | Gap difference | Partially filled small-bin slabs | Heap-profiling sampled allocations | Large-extent page pad | |---|---:|---:|---:|---:| | environmentd | 318 | 304 | 16 | -2 | | `mz_catalog_server` | 217 | 164 | 59 | -6 | | `quickstart` | 339 | 285 | 52 | 2 | - Small-bin slabs are at least one page, so with a 64 KiB page each slab is 320 to 448 KiB instead of 20 to 28 KiB. Each of the 48 arenas holds about one partly used slab per size class, mostly under 25% full. - With `prof:true` and `lg_prof_sample:19`, every sampled small allocation gets its own page-aligned extent. That extent counts as active and not as allocated, so a 64 KiB page makes each sample cost 64 KiB or more. With `prof_active:false` that term is zero. - Cutting the arena count from 48 to 16 cut the slab term by 48 to 61%, in single runs. jemalloc creates four arenas per CPU in the process's affinity mask, which would explain why the production gap scales with pod and node size. That link is inferred and not measured in production. ## How the cost scales The two terms in step 4 scale differently, so the effect on small and large pods differs. **Partly filled slabs: roughly a fixed cost per process shape.** - Each arena keeps about one partly used slab per size class in use, so this term follows arenas times size classes in use, not heap size. - jemalloc creates four arenas per CPU in the process's affinity mask. Swap-enabled replica sizes do not get exclusive CPUs, so a clusterd pod's arena count follows its node's vCPU count. That last step is inferred from the replica-size config and was not measured on a pod. - Locally, cutting the arena count from 48 to 16 cut this term by 48 to 61%. - On the production canaries, the clusterd gap as a share of `mz_memory_limiter_memory_limit_bytes` falls as pods grow. This is a snapshot at 2026-10-04 05:40 UTC across different workloads, not a controlled scaling test: | Replica sizes | Limiter limit | Gap / limit | |---|---:|---:| | `mz_probe` | 1.9 GiB | 3.9 to 6.0% | | 25cc | 9.5 GiB | 0.8 to 2.4% | | 50cc | 19.0 GiB | 0.5% | | M.1-nano, M.1-micro | 26.5 to 53.1 GiB | 0.2 to 0.7% | | 100cc to 400cc, M.1-xsmall | 37.9 to 151.6 GiB | 0.2 to 0.5% | | M.1-small, M.1-medium, 600cc | 212 to 318 GiB | 0.3 to 1.4% | | M.1-1.5xlarge, M.1-3xlarge | 637 to 1274 GiB | 0.1% | The largest pod's gap, 1.58 GiB, is about 22 times the smallest's, for about 670 times the limit. - Not measured: frees in a different order from allocation also leave partly empty slabs behind, and a 64 KiB-page slab holds 16 times as many regions. A heap with heavy churn could therefore grow this part of the term with its size. **Heap-profiling samples: linear in live small-object bytes.** - With `lg_prof_sample:19`, about one small allocation is sampled per 512 KiB allocated. The live sampled allocations therefore number about live small-object bytes divided by 512 KiB, and each takes at least one page. - That puts the term at roughly 12 to 25% of live small-object bytes with a 64 KiB page, against about 1% with a 4 KiB page. This is an estimate from the sampling rate. The measured term is 7 to 22% of `jemalloc_allocated` on the three local processes. - Large allocations are unaffected, so pods whose heap is mostly large allocations see a smaller share. **Resident memory, which the clusterd memory limiter counts, rises less than either term.** - The kernel page stays 4 KiB, so a sparsely used 64 KiB slab, or a 64 KiB extent holding one small sampled object, becomes resident only in the 4 KiB pages actually touched. - Locally, VmRSS was 32% higher on environmentd and 9 to 18% higher on the clusterd processes with the 64 KiB page, while the gap was about ten times higher. - That explanation fits the numbers but was not measured per term. ## Why the target-prefixed variable tikv-jemalloc-sys reads `<TARGET>_JEMALLOC_SYS_WITH_LG_PAGE` before `JEMALLOC_SYS_WITH_LG_PAGE` ([tikv/jemallocator#91](tikv/jemallocator#91), used this way in [tikv/jemallocator#122](tikv/jemallocator#122)). The unprefixed variable would also apply to aarch64 macOS builds, where the kernel page is 16 KiB. There jemalloc aborts at startup with "Unsupported system page size" ([tikv/jemallocator#80](tikv/jemallocator#80)). The prefixed form changes only the aarch64 Linux build. x86_64 already uses a 4 KiB page. Upstream moved to 64 KiB so one aarch64 binary runs on 4, 16 and 64 KiB kernels. The thread it closed, [jemalloc/jemalloc#2639](jemalloc/jemalloc#2639), notes the costs, including heap profiling. With this change, a Materialize aarch64 binary again fails to start on a kernel with pages larger than 4 KiB. v26.44.0 and earlier had the same constraint. Their 4 KiB-page jemalloc runs on our arm64 nodes, so those kernels use 4 KiB pages. ## Verification - `cargo build -p tikv-jemalloc-sys -vv` in the ci-builder on this branch passes `--with-lg-page=12` to jemalloc's configure, which reports `LG_PAGE : 12` for `aarch64-unknown-linux-gnu`. - The rc.3 build with the equivalent unprefixed variable reports `Page size: 4096` and gives the measurements above. No tests were added. Whether the published arm64 images report `Page size: 4096` can be checked by running the next main image with `MALLOC_CONF=stats_print:true clusterd --version`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This fixes #90.
It allows all environment variables that detected by
read_and_watch_envandread_and_watch_env_osto be general or target-specific, which allows some unavoidable cross-compiling scene.When compiling jemalloc-sys under Cargo projects,
TARGETis set. Variables can be set as target-specific in the form of<TARGET_TRIPLE>_MYENV. For example,AARCH64_APPLE_IOS_JEMALLOC_OVERRIDEwill setJEMALLOC_OVERRIDEwhen cargo is targeting aarch64-apple-ios, which is not affecting other targets.When
AARCH64_APPLE_IOS_JEMALLOC_OVERRIDEandJEMALLOC_OVERRIDEare both set, the build.rs will prefer prefixed one instead of the non-prefixed one. The unprefixed one would only be used when target-specific one is not set.