Skip to content

allow per-target env detection - #91

Merged
BusyJay merged 4 commits into
tikv:mainfrom
Torrekie:patch-per-target-env
Jul 14, 2024
Merged

BusyJay merged 4 commits into
tikv:mainfrom
Torrekie:patch-per-target-env

Conversation

@Torrekie

Copy link
Copy Markdown

This fixes #90.

It allows all environment variables that detected by read_and_watch_env and read_and_watch_env_os to be general or target-specific, which allows some unavoidable cross-compiling scene.

When compiling jemalloc-sys under Cargo projects, TARGET is set. Variables can be set as target-specific in the form of <TARGET_TRIPLE>_MYENV. For example, AARCH64_APPLE_IOS_JEMALLOC_OVERRIDE will set JEMALLOC_OVERRIDE when cargo is targeting aarch64-apple-ios, which is not affecting other targets.

When AARCH64_APPLE_IOS_JEMALLOC_OVERRIDE and JEMALLOC_OVERRIDE are both set, the build.rs will prefer prefixed one instead of the non-prefixed one. The unprefixed one would only be used when target-specific one is not set.

@Torrekie
Torrekie force-pushed the patch-per-target-env branch from b316bf7 to c2234c2 Compare June 17, 2024 10:00
Comment thread jemalloc-sys/build.rs Resolved
Torrekie and others added 2 commits June 17, 2024 18:06
Signed-off-by: Torrekie <[email protected]>
@Torrekie
Torrekie force-pushed the patch-per-target-env branch from c2234c2 to 88b54a4 Compare June 17, 2024 10:08
Comment thread jemalloc-sys/build.rs Resolved
Comment thread jemalloc-sys/build.rs Resolved
@BusyJay

BusyJay commented Jun 21, 2024

Copy link
Copy Markdown
Member

You can ignore the aarch64 failure. The clippy failure is not related to your change, but if you can fix it in this PR we can merge it all together.

@BusyJay BusyJay left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@BusyJay
BusyJay merged commit 2fde3d8 into tikv:main Jul 14, 2024
bosconi added a commit to MaterializeInc/materialize that referenced this pull request Oct 5, 2026
## Summary

Pin jemalloc's allocator page to 4 KiB for `aarch64-unknown-linux-gnu`
builds by setting `AARCH64_UNKNOWN_LINUX_GNU_JEMALLOC_SYS_WITH_LG_PAGE =
"12"` in `.cargo/config.toml`.

[#39339](#39339)
bumped tikv-jemalloc-sys from 0.6.1 (jemalloc 5.3.0) to 0.7.1 (jemalloc
5.3.1). jemalloc 5.3.1 changed its default allocator page on aarch64
Linux from the build host's page to 64 KiB
([meta/jemalloc#34](meta/jemalloc#34)).
Neither #39339 nor the tikv-jemallocator [0.7.0 release
notes](https://github.com/tikv/jemallocator/releases/tag/0.7.0) mention
it, and I found no discussion of it in code, issues or Slack.

@antiguru confirmed on this PR that the move to 64 KiB was [not a
deliberate
choice](#39549 (comment)).
Every production and staging environmentd and clusterd pod runs on arm64
nodes, so rc.3 runs with a 64 KiB allocator page on 4 KiB-page kernels.

The effect is a step in `jemalloc_active` minus `jemalloc_allocated` in
every Materialize process. This PR restores the 4 KiB page that v26.44.0
and earlier had on aarch64, and keeps everything else in jemalloc 5.3.1.

## What was observed

The v26.45.0-rc.3 sign-off sweeps on staging and the production canaries
flagged `jemalloc_active` rising while `jemalloc_allocated` stayed flat
or fell, from process start, in environmentd and clusterd:

- environmentd on the us-east-1 canaries, v26.44.0 against rc.3 at equal
post-restart age: active minus allocated went from 55 MB to 490 MB on
one environment and from 84 MB to 558 MB on another.
- clusterd on the largest canary pods: from 0.19 GB to 1.70 GB, scaling
with pod size.
- No OOMs and no limit was hit.

## How it was bisected

All of this ran locally on an Apple Silicon Mac under OrbStack:
linux/arm64 natively, 4 KiB kernel page.

**1. Page size per image.** Running each image with
`MALLOC_CONF=stats_print:true <binary> --version` prints jemalloc's
compiled page size:

| Image | Arch | jemalloc | Page size |
|---|---|---|---|
| `environmentd`, `clusterd`, `materialized` `v26.44.0` | arm64 | 5.3.0
| 4096 |
| `environmentd`, `clusterd`, `materialized` `v26.44.0` | amd64 | 5.3.0
| 4096 |
| `environmentd`, `clusterd`, `materialized` `v26.45.0-rc.3` | arm64 |
5.3.1 | 65536 |
| `environmentd`, `clusterd` `v26.45.0-rc.3` | amd64 | 5.3.1 | 4096 |
|
`materialized:v26.45.0-dev.0--main.geb032636b571a61c1016d2a13f7398b5edceb8ff`
(parent of #39339) | arm64 | 5.3.0 | 4096 |
|
`materialized:v26.45.0-dev.0--main.g0d478e2dc6ba5361b948bc222c61968878508625`
(#39339) | arm64 | 5.3.1 | 65536 |

`kube_pod_container_info` shows every rc.3 environmentd and clusterd
container on the production canaries (us-east-1, eu-west-1, us-west-2)
and on staging (us-east-1, eu-west-1) running the Docker Hub index
digests that were tested.

**2. Workload bisection on per-commit images.** 150 of the 232 main
commits between the v26.44.0 and rc.3 branch points have `--main.g<sha>`
images on Docker Hub, including #39339 and its parent. A harness started
one fresh `materialized` container per run on a fixed timeline: 40 s
idle, then a fixed SQL workload on `quickstart` (a 1.5M-row table, an
index, an aggregating and a joining materialized view, deletes, some
catalog churn), then a 60 s settle and six samples 10 s apart. It
recorded jemalloc stats from environmentd's `/metrics` and from each
clusterd's internal HTTP socket, plus VmRSS. Each arm64 configuration
ran twice, interleaved.

Active minus allocated, after the workload, in MiB:

| Configuration | environmentd | `mz_catalog_server` | `quickstart` |
|---|---:|---:|---:|
| `v26.44.0` | 40 | 23 | 36 |
| main, parent of #39339 | 42 | 22 | 36 |
| main, #39339 | 366 | 233 | 390 |
| `v26.45.0-rc.3` | 363 | 222 | 373 |
| rc.3, `experimental_tcache_gc:false` | 393 | 235 | 372 |
| rc.3, `disable_large_size_classes:false` | 371 | 234 | 380 |
| rc.3, both of the above | 382 | 244 | 368 |
| `v26.44.0`, amd64 | 43 | 22 | 35 |
| rc.3, amd64 | 41 | 24 | 38 |

The step is in #39339, the only first-parent commit between the two main
images. The two new jemalloc 5.3.1 defaults, set back through
`MALLOC_CONF`, do not move it. rc.3 on amd64 runs the same jemalloc
5.3.1 with a 4 KiB page and has v26.44.0's gap.

**3. Page size alone.** rc.3 was rebuilt twice with `cargo build
--profile optimized --bin materialized` in the ci-builder: once
unchanged and once with `JEMALLOC_SYS_WITH_LG_PAGE=12`. Each binary
replaced `/usr/local/bin/materialized` in the released arm64 rc.3 image.
The 4 KiB build keeps `experimental_tcache_gc` and
`disable_large_size_classes` on.

| Build | environmentd gap | environmentd VmRSS | `mz_catalog_server`
gap | `mz_catalog_server` VmRSS | `quickstart` gap | `quickstart` VmRSS
|
|---|---:|---:|---:|---:|---:|---:|
| rc.3, 64 KiB page | 362 | 434 | 246 | 432 | 382 | 869 |
| rc.3, 4 KiB page | 43 | 328 | 24 | 395 | 36 | 739 |

Values in MiB, two runs each.

**4. Where the extra active memory sits.** Full `malloc_stats_print`
output was dumped from the live processes after the same workload, for
both rebuilt images. The table splits the 64 KiB minus 4 KiB gap
difference, in MiB:

| Process | Gap difference | Partially filled small-bin slabs |
Heap-profiling sampled allocations | Large-extent page pad |
|---|---:|---:|---:|---:|
| environmentd | 318 | 304 | 16 | -2 |
| `mz_catalog_server` | 217 | 164 | 59 | -6 |
| `quickstart` | 339 | 285 | 52 | 2 |

- Small-bin slabs are at least one page, so with a 64 KiB page each slab
is 320 to 448 KiB instead of 20 to 28 KiB. Each of the 48 arenas holds
about one partly used slab per size class, mostly under 25% full.
- With `prof:true` and `lg_prof_sample:19`, every sampled small
allocation gets its own page-aligned extent. That extent counts as
active and not as allocated, so a 64 KiB page makes each sample cost 64
KiB or more. With `prof_active:false` that term is zero.
- Cutting the arena count from 48 to 16 cut the slab term by 48 to 61%,
in single runs. jemalloc creates four arenas per CPU in the process's
affinity mask, which would explain why the production gap scales with
pod and node size. That link is inferred and not measured in production.

## How the cost scales

The two terms in step 4 scale differently, so the effect on small and
large pods differs.

**Partly filled slabs: roughly a fixed cost per process shape.**
- Each arena keeps about one partly used slab per size class in use, so
this term follows arenas times size classes in use, not heap size.
- jemalloc creates four arenas per CPU in the process's affinity mask.
Swap-enabled replica sizes do not get exclusive CPUs, so a clusterd
pod's arena count follows its node's vCPU count. That last step is
inferred from the replica-size config and was not measured on a pod.
- Locally, cutting the arena count from 48 to 16 cut this term by 48 to
61%.
- On the production canaries, the clusterd gap as a share of
`mz_memory_limiter_memory_limit_bytes` falls as pods grow. This is a
snapshot at 2026-10-04 05:40 UTC across different workloads, not a
controlled scaling test:

  | Replica sizes | Limiter limit | Gap / limit |
  |---|---:|---:|
  | `mz_probe` | 1.9 GiB | 3.9 to 6.0% |
  | 25cc | 9.5 GiB | 0.8 to 2.4% |
  | 50cc | 19.0 GiB | 0.5% |
  | M.1-nano, M.1-micro | 26.5 to 53.1 GiB | 0.2 to 0.7% |
  | 100cc to 400cc, M.1-xsmall | 37.9 to 151.6 GiB | 0.2 to 0.5% |
  | M.1-small, M.1-medium, 600cc | 212 to 318 GiB | 0.3 to 1.4% |
  | M.1-1.5xlarge, M.1-3xlarge | 637 to 1274 GiB | 0.1% |

The largest pod's gap, 1.58 GiB, is about 22 times the smallest's, for
about 670 times the limit.
- Not measured: frees in a different order from allocation also leave
partly empty slabs behind, and a 64 KiB-page slab holds 16 times as many
regions. A heap with heavy churn could therefore grow this part of the
term with its size.

**Heap-profiling samples: linear in live small-object bytes.**
- With `lg_prof_sample:19`, about one small allocation is sampled per
512 KiB allocated. The live sampled allocations therefore number about
live small-object bytes divided by 512 KiB, and each takes at least one
page.
- That puts the term at roughly 12 to 25% of live small-object bytes
with a 64 KiB page, against about 1% with a 4 KiB page. This is an
estimate from the sampling rate. The measured term is 7 to 22% of
`jemalloc_allocated` on the three local processes.
- Large allocations are unaffected, so pods whose heap is mostly large
allocations see a smaller share.

**Resident memory, which the clusterd memory limiter counts, rises less
than either term.**
- The kernel page stays 4 KiB, so a sparsely used 64 KiB slab, or a 64
KiB extent holding one small sampled object, becomes resident only in
the 4 KiB pages actually touched.
- Locally, VmRSS was 32% higher on environmentd and 9 to 18% higher on
the clusterd processes with the 64 KiB page, while the gap was about ten
times higher.
- That explanation fits the numbers but was not measured per term.

## Why the target-prefixed variable

tikv-jemalloc-sys reads `<TARGET>_JEMALLOC_SYS_WITH_LG_PAGE` before
`JEMALLOC_SYS_WITH_LG_PAGE`
([tikv/jemallocator#91](tikv/jemallocator#91),
used this way in
[tikv/jemallocator#122](tikv/jemallocator#122)).
The unprefixed variable would also apply to aarch64 macOS builds, where
the kernel page is 16 KiB. There jemalloc aborts at startup with
"Unsupported system page size"
([tikv/jemallocator#80](tikv/jemallocator#80)).
The prefixed form changes only the aarch64 Linux build. x86_64 already
uses a 4 KiB page.

Upstream moved to 64 KiB so one aarch64 binary runs on 4, 16 and 64 KiB
kernels. The thread it closed,
[jemalloc/jemalloc#2639](jemalloc/jemalloc#2639),
notes the costs, including heap profiling. With this change, a
Materialize aarch64 binary again fails to start on a kernel with pages
larger than 4 KiB. v26.44.0 and earlier had the same constraint. Their 4
KiB-page jemalloc runs on our arm64 nodes, so those kernels use 4 KiB
pages.

## Verification

- `cargo build -p tikv-jemalloc-sys -vv` in the ci-builder on this
branch passes `--with-lg-page=12` to jemalloc's configure, which reports
`LG_PAGE : 12` for `aarch64-unknown-linux-gnu`.
- The rc.3 build with the equivalent unprefixed variable reports `Page
size: 4096` and gives the measurements above.

No tests were added. Whether the published arm64 images report `Page
size: 4096` can be checked by running the next main image with
`MALLOC_CONF=stats_print:true clusterd --version`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Allow per-target JEMALLOC_OVERRIDE env

2 participants