Skip to content

Use inline storage for value iterators#502

Merged
cataphract merged 3 commits into
masterfrom
glopes/reduce-malloc-contention
Jul 2, 2026
Merged

Use inline storage for value iterators#502
cataphract merged 3 commits into
masterfrom
glopes/reduce-malloc-contention

Conversation

@cataphract

Copy link
Copy Markdown
Contributor

Dramatically reduce malloc contention under musl. glibc also has significant improvements. See commit message for more details.

@cataphract
cataphract requested a review from a team as a code owner June 30, 2026 17:01
@datadog-datadog-prod-us1

This comment has been minimized.

The hot value_iterator path allocates two small vectors for every
iterator: the traversal stack and the key path prefix. On musl, those
allocations show up as allocator lock contention under the Rust helper,
especially when many worker requests run the WAF concurrently.

Replace those vectors with a local small_vector that stores the common
case inline and spills to std::vector only when the inline capacity is
exceeded.  The iterator stack is reduced to an initial capacity of 16
entries inline.  The key path prefix keeps 8 entries inline.

Methodology
===========

Benchmark host:
- ubuntu vm on Ryzen 9 9950X with 14 processors and 64 GB RAM dedicated
- Docker-based reproduction using the minimal FrankenPHP worker app from
  the reported issue
  (DataDog/dd-trace-php#3896 (comment)).
  The app runs in FrankenPHP worker mode with opcache preload and
  returns "ok" for every request.
- dd-trace 1.22.0 with AppSec enabled and Rust helper redirection enabled:
  DD_APPSEC_ENABLED=true
  DD_APPSEC_HELPER_RUST_REDIRECTION=true

Build inputs:
- Both baseline and small_vector builds use the no-exception user_resource
  change: user_resource is final and memory_resource equality uses pointer
  dynamic_cast instead of a throwing reference dynamic_cast.
  See #501
- libddwaf was built from source with docker/libddwaf/build/Dockerfile for
  x86_64.
- The produced libddwaf musl archive was extracted into a LIBDDWAF_PREFIX.
- When linking helper-rust through LIBDDWAF_PREFIX, Unwind* archive members
  were removed from libddwaf.a so the helper link uses the same unwinder input
  as the intended helper build path.
- helper-rust was built in datadog/dd-appsec-php-ci:nginx-fpm-php-8.5-release-musl
  with:
    LIBDDWAF_PREFIX=/libddwaf-prefix
    cargo +nightly-$(uname -m)-unknown-linux-musl build --release \
      -Zhost-config -Ztarget-applies-to-host \
      --target $(uname -m)-unknown-linux-musl
- The resulting libddappsec-helper-rust.so replaced the helper in the 1.22.0
  test images.

Load generation and perf collection:
- Each case starts a fresh container and discovers the helper pid with
  docker top.
- perf stat is attached to the helper pid only:
    perf stat -x, -p $helper_pid \
      -e task-clock,context-switches,cpu-migrations,cycles,instructions,\
         syscalls:sys_enter_futex,syscalls:sys_enter_recvfrom,\
         syscalls:sys_enter_sendto,syscalls:sys_enter_mmap,\
         syscalls:sys_enter_munmap \
      -- sleep 15
- The harness starts perf, waits 1 second, then starts k6. The perf window is
  therefore a 15 second helper-only sample overlapping the 25 second k6 run.
- Fixed-rate k6 case:
    constant-arrival-rate, rate=1000/s, duration=25s,
    preAllocatedVUs=100, maxVUs=1000.
- Saturated k6 case:
    vus=128, duration=25s.

Results
=======

Baseline numbers below are no-exception baseline values. The
small_vector rows are from a rerun with a worktree identical to this
comit.

Summary deltas:
- glibc fixed 1000 rps: avg latency -8.4%, p95 latency -7.6%, helper
  task-clock -4.2%, instructions -7.8%.
- glibc saturated: throughput +6.9%, avg latency -6.4%, helper task-clock
  -4.3%, instructions -8.3%.
- musl fixed 1000 rps: avg latency -9.7%, p95 latency -14.9%, helper
  task-clock -9.1%, instructions -14.7%, futex syscalls -50.1%.
- musl saturated: throughput +167.4%, avg latency -62.6%, p95 latency
  -65.5%, helper task-clock/request -70.8%, futex syscalls/request -75.0%.

glibc fixed 1000 rps, no-exception baseline:
- k6:
  - http_req_duration: avg=543.17us min=350.82us med=518.75us max=19.67ms
    p90=615.11us p95=662.81us
  - http_reqs: 25000, 999.973757/s
  - iterations: 25000, 999.973757/s
  - vus_max: 100 min=100 max=100
- perf stat, helper pid only:
  - task-clock: 5243.58 ms
  - context-switches: 132416
  - cpu-migrations: 203
  - cycles: 15630911903
  - instructions: 15720001601
  - syscalls:sys_enter_futex: 88319
  - syscalls:sys_enter_recvfrom: 28742
  - syscalls:sys_enter_sendto: 34
  - syscalls:sys_enter_mmap: 13909
  - syscalls:sys_enter_munmap: 9264

glibc fixed 1000 rps, no-exception + small_vector:
- k6:
  - http_req_duration: avg=497.74us min=328.25us med=475.44us max=15.24ms
    p90=572.33us p95=612.62us
  - http_reqs: 25001, 999.977188/s
  - iterations: 25001, 999.977188/s
  - vus_max: 100 min=100 max=100
- perf stat, helper pid only:
  - task-clock: 5021.74 ms
  - context-switches: 134145
  - cpu-migrations: 129
  - cycles: 14471814253
  - instructions: 14499136714
  - syscalls:sys_enter_futex: 89303
  - syscalls:sys_enter_recvfrom: 28739
  - syscalls:sys_enter_sendto: 34
  - syscalls:sys_enter_mmap: 13897
  - syscalls:sys_enter_munmap: 9236

Comparison, glibc fixed 1000 rps:
- avg latency: 543.17us -> 497.74us (-8.4%)
- p95 latency: 662.81us -> 612.62us (-7.6%)
- task-clock: 5243.58 ms -> 5021.74 ms (-4.2%)
- instructions: 15720001601 -> 14499136714 (-7.8%)
- futex syscalls: 88319 -> 89303 (+1.1%)

glibc saturated, no-exception baseline:
- k6:
  - http_req_duration: avg=9.94ms min=680.73us med=8.96ms max=2.05s
    p90=11.76ms p95=12.84ms
  - http_reqs: 330401, 12810.603878/s
  - iterations: 330401, 12810.603878/s
  - vus_max: 128 min=128 max=128
- perf stat, helper pid only:
  - task-clock: 68135.12 ms
  - context-switches: 1237062
  - cpu-migrations: 21476
  - cycles: 225350789941
  - instructions: 197768864688
  - syscalls:sys_enter_futex: 1360957
  - syscalls:sys_enter_recvfrom: 369385
  - syscalls:sys_enter_sendto: 96
  - syscalls:sys_enter_mmap: 178627
  - syscalls:sys_enter_munmap: 60535

glibc saturated, no-exception + small_vector:
- k6:
  - http_req_duration: avg=9.3ms min=436.15us med=8.85ms max=777.96ms
    p90=11.97ms p95=13.19ms
  - http_reqs: 342555, 13699.583225/s
  - iterations: 342555, 13699.583225/s
  - vus_max: 128 min=128 max=128
- perf stat, helper pid only:
  - task-clock: 65193.36 ms
  - context-switches: 1218744
  - cpu-migrations: 18246
  - cycles: 212753962548
  - instructions: 181374724551
  - syscalls:sys_enter_futex: 1354117
  - syscalls:sys_enter_recvfrom: 369160
  - syscalls:sys_enter_sendto: 96
  - syscalls:sys_enter_mmap: 178496
  - syscalls:sys_enter_munmap: 66577

Comparison, glibc saturated:
- throughput: 12810.603878/s -> 13699.583225/s (+6.9%)
- avg latency: 9.94ms -> 9.3ms (-6.4%)
- p95 latency: 12.84ms -> 13.19ms (+2.7%)
- task-clock: 68135.12 ms -> 65193.36 ms (-4.3%)
- instructions: 197768864688 -> 181374724551 (-8.3%)
- futex syscalls: 1360957 -> 1354117 (-0.5%)

musl fixed 1000 rps, no-exception baseline:
- k6:
  - http_req_duration: avg=610.45us min=390.08us med=563.78us max=25.08ms
    p90=710.34us p95=793.91us
  - http_reqs: 25000, 999.951751/s
  - iterations: 25000, 999.951751/s
  - vus_max: 100 min=100 max=100
- perf stat, helper pid only:
  - task-clock: 5974.15 ms
  - context-switches: 138573
  - cpu-migrations: 248
  - cycles: 18865332259
  - instructions: 18781507893
  - syscalls:sys_enter_futex: 292064
  - syscalls:sys_enter_recvfrom: 28760
  - syscalls:sys_enter_sendto: 0
  - syscalls:sys_enter_mmap: 19454
  - syscalls:sys_enter_munmap: 14037

musl fixed 1000 rps, no-exception + small_vector:
- k6:
  - http_req_duration: avg=551.06us min=357.95us med=524.59us max=19.96ms
    p90=629.56us p95=675.68us
  - http_reqs: 25001, 1000.008088/s
  - iterations: 25001, 1000.008088/s
  - vus_max: 100 min=100 max=100
- perf stat, helper pid only:
  - task-clock: 5432.17 ms
  - context-switches: 136657
  - cpu-migrations: 153
  - cycles: 16235435272
  - instructions: 16015330444
  - syscalls:sys_enter_futex: 145783
  - syscalls:sys_enter_recvfrom: 28760
  - syscalls:sys_enter_sendto: 0
  - syscalls:sys_enter_mmap: 20023
  - syscalls:sys_enter_munmap: 14594

Comparison, musl fixed 1000 rps:
- avg latency: 610.45us -> 551.06us (-9.7%)
- p95 latency: 793.91us -> 675.68us (-14.9%)
- task-clock: 5974.15 ms -> 5432.17 ms (-9.1%)
- instructions: 18781507893 -> 16015330444 (-14.7%)
- futex syscalls: 292064 -> 145783 (-50.1%)

musl saturated, no-exception baseline:
- k6:
  - http_req_duration: avg=49.5ms min=9.61ms med=51.98ms max=82.96ms
    p90=63.65ms p95=66.41ms
  - http_reqs: 64648, 2580.320337/s
  - iterations: 64648, 2580.320337/s
  - vus_max: 128 min=128 max=128
- perf stat, helper pid only:
  - task-clock: 135302.10 ms
  - context-switches: 2864560
  - cpu-migrations: 4984
  - cycles: 446011049906
  - instructions: 151276290080
  - syscalls:sys_enter_futex: 45470580
  - syscalls:sys_enter_recvfrom: 82553
  - syscalls:sys_enter_sendto: 0
  - syscalls:sys_enter_mmap: 63221
  - syscalls:sys_enter_munmap: 40523

musl saturated, no-exception + small_vector:
- k6:
  - http_req_duration: avg=18.51ms min=7.39ms med=18.23ms max=65.77ms
    p90=21.74ms p95=22.93ms
  - http_reqs: 172599, 6899.350591/s
  - iterations: 172599, 6899.350591/s
  - vus_max: 128 min=128 max=128
- perf stat, helper pid only:
  - task-clock: 105390.34 ms
  - context-switches: 2277070
  - cpu-migrations: 12171
  - cycles: 340696185207
  - instructions: 153580862953
  - syscalls:sys_enter_futex: 30372913
  - syscalls:sys_enter_recvfrom: 187451
  - syscalls:sys_enter_sendto: 0
  - syscalls:sys_enter_mmap: 30204
  - syscalls:sys_enter_munmap: 26087

Comparison, musl saturated:
- throughput: 2580.320337/s -> 6899.350591/s (+167.4%)
- avg latency: 49.5ms -> 18.51ms (-62.6%)
- p95 latency: 66.41ms -> 22.93ms (-65.5%)
- task-clock: 135302.10 ms -> 105390.34 ms (-22.1%)
- task-clock/request: 2.0929 ms -> 0.6106 ms (-70.8%)
- instructions/request: 2.3400M -> 0.8898M (-62.0%)
- futex syscalls/request: 703.4 -> 176.0 (-75.0%)
@cataphract
cataphract force-pushed the glopes/reduce-malloc-contention branch from 074c92e to f110f40 Compare June 30, 2026 17:05

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 074c92ef8e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/iterator.hpp
Comment thread src/iterator.hpp Dismissed
Comment thread src/iterator.hpp Dismissed
@github-actions

github-actions Bot commented Jun 30, 2026

Copy link
Copy Markdown

Dynamic Artifact Size Comparison 📦

Artifact Previous Release This PR Difference
darwin-arm64::libddwaf.dylib 2020576 2020272 -0.01%
darwin-universal::libddwaf.dylib 4248800 4232112 -0.39%
darwin-x86_64::libddwaf.dylib 2208040 2207728 -0.01%
linux-aarch64::libddwaf.so 2388576 2387392 -0.04%
linux-armv7::libddwaf.so 2056400 2056572 0.00%
linux-i386::libddwaf.so 2312900 2308204 -0.20%
linux-x86_64::libddwaf.so 2581376 2581096 -0.01%
windows-arm64::ddwaf.dll 6731264 6726144 -0.07%
windows-win32::ddwaf.dll 3330560 3328000 -0.07%
windows-x64::ddwaf.dll 4061184 4066816 0.13%

Static Artifact Size Comparison 📦

Artifact Previous Release This PR Difference
darwin-arm64::libddwaf.a 92531240 92555960 0.02%
darwin-arm64::libddwaf.a.stripped 4715920 4717512 0.03%
darwin-universal::libddwaf.a 186577856 186638424 0.03%
darwin-universal::libddwaf.a.stripped 10041896 10047056 0.05%
darwin-x86_64::libddwaf.a 94046568 94082416 0.03%
darwin-x86_64::libddwaf.a.stripped 5325928 5329496 0.06%
linux-aarch64::libddwaf.a 75455020 75526438 0.09%
linux-aarch64::libddwaf.a.stripped 12184592 12185562 0.00%
linux-armv7::libddwaf.a 66674258 66734102 0.08%
linux-armv7::libddwaf.a.stripped 11189582 11193482 0.03%
linux-i386::libddwaf.a 64858564 64921342 0.09%
linux-i386::libddwaf.a.stripped 9676948 9680046 0.03%
linux-x86_64::libddwaf.a 75942542 76013242 0.09%
linux-x86_64::libddwaf.a.stripped 11985466 11989382 0.03%
windows-arm64::ddwaf.lib 16902 16902 0.00%
windows-arm64::ddwaf_static.lib 135254130 134713692 -0.39%
windows-win32::ddwaf.lib 17228 17228 0.00%
windows-win32::ddwaf_static.lib 47779818 47488650 -0.60%
windows-x64::ddwaf.lib 16902 16902 0.00%
windows-x64::ddwaf_static.lib 55616486 55265732 -0.63%

Comment thread src/iterator.hpp Dismissed
@cataphract
cataphract requested a review from Taiki-San July 2, 2026 09:24

@Taiki-San Taiki-San left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good.

The pointer arithmetic around the access to inline_ptr is a little bit scary/brittle (I'm thinking about race conditions or new call sites without a boundary check) but the current implementation/usage is fine

@cataphract
cataphract enabled auto-merge (squash) July 2, 2026 12:32
@cataphract
cataphract merged commit 7c0349a into master Jul 2, 2026
55 of 56 checks passed
@cataphract
cataphract deleted the glopes/reduce-malloc-contention branch July 2, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants