Skip to content

feat(profiling): add heap-live profiling for memory leak detection#3623

Merged
realFlowControl merged 30 commits into
masterfrom
florian/heap-live-profiling
Jun 12, 2026
Merged

feat(profiling): add heap-live profiling for memory leak detection#3623
realFlowControl merged 30 commits into
masterfrom
florian/heap-live-profiling

Conversation

@realFlowControl

@realFlowControl realFlowControl commented Feb 4, 2026

Copy link
Copy Markdown
Member

Description

Track allocations that survive across profile exports using heap-live-samples and heap-live-size sample types. Samples are emitted in batches at export time.

Enable via DD_PROFILING_EXPERIMENTAL_HEAP_LIVE_ENABLED or datadog.profiling.experimental_heap_live_enabled (default disabled), only works when allocation profiling is active.

Note

This feature is default disabled and if you are running a SAPI like PHP-FPM or mod_php (Apache) that is purely request/response based, you do not get any value out of this. This feature is meant for long running process (CLI workers consuming messages, ReactPHP applications, FrankenPHP Worker Mode, ...).

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

PROF-13688

@github-actions github-actions Bot added the profiling Relates to the Continuous Profiler label Feb 4, 2026
@datadog-official

datadog-official Bot commented Feb 4, 2026

Copy link
Copy Markdown

Pipelines  Tests

Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🚦 11 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-php | ASAN Opcache tests: [8.5]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-php | ASAN test_c with multiple observers: [8.4]   View in Datadog   GitLab

DataDog/apm-reliability/dd-trace-php | ASAN test_c with multiple observers: [8.5]   View in Datadog   GitLab

View all 11 failed jobs.

❄️ 1 New flaky test detected

profiling/tests/phpt/heap_live_phpinfo_04.phpt ([profiling] heap live profiling with active JIT) from php.profiling.tests.phpt   View in Datadog (Fix with Cursor)
     string(%d) "%s"
     string(%d) "%s"
     Done.

New test introduced in this PR is flaky.

View in Flaky Test Management

ℹ️ Info

No other issues found (see more)

🧪 All tests passed

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 54.12% (+0.00%)

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 446a8d9 | Docs | Datadog PR Page | Give us feedback!

@pr-commenter

pr-commenter Bot commented Feb 4, 2026

Copy link
Copy Markdown

Benchmarks [ profiler ]

Benchmark execution time: 2026-06-10 11:12:20

Comparing candidate commit 446a8d9 in PR branch florian/heap-live-profiling with baseline commit c05c43c in branch master.

Found 0 performance improvements and 9 performance regressions! Performance is the same for 19 metrics, 8 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:php-profiler-exceptions-with-profiler

  • 🟥 execution_time [+8.303ms; +13.109ms] or [+9.044%; +14.279%]

scenario:php-profiler-exceptions-with-profiler-and-timeline

  • 🟥 cpu_usage_percentage [+2.215%; +3.844%]
  • 🟥 execution_time [+8.627ms; +15.562ms] or [+4.488%; +8.096%]

scenario:php-profiler-timeline-memory-control

  • 🟥 cpu_user_time [+34.405ms; +42.512ms] or [+5.733%; +7.083%]
  • 🟥 execution_time [+36.059ms; +40.262ms] or [+5.740%; +6.409%]

scenario:php-profiler-timeline-memory-with-profiler

  • 🟥 cpu_user_time [+106.449ms; +151.601ms] or [+8.750%; +12.462%]
  • 🟥 execution_time [+88.117ms; +106.896ms] or [+8.886%; +10.779%]

scenario:php-profiler-timeline-memory-with-profiler-and-timeline

  • 🟥 cpu_user_time [+61.652ms; +109.016ms] or [+4.451%; +7.871%]
  • 🟥 execution_time [+49.706ms; +71.891ms] or [+4.048%; +5.854%]

@realFlowControl
realFlowControl force-pushed the florian/heap-live-profiling branch from bc087f2 to 817465a Compare February 4, 2026 21:03
@codecov-commenter

codecov-commenter commented Feb 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.20%. Comparing base (de9c414) to head (91f9d71).

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3623      +/-   ##
==========================================
+ Coverage   62.11%   62.20%   +0.08%     
==========================================
  Files         141      141              
  Lines       13387    13387              
  Branches     1753     1753              
==========================================
+ Hits         8315     8327      +12     
+ Misses       4273     4263      -10     
+ Partials      799      797       -2     

see 2 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update de9c414...91f9d71. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pr-commenter

pr-commenter Bot commented Feb 5, 2026

Copy link
Copy Markdown

Benchmarks [ tracer ]

Benchmark execution time: 2026-06-10 08:43:23

Comparing candidate commit 703fec2 in PR branch florian/heap-live-profiling with baseline commit 6a0bbf7 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 194 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@realFlowControl
realFlowControl force-pushed the florian/heap-live-profiling branch 5 times, most recently from 4aaffbf to 540cc19 Compare February 10, 2026 15:22
@realFlowControl
realFlowControl force-pushed the florian/heap-live-profiling branch 2 times, most recently from 0ecb066 to 79ff80d Compare February 11, 2026 17:55
realFlowControl and others added 3 commits February 12, 2026 08:19
Track allocations that survive across profile exports using heap-live-samples
and heap-live-size sample types. Samples are emitted in batches at export time.

Enabled via DD_PROFILING_HEAP_LIVE_ENABLED when allocation profiling is active.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
- Use functional style (map + match) in collect_batched_heap_live_samples
- Only create ProfileIndex when heap-live tracking is enabled
- Replace 32 repetitive I/O profiling lines with a loop
- Use filter_map in sample type filter method
- Add early bail-out in free_allocation when heap-live is disabled

Co-Authored-By: Claude Opus 4.5 <[email protected]>
Replace default SipHash with a simple bit-mixing hasher optimized for
pointer addresses. Since pointers are already well-distributed, we use
`ptr ^ (ptr >> 4)` instead of expensive cryptographic hashing.

This reduces overhead in untrack_allocation() which is called on every free.

Co-Authored-By: Claude Opus 4.5 <[email protected]>
@realFlowControl
realFlowControl force-pushed the florian/heap-live-profiling branch 2 times, most recently from d87cdf9 to 91f9d71 Compare February 12, 2026 07:46
realFlowControl and others added 6 commits April 30, 2026 12:14
- Cargo.lock: drop unrelated lockfile churn so the diff vs master
  only contains entries cargo actually pulled in for dashmap.
- SampleData / LiveHeapSample: Arc-share frames and labels so the
  in-flight sample message and the heap-live tracker share one
  allocation each, and re-emitting tracked entries on every export
  becomes Arc::clone instead of deep clone.
- realloc: guard against NULL return so we don't free-track a still-
  live prev_ptr.
- visibility: tighten heap-live helpers to pub(crate); drop the
  unused clear_live_heap_tracker / live_heap_tracker_len helpers.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
ProfileIndex is effectively a singleton per request (sample_types fixed at
startup, tags resolved once per rinit), but it was being deep-cloned on
every prepare_sample_message, every HashMap insert, every heap-live track,
and every heap-live emission.

Wrap it in Arc so:
 - SampleMessage::key and LiveHeapSample::key share one allocation.
 - HashMap<Arc<ProfileIndex>, InternalProfile> keys via Arc's Hash/Eq
   (which delegate to ProfileIndex), so behavior is unchanged.
 - The heap-live re-emission path becomes a pure Arc::clone instead of a
   deep Vec<ValueType> clone.

Behavior-preserving refactor. Pairs with the existing Arc<Backtrace> /
Arc<Vec<Label>> work on SampleData.
The previous implementation paid for a Profiler::get() (OnceLock atomic
load) + AtomicPtr load + SystemSettings deref on every efree/erealloc,
even when heap-live profiling was disabled. That overhead is likely
behind the timeline-memory bench regression observed in the PoC, since
that benchmark exercises the free path heavily without enabling
heap-live.

Add a static AtomicBool that mirrors profiling_heap_live_enabled and
gate the alloc handlers (alloc_prof_free, alloc_prof_realloc) on it
*before* touching Profiler::get(). When heap-live is off the off-path
collapses to a single relaxed load + branch.

The static is set in Profiler::new and cleared in Profiler::stop /
Profiler::kill, so teardown is also safe (any late free during shutdown
short-circuits cleanly).

Profiler::is_heap_live_enabled now reads the same static so all gates
agree on one source of truth, dropping the AtomicPtr/deref dance on the
allocation-sampling hot path as well.
@realFlowControl realFlowControl changed the title PoC: feat(profiling): add heap-live profiling for memory leak detection feat(profiling): add heap-live profiling for memory leak detection May 26, 2026
@realFlowControl
realFlowControl marked this pull request as ready for review May 26, 2026 10:53
@realFlowControl
realFlowControl requested review from a team as code owners May 26, 2026 10:53
realFlowControl and others added 5 commits May 27, 2026 13:19
…unter

DashMap::len() acquires a read-lock on all 16 shards per call, paying
16 lock acquisitions on every sampled allocation just for a racy cap
check. Replace with a shared AtomicUsize that is incremented on insert
and decremented on remove; a single Relaxed load serves the cap check
with identical TOCTOU semantics.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
…ce lock contention

DashMap::iter() holds a per-shard read lock for each entry while in
scope. With the previous code, handle_sample_message was called while
the lock was still held, blocking concurrent efree calls on PHP threads
for the full duration of a 4096-entry export iteration.

Snapshot all values into a Vec first (cheap: just Arc::clone per field),
then process with no locks held.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
…n free/update methods

Call sites in the allocation hooks already gate on HEAP_LIVE_ENABLED
before calling free_allocation and update_allocation_size. The inner
checks were dead code and misleading. Replace with debug_assert and a
precondition doc comment.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
The XOR-shift mix (ptr ^ (ptr >> 4)) leaves sequential bump-allocator
addresses correlated in the high bits, clustering them onto the same
DashMap shards under ZTS workloads. FxHasher's multiply-rotate mix
fully avalanches bits, spreading sequential pointers evenly across all
16 shards. rustc-hash is already a direct dependency.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
…eap-live sample values

Replace fragile string matching on "heap-live-samples"/"heap-live-size"
in collect_batched_heap_live_samples with SampleTypeFilter::filter(). A
rename or typo in the type strings previously produced silent all-zero
heap-live exports. Derive Clone on SampleTypeFilter and pass a clone to
TimeCollector.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <[email protected]>
Comment thread profiling/src/allocation/mod.rs Outdated
realFlowControl and others added 8 commits May 27, 2026 20:30
Previously, in-place reallocs called update_allocation_size() which
mutated the tracked allocation_size from S0 to S1 in place. This broke
the upscaling invariant: the Poisson estimator requires that the
sampling decision was made at the size used in the upscale formula, but
the allocation was sampled at S0 while the upscaler saw S1.

Fix by treating in-place reallocs the same as pointer-moving reallocs:
untrack the old allocation and re-sample at the new size. The sampling
decision is then always made at the reported size, keeping the estimator
unbiased. In-place reallocs are rare in PHP, so the performance impact
is negligible.

Remove update_allocation_size() from Profiler, allocation/mod.rs, and
both allocation handler files entirely.
Comment thread profiling/src/allocation/mod.rs Outdated

@morrisonlevi morrisonlevi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the expectation for heap profiles and correlation to traces, spans, and endpoints? Should we even be gathering the local root span id/span id for these?

@realFlowControl
realFlowControl force-pushed the florian/heap-live-profiling branch from 451c010 to 08f80e1 Compare June 10, 2026 07:04
@realFlowControl
realFlowControl force-pushed the florian/heap-live-profiling branch from 08f80e1 to 703fec2 Compare June 10, 2026 07:25
@realFlowControl

Copy link
Copy Markdown
Member Author

What's the expectation for heap profiles and correlation to traces, spans, and endpoints? Should we even be gathering the local root span id/span id for these?

I think we should yes. The Heap Live profiles show what triggered that allocation. Also: we are sharing the sample message with the allocation profiler, so once all that information is collected, we already have it anyway (it would be extra work to remove that). What would be nice is if in the backend we could go from a stack trace to the span in the tracer (it is possible the other way round, just not back), but I do not know if that is too much important at all.

So TLDR: I am in favor of keeping that information, also because it is consistent with the allocation profiler.

@morrisonlevi morrisonlevi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code wise, it's looking okay to me. I've still not played much with its functionality. However, it's off by default so at this stage I'm willing to approve.

@realFlowControl
realFlowControl merged commit a5a1032 into master Jun 12, 2026
2123 of 2135 checks passed
@realFlowControl
realFlowControl deleted the florian/heap-live-profiling branch June 12, 2026 07:09
@github-actions github-actions Bot added this to the 1.22.0 milestone Jun 12, 2026
bwoebi added a commit that referenced this pull request Jun 12, 2026
…set-ddtrace-retry-interval

* 'master' of github.com:DataDog/dd-trace-php:
  libdatadog update to 6760faae (#3944)
  feat(profiling): add heap-live profiling for memory leak detection (#3623)
  chore: add kube pod annotation to debug request-replayer (#3978)
@realFlowControl

Copy link
Copy Markdown
Member Author

@ajohnston-knak version 1.22.0 has been released bringing a public preview of heap live profiling for PHP.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

profiling Relates to the Continuous Profiler

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants