Skip to content

feat(profiler): support the experimental goroutine leak profile#4420

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
nick.ripley/goroutine-leak-profile
Mar 11, 2026
Merged

feat(profiler): support the experimental goroutine leak profile#4420
gh-worker-dd-mergequeue-cf854d[bot] merged 2 commits into
mainfrom
nick.ripley/goroutine-leak-profile

Conversation

@nsrip-dd

@nsrip-dd nsrip-dd commented Feb 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Go 1.26 added an experimental goroutine leak profiler. See the proposal
in https://go.dev/issue/74609. This profiler collects tracebacks for
goroutines which are permanently blocked on synchronization, e.g. a
goroutine blocked on acquiring a locked mutex where that goroutine has
the only remaining reference to the mutex. This profiler is only
available if the program is built with GOEXPERIMENT=goroutineleakprofile

This commit updates the profiler to collect this profile if it is
available. Currently, the profile will be collected unconditionally if
the program is built with that GOEXPERIMENT, under the assumption that
the presence of that GOEXPERIMENT means the user wants that data. This
commit doesn't introduce any other knobs to toggle the profile type.
Since the profile type is experimental and the runtime API isn't
finalized, we shouldn't commit to a public API in our library.

@codecov

codecov Bot commented Feb 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.00000% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 60.03%. Comparing base (cf61946) to head (fe0699a).

Files with missing lines Patch % Lines
profiler/profiler.go 53.84% 4 Missing and 2 partials ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
profiler/compression.go 86.66% <100.00%> (ø)
profiler/profile.go 84.58% <ø> (ø)
profiler/telemetry.go 88.37% <100.00%> (ø)
profiler/profiler.go 88.76% <53.84%> (ø)

... and 429 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@nsrip-dd
nsrip-dd force-pushed the nick.ripley/goroutine-leak-profile branch from 3e084dc to a3b34cb Compare February 13, 2026 20:03
@pr-commenter

pr-commenter Bot commented Feb 13, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-03-09 15:13:17

Comparing candidate commit fe0699a in PR branch nick.ripley/goroutine-leak-profile with baseline commit cf61946 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 156 metrics, 8 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@nsrip-dd
feat(profiler): support the experimental goroutine leak profile
6b793d7 
Go 1.26 added an experimental goroutine leak profiler. See the proposal
in https://go.dev/issue/74609. This profiler collects tracebacks for
goroutines which are permanently blocked on synchronization, e.g. a
goroutine blocked on acquiring a locked mutex where that goroutine has
the only remaining reference to the mutex. This profiler is only
available if the program is built with GOEXPERIMENT=goroutineleakprofile

This commit updates the profiler to collect this profile if it is
available. Currently, the profile will be collected unconditionally if
the program is built with that GOEXPERIMENT, under the assumption that
the presence of that GOEXPERIMENT means the user wants that data. This
commit doesn't introduce any other knobs to toggle the profile type.
Since the profile type is experimental and the runtime API isn't
finalized, we shouldn't commit to a public API in our library.
@nsrip-dd
nsrip-dd force-pushed the nick.ripley/goroutine-leak-profile branch from a3b34cb to 6b793d7 Compare March 9, 2026 13:09
@nsrip-dd nsrip-dd changed the title WIP: feat(profiler): support the experimental goroutine leak profile feat(profiler): support the experimental goroutine leak profile Mar 9, 2026
@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Mar 9, 2026 •
edited by datadog-official Bot
Loading

Copy link
Copy Markdown

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

❄️ 3 New flaky tests detected

TestGoroutineLeakProfile from github.com/DataDog/dd-trace-go/v2/profiler (Datadog) (Fix with Cursor) 
Failed

=== RUN   TestGoroutineLeakProfile
--- FAIL: TestGoroutineLeakProfile (46.43s)
TestGoroutineLeakProfile/with_experiment from github.com/DataDog/dd-trace-go/v2/profiler (Datadog) (Fix with Cursor) 
Failed

=== RUN   TestGoroutineLeakProfile/with_experiment
    testing.go:1464: TempDir RemoveAll cleanup: unlinkat C: The process cannot access the file because it is being used by another process.
    --- FAIL: TestGoroutineLeakProfile/with_experiment (38.91s)
TestGoroutineLeakProfile/without_experiment from github.com/DataDog/dd-trace-go/v2/profiler (Datadog) (Fix with Cursor) 
Failed

=== RUN   TestGoroutineLeakProfile/without_experiment
    testing.go:1464: TempDir RemoveAll cleanup: unlinkat C: The process cannot access the file because it is being used by another process.
    --- FAIL: TestGoroutineLeakProfile/without_experiment (7.52s)

ℹ️ Info

🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 60.00%
• Overall Coverage: 59.15% (+3.41%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fe0699a | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@nsrip-dd
nsrip-dd force-pushed the nick.ripley/goroutine-leak-profile branch from 6d35d80 to fde1f15 Compare March 9, 2026 14:26
@nsrip-dd
chore(profiler): test the goroutine leak profile support
fe0699a 
Test that a program built with the goroutine leak experiment produces a
goroutine leak profiler when instrumented with our library. And test
that a program build without that experiment doesn't produce the
profile. The first draft of this was generated using Sonnet 4.6 and I
tidied it up.
@nsrip-dd
nsrip-dd force-pushed the nick.ripley/goroutine-leak-profile branch from fde1f15 to fe0699a Compare March 9, 2026 14:52
@nsrip-dd

nsrip-dd commented Mar 9, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

I will merge this myself once backend support lands.

@nsrip-dd
nsrip-dd marked this pull request as ready for review March 9, 2026 17:45
@nsrip-dd
nsrip-dd requested a review from a team as a code owner March 9, 2026 17:45
felixge
felixge approved these changes Mar 10, 2026
View reviewed changes

@felixge felixge left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - thanks for building this @nsrip-dd 🙇

Comment thread profiler/profiler.go
Comment thread profiler/profile.go
Comment thread profiler/profile.go
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 5160dbf into main Mar 11, 2026
203 of 370 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the nick.ripley/goroutine-leak-profile branch March 11, 2026 11:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@felixge felixge felixge approved these changes

Assignees

No one assigned

Labels

mergequeue-status: done

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nsrip-dd @felixge