Skip to content

feat(ddtrace/tracer): reduce allocations on ETP/v1 trace protocol#4497

Merged
darccio merged 5 commits into
mainfrom
dario.castane/apmlp-856/reduce-allocations
Mar 19, 2026
Merged

feat(ddtrace/tracer): reduce allocations on ETP/v1 trace protocol#4497
darccio merged 5 commits into
mainfrom
dario.castane/apmlp-856/reduce-allocations

Conversation

@darccio

@darccio darccio commented Mar 4, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?

Adds a payload pool and reuse logic for ETP/v1 payload internals.

Motivation

Reduce allocations as described below.

Here's the benchmark comparison table extracted from the PR:

ETP/v1 Payload Benchmark: Before vs After (with pool)

Scenario ns/op (before) ns/op (after) Speedup B/op (before) B/op (after) Reduction allocs/op (before) allocs/op (after) Reduction
simple / 1 span 2,053 715.6 2.87x 2,352 16 147x 20 1 20x
detailed / 1 span 2,380 1,015 2.34x 3,112 32 97x 25 2 12.5x
simple / 10 spans 4,734 2,556 1.85x 4,728 16 296x 26 1 26x
detailed / 10 spans 10,738 5,320 2.02x 10,728 32 335x 33 2 16.5x
simple / 100 spans 28,069 18,560 1.51x 25,208 16 1,575x 31 1 31x
detailed / 100 spans 61,775 41,393 1.49x 48,617 33 1,473x 37 2 18.5x
simple / 1,000 spans 219,828 178,463 1.23x 181,497 61 2,975x 36 1 36x
detailed / 1,000 spans 532,561 405,264 1.31x 500,462 136 3,679x 44 2 22x

v0.4 baseline (reference)

Scenario ns/op B/op allocs/op
simple / 1 span 498.0 512 4
detailed / 1 span 930.6 833 4
simple / 10 spans 2,119 2,819 4
detailed / 10 spans 4,570 6,280 4
simple / 100 spans 17,568 24,736 4
detailed / 100 spans 38,881 57,552 4
simple / 1,000 spans 148,768 237,837 4
detailed / 1,000 spans 339,564 541,250 4

Key takeaway: the pool brings v1.0 allocations from 20–44 allocs/op down to just 1–2 allocs/op, and cuts per-op heap allocation by 97x–3,679x depending on trace size. Latency also improves 1.2–2.9x. The v1.0-with-pool now closely matches the v0.4 baseline in allocation count, while the bytes-per-op is far lower for large traces (61 B vs 237 KB for 1,000 spans).

Original results

Baseline

goos: darwin
goarch: arm64
pkg: github.com/DataDog/dd-trace-go/v2/ddtrace/tracer
cpu: Apple M1 Max

BenchmarkPayloadVersions/simple_1spans/v0.4-10           2440736               498.0 ns/op           512 B/op          4 allocs/op
BenchmarkPayloadVersions/detailed_1spans/v0.4-10         1415994               930.6 ns/op           833 B/op          4 allocs/op
BenchmarkPayloadVersions/simple_10spans/v0.4-10           582498              2119 ns/op            2819 B/op          4 allocs/op
BenchmarkPayloadVersions/detailed_10spans/v0.4-10         225103              4570 ns/op            6280 B/op          4 allocs/op
BenchmarkPayloadVersions/simple_100spans/v0.4-10           66343             17568 ns/op           24736 B/op          4 allocs/op
BenchmarkPayloadVersions/detailed_100spans/v0.4-10         30717             38881 ns/op           57552 B/op          4 allocs/op
BenchmarkPayloadVersions/simple_1000spans/v0.4-10           7592            148768 ns/op          237837 B/op          4 allocs/op
BenchmarkPayloadVersions/detailed_1000spans/v0.4-10         3399            339564 ns/op          541250 B/op          4 allocs/op

BenchmarkPayloadVersions/simple_1spans/v1.0-10            558800              2053 ns/op            2352 B/op         20 allocs/op
BenchmarkPayloadVersions/detailed_1spans/v1.0-10          498726              2380 ns/op            3112 B/op         25 allocs/op
BenchmarkPayloadVersions/simple_10spans/v1.0-10           265585              4734 ns/op            4728 B/op         26 allocs/op
BenchmarkPayloadVersions/detailed_10spans/v1.0-10         107678             10738 ns/op           10728 B/op         33 allocs/op
BenchmarkPayloadVersions/simple_100spans/v1.0-10           42873             28069 ns/op           25208 B/op         31 allocs/op
BenchmarkPayloadVersions/detailed_100spans/v1.0-10         20002             61775 ns/op           48617 B/op         37 allocs/op
BenchmarkPayloadVersions/simple_1000spans/v1.0-10           5454            219828 ns/op          181497 B/op         36 allocs/op
BenchmarkPayloadVersions/detailed_1000spans/v1.0-10         2242            532561 ns/op          500462 B/op         44 allocs/op

Optimized (pooled):

BenchmarkPayloadVersions/simple_1spans/v1.0-10           1680595               715.6 ns/op            16 B/op          1 allocs/op
BenchmarkPayloadVersions/detailed_1spans/v1.0-10         1000000              1015 ns/op              32 B/op          2 allocs/op
BenchmarkPayloadVersions/simple_10spans/v1.0-10           458776              2556 ns/op              16 B/op          1 allocs/op
BenchmarkPayloadVersions/detailed_10spans/v1.0-10         220927              5320 ns/op              32 B/op          2 allocs/op
BenchmarkPayloadVersions/simple_100spans/v1.0-10           64609             18560 ns/op              16 B/op          1 allocs/op
BenchmarkPayloadVersions/detailed_100spans/v1.0-10         29056             41393 ns/op              33 B/op          2 allocs/op
BenchmarkPayloadVersions/simple_1000spans/v1.0-10           6758            178463 ns/op              61 B/op          1 allocs/op
BenchmarkPayloadVersions/detailed_1000spans/v1.0-10         2965            405264 ns/op             136 B/op          2 allocs/op

Optimized (unpooled):

BenchmarkPayloadVersions/simple_1spans/v1.0-10            454594              2378 ns/op            6352 B/op         16 allocs/op
BenchmarkPayloadVersions/detailed_1spans/v1.0-10          389655              3087 ns/op            7008 B/op         18 allocs/op
BenchmarkPayloadVersions/simple_10spans/v1.0-10           239406              5032 ns/op            9104 B/op         16 allocs/op
BenchmarkPayloadVersions/detailed_10spans/v1.0-10         155892              7578 ns/op            9120 B/op         17 allocs/op
BenchmarkPayloadVersions/simple_100spans/v1.0-10           40876             29080 ns/op           38800 B/op         16 allocs/op
BenchmarkPayloadVersions/detailed_100spans/v1.0-10         23526             50942 ns/op           38816 B/op         17 allocs/op
BenchmarkPayloadVersions/simple_1000spans/v1.0-10           5241            222815 ns/op          309138 B/op         16 allocs/op
BenchmarkPayloadVersions/detailed_1000spans/v1.0-10         2616            497754 ns/op          309154 B/op         17 allocs/op

Reviewer's Checklist

  • Changed code has unit tests for its functionality at or near 100% coverage.
  • There is a benchmark for any new code, or changes to existing code.
  • New code is free of linting errors. You can check this by running make lint locally.
  • New code doesn't break existing tests. You can check this by running make test locally.
  • Add an appropriate team label so this PR gets put in the right place for the release notes.

Unsure? Have a question? Request a review!

@darccio
darccio requested a review from a team as a code owner March 4, 2026 11:16
@pr-commenter

pr-commenter Bot commented Mar 4, 2026 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Benchmark execution time: 2026-03-17 12:44:54

Comparing candidate commit 78b1651 in PR branch dario.castane/apmlp-856/reduce-allocations with baseline commit e8c42f7 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 155 metrics, 9 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

@darccio
darccio requested review from a team as code owners March 4, 2026 11:32
@codecov

codecov Bot commented Mar 4, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.31373% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 59.92%. Comparing base (5793394) to head (0055256).
⚠️ Report is 4 commits behind head on main.

Files with missing lines Patch % Lines
ddtrace/tracer/payload_v1.go 82.22% 7 Missing and 1 partial ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
ddtrace/tracer/payload.go 100.00% <100.00%> (ø)
ddtrace/tracer/writer.go 91.83% <100.00%> (ø)
ddtrace/tracer/payload_v1.go 68.47% <82.22%> (ø)

... and 427 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@darccio darccio added the AI Assisted AI/LLM assistance used in this PR (partially or fully) label Mar 4, 2026
kakkoyun
kakkoyun reviewed Mar 10, 2026
View reviewed changes

@kakkoyun kakkoyun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

Could we add a comparison table to the PR description? It would be easier to reason about the changes.

@darccio

darccio commented Mar 16, 2026

Copy link
Copy Markdown
Member Author

@kakkoyun Done!

@darccio
darccio requested a review from kakkoyun March 16, 2026 16:08
@darccio

darccio commented Mar 16, 2026

Copy link
Copy Markdown
Member Author

BenchmarkPayloadVersions is failing because the benchmark doesn't run on main (until this PR is merged).

@datadog-official

datadog-official Bot commented Mar 16, 2026 •
edited by datadog-datadog-prod-us1 Bot
Loading

Copy link
Copy Markdown
Contributor

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 85.42%
• Overall Coverage: 59.33% (+3.55%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 78b1651 | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!

@kakkoyun

kakkoyun commented Mar 19, 2026

Copy link
Copy Markdown
Member

@darccio Feel free to merge this as admin.

@darccio
darccio merged commit a0a8836 into main Mar 19, 2026
110 of 113 checks passed
@darccio
darccio deleted the dario.castane/apmlp-856/reduce-allocations branch March 19, 2026 09:12
@darccio

darccio commented Mar 19, 2026

Copy link
Copy Markdown
Member Author

sudo merge done 😁 I usually prefer to wait for all the approvals but this one is ok.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@kakkoyun kakkoyun kakkoyun approved these changes

Assignees

No one assigned

Labels

AI Assisted AI/LLM assistance used in this PR (partially or fully) team:apm-go

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@darccio @kakkoyun