Add span-creation JMH benchmarks (front-half allocation) by dougqh · Pull Request #11915 · DataDog/dd-trace-java

dougqh · 2026-07-10T21:00:53Z

What Does This Do

Adds application-thread ("front-half") allocation benchmarks for the full span-creation lifecycle (create → tag → finish), to ground the TagMap 2.0 / SpanPrototype allocation work against the real span path.

SpanCreationBenchmark — single span. Bare baseline (startSpan vs buildSpan), two known-tag shapes set after start — web-server (7 tags) and JDBC/DB client (9 tags) — plus a builder-tag-path arm (withTag before start(), the OTel-bridge shape).
TraceAssemblyBenchmark — web-shape root + N children (childCount 1/5/20), exercising the per-span baseline-tag copy that map-to-map copy (TagMap 1.0) and the trace/span tag split (level-split) target.
SpanCreationVirtualThreadBenchmark — startSpan on a virtual thread (JDK 21+ via reflection), the regime where SpanBuilder reuse is disabled.
DropWriter — shared no-op Writer so finish() excludes serialization / agent I/O, isolating front-half allocation for -prof gc.

Motivation

gc.alloc.rate.norm (B/op) is deterministic and is the primary signal; throughput is directional. The benchmarks are deliberately drift-stable (only API byte-identical v1.53→master), so the same file can be grafted onto past release tags to reconstruct a historical allocation curve.

Additional Notes

Measurement infrastructure only — no product change.
Run with -prof gc, and force tracer logging to WARN (unsuppressed DEBUG logging allocates and corrupts the numbers).

🤖 Generated with Claude Code

Validation — rigorous 8-thread version sweep (1.53 → 1.64)

These benchmarks were run across every release 1.53→1.64 (-f3 -wi5 -i5 -t8 -prof gc) to validate the allocation trend. Allocation is the trustworthy axis; throughput is directional (build-to-build inlining-bimodal on a laptop — read trends, not points).

Allocation (B/op) — trustworthy axis

benchmark	1.53	1.54	1.55	1.56	1.57	1.58	1.59	1.60	1.61	1.62	1.63	1.64
`bareStartSpan`	1331	1424	1309	1322	1322	1313	1175	1181	959	948	927	923
`bareBuildSpan`	1331	1424	1422	1393	1363	1310	1166	1192	952	949	927	958
`webServerSpan`	2059	2131	2098	2131	2063	2018	1869	1859	1639	1675	1627	1637
`webServerSpanViaBuilder`	2435	2491	2477	2464	2411	2442	1987	1963	1774	1787	1738	1751
`jdbcClientSpan`	1821	2038	2029	2017	2074	2065	1866	1818	1619	1614	1430	1421
`webServerTrace[1]`	2792	2875	2817	2827	2747	2640	2504	2519	2240	2256	2239	2241
`webServerTrace[5]`	5701	5773	5752	5763	5301	5643	5376	5264	4635	4651	4869	4721
`webServerTrace[20]`	16675	16728	16733	16728	15059	15072	15371	14987	14259	13689	14992	14592

Cumulative 1.53 → 1.64:

benchmark	1.53	1.64	Δ
`bareStartSpan`	1331	923	-31%
`bareBuildSpan`	1331	958	-28%
`webServerSpan`	2059	1637	-21%
`webServerSpanViaBuilder`	2435	1751	-28%
`jdbcClientSpan`	1821	1421	-22%
`webServerTrace[1]`	2792	2241	-20%
`webServerTrace[5]`	5701	4721	-17%
`webServerTrace[20]`	16675	14592	-12%

Two structural steps carry the curve: 1.59 (TagMap 1.0 default → Entry-sharing) and 1.61 (interceptor/links elimination cluster), plus a 1.63 jdbc-specific drop. Single-span −28-31% (triple-confirmed: this sweep, the 4-thread archaeology, and the PetClinic macro stress test at −29%). Trace path smaller (−12-20%) — the per-child fresh-tag insertion cost that TagMap 2.0 (dense + level-split) targets next.

Throughput (ops/us) — directional only

benchmark	1.53	1.54	1.55	1.56	1.57	1.58	1.59	1.60	1.61	1.62	1.63	1.64
`bareStartSpan`	5.69	4.26	3.87	3.92	4.21	4.16	4.20	5.48	4.17	5.53	6.03	5.37
`bareBuildSpan`	5.49	4.25	4.13	4.03	4.10	4.19	4.66	5.66	4.31	5.72	5.97	5.26
`webServerSpan`	4.00	3.47	3.37	3.78	3.31	3.34	3.43	3.41	3.46	4.19	4.66	4.62
`webServerSpanViaBuilder`	3.99	3.55	3.52	3.32	3.49	3.52	3.52	3.54	3.58	4.37	5.13	5.29
`jdbcClientSpan`	5.33	3.15	3.14	3.15	3.37	3.27	3.45	3.78	3.46	4.49	4.49	5.06
`webServerTrace[1]`	2.43	2.32	2.35	2.52	2.30	2.48	2.29	2.30	2.37	2.34	2.91	2.42
`webServerTrace[5]`	0.78	0.84	0.93	0.90	0.95	1.05	1.02	0.98	0.88	1.14	1.04	0.94
`webServerTrace[20]`	0.25	0.26	0.25	0.28	0.31	0.30	0.28	0.30	0.26	0.30	0.25	0.25

Caveat: throughput on this harness is build-to-build inlining-bimodal (see the bareStart column alternating ~4.2 / ~5.5) — reproducible across clean re-runs, not thermal or contamination, and not stabilized by @Fork(3) (which averages within a build, not across builds). Read trends, not points. The one real-ish signal is the 1.62→1.64 climb on the tagged arms (web 3.4→4.6, webBuilder 3.5→5.3), corroborating the 1.62 isOutbound/span.kind CPU win (#11116) + client-stats changes. Allocation is the axis; throughput corroborates large effects only. A CPU win should be sized from a profile (cycles), not this axis.

Measured a real change (#11701)

TraceAssemblyBenchmark was used to size #11701 (local children share the parent's PropagationTags). Isolating just that one-line toggle (share vs empty()-per-child, all else identical), the shared path drops gc.alloc.rate.norm by ~96 B per child (JOL-confirmed shallow size of one empty() PTags), scaling with fan-out: trace[20] −2031 B/op (≈14.5%), trace[5] −442 B/op, single-span control flat. A concrete consumer validating the per-child slope this harness exists to expose.

Adds application-thread (front-half) allocation benchmarks that ground the TagMap 2.0 / SpanPrototype work against the real span lifecycle: - SpanCreationBenchmark (front-A): single span create -> tag -> finish. Bare baseline (startSpan vs buildSpan), two known-tag shapes set after start — web-server (7 tags) and JDBC/DB client (9 tags) — plus a builder-tag-path arm (withTag before start, the OTel-bridge shape) to track how the startSpan / buildSpan lineages diverge across releases. - TraceAssemblyBenchmark (front-B): web-shape root + N children (childCount 1/5/20), exercising the per-span baseline-tag copy that map-to-map copy (TagMap 1.0) and the trace/span tag split (level-split) target. - DropWriter: shared no-op Writer so finish() excludes serialization / agent I/O, isolating front-half allocation for -prof gc. Deliberately drift-stable (v1.53->master API) so the same file can be grafted onto old release tags for a historical allocation curve. Co-Authored-By: Claude Opus 4.8 <[email protected]>

Runs startSpan create+finish on a virtual thread — the regime the platform-thread SpanCreationBenchmark is blind to. startSpan's thread-local SpanBuilder reuse (#9537) is disabled on virtual threads, so a builder is allocated per span there; the builder bypass (#9998) removes it. This bench is where that difference shows. Starts the virtual thread via reflection (Thread.startVirtualThread) so the jmh source set still compiles on pre-21 toolchains; requires a 21+ JDK at run time. The per-op vthread spawn/join cost is constant across tracer versions, so it cancels in the version-over-version delta. Co-Authored-By: Claude Opus 4.8 <[email protected]>

dd-octo-sts · 2026-07-10T21:23:41Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.98 s	13.93 s	[-0.4%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent	12.87 s	12.88 s	[-0.8%; +0.6%] (no difference)
startup:petclinic:appsec:Agent	16.88 s	16.78 s	[-0.4%; +1.6%] (no difference)
startup:petclinic:iast:Agent	16.83 s	16.90 s	[-1.2%; +0.3%] (no difference)
startup:petclinic:profiling:Agent	16.73 s	16.00 s	[+0.2%; +8.9%] (maybe worse)
startup:petclinic:sca:Agent	16.92 s	16.78 s	[+0.0%; +1.6%] (maybe worse)
startup:petclinic:tracing:Agent	15.95 s	16.26 s	[-2.8%; -1.0%] (maybe better)

Commit: de9dcb31 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

Fold the v1.53.0 -> 1.64.0 sweep results into the SpanCreationBenchmark and TraceAssemblyBenchmark javadoc so the trend is discoverable from the source without re-running. Allocation (gc.alloc.rate.norm B/op) carries a net Δ%; throughput (ops/us) is included but flagged directional-only and given no Δ% since it has no reliable trend (laptop thermals + per-fork inlining bimodality). Co-Authored-By: Claude Opus 4.8 <[email protected]>

datadog-datadog-prod-us1

More details

Pure benchmark-only PR (no production code changes): all four new JMH files are measurement infrastructure. Every critical API contract was verified statically — DropWriter correctly implements all five Writer methods, all Tags constants exist, asChildOf(root) compiles because CoreTracer.buildSpan() returns the concrete CoreSpanBuilder (which has asChildOf(AgentSpan)) not the SpanBuilder interface (which has only asChildOf(AgentSpanContext)), and the Thread.startVirtualThread MethodHandle pattern was confirmed correct on JDK 21. No behavioral regressions are possible: this PR adds only benchmarks.

Was this helpful? React 👍 or 👎

_{📊 Validated against 6 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit e92967b · What is Autotest? · Any feedback? Reach out in #autotest}

A real jdbc span is never "servlet.request"; using a distinct "database.query" operation name keeps the web- and DB-shaped span benchmarks from being conflated by operation. Co-Authored-By: Claude Opus 4.8 <[email protected]>

Now that the DB-shaped span has its own JDBC_OPERATION_NAME, name the web/server operation constant symmetrically for clarity. Co-Authored-By: Claude Opus 4.8 <[email protected]>

sarahchen6

Left a few comments, but otherwise looks good. Pre-approving !

sarahchen6 · 2026-07-21T15:10:19Z

+ */
+final class DropWriter implements Writer {
+  @Override
+  public void write(List<DDSpan> trace) {}


should this writer leverage JMH's blackhole like PendingTraceWrite and BlackholeWriter? IIUC this will keep unused code active for the benchmarks

Yeah, I think that makes sense. I'd held off initially because I was debating having a way to cover both the foreground / application thread work and the background / serializer & stats thread work.

But either way, using a Blackhole here probably makes sense.
I'm confident that spans weren't escaping, so the results shouldn't change.

So I think I'm going to land the PR as is -- then I'll do a re-run over night just to confirm that the results haven't changed significantly. If they have I'll do another PR to refresh the results.

sarahchen6 · 2026-07-21T15:25:31Z

+ *
+ * <p>Same conventions as {@link SpanCreationBenchmark}: {@link DropWriter} isolates front-half
+ * allocation; read alloc ({@code -prof gc}) as the anchor, throughput as directional; logging must
+ * be forced to WARN or DEBUG-line allocation corrupts the numbers. Drift-stable v1.53→master


A point brought up by Codex is that all of these benchmarks inherit from https://github.com/DataDog/dd-trace-java/blob/master/dd-trace-core/src/test/resources/logback.xml#L12 which sets the logging level to DEBUG, but at least TraceAssemblyBenchmark here expects logging to be at the WARN level to not corrupt numbers. The recommended fix is to neutralize the logging setting for all of these benchmarks with:

@Fork(value = 3, jvmArgsAppend = "-DTEST_LOG_LEVEL=warn")

WDYT?

Yeah, that's a good point -- and is a problem that I ended up hitting in another Claude session. Might as well just fix it for good.

sarahchen6 · 2026-07-21T15:32:07Z

+ * no serialization or agent I/O leaking into the {@code -prof gc} number.
+ *
+ * <p>Drift-stable: implements only the five-method {@link Writer} interface, unchanged
+ * v1.53→master.


"master" is used in descriptions throughout, but this reference will always be changing -- I think we should specify 1.64 instead to be clear that that's what the latest tested version is, or something like "master at the time of this commit"

The reference to master is largely intentional. My aim is that this benchmark stays valid as a way compare performance over time -- basically as long as we maintain binary compatibility.

Benchmarks that depend on newer APIs will go in their own benchmark classes, since they won't work on older dd-trace-java versions.

… benchmarks DropWriter.write() was a true no-op, letting the JIT treat the finish()-triggered write as dead code; hand the trace to a Blackhole instead. Also force -DTEST_LOG_LEVEL=warn on all three benchmark classes' forks, since logback.xml defaults to DEBUG and the benchmarks' own javadoc requires WARN to keep DEBUG logging from corrupting the -prof gc allocation numbers. Co-Authored-By: Claude Sonnet 5 <[email protected]>

dougqh and others added 2 commits July 10, 2026 14:27

dougqh added comp: core Tracer core type: refactoring tag: no release note tag: ai generated Largely based on code generated by an AI or LLM labels Jul 10, 2026

This comment has been minimized.

Sign in to view

dougqh marked this pull request as ready for review July 13, 2026 15:14

dougqh requested a review from a team as a code owner July 13, 2026 15:14

dougqh requested a review from ygree July 13, 2026 15:14

datadog-datadog-prod-us1 Bot reviewed Jul 13, 2026

View reviewed changes

dougqh mentioned this pull request Jul 14, 2026

Share parent's PropagationTags for local child spans #11701

Draft

Merge branch 'master' into dougqh/span-creation-bench

0a8eed5

This was referenced Jul 15, 2026

Wire mergedTracerTags as a read-through parent at span build (level-split) (phase 1a) #11932

Open

Store known span tags densely in TagMap by tag-id (phase 2) #11814

Draft

dougqh and others added 2 commits July 16, 2026 11:15

Give the jdbc-shaped benchmark span its own operation name

be92105

A real jdbc span is never "servlet.request"; using a distinct "database.query" operation name keeps the web- and DB-shaped span benchmarks from being conflated by operation. Co-Authored-By: Claude Opus 4.8 <[email protected]>

Rename OPERATION_NAME to SERVER_OPERATION_NAME in the span benchmark

db4a7b4

Now that the DB-shaped span has its own JDBC_OPERATION_NAME, name the web/server operation constant symmetrically for clarity. Co-Authored-By: Claude Opus 4.8 <[email protected]>

PerfectSlayer added tag: no release notes Changes to exclude from release notes and removed tag: no release note labels Jul 20, 2026

sarahchen6 approved these changes Jul 21, 2026

View reviewed changes

dougqh and others added 2 commits July 21, 2026 12:08

Merge branch 'master' into dougqh/span-creation-bench

de9dcb3

dougqh enabled auto-merge July 21, 2026 16:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add span-creation JMH benchmarks (front-half allocation)#11915

Add span-creation JMH benchmarks (front-half allocation)#11915
dougqh wants to merge 8 commits into
masterfrom
dougqh/span-creation-bench

dougqh commented Jul 10, 2026 •

edited

Loading

Uh oh!

dd-octo-sts Bot commented Jul 10, 2026 •

edited

Loading

Uh oh!

This comment has been minimized.

datadog-datadog-prod-us1 Bot left a comment

Uh oh!

sarahchen6 left a comment

Uh oh!

sarahchen6 Jul 21, 2026

Uh oh!

dougqh Jul 21, 2026 •

edited

Loading

Uh oh!

sarahchen6 Jul 21, 2026

Uh oh!

dougqh Jul 21, 2026

Uh oh!

sarahchen6 Jul 21, 2026

Uh oh!

dougqh Jul 21, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

dougqh commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Validation — rigorous 8-thread version sweep (1.53 → 1.64)

Allocation (B/op) — trustworthy axis

Throughput (ops/us) — directional only

Measured a real change (#11701)

Uh oh!

dd-octo-sts Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

This comment has been minimized.

datadog-datadog-prod-us1 Bot left a comment

Choose a reason for hiding this comment

Uh oh!

sarahchen6 left a comment

Choose a reason for hiding this comment

Uh oh!

sarahchen6 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

dougqh Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sarahchen6 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

dougqh Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

sarahchen6 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

dougqh Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dougqh commented Jul 10, 2026 •

edited

Loading

dd-octo-sts Bot commented Jul 10, 2026 •

edited

Loading

dougqh Jul 21, 2026 •

edited

Loading

dougqh Jul 21, 2026 •

edited

Loading