Skip to content

Support aot_training mode to be used when creating AOT caches #10166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mcculls merged 4 commits into from
Dec 15, 2025
Merged

Support aot_training mode to be used when creating AOT caches #10166

mcculls merged 4 commits into from
Dec 15, 2025

Conversation

@mcculls
Copy link
Contributor

@mcculls mcculls commented Dec 11, 2025 •
edited
Loading

What Does This Do

This PR adds support for a special "AOT training" mode where only the agent jar is added to the boot classpath, no services are started.

This mode is automatically applied on Java 25 and above when we detect the JVM itself is in AOT training mode.
We do this by reflectively checking whether the JVM's CDS.isDumpingArchive() helper method returns true.

It can also be explicitly enabled by adding =aot_training to the end of the -javaagent option:

-javaagent:dd-java-agent.jar=aot_training

If necessary the automatic detection can be explicitly turned off using this environment variable:

DD_DETECT_AOT_TRAINING_MODE=false

This PR includes a workaround for a potential AOT bug with Java 25 where TraceInterceptor is mistakenly restored from the system class-loader in production, even though it was visible from the boot class-loader during training, resulting in LinkageErrors. Any call to Tracer.addTraceInterceptor from application code in the system class-loader appears to trigger this bug. The workaround is to replace these calls during training with opcodes that pop the tracer and argument, and push the expected return value. This transformation is not persisted, so in production the original method is invoked.

Note: early access builds of Java 26 do not appear to suffer from this bug.

Motivation

Improves the experience when using AOT caching on Java 25:

# populate cache
java -XX:AOTCacheOutput=petclinic.aot -javaagent:dd-java-agent.jar \
    -Dspring.context.exit=onRefresh -jar application/spring-petclinic-4.0.0-SNAPSHOT.jar

# use cache
java -XX:AOTCache=petclinic.aot -javaagent:dd-java-agent.jar \
    -Dspring.context.exit=onRefresh -jar application/spring-petclinic-4.0.0-SNAPSHOT.jar

You can still get a large performance boost using this approach, even though the classes aren't transformed during AOT cache creation. With Spring Petclinic the startup time can drop by 30% - from 13s to 8.5s

Additional Notes

The new mode avoids crashes in the Java 25 JDK when adding the Java agent during creation of the AOT cache, such as:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000000108dea73c, pid=69345, tid=5379
#
# JRE version:  (25.0.1+8) (build )
# Java VM: OpenJDK 64-Bit Server VM (25.0.1+8-LTS, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# V  [libjvm.dylib+0xb8673c]  vmClasses::resolve_shared_class(InstanceKlass*, ClassLoaderData*, Handle, JavaThread*)+0x18

It especially helps when the application consumes dd-trace-api because if you don't add the Java agent during AOT cache creation then you can get a linkage error applying the cache in production because the location of the API classes has changed:

Caused by: java.lang.LinkageError: loader constraint violation: when resolving interface method 'boolean datadog.trace.api.Tracer.addTraceInterceptor(datadog.trace.api.interceptor.TraceInterceptor)' 
the class loader 'app' of the current class, ****/DatadogConfiguration$TraceInterceptorsConfiguration, and the class loader 'bootstrap' for the method's defining class, datadog/trace/api/Tracer, 
have different Class objects for the type datadog/trace/api/interceptor/TraceInterceptor used in the signature (****.DatadogConfiguration$TraceInterceptorsConfiguration is in unnamed module of loader 'app'; 
datadog.trace.api.Tracer is in unnamed module of loader 'bootstrap')
at ****.DatadogConfiguration$TraceInterceptorsConfiguration.afterPropertiesSet(DatadogConfiguration.java:98)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1873)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1822)
... 16 common frames omitted

I'll add a smoke test to cover this use-case in a separate PR, but would like to get this change merged first.

Contributor Checklist

Jira ticket: APMS-18027

@mcculls mcculls requested a review from a team as a code owner December 11, 2025 09:02
@mcculls mcculls added the type: enhancement Enhancements and improvements label Dec 11, 2025
@mcculls mcculls added comp: core Tracer core tag: performance Performance related changes labels Dec 11, 2025
@jpbempel
Copy link
Member

jpbempel commented Dec 11, 2025

Bike shedding: bootclasspath

PerfectSlayer
PerfectSlayer approved these changes Dec 11, 2025
View reviewed changes
Copy link
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. Proposed an alternative name for the argument.
Make sure to link the test follow up PR 🙏

manuel-alvarez-alvarez
manuel-alvarez-alvarez approved these changes Dec 11, 2025
View reviewed changes
Copy link
Member

@manuel-alvarez-alvarez manuel-alvarez-alvarez left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Does it make sense to try to detect -XX:AOTCacheOutput on JDK 9+ using ProcessHandle to enable the mode automatically? (maybe this is too heavy during startup)

@mcculls
Copy link
Contributor Author

mcculls commented Dec 11, 2025

Does it make sense to try to detect -XX:AOTCacheOutput on JDK 9+ using ProcessHandle to enable the mode automatically? (maybe this is to heavy during startup)

That was my first thought, but I was worried it would be too heavy - need to do some benchmarking to see what it adds (also need to do some build-wrangling to avoid any reflection overhead)

@PerfectSlayer
Copy link
Contributor

PerfectSlayer commented Dec 11, 2025 •
edited
Loading

Maybe datadog.environment.JavaVirtualMachine#getVmOptions() could help too to discover the -XX:AOTCacheOutput option.

@mcculls
Copy link
Contributor Author

mcculls commented Dec 11, 2025

Maybe datadog.environment.JavaVirtualMachine#getVmOptions() could help too to discover the -XX:AOTCacheOutput option.

IIRC that can still end up calling ManagementFactory.getRuntimeMXBean().getInputArguments() as a last resort, which would load too many classes and break custom log managers - maybe there's a lighter compromise...

@PerfectSlayer
Copy link
Contributor

PerfectSlayer commented Dec 11, 2025 •
edited
Loading

Yes, sadly it might. About ProcessHandle, it is in my task list as improvement for Java 9+.

About relying on ProcessHandle only, it might fail is to handle is JAVA_TOOL_OPTIONS and JDK_JAVA_OPTIONS. So if users are using env vars to control their training runs, ProcessHandle detection might not be enough.

@mcculls mcculls force-pushed the mcculls/support-classpath-only-mode branch from f07d03d to e17e8ad Compare December 13, 2025 14:57
@mcculls mcculls changed the title Support 'classpath' mode to be used when creating AOT caches Support aot_training mode to be used when creating AOT caches Dec 13, 2025
@pr-commenter
Copy link

pr-commenter bot commented Dec 13, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/support-classpath-only-mode
git_commit_date 1765661039 1765660484
git_commit_sha ef9d162d84 b79d752
release_version 1.57.0-SNAPSHOT~fef9d162d84 1.57.0-SNAPSHOT~b79d752021
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1765662314 1765662314
ci_job_id 1293306792 1293306792
ci_pipeline_id 86652142 86652142
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-lodutds5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-lodutds5 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 6 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.57.0-SNAPSHOT~b79d752021, baseline=1.57.0-SNAPSHOT~fef9d162d84

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.086 s) : 0, 1085818
Total [baseline] (10.876 s) : 0, 10876217
Agent [candidate] (1.082 s) : 0, 1082443
Total [candidate] (10.811 s) : 0, 10811144
section appsec
Agent [baseline] (1.269 s) : 0, 1268564
Total [baseline] (11.003 s) : 0, 11002885
Agent [candidate] (1.271 s) : 0, 1270940
Total [candidate] (10.978 s) : 0, 10977695
section iast
Agent [baseline] (1.223 s) : 0, 1222689
Total [baseline] (11.191 s) : 0, 11191170
Agent [candidate] (1.221 s) : 0, 1221430
Total [candidate] (11.208 s) : 0, 11207779
section profiling
Agent [baseline] (1.212 s) : 0, 1211600
Total [baseline] (10.909 s) : 0, 10908956
Agent [candidate] (1.21 s) : 0, 1210271
Total [candidate] (11.047 s) : 0, 11046802
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.086 s -
Agent appsec 1.269 s 182.746 ms (16.8%)
Agent iast 1.223 s 136.871 ms (12.6%)
Agent profiling 1.212 s 125.782 ms (11.6%)
Total tracing 10.876 s -
Total appsec 11.003 s 126.668 ms (1.2%)
Total iast 11.191 s 314.953 ms (2.9%)
Total profiling 10.909 s 32.739 ms (0.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.082 s -
Agent appsec 1.271 s 188.497 ms (17.4%)
Agent iast 1.221 s 138.987 ms (12.8%)
Agent profiling 1.21 s 127.828 ms (11.8%)
Total tracing 10.811 s -
Total appsec 10.978 s 166.551 ms (1.5%)
Total iast 11.208 s 396.635 ms (3.7%)
Total profiling 11.047 s 235.658 ms (2.2%) 
gantt
    title petclinic - break down per module: candidate=1.57.0-SNAPSHOT~b79d752021, baseline=1.57.0-SNAPSHOT~fef9d162d84

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.182 ms) : 0, 1182
crashtracking [candidate] (1.173 ms) : 0, 1173
BytebuddyAgent [baseline] (652.06 ms) : 0, 652060
BytebuddyAgent [candidate] (649.119 ms) : 0, 649119
GlobalTracer [baseline] (282.752 ms) : 0, 282752
GlobalTracer [candidate] (282.631 ms) : 0, 282631
AppSec [baseline] (32.642 ms) : 0, 32642
AppSec [candidate] (32.413 ms) : 0, 32413
Debugger [baseline] (68.156 ms) : 0, 68156
Debugger [candidate] (68.258 ms) : 0, 68258
Remote Config [baseline] (616.938 µs) : 0, 617
Remote Config [candidate] (641.778 µs) : 0, 642
Telemetry [baseline] (8.969 ms) : 0, 8969
Telemetry [candidate] (8.965 ms) : 0, 8965
Flare Poller [baseline] (3.725 ms) : 0, 3725
Flare Poller [candidate] (3.782 ms) : 0, 3782
section appsec
crashtracking [baseline] (1.185 ms) : 0, 1185
crashtracking [candidate] (1.192 ms) : 0, 1192
BytebuddyAgent [baseline] (692.343 ms) : 0, 692343
BytebuddyAgent [candidate] (692.661 ms) : 0, 692661
GlobalTracer [baseline] (258.753 ms) : 0, 258753
GlobalTracer [candidate] (260.762 ms) : 0, 260762
AppSec [baseline] (175.181 ms) : 0, 175181
AppSec [candidate] (175.598 ms) : 0, 175598
Debugger [baseline] (67.015 ms) : 0, 67015
Debugger [candidate] (66.716 ms) : 0, 66716
Remote Config [baseline] (740.455 µs) : 0, 740
Remote Config [candidate] (722.426 µs) : 0, 722
Telemetry [baseline] (9.182 ms) : 0, 9182
Telemetry [candidate] (9.143 ms) : 0, 9143
Flare Poller [baseline] (3.884 ms) : 0, 3884
Flare Poller [candidate] (4.028 ms) : 0, 4028
IAST [baseline] (24.64 ms) : 0, 24640
IAST [candidate] (24.69 ms) : 0, 24690
section iast
crashtracking [baseline] (1.185 ms) : 0, 1185
crashtracking [candidate] (1.181 ms) : 0, 1181
BytebuddyAgent [baseline] (790.793 ms) : 0, 790793
BytebuddyAgent [candidate] (789.273 ms) : 0, 789273
GlobalTracer [baseline] (255.7 ms) : 0, 255700
GlobalTracer [candidate] (255.905 ms) : 0, 255905
AppSec [baseline] (34.996 ms) : 0, 34996
AppSec [candidate] (34.173 ms) : 0, 34173
Debugger [baseline] (65.096 ms) : 0, 65096
Debugger [candidate] (65.884 ms) : 0, 65884
Remote Config [baseline] (531.416 µs) : 0, 531
Remote Config [candidate] (578.868 µs) : 0, 579
Telemetry [baseline] (8.408 ms) : 0, 8408
Telemetry [candidate] (8.495 ms) : 0, 8495
Flare Poller [baseline] (3.473 ms) : 0, 3473
Flare Poller [candidate] (3.545 ms) : 0, 3545
IAST [baseline] (27.02 ms) : 0, 27020
IAST [candidate] (27.116 ms) : 0, 27116
section profiling
ProfilingAgent [baseline] (96.965 ms) : 0, 96965
ProfilingAgent [candidate] (97.719 ms) : 0, 97719
crashtracking [baseline] (1.215 ms) : 0, 1215
crashtracking [candidate] (1.214 ms) : 0, 1214
BytebuddyAgent [baseline] (706.712 ms) : 0, 706712
BytebuddyAgent [candidate] (704.669 ms) : 0, 704669
GlobalTracer [baseline] (221.932 ms) : 0, 221932
GlobalTracer [candidate] (222.472 ms) : 0, 222472
AppSec [baseline] (32.246 ms) : 0, 32246
AppSec [candidate] (32.44 ms) : 0, 32440
Debugger [baseline] (68.844 ms) : 0, 68844
Debugger [candidate] (68.242 ms) : 0, 68242
Remote Config [baseline] (652.165 µs) : 0, 652
Remote Config [candidate] (628.732 µs) : 0, 629
Telemetry [baseline] (8.99 ms) : 0, 8990
Telemetry [candidate] (9.002 ms) : 0, 9002
Flare Poller [baseline] (3.74 ms) : 0, 3740
Flare Poller [candidate] (3.803 ms) : 0, 3803
Profiling [baseline] (97.569 ms) : 0, 97569
Profiling [candidate] (98.299 ms) : 0, 98299
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.57.0-SNAPSHOT~b79d752021, baseline=1.57.0-SNAPSHOT~fef9d162d84

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.078 s) : 0, 1078397
Total [baseline] (8.747 s) : 0, 8747293
Agent [candidate] (1.091 s) : 0, 1090515
Total [candidate] (8.787 s) : 0, 8787348
section iast
Agent [baseline] (1.231 s) : 0, 1230871
Total [baseline] (9.376 s) : 0, 9375933
Agent [candidate] (1.22 s) : 0, 1219526
Total [candidate] (9.332 s) : 0, 9332364
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.078 s -
Agent iast 1.231 s 152.473 ms (14.1%)
Total tracing 8.747 s -
Total iast 9.376 s 628.64 ms (7.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.091 s -
Agent iast 1.22 s 129.012 ms (11.8%)
Total tracing 8.787 s -
Total iast 9.332 s 545.016 ms (6.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.57.0-SNAPSHOT~b79d752021, baseline=1.57.0-SNAPSHOT~fef9d162d84

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.175 ms) : 0, 1175
crashtracking [candidate] (1.203 ms) : 0, 1203
BytebuddyAgent [baseline] (647.596 ms) : 0, 647596
BytebuddyAgent [candidate] (655.958 ms) : 0, 655958
GlobalTracer [baseline] (280.838 ms) : 0, 280838
GlobalTracer [candidate] (284.213 ms) : 0, 284213
AppSec [baseline] (32.191 ms) : 0, 32191
AppSec [candidate] (32.602 ms) : 0, 32602
Debugger [baseline] (67.69 ms) : 0, 67690
Debugger [candidate] (67.339 ms) : 0, 67339
Remote Config [baseline] (653.634 µs) : 0, 654
Remote Config [candidate] (608.059 µs) : 0, 608
Telemetry [baseline] (9.036 ms) : 0, 9036
Telemetry [candidate] (9.103 ms) : 0, 9103
Flare Poller [baseline] (3.751 ms) : 0, 3751
Flare Poller [candidate] (3.814 ms) : 0, 3814
section iast
crashtracking [baseline] (1.191 ms) : 0, 1191
crashtracking [candidate] (1.18 ms) : 0, 1180
BytebuddyAgent [baseline] (796.902 ms) : 0, 796902
BytebuddyAgent [candidate] (789.343 ms) : 0, 789343
GlobalTracer [baseline] (257.585 ms) : 0, 257585
GlobalTracer [candidate] (255.479 ms) : 0, 255479
IAST [baseline] (27.092 ms) : 0, 27092
IAST [candidate] (26.899 ms) : 0, 26899
AppSec [baseline] (35.594 ms) : 0, 35594
AppSec [candidate] (35.363 ms) : 0, 35363
Debugger [baseline] (64.517 ms) : 0, 64517
Debugger [candidate] (63.46 ms) : 0, 63460
Remote Config [baseline] (553.149 µs) : 0, 553
Remote Config [candidate] (626.818 µs) : 0, 627
Telemetry [baseline] (8.359 ms) : 0, 8359
Telemetry [candidate] (8.407 ms) : 0, 8407
Flare Poller [baseline] (3.487 ms) : 0, 3487
Flare Poller [candidate] (3.476 ms) : 0, 3476
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/support-classpath-only-mode
git_commit_date 1765660958 1765660484
git_commit_sha ef9d162d84 b79d752
release_version 1.57.0-SNAPSHOT~fef9d162d84 1.57.0-SNAPSHOT~b79d752021
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1765662715 1765662715
ci_job_id 1293306793 1293306793
ci_pipeline_id 86652142 86652142
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-7bk4baq6 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-7bk4baq6 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 4 performance improvements and 0 performance regressions! Performance is the same for 15 metrics, 17 unstable metrics.

scenario Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p95 Δ mean throughput candidate mean agg_http_req_duration_p50 candidate mean agg_http_req_duration_p95 candidate mean throughput baseline mean agg_http_req_duration_p50 baseline mean agg_http_req_duration_p95 baseline mean throughput
scenario:load:insecure-bank:iast_GLOBAL:high_load better
[-218.330µs; -121.941µs] or [-7.575%; -4.231%]		 unsure
[-551.831µs; -84.848µs] or [-6.845%; -1.052%]		 unstable
[-74.157op/s; +216.845op/s] or [-5.922%; +17.317%]		 2.712ms 7.744ms 1323.562op/s 2.882ms 8.062ms 1252.219op/s
scenario:load:petclinic:appsec:high_load better
[-1234.849µs; -547.944µs] or [-6.417%; -2.847%]		 better
[-2.105ms; -0.694ms] or [-6.746%; -2.223%]		 unstable
[-14.992op/s; +32.305op/s] or [-6.251%; +13.469%]		 18.352ms 29.804ms 248.500op/s 19.244ms 31.204ms 239.844op/s
scenario:load:petclinic:no_agent:high_load better
[-2.369ms; -0.695ms] or [-12.316%; -3.612%]		 unstable
[-3.501ms; -0.275ms] or [-11.038%; -0.866%]		 unstable
[-8.760op/s; +43.947op/s] or [-3.659%; +18.357%]		 17.705ms 29.831ms 257.000op/s 19.237ms 31.719ms 239.406op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~b79d752021, baseline=1.57.0-SNAPSHOT~fef9d162d84
    dateFormat X
    axisFormat %s
section baseline
no_agent (19.5 ms) : 19297, 19703
.   : milestone, 19500,
appsec (19.465 ms) : 19264, 19666
.   : milestone, 19465,
code_origins (17.725 ms) : 17549, 17900
.   : milestone, 17725,
iast (17.942 ms) : 17765, 18119
.   : milestone, 17942,
profiling (19.729 ms) : 19528, 19929
.   : milestone, 19729,
tracing (17.927 ms) : 17750, 18105
.   : milestone, 17927,
section candidate
no_agent (18.152 ms) : 17967, 18338
.   : milestone, 18152,
appsec (18.781 ms) : 18593, 18969
.   : milestone, 18781,
code_origins (17.68 ms) : 17505, 17854
.   : milestone, 17680,
iast (17.864 ms) : 17684, 18044
.   : milestone, 17864,
profiling (19.875 ms) : 19669, 20080
.   : milestone, 19875,
tracing (18.096 ms) : 17917, 18276
.   : milestone, 18096,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 19.5 ms [19.297 ms, 19.703 ms] -
appsec 19.465 ms [19.264 ms, 19.666 ms] -35.128 µs (-0.2%)
code_origins 17.725 ms [17.549 ms, 17.9 ms] -1.775 ms (-9.1%)
iast 17.942 ms [17.765 ms, 18.119 ms] -1.558 ms (-8.0%)
profiling 19.729 ms [19.528 ms, 19.929 ms] 228.847 µs (1.2%)
tracing 17.927 ms [17.75 ms, 18.105 ms] -1.573 ms (-8.1%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 18.152 ms [17.967 ms, 18.338 ms] -
appsec 18.781 ms [18.593 ms, 18.969 ms] 628.763 µs (3.5%)
code_origins 17.68 ms [17.505 ms, 17.854 ms] -472.542 µs (-2.6%)
iast 17.864 ms [17.684 ms, 18.044 ms] -288.265 µs (-1.6%)
profiling 19.875 ms [19.669 ms, 20.08 ms] 1.723 ms (9.5%)
tracing 18.096 ms [17.917 ms, 18.276 ms] -55.941 µs (-0.3%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.57.0-SNAPSHOT~b79d752021, baseline=1.57.0-SNAPSHOT~fef9d162d84
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.182 ms) : 1171, 1194
.   : milestone, 1182,
iast (3.382 ms) : 3332, 3433
.   : milestone, 3382,
iast_FULL (5.735 ms) : 5677, 5793
.   : milestone, 5735,
iast_GLOBAL (3.664 ms) : 3597, 3731
.   : milestone, 3664,
profiling (2.006 ms) : 1989, 2024
.   : milestone, 2006,
tracing (1.852 ms) : 1837, 1868
.   : milestone, 1852,
section candidate
no_agent (1.209 ms) : 1198, 1221
.   : milestone, 1209,
iast (3.202 ms) : 3162, 3242
.   : milestone, 3202,
iast_FULL (5.569 ms) : 5514, 5624
.   : milestone, 5569,
iast_GLOBAL (3.461 ms) : 3411, 3511
.   : milestone, 3461,
profiling (2.144 ms) : 2123, 2165
.   : milestone, 2144,
tracing (1.882 ms) : 1865, 1899
.   : milestone, 1882,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.182 ms [1.171 ms, 1.194 ms] -
iast 3.382 ms [3.332 ms, 3.433 ms] 2.2 ms (186.1%)
iast_FULL 5.735 ms [5.677 ms, 5.793 ms] 4.553 ms (385.1%)
iast_GLOBAL 3.664 ms [3.597 ms, 3.731 ms] 2.482 ms (209.9%)
profiling 2.006 ms [1.989 ms, 2.024 ms] 823.937 µs (69.7%)
tracing 1.852 ms [1.837 ms, 1.868 ms] 670.005 µs (56.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.209 ms [1.198 ms, 1.221 ms] -
iast 3.202 ms [3.162 ms, 3.242 ms] 1.993 ms (164.8%)
iast_FULL 5.569 ms [5.514 ms, 5.624 ms] 4.36 ms (360.5%)
iast_GLOBAL 3.461 ms [3.411 ms, 3.511 ms] 2.252 ms (186.2%)
profiling 2.144 ms [2.123 ms, 2.165 ms] 934.853 µs (77.3%)
tracing 1.882 ms [1.865 ms, 1.899 ms] 672.658 µs (55.6%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master mcculls/support-classpath-only-mode
git_commit_date 1765661216 1765660484
git_commit_sha ef9d162d84 b79d752
release_version 1.57.0-SNAPSHOT~fef9d162d84 1.57.0-SNAPSHOT~b79d752021
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1765662742 1765662742
ci_job_id 1293306794 1293306794
ci_pipeline_id 86652142 86652142
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-1-pnbzkkic 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-1-pnbzkkic 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~b79d752021, baseline=1.57.0-SNAPSHOT~fef9d162d84
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.442 s) : 15442000, 15442000
.   : milestone, 15442000,
appsec (14.521 s) : 14521000, 14521000
.   : milestone, 14521000,
iast (18.023 s) : 18023000, 18023000
.   : milestone, 18023000,
iast_GLOBAL (17.815 s) : 17815000, 17815000
.   : milestone, 17815000,
profiling (15.322 s) : 15322000, 15322000
.   : milestone, 15322000,
tracing (14.987 s) : 14987000, 14987000
.   : milestone, 14987000,
section candidate
no_agent (15.248 s) : 15248000, 15248000
.   : milestone, 15248000,
appsec (14.791 s) : 14791000, 14791000
.   : milestone, 14791000,
iast (18.509 s) : 18509000, 18509000
.   : milestone, 18509000,
iast_GLOBAL (17.84 s) : 17840000, 17840000
.   : milestone, 17840000,
profiling (14.698 s) : 14698000, 14698000
.   : milestone, 14698000,
tracing (14.821 s) : 14821000, 14821000
.   : milestone, 14821000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.442 s [15.442 s, 15.442 s] -
appsec 14.521 s [14.521 s, 14.521 s] -921.0 ms (-6.0%)
iast 18.023 s [18.023 s, 18.023 s] 2.581 s (16.7%)
iast_GLOBAL 17.815 s [17.815 s, 17.815 s] 2.373 s (15.4%)
profiling 15.322 s [15.322 s, 15.322 s] -120.0 ms (-0.8%)
tracing 14.987 s [14.987 s, 14.987 s] -455.0 ms (-2.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.248 s [15.248 s, 15.248 s] -
appsec 14.791 s [14.791 s, 14.791 s] -457.0 ms (-3.0%)
iast 18.509 s [18.509 s, 18.509 s] 3.261 s (21.4%)
iast_GLOBAL 17.84 s [17.84 s, 17.84 s] 2.592 s (17.0%)
profiling 14.698 s [14.698 s, 14.698 s] -550.0 ms (-3.6%)
tracing 14.821 s [14.821 s, 14.821 s] -427.0 ms (-2.8%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.57.0-SNAPSHOT~b79d752021, baseline=1.57.0-SNAPSHOT~fef9d162d84
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.483 ms) : 1471, 1494
.   : milestone, 1483,
appsec (3.679 ms) : 3464, 3893
.   : milestone, 3679,
iast (2.217 ms) : 2153, 2281
.   : milestone, 2217,
iast_GLOBAL (2.258 ms) : 2193, 2323
.   : milestone, 2258,
profiling (2.093 ms) : 2039, 2146
.   : milestone, 2093,
tracing (2.05 ms) : 1999, 2100
.   : milestone, 2050,
section candidate
no_agent (1.479 ms) : 1468, 1491
.   : milestone, 1479,
appsec (3.737 ms) : 3516, 3958
.   : milestone, 3737,
iast (2.217 ms) : 2152, 2281
.   : milestone, 2217,
iast_GLOBAL (2.255 ms) : 2191, 2320
.   : milestone, 2255,
profiling (2.072 ms) : 2020, 2125
.   : milestone, 2072,
tracing (2.053 ms) : 2002, 2104
.   : milestone, 2053,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.483 ms [1.471 ms, 1.494 ms] -
appsec 3.679 ms [3.464 ms, 3.893 ms] 2.196 ms (148.1%)
iast 2.217 ms [2.153 ms, 2.281 ms] 734.134 µs (49.5%)
iast_GLOBAL 2.258 ms [2.193 ms, 2.323 ms] 775.524 µs (52.3%)
profiling 2.093 ms [2.039 ms, 2.146 ms] 609.922 µs (41.1%)
tracing 2.05 ms [1.999 ms, 2.1 ms] 567.049 µs (38.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.479 ms [1.468 ms, 1.491 ms] -
appsec 3.737 ms [3.516 ms, 3.958 ms] 2.257 ms (152.6%)
iast 2.217 ms [2.152 ms, 2.281 ms] 737.2 µs (49.8%)
iast_GLOBAL 2.255 ms [2.191 ms, 2.32 ms] 775.913 µs (52.4%)
profiling 2.072 ms [2.02 ms, 2.125 ms] 592.825 µs (40.1%)
tracing 2.053 ms [2.002 ms, 2.104 ms] 573.788 µs (38.8%)

@mcculls mcculls force-pushed the mcculls/support-classpath-only-mode branch from e17e8ad to 9925a51 Compare December 13, 2025 16:33
@mcculls
Support 'aot_training' mode to be used when creating AOT caches
07cf114 
In this mode only the agent jar is added to the boot classpath, no services are started.

Also workaround potential AOT bug where TraceInterceptor is mistakenly restored from the
system class-loader in production, even though it was visible from the boot class-loader
during training, resulting in LinkageErrors.

Any call to Tracer.addTraceInterceptor from application code in the system class-loader
appears to trigger this bug. The workaround is to replace these calls during training
with opcodes that pop the tracer and argument, and push the expected return value.

This transformation is not persisted, so in production the original method is invoked.
@mcculls
Add automatic check for AOT training mode, based on CDS.isDumpingArch…
c234bb3 
…ive()
@mcculls mcculls force-pushed the mcculls/support-classpath-only-mode branch from 2d191f2 to 7e95bec Compare December 13, 2025 21:00
@mcculls mcculls force-pushed the mcculls/support-classpath-only-mode branch from 7e95bec to b79d752 Compare December 13, 2025 21:15
@mcculls mcculls requested a review from a team as a code owner December 13, 2025 21:15
@mcculls mcculls requested review from AlexeyKuznetsov-DD, PerfectSlayer and amarziali and removed request for a team December 13, 2025 21:15
@mcculls
Copy link
Contributor Author

mcculls commented Dec 13, 2025

wrt. dd-gitlab/validate_supported_configurations_v2_local_file - I'll add the new DD_AOT_TRAINING envar to the dashboard once people are ok with it

PerfectSlayer
PerfectSlayer reviewed Dec 15, 2025
View reviewed changes
Copy link
Contributor

@PerfectSlayer PerfectSlayer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 suggestion: ‏That would be interesting to include this mode into benchmarks too 😉

@mcculls mcculls force-pushed the mcculls/support-classpath-only-mode branch from f9564a7 to 436305b Compare December 15, 2025 09:31
@mcculls mcculls merged commit 81f2326 into master Dec 15, 2025
561 checks passed
@mcculls mcculls deleted the mcculls/support-classpath-only-mode branch December 15, 2025 10:28
@github-actions github-actions bot added this to the 1.57.0 milestone Dec 15, 2025
This was referenced Dec 15, 2025
Open
Open
@yagrxu yagrxu mentioned this pull request Dec 15, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@bric3 bric3 bric3 approved these changes

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD Awaiting requested review from AlexeyKuznetsov-DD AlexeyKuznetsov-DD is a code owner automatically assigned from DataDog/apm-lang-platform-java

@amarziali amarziali Awaiting requested review from amarziali

@manuel-alvarez-alvarez manuel-alvarez-alvarez Awaiting requested review from manuel-alvarez-alvarez

Assignees

No one assigned

Labels

comp: core Tracer core tag: performance Performance related changes type: enhancement Enhancements and improvements

Projects

None yet

Milestone

1.57.0

Development

Successfully merging this pull request may close these issues.

7 participants

@mcculls @jpbempel @PerfectSlayer @bric3 @manuel-alvarez-alvarez @amarziali