Implement debug context propagation to enable live debugging of java applications by evanchooly · Pull Request #7286 · DataDog/dd-trace-java

evanchooly · 2024-07-05T15:53:13Z

This PR enhances the exist span origin feature with optional snapshot captures. It introduces variable configuration via the _dd.p.debug tag currently set via propagation tags. It's a follow on implementation of two RFCs:

Both specs are currently in flux, however, so to some extent this PR is ahead of the curve. There are some aspects that could likely be cleaned up and I will try to call out the one specific case I'd like some feedback on in case a better option presents itself.

pr-commenter · 2024-07-05T18:50:58Z

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	evanchooly/DEBUG-2434
git_commit_date	1722520765	1722526332
git_commit_sha	`1d41c00`	`c126dd8`
release_version	1.38.0-SNAPSHOT~1d41c00244	1.38.0-SNAPSHOT~c126dd86e3

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1722528896	1722528896
ci_job_id	592089971	592089971
ci_pipeline_id	40692417	40692417
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 2 performance improvements and 3 performance regressions! Performance is the same for 49 metrics, 9 unstable metrics.

scenario	Δ mean execution_time	candidate mean execution_time	baseline mean execution_time
scenario:startup:insecure-bank:iast:AppSec	better [-7.605ms; -4.008ms] or [-14.270%; -7.521%]	47.488ms	53.295ms
scenario:startup:insecure-bank:iast:Remote Config	worse [+11.473µs; +49.018µs] or [+2.043%; +8.726%]	591.967µs	561.721µs
scenario:startup:insecure-bank:tracing:Telemetry	worse [+206.664µs; +456.886µs] or [+2.823%; +6.241%]	7.652ms	7.320ms
scenario:startup:petclinic:profiling:Remote Config	better [-93.040µs; -45.396µs] or [-12.963%; -6.325%]	648.497µs	717.716µs
scenario:startup:petclinic:tracing:Telemetry	worse [+208.517µs; +540.839µs] or [+2.862%; +7.422%]	7.662ms	7.287ms

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.38.0-SNAPSHOT~c126dd86e3, baseline=1.38.0-SNAPSHOT~1d41c00244

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.046 s) : 0, 1046099
Total [baseline] (10.314 s) : 0, 10313630
Agent [candidate] (1.049 s) : 0, 1049483
Total [candidate] (10.348 s) : 0, 10347632
section appsec
Agent [baseline] (1.168 s) : 0, 1167860
Total [baseline] (10.552 s) : 0, 10551528
Agent [candidate] (1.161 s) : 0, 1161174
Total [candidate] (10.488 s) : 0, 10487787
section iast
Agent [baseline] (1.173 s) : 0, 1173250
Total [baseline] (10.782 s) : 0, 10782351
Agent [candidate] (1.173 s) : 0, 1172564
Total [candidate] (10.77 s) : 0, 10769701
section profiling
Agent [baseline] (1.244 s) : 0, 1243684
Total [baseline] (10.567 s) : 0, 10567013
Agent [candidate] (1.24 s) : 0, 1239933
Total [candidate] (10.622 s) : 0, 10622215

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.046 s	-
Agent	appsec	1.168 s	121.761 ms (11.6%)
Agent	iast	1.173 s	127.151 ms (12.2%)
Agent	profiling	1.244 s	197.585 ms (18.9%)
Total	tracing	10.314 s	-
Total	appsec	10.552 s	237.898 ms (2.3%)
Total	iast	10.782 s	468.721 ms (4.5%)
Total	profiling	10.567 s	253.383 ms (2.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.049 s	-
Agent	appsec	1.161 s	111.691 ms (10.6%)
Agent	iast	1.173 s	123.081 ms (11.7%)
Agent	profiling	1.24 s	190.45 ms (18.1%)
Total	tracing	10.348 s	-
Total	appsec	10.488 s	140.155 ms (1.4%)
Total	iast	10.77 s	422.069 ms (4.1%)
Total	profiling	10.622 s	274.582 ms (2.7%)

gantt
    title petclinic - break down per module: candidate=1.38.0-SNAPSHOT~c126dd86e3, baseline=1.38.0-SNAPSHOT~1d41c00244

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (668.85 ms) : 0, 668850
BytebuddyAgent [candidate] (670.884 ms) : 0, 670884
GlobalTracer [baseline] (305.443 ms) : 0, 305443
GlobalTracer [candidate] (306.571 ms) : 0, 306571
AppSec [baseline] (50.379 ms) : 0, 50379
AppSec [candidate] (50.198 ms) : 0, 50198
Remote Config [baseline] (674.768 µs) : 0, 675
Remote Config [candidate] (677.316 µs) : 0, 677
Telemetry [baseline] (7.287 ms) : 0, 7287
Telemetry [candidate] (7.662 ms) : 0, 7662
section appsec
BytebuddyAgent [baseline] (680.437 ms) : 0, 680437
BytebuddyAgent [candidate] (676.298 ms) : 0, 676298
GlobalTracer [baseline] (299.448 ms) : 0, 299448
GlobalTracer [candidate] (298.742 ms) : 0, 298742
AppSec [baseline] (155.371 ms) : 0, 155371
AppSec [candidate] (153.628 ms) : 0, 153628
Remote Config [baseline] (603.56 µs) : 0, 604
Remote Config [candidate] (626.926 µs) : 0, 627
Telemetry [baseline] (8.224 ms) : 0, 8224
Telemetry [candidate] (7.906 ms) : 0, 7906
IAST [baseline] (20.405 ms) : 0, 20405
IAST [candidate] (20.928 ms) : 0, 20928
section iast
BytebuddyAgent [baseline] (783.846 ms) : 0, 783846
BytebuddyAgent [candidate] (781.293 ms) : 0, 781293
GlobalTracer [baseline] (296.228 ms) : 0, 296228
GlobalTracer [candidate] (295.801 ms) : 0, 295801
AppSec [baseline] (48.267 ms) : 0, 48267
AppSec [candidate] (47.475 ms) : 0, 47475
Remote Config [baseline] (587.162 µs) : 0, 587
Remote Config [candidate] (591.926 µs) : 0, 592
Telemetry [baseline] (7.87 ms) : 0, 7870
Telemetry [candidate] (7.102 ms) : 0, 7102
IAST [baseline] (22.9 ms) : 0, 22900
IAST [candidate] (26.821 ms) : 0, 26821
section profiling
ProfilingAgent [baseline] (95.118 ms) : 0, 95118
ProfilingAgent [candidate] (94.602 ms) : 0, 94602
BytebuddyAgent [baseline] (663.777 ms) : 0, 663777
BytebuddyAgent [candidate] (661.464 ms) : 0, 661464
GlobalTracer [baseline] (388.526 ms) : 0, 388526
GlobalTracer [candidate] (388.178 ms) : 0, 388178
AppSec [baseline] (51.277 ms) : 0, 51277
AppSec [candidate] (51.601 ms) : 0, 51601
Remote Config [baseline] (717.716 µs) : 0, 718
Remote Config [candidate] (648.497 µs) : 0, 648
Telemetry [baseline] (7.28 ms) : 0, 7280
Telemetry [candidate] (7.395 ms) : 0, 7395
Profiling [baseline] (95.142 ms) : 0, 95142
Profiling [candidate] (94.626 ms) : 0, 94626

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.38.0-SNAPSHOT~c126dd86e3, baseline=1.38.0-SNAPSHOT~1d41c00244

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.045 s) : 0, 1044625
Total [baseline] (8.524 s) : 0, 8523617
Agent [candidate] (1.044 s) : 0, 1044083
Total [candidate] (8.488 s) : 0, 8488347
section iast
Agent [baseline] (1.191 s) : 0, 1191207
Total [baseline] (9.034 s) : 0, 9033947
Agent [candidate] (1.174 s) : 0, 1173972
Total [candidate] (8.987 s) : 0, 8986711
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.173 s) : 0, 1172774
Total [baseline] (8.963 s) : 0, 8962999
Agent [candidate] (1.171 s) : 0, 1170777
Total [candidate] (8.946 s) : 0, 8945814
section iast_TELEMETRY_OFF
Agent [baseline] (1.171 s) : 0, 1170919
Total [baseline] (8.997 s) : 0, 8996528
Agent [candidate] (1.18 s) : 0, 1179928
Total [candidate] (9.015 s) : 0, 9014604

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.045 s	-
Agent	iast	1.191 s	146.582 ms (14.0%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.173 s	128.149 ms (12.3%)
Agent	iast_TELEMETRY_OFF	1.171 s	126.294 ms (12.1%)
Total	tracing	8.524 s	-
Total	iast	9.034 s	510.33 ms (6.0%)
Total	iast_HARDCODED_SECRET_DISABLED	8.963 s	439.382 ms (5.2%)
Total	iast_TELEMETRY_OFF	8.997 s	472.911 ms (5.5%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.044 s	-
Agent	iast	1.174 s	129.889 ms (12.4%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.171 s	126.693 ms (12.1%)
Agent	iast_TELEMETRY_OFF	1.18 s	135.845 ms (13.0%)
Total	tracing	8.488 s	-
Total	iast	8.987 s	498.364 ms (5.9%)
Total	iast_HARDCODED_SECRET_DISABLED	8.946 s	457.467 ms (5.4%)
Total	iast_TELEMETRY_OFF	9.015 s	526.257 ms (6.2%)

gantt
    title insecure-bank - break down per module: candidate=1.38.0-SNAPSHOT~c126dd86e3, baseline=1.38.0-SNAPSHOT~1d41c00244

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (667.384 ms) : 0, 667384
BytebuddyAgent [candidate] (666.715 ms) : 0, 666715
GlobalTracer [baseline] (305.657 ms) : 0, 305657
GlobalTracer [candidate] (305.54 ms) : 0, 305540
AppSec [baseline] (50.176 ms) : 0, 50176
AppSec [candidate] (50.073 ms) : 0, 50073
Remote Config [baseline] (682.375 µs) : 0, 682
Remote Config [candidate] (674.027 µs) : 0, 674
Telemetry [baseline] (7.32 ms) : 0, 7320
Telemetry [candidate] (7.652 ms) : 0, 7652
section iast
BytebuddyAgent [baseline] (795.107 ms) : 0, 795107
BytebuddyAgent [candidate] (781.275 ms) : 0, 781275
GlobalTracer [baseline] (299.206 ms) : 0, 299206
GlobalTracer [candidate] (296.127 ms) : 0, 296127
AppSec [baseline] (53.295 ms) : 0, 53295
AppSec [candidate] (47.488 ms) : 0, 47488
IAST [baseline] (22.378 ms) : 0, 22378
IAST [candidate] (28.039 ms) : 0, 28039
Remote Config [baseline] (561.721 µs) : 0, 562
Remote Config [candidate] (591.967 µs) : 0, 592
Telemetry [baseline] (6.964 ms) : 0, 6964
Telemetry [candidate] (6.97 ms) : 0, 6970
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (780.878 ms) : 0, 780878
BytebuddyAgent [candidate] (779.766 ms) : 0, 779766
GlobalTracer [baseline] (295.549 ms) : 0, 295549
GlobalTracer [candidate] (295.277 ms) : 0, 295277
AppSec [baseline] (52.105 ms) : 0, 52105
AppSec [candidate] (48.09 ms) : 0, 48090
IAST [baseline] (21.676 ms) : 0, 21676
IAST [candidate] (26.548 ms) : 0, 26548
Remote Config [baseline] (563.051 µs) : 0, 563
Remote Config [candidate] (587.114 µs) : 0, 587
Telemetry [baseline] (8.525 ms) : 0, 8525
Telemetry [candidate] (6.982 ms) : 0, 6982
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (781.132 ms) : 0, 781132
BytebuddyAgent [candidate] (788.326 ms) : 0, 788326
GlobalTracer [baseline] (296.095 ms) : 0, 296095
GlobalTracer [candidate] (296.634 ms) : 0, 296634
AppSec [baseline] (46.973 ms) : 0, 46973
AppSec [candidate] (47.451 ms) : 0, 47451
IAST [baseline] (25.602 ms) : 0, 25602
IAST [candidate] (26.344 ms) : 0, 26344
Remote Config [baseline] (607.949 µs) : 0, 608
Remote Config [candidate] (573.237 µs) : 0, 573
Telemetry [baseline] (6.97 ms) : 0, 6970
Telemetry [candidate] (6.93 ms) : 0, 6930

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-08-01T15:45:41	2024-08-01T15:52:31
git_branch	master	evanchooly/DEBUG-2434
git_commit_date	1722520765	1722526332
git_commit_sha	`1d41c00`	`c126dd8`
release_version	1.38.0-SNAPSHOT~1d41c00244	1.38.0-SNAPSHOT~c126dd86e3
start_time	2024-08-01T15:45:27	2024-08-01T15:52:18

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1722527897	1722527897
ci_job_id	592089972	592089972
ci_pipeline_id	40692417	40692417
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.38.0-SNAPSHOT~c126dd86e3, baseline=1.38.0-SNAPSHOT~1d41c00244
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.339 ms) : 1320, 1359
.   : milestone, 1339,
appsec (1.725 ms) : 1701, 1748
.   : milestone, 1725,
appsec_no_iast (1.721 ms) : 1696, 1746
.   : milestone, 1721,
iast (1.473 ms) : 1450, 1496
.   : milestone, 1473,
profiling (1.513 ms) : 1487, 1538
.   : milestone, 1513,
tracing (1.461 ms) : 1436, 1486
.   : milestone, 1461,
section candidate
no_agent (1.356 ms) : 1336, 1375
.   : milestone, 1356,
appsec (1.728 ms) : 1704, 1751
.   : milestone, 1728,
appsec_no_iast (1.701 ms) : 1677, 1726
.   : milestone, 1701,
iast (1.489 ms) : 1467, 1511
.   : milestone, 1489,
profiling (1.511 ms) : 1485, 1537
.   : milestone, 1511,
tracing (1.471 ms) : 1447, 1495
.   : milestone, 1471,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.339 ms [1.32 ms, 1.359 ms]	-
appsec	1.725 ms [1.701 ms, 1.748 ms]	385.249 µs (28.8%)
appsec_no_iast	1.721 ms [1.696 ms, 1.746 ms]	381.162 µs (28.5%)
iast	1.473 ms [1.45 ms, 1.496 ms]	133.668 µs (10.0%)
profiling	1.513 ms [1.487 ms, 1.538 ms]	173.207 µs (12.9%)
tracing	1.461 ms [1.436 ms, 1.486 ms]	121.588 µs (9.1%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.356 ms [1.336 ms, 1.375 ms]	-
appsec	1.728 ms [1.704 ms, 1.751 ms]	371.955 µs (27.4%)
appsec_no_iast	1.701 ms [1.677 ms, 1.726 ms]	345.814 µs (25.5%)
iast	1.489 ms [1.467 ms, 1.511 ms]	133.543 µs (9.9%)
profiling	1.511 ms [1.485 ms, 1.537 ms]	155.418 µs (11.5%)
tracing	1.471 ms [1.447 ms, 1.495 ms]	115.671 µs (8.5%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.38.0-SNAPSHOT~c126dd86e3, baseline=1.38.0-SNAPSHOT~1d41c00244
    dateFormat X
    axisFormat %s
section baseline
no_agent (371.471 µs) : 352, 391
.   : milestone, 371,
iast (493.571 µs) : 472, 515
.   : milestone, 494,
iast_FULL (553.233 µs) : 532, 574
.   : milestone, 553,
iast_GLOBAL (516.3 µs) : 494, 539
.   : milestone, 516,
iast_HARDCODED_SECRET_DISABLED (484.967 µs) : 464, 506
.   : milestone, 485,
iast_INACTIVE (460.514 µs) : 438, 483
.   : milestone, 461,
iast_TELEMETRY_OFF (477.044 µs) : 455, 499
.   : milestone, 477,
tracing (443.898 µs) : 423, 465
.   : milestone, 444,
section candidate
no_agent (378.015 µs) : 358, 398
.   : milestone, 378,
iast (485.065 µs) : 464, 506
.   : milestone, 485,
iast_FULL (552.785 µs) : 532, 574
.   : milestone, 553,
iast_GLOBAL (512.236 µs) : 491, 534
.   : milestone, 512,
iast_HARDCODED_SECRET_DISABLED (481.81 µs) : 461, 503
.   : milestone, 482,
iast_INACTIVE (454.916 µs) : 434, 476
.   : milestone, 455,
iast_TELEMETRY_OFF (473.866 µs) : 453, 495
.   : milestone, 474,
tracing (450.919 µs) : 430, 472
.   : milestone, 451,

baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	371.471 µs [351.637 µs, 391.305 µs]	-
iast	493.571 µs [472.155 µs, 514.987 µs]	122.1 µs (32.9%)
iast_FULL	553.233 µs [532.15 µs, 574.315 µs]	181.762 µs (48.9%)
iast_GLOBAL	516.3 µs [494.004 µs, 538.596 µs]	144.829 µs (39.0%)
iast_HARDCODED_SECRET_DISABLED	484.967 µs [464.058 µs, 505.877 µs]	113.496 µs (30.6%)
iast_INACTIVE	460.514 µs [438.363 µs, 482.664 µs]	89.043 µs (24.0%)
iast_TELEMETRY_OFF	477.044 µs [455.271 µs, 498.817 µs]	105.573 µs (28.4%)
tracing	443.898 µs [422.922 µs, 464.874 µs]	72.427 µs (19.5%)

candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	378.015 µs [357.544 µs, 398.485 µs]	-
iast	485.065 µs [463.834 µs, 506.296 µs]	107.05 µs (28.3%)
iast_FULL	552.785 µs [531.553 µs, 574.018 µs]	174.77 µs (46.2%)
iast_GLOBAL	512.236 µs [490.502 µs, 533.97 µs]	134.221 µs (35.5%)
iast_HARDCODED_SECRET_DISABLED	481.81 µs [460.711 µs, 502.91 µs]	103.795 µs (27.5%)
iast_INACTIVE	454.916 µs [433.891 µs, 475.941 µs]	76.901 µs (20.3%)
iast_TELEMETRY_OFF	473.866 µs [452.505 µs, 495.227 µs]	95.851 µs (25.4%)
tracing	450.919 µs [430.053 µs, 471.785 µs]	72.904 µs (19.3%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	evanchooly/DEBUG-2434
git_commit_date	1722520765	1722526332
git_commit_sha	`1d41c00`	`c126dd8`
release_version	1.38.0-SNAPSHOT~1d41c00244	1.38.0-SNAPSHOT~c126dd86e3

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1722528432	1722528432
ci_job_id	592089973	592089973
ci_pipeline_id	40692417	40692417
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.38.0-SNAPSHOT~c126dd86e3, baseline=1.38.0-SNAPSHOT~1d41c00244
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.456 ms) : 1445, 1467
.   : milestone, 1456,
appsec (2.202 ms) : 2168, 2236
.   : milestone, 2202,
iast (1.96 ms) : 1919, 2002
.   : milestone, 1960,
iast_GLOBAL (2.005 ms) : 1963, 2048
.   : milestone, 2005,
profiling (1.861 ms) : 1826, 1897
.   : milestone, 1861,
tracing (1.835 ms) : 1802, 1868
.   : milestone, 1835,
section candidate
no_agent (1.455 ms) : 1443, 1466
.   : milestone, 1455,
appsec (2.207 ms) : 2172, 2242
.   : milestone, 2207,
iast (1.956 ms) : 1915, 1997
.   : milestone, 1956,
iast_GLOBAL (2.0 ms) : 1959, 2041
.   : milestone, 2000,
profiling (1.844 ms) : 1811, 1877
.   : milestone, 1844,
tracing (1.837 ms) : 1804, 1870
.   : milestone, 1837,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.456 ms [1.445 ms, 1.467 ms]	-
appsec	2.202 ms [2.168 ms, 2.236 ms]	746.14 µs (51.2%)
iast	1.96 ms [1.919 ms, 2.002 ms]	504.18 µs (34.6%)
iast_GLOBAL	2.005 ms [1.963 ms, 2.048 ms]	549.38 µs (37.7%)
profiling	1.861 ms [1.826 ms, 1.897 ms]	405.307 µs (27.8%)
tracing	1.835 ms [1.802 ms, 1.868 ms]	378.665 µs (26.0%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.455 ms [1.443 ms, 1.466 ms]	-
appsec	2.207 ms [2.172 ms, 2.242 ms]	752.514 µs (51.7%)
iast	1.956 ms [1.915 ms, 1.997 ms]	501.501 µs (34.5%)
iast_GLOBAL	2.0 ms [1.959 ms, 2.041 ms]	545.764 µs (37.5%)
profiling	1.844 ms [1.811 ms, 1.877 ms]	389.489 µs (26.8%)
tracing	1.837 ms [1.804 ms, 1.87 ms]	382.276 µs (26.3%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.38.0-SNAPSHOT~c126dd86e3, baseline=1.38.0-SNAPSHOT~1d41c00244
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.001 s) : 15001000, 15001000
.   : milestone, 15001000,
appsec (15.222 s) : 15222000, 15222000
.   : milestone, 15222000,
iast (18.772 s) : 18772000, 18772000
.   : milestone, 18772000,
iast_GLOBAL (17.71 s) : 17710000, 17710000
.   : milestone, 17710000,
profiling (15.003 s) : 15003000, 15003000
.   : milestone, 15003000,
tracing (15.248 s) : 15248000, 15248000
.   : milestone, 15248000,
section candidate
no_agent (15.419 s) : 15419000, 15419000
.   : milestone, 15419000,
appsec (14.89 s) : 14890000, 14890000
.   : milestone, 14890000,
iast (18.943 s) : 18943000, 18943000
.   : milestone, 18943000,
iast_GLOBAL (17.926 s) : 17926000, 17926000
.   : milestone, 17926000,
profiling (15.205 s) : 15205000, 15205000
.   : milestone, 15205000,
tracing (15.081 s) : 15081000, 15081000
.   : milestone, 15081000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.001 s [15.001 s, 15.001 s]	-
appsec	15.222 s [15.222 s, 15.222 s]	221.0 ms (1.5%)
iast	18.772 s [18.772 s, 18.772 s]	3.771 s (25.1%)
iast_GLOBAL	17.71 s [17.71 s, 17.71 s]	2.709 s (18.1%)
profiling	15.003 s [15.003 s, 15.003 s]	2.0 ms (0.0%)
tracing	15.248 s [15.248 s, 15.248 s]	247.0 ms (1.6%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.419 s [15.419 s, 15.419 s]	-
appsec	14.89 s [14.89 s, 14.89 s]	-529.0 ms (-3.4%)
iast	18.943 s [18.943 s, 18.943 s]	3.524 s (22.9%)
iast_GLOBAL	17.926 s [17.926 s, 17.926 s]	2.507 s (16.3%)
profiling	15.205 s [15.205 s, 15.205 s]	-214.0 ms (-1.4%)
tracing	15.081 s [15.081 s, 15.081 s]	-338.0 ms (-2.2%)

pr-commenter · 2024-07-15T16:59:54Z

Debugger benchmarks

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
ci_job_date	1722527078	1722527465
end_time	2024-08-01T15:45:51	2024-08-01T15:52:19
git_branch	master	evanchooly/DEBUG-2434
git_commit_sha	`1d41c00`	`c126dd8`
start_time	2024-08-01T15:44:39	2024-08-01T15:51:06

See matching parameters

	Baseline	Candidate
ci_job_id	592089977	592089977
ci_pipeline_id	40692417	40692417
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date	1722526332	1722526332

Summary

Found 5 performance improvements and 0 performance regressions! Performance is the same for 4 metrics, 6 unstable metrics.

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:loop	better [-468.817µs; -455.314µs] or [-4.346%; -4.221%]	better [-482.056µs; -418.538µs] or [-4.402%; -3.822%]	better [-479.674µs; -362.760µs] or [-4.352%; -3.291%]	better [-653.230µs; -165.595µs] or [-5.734%; -1.454%]	better [+2.414op/s; +4.384op/s] or [+2.667%; +4.845%]

See unchanged results

scenario	Δ mean agg_http_req_duration_min	Δ mean agg_http_req_duration_p50	Δ mean agg_http_req_duration_p75	Δ mean agg_http_req_duration_p99	Δ mean throughput
scenario:noprobe	unstable [-15.307µs; +35.929µs] or [-6.073%; +14.255%]	unstable [-20.969µs; +53.710µs] or [-7.422%; +19.012%]	unstable [-30.217µs; +66.397µs] or [-10.250%; +22.522%]	unstable [-120.641µs; +106.372µs] or [-18.243%; +16.085%]	same
scenario:basic	same	same	same	unstable [-30.785µs; +52.432µs] or [-5.339%; +9.093%]	unstable [-311.701op/s; +34.347op/s] or [-11.533%; +1.271%]

Request duration reports for reports

gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (282.503 µs) : 260, 305
.   : milestone, 283,
basic (287.577 µs) : 280, 296
.   : milestone, 288,
loop (10.95 ms) : 10925, 10975
.   : milestone, 10950,
section candidate
noprobe (298.874 µs) : 254, 344
.   : milestone, 299,
basic (295.32 µs) : 286, 304
.   : milestone, 295,
loop (10.5 ms) : 10466, 10533
.   : milestone, 10500,

baseline results

Scenario	Request median duration [CI 0.99]
noprobe	282.503 µs [259.533 µs, 305.473 µs]
basic	287.577 µs [279.588 µs, 295.566 µs]
loop	10.95 ms [10.925 ms, 10.975 ms]

candidate results

Scenario	Request median duration [CI 0.99]
noprobe	298.874 µs [254.022 µs, 343.725 µs]
basic	295.32 µs [286.226 µs, 304.415 µs]
loop	10.5 ms [10.466 ms, 10.533 ms]

datadog-datadog-prod-us1 · 2024-07-18T16:05:48Z

Library Vulnerabilities

clean up varargs warning

mcculls

The CI tests show several instrumentation errors related to not finding the SpanOriginInfo class - those will need to be fixed before this can be merged.

Also was it intentional to remove the InstrumenterConfig.get().isSpanOriginEnabled() checks? Without those the span-origin code will now run everywhere - even when not using live debugging.

mcculls · 2024-08-02T18:21:35Z

  public EntrySpanOriginInstrumentation(String instrumentationName) {
    super(instrumentationName);
-    this.matcher = namedOneOf(getAnnotations());
+    this.matcher = NameMatchers.namedOneOf(getAnnotations());


side note: we prefer to static import NameMatchers / HierarchyMatchers methods in the rest of the codebase

mcculls

+1, thanks for your patience!

evanchooly requested a review from jpbempel July 5, 2024 15:53

evanchooly commented Jul 5, 2024

View reviewed changes

Comment thread ...ugger/debugger-bootstrap/src/main/java/datadog/trace/bootstrap/debugger/DebuggerContext.java Outdated

jpbempel reviewed Jul 8, 2024

View reviewed changes

evanchooly force-pushed the evanchooly/DEBUG-2434 branch from e79a48e to 612933e Compare July 15, 2024 16:35

jpbempel reviewed Jul 16, 2024

View reviewed changes

Comment thread dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/probe/SpanDebuggerProbe.java Outdated

Comment thread ...ent/agent-debugger/src/main/java/com/datadog/debugger/snapshot/SpanDebuggerProbeManager.java Outdated

evanchooly force-pushed the evanchooly/DEBUG-2434 branch from cab3872 to 805e31e Compare July 18, 2024 16:03

evanchooly force-pushed the evanchooly/DEBUG-2434 branch from 805e31e to 20c5129 Compare July 18, 2024 16:08

evanchooly marked this pull request as ready for review July 18, 2024 16:27

evanchooly requested review from a team as code owners July 18, 2024 16:27

evanchooly requested review from jpbempel, mcculls and ygree and removed request for a team July 18, 2024 16:27

jpbempel reviewed Jul 19, 2024

View reviewed changes

evanchooly force-pushed the evanchooly/DEBUG-2434 branch from 1b15fd9 to cdd9a02 Compare July 19, 2024 14:42

implement distributed debugging

62db520

evanchooly force-pushed the evanchooly/DEBUG-2434 branch from cdd9a02 to 62db520 Compare July 19, 2024 15:04

jpbempel approved these changes Jul 19, 2024

View reviewed changes

evanchooly added 3 commits July 22, 2024 14:13

basic fingerprinting test

13e77ac

dummy up some spans and invoke the probes directly

0e860d2

add some branch coverage

a624690

clean up varargs warning

shatzi reviewed Jul 23, 2024

View reviewed changes

some renaming suggestions from Idan

e6496a1

mcculls requested changes Jul 24, 2024

View reviewed changes

add enablement check before instrumenting code origin info

4c806b3

evanchooly added 4 commits July 26, 2024 20:17

CI test failure fixes

3c64ab9

let's try this on CI

1c06bda

add an additional check for testing

291d307

remove unused method

1c563bd

evanchooly requested a review from mcculls July 30, 2024 20:49

PR clean ups

2dc8e9b

evanchooly enabled auto-merge July 30, 2024 20:50

mcculls reviewed Jul 31, 2024

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/propagation/ptags/PTagsFactory.java Outdated

mcculls reviewed Jul 31, 2024

View reviewed changes

Comment thread dd-trace-core/src/main/java/datadog/trace/core/taginterceptor/TagInterceptor.java

mcculls reviewed Jul 31, 2024

View reviewed changes

Comment thread ...gger-bootstrap/src/main/java/datadog/trace/bootstrap/debugger/spanorigin/SpanOriginInfo.java Outdated

mcculls reviewed Jul 31, 2024

View reviewed changes

Comment thread ...gger-bootstrap/src/main/java/datadog/trace/bootstrap/debugger/spanorigin/SpanOriginInfo.java Outdated

dougqh reviewed Aug 1, 2024

View reviewed changes

Comment thread dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/exception/Fingerprinter.java Outdated

PR clean ups

c126dd8

evanchooly requested a review from mcculls August 1, 2024 16:36

mcculls reviewed Aug 2, 2024

View reviewed changes

mcculls approved these changes Aug 2, 2024

View reviewed changes

evanchooly merged commit c08dc4f into master Aug 2, 2024

evanchooly deleted the evanchooly/DEBUG-2434 branch August 2, 2024 18:25

github-actions Bot added this to the 1.38.0 milestone Aug 2, 2024

nayeem-kamal added the comp: debugger Dynamic Instrumentation label Aug 5, 2024

Conversation

evanchooly commented Jul 5, 2024

Uh oh!

Uh oh!

pr-commenter Bot commented Jul 5, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

pr-commenter Bot commented Jul 15, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Debugger benchmarks

Parameters

Summary

Uh oh!

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Jul 18, 2024

Library Vulnerabilities

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcculls left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mcculls Aug 2, 2024

Choose a reason for hiding this comment

Uh oh!

mcculls left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

pr-commenter Bot commented Jul 5, 2024 •

edited

Loading

pr-commenter Bot commented Jul 15, 2024 •

edited

Loading