Skip to content

Reports instrumentation failure#5795

Merged
jpbempel merged 2 commits into
masterfrom
jpbempel/report-instrumentation-fails
Aug 30, 2023
Merged

Reports instrumentation failure#5795
jpbempel merged 2 commits into
masterfrom
jpbempel/report-instrumentation-fails

Conversation

@jpbempel

@jpbempel jpbempel commented Aug 28, 2023

Copy link
Copy Markdown
Member

What Does This Do

Send probe status error for all probes applied on the same class indicating instrumentation could not be applied for this class.

Motivation

Additional Notes

@jpbempel
jpbempel requested a review from a team as a code owner August 28, 2023 16:30
@jpbempel
jpbempel requested review from shatzi and removed request for a team August 28, 2023 16:30
@pr-commenter

pr-commenter Bot commented Aug 28, 2023 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~4818319bb7 1.20.0-SNAPSHOT~b6206d14c9
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.20.0-SNAPSHOT~b6206d14c9, baseline=1.20.0-SNAPSHOT~4818319bb7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.49 s) : 0, 1489582
Total [baseline] (12.124 s) : 0, 12123722
Agent [candidate] (1.49 s) : 0, 1489809
Total [candidate] (12.098 s) : 0, 12098362
section appsec
Agent [baseline] (1.624 s) : 0, 1623964
Total [baseline] (12.194 s) : 0, 12194139
Agent [candidate] (1.629 s) : 0, 1629257
Total [candidate] (12.23 s) : 0, 12230084
section iast
Agent [baseline] (1.63 s) : 0, 1629998
Total [baseline] (12.887 s) : 0, 12886575
Agent [candidate] (1.638 s) : 0, 1638231
Total [candidate] (12.917 s) : 0, 12917397
section profiling
Agent [baseline] (1.775 s) : 0, 1775243
Total [baseline] (12.426 s) : 0, 12426359
Agent [candidate] (1.785 s) : 0, 1785194
Total [candidate] (12.475 s) : 0, 12475257
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.49 s -
Agent appsec 1.624 s 134.382 ms (9.0%)
Agent iast 1.63 s 140.416 ms (9.4%)
Agent profiling 1.775 s 285.661 ms (19.2%)
Total tracing 12.124 s -
Total appsec 12.194 s 70.417 ms (0.6%)
Total iast 12.887 s 762.852 ms (6.3%)
Total profiling 12.426 s 302.636 ms (2.5%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.49 s -
Agent appsec 1.629 s 139.447 ms (9.4%)
Agent iast 1.638 s 148.422 ms (10.0%)
Agent profiling 1.785 s 295.385 ms (19.8%)
Total tracing 12.098 s -
Total appsec 12.23 s 131.722 ms (1.1%)
Total iast 12.917 s 819.035 ms (6.8%)
Total profiling 12.475 s 376.895 ms (3.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.20.0-SNAPSHOT~b6206d14c9, baseline=1.20.0-SNAPSHOT~4818319bb7

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (919.133 ms) : 0, 919133
BytebuddyAgent [candidate] (919.881 ms) : 0, 919881
GlobalTracer [baseline] (431.309 ms) : 0, 431309
GlobalTracer [candidate] (430.952 ms) : 0, 430952
AppSec [baseline] (70.568 ms) : 0, 70568
AppSec [candidate] (70.283 ms) : 0, 70283
Remote Config [baseline] (814.08 µs) : 0, 814
Remote Config [candidate] (813.14 µs) : 0, 813
Telemetry [baseline] (7.76 ms) : 0, 7760
Telemetry [candidate] (7.746 ms) : 0, 7746
section appsec
BytebuddyAgent [baseline] (921.006 ms) : 0, 921006
BytebuddyAgent [candidate] (924.86 ms) : 0, 924860
GlobalTracer [baseline] (432.142 ms) : 0, 432142
GlobalTracer [candidate] (432.364 ms) : 0, 432364
AppSec [baseline] (202.245 ms) : 0, 202245
AppSec [candidate] (203.22 ms) : 0, 203220
Remote Config [baseline] (889.061 µs) : 0, 889
Remote Config [candidate] (887.687 µs) : 0, 888
Telemetry [baseline] (7.522 ms) : 0, 7522
Telemetry [candidate] (7.565 ms) : 0, 7565
section iast
BytebuddyAgent [baseline] (1.069 s) : 0, 1068858
BytebuddyAgent [candidate] (1.074 s) : 0, 1073544
GlobalTracer [baseline] (407.212 ms) : 0, 407212
GlobalTracer [candidate] (409.814 ms) : 0, 409814
AppSec [baseline] (67.276 ms) : 0, 67276
AppSec [candidate] (67.695 ms) : 0, 67695
Remote Config [baseline] (715.454 µs) : 0, 715
Remote Config [candidate] (717.546 µs) : 0, 718
Telemetry [baseline] (7.625 ms) : 0, 7625
Telemetry [candidate] (7.666 ms) : 0, 7666
IAST [baseline] (18.291 ms) : 0, 18291
IAST [candidate] (18.445 ms) : 0, 18445
section profiling
ProfilingAgent [baseline] (134.958 ms) : 0, 134958
ProfilingAgent [candidate] (136.604 ms) : 0, 136604
BytebuddyAgent [baseline] (941.533 ms) : 0, 941533
BytebuddyAgent [candidate] (946.74 ms) : 0, 946740
GlobalTracer [baseline] (531.503 ms) : 0, 531503
GlobalTracer [candidate] (534.256 ms) : 0, 534256
AppSec [baseline] (71.048 ms) : 0, 71048
AppSec [candidate] (71.135 ms) : 0, 71135
Remote Config [baseline] (815.5 µs) : 0, 816
Remote Config [candidate] (818.901 µs) : 0, 819
Telemetry [baseline] (7.866 ms) : 0, 7866
Telemetry [candidate] (7.906 ms) : 0, 7906
Profiling [baseline] (135.007 ms) : 0, 135007
Profiling [candidate] (136.652 ms) : 0, 136652
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.20.0-SNAPSHOT~b6206d14c9, baseline=1.20.0-SNAPSHOT~4818319bb7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.49 s) : 0, 1490325
Total [baseline] (12.65 s) : 0, 12650477
Agent [candidate] (1.489 s) : 0, 1488856
Total [candidate] (12.592 s) : 0, 12592344
section appsec
Agent [baseline] (1.619 s) : 0, 1619061
Total [baseline] (12.761 s) : 0, 12761400
Agent [candidate] (1.621 s) : 0, 1621298
Total [candidate] (12.757 s) : 0, 12757200
section iast
Agent [baseline] (1.638 s) : 0, 1637575
Total [baseline] (13.09 s) : 0, 13090470
Agent [candidate] (1.637 s) : 0, 1636820
Total [candidate] (13.121 s) : 0, 13120711
section profiling
Agent [baseline] (1.766 s) : 0, 1766094
Total [baseline] (12.98 s) : 0, 12979566
Agent [candidate] (1.766 s) : 0, 1765711
Total [candidate] (13.022 s) : 0, 13021535
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.49 s -
Agent appsec 1.619 s 128.735 ms (8.6%)
Agent iast 1.638 s 147.25 ms (9.9%)
Agent profiling 1.766 s 275.769 ms (18.5%)
Total tracing 12.65 s -
Total appsec 12.761 s 110.923 ms (0.9%)
Total iast 13.09 s 439.993 ms (3.5%)
Total profiling 12.98 s 329.09 ms (2.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.489 s -
Agent appsec 1.621 s 132.442 ms (8.9%)
Agent iast 1.637 s 147.964 ms (9.9%)
Agent profiling 1.766 s 276.855 ms (18.6%)
Total tracing 12.592 s -
Total appsec 12.757 s 164.855 ms (1.3%)
Total iast 13.121 s 528.367 ms (4.2%)
Total profiling 13.022 s 429.19 ms (3.4%) 
gantt
    title petclinic - break down per module: candidate=1.20.0-SNAPSHOT~b6206d14c9, baseline=1.20.0-SNAPSHOT~4818319bb7

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (920.293 ms) : 0, 920293
BytebuddyAgent [candidate] (918.134 ms) : 0, 918134
GlobalTracer [baseline] (430.517 ms) : 0, 430517
GlobalTracer [candidate] (431.694 ms) : 0, 431694
AppSec [baseline] (71.049 ms) : 0, 71049
AppSec [candidate] (70.552 ms) : 0, 70552
Remote Config [baseline] (825.007 µs) : 0, 825
Remote Config [candidate] (815.502 µs) : 0, 816
Telemetry [baseline] (7.767 ms) : 0, 7767
Telemetry [candidate] (7.79 ms) : 0, 7790
section appsec
BytebuddyAgent [baseline] (918.401 ms) : 0, 918401
BytebuddyAgent [candidate] (920.461 ms) : 0, 920461
GlobalTracer [baseline] (430.212 ms) : 0, 430212
GlobalTracer [candidate] (430.05 ms) : 0, 430050
AppSec [baseline] (202.281 ms) : 0, 202281
AppSec [candidate] (202.522 ms) : 0, 202522
Remote Config [baseline] (882.883 µs) : 0, 883
Remote Config [candidate] (882.9 µs) : 0, 883
Telemetry [baseline] (7.496 ms) : 0, 7496
Telemetry [candidate] (7.547 ms) : 0, 7547
section iast
BytebuddyAgent [baseline] (1.074 s) : 0, 1074420
BytebuddyAgent [candidate] (1.074 s) : 0, 1074435
GlobalTracer [baseline] (408.988 ms) : 0, 408988
GlobalTracer [candidate] (408.324 ms) : 0, 408324
AppSec [baseline] (67.168 ms) : 0, 67168
AppSec [candidate] (67.343 ms) : 0, 67343
Remote Config [baseline] (726.697 µs) : 0, 727
Remote Config [candidate] (716.416 µs) : 0, 716
Telemetry [baseline] (7.659 ms) : 0, 7659
Telemetry [candidate] (7.571 ms) : 0, 7571
IAST [baseline] (18.472 ms) : 0, 18472
IAST [candidate] (18.232 ms) : 0, 18232
section profiling
ProfilingAgent [baseline] (118.818 ms) : 0, 118818
ProfilingAgent [candidate] (119.689 ms) : 0, 119689
BytebuddyAgent [baseline] (946.045 ms) : 0, 946045
BytebuddyAgent [candidate] (944.194 ms) : 0, 944194
GlobalTracer [baseline] (533.674 ms) : 0, 533674
GlobalTracer [candidate] (534.28 ms) : 0, 534280
AppSec [baseline] (71.44 ms) : 0, 71440
AppSec [candidate] (71.796 ms) : 0, 71796
Remote Config [baseline] (815.402 µs) : 0, 815
Remote Config [candidate] (813.761 µs) : 0, 814
Telemetry [baseline] (7.823 ms) : 0, 7823
Telemetry [candidate] (7.819 ms) : 0, 7819
Profiling [baseline] (118.867 ms) : 0, 118867
Profiling [candidate] (119.738 ms) : 0, 119738
Loading

Load

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~4818319bb7 1.20.0-SNAPSHOT~b6206d14c9
config baseline candidate
end_time 2023-08-30T07:41:25 2023-08-30T08:06:14
start_time 2023-08-30T07:41:03 2023-08-30T08:05:53
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~b6206d14c9, baseline=1.20.0-SNAPSHOT~4818319bb7
    dateFormat X
    axisFormat %s
section baseline
no_agent (440.543 µs) : 411, 470
.   : milestone, 441,
appsec (810.658 µs) : 779, 843
.   : milestone, 811,
iast (547.198 µs) : 516, 578
.   : milestone, 547,
iast_FULL (654.518 µs) : 624, 685
.   : milestone, 655,
iast_INACTIVE (524.334 µs) : 493, 556
.   : milestone, 524,
profiling (541.509 µs) : 510, 573
.   : milestone, 542,
tracing (528.3 µs) : 497, 560
.   : milestone, 528,
section candidate
no_agent (446.314 µs) : 417, 476
.   : milestone, 446,
appsec (821.193 µs) : 790, 852
.   : milestone, 821,
iast (553.469 µs) : 522, 585
.   : milestone, 553,
iast_FULL (637.972 µs) : 607, 668
.   : milestone, 638,
iast_INACTIVE (523.861 µs) : 494, 554
.   : milestone, 524,
profiling (552.844 µs) : 522, 584
.   : milestone, 553,
tracing (519.789 µs) : 490, 549
.   : milestone, 520,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 440.543 µs [411.295 µs, 469.792 µs] -
appsec 810.658 µs [778.744 µs, 842.573 µs] 370.115 µs (84.0%)
iast 547.198 µs [516.153 µs, 578.242 µs] 106.654 µs (24.2%)
iast_FULL 654.518 µs [624.245 µs, 684.792 µs] 213.975 µs (48.6%)
iast_INACTIVE 524.334 µs [492.946 µs, 555.721 µs] 83.79 µs (19.0%)
profiling 541.509 µs [510.323 µs, 572.695 µs] 100.965 µs (22.9%)
tracing 528.3 µs [496.943 µs, 559.656 µs] 87.756 µs (19.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 446.314 µs [417.02 µs, 475.609 µs] -
appsec 821.193 µs [789.997 µs, 852.389 µs] 374.878 µs (84.0%)
iast 553.469 µs [521.77 µs, 585.167 µs] 107.154 µs (24.0%)
iast_FULL 637.972 µs [607.496 µs, 668.448 µs] 191.658 µs (42.9%)
iast_INACTIVE 523.861 µs [493.968 µs, 553.755 µs] 77.547 µs (17.4%)
profiling 552.844 µs [521.536 µs, 584.151 µs] 106.529 µs (23.9%)
tracing 519.789 µs [490.182 µs, 549.397 µs] 73.475 µs (16.5%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~b6206d14c9, baseline=1.20.0-SNAPSHOT~4818319bb7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.71 ms) : 1686, 1734
.   : milestone, 1710,
appsec (2.099 ms) : 2067, 2132
.   : milestone, 2099,
iast (1.814 ms) : 1781, 1847
.   : milestone, 1814,
profiling (1.842 ms) : 1804, 1879
.   : milestone, 1842,
tracing (1.838 ms) : 1805, 1870
.   : milestone, 1838,
section candidate
no_agent (1.688 ms) : 1664, 1712
.   : milestone, 1688,
appsec (2.098 ms) : 2066, 2131
.   : milestone, 2098,
iast (1.83 ms) : 1798, 1863
.   : milestone, 1830,
profiling (1.819 ms) : 1785, 1852
.   : milestone, 1819,
tracing (1.814 ms) : 1782, 1846
.   : milestone, 1814,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.71 ms [1.686 ms, 1.734 ms] -
appsec 2.099 ms [2.067 ms, 2.132 ms] 389.405 µs (22.8%)
iast 1.814 ms [1.781 ms, 1.847 ms] 104.381 µs (6.1%)
profiling 1.842 ms [1.804 ms, 1.879 ms] 131.506 µs (7.7%)
tracing 1.838 ms [1.805 ms, 1.87 ms] 127.797 µs (7.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.688 ms [1.664 ms, 1.712 ms] -
appsec 2.098 ms [2.066 ms, 2.131 ms] 410.427 µs (24.3%)
iast 1.83 ms [1.798 ms, 1.863 ms] 142.071 µs (8.4%)
profiling 1.819 ms [1.785 ms, 1.852 ms] 130.548 µs (7.7%)
tracing 1.814 ms [1.782 ms, 1.846 ms] 125.785 µs (7.5%)

@jpbempel jpbempel added the comp: debugger Dynamic Instrumentation label Aug 29, 2023
@jpbempel
Reports instrumentation failure
17ff482 
Send probe status error for all probes applied on the same class
indicating instrumentation could not be applied for this class.
@jpbempel
jpbempel force-pushed the jpbempel/report-instrumentation-fails branch from e216769 to 17ff482 Compare August 29, 2023 16:00
shatzi
shatzi approved these changes Aug 30, 2023
View reviewed changes

@shatzi shatzi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks awesome.

Comment thread dd-java-agent/agent-debugger/src/main/java/com/datadog/debugger/agent/DebuggerTransformer.java Outdated
// on a separate class files because probe was set on an inner/top-level class
return;
}
reportError(definitions, format, className, location);

@shatzi shatzi Aug 30, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can be simplifed:

private void reportLocationNotFound(List<ProbeDefinition> definitions, String className, String methodName) {
    if (methodName != null) {
      reportError(definitions, CANNOT_FIND_METHOD, className, methodName);
    } else {
      // This is a line probe, so we don't report line not found because the line may be found later
      // on a separate class files because probe was set on an inner/top-level class
      return;
    }
}

@jpbempel
jpbempel merged commit ac2c282 into master Aug 30, 2023
@jpbempel
jpbempel deleted the jpbempel/report-instrumentation-fails branch August 30, 2023 08:18
@github-actions github-actions Bot added this to the 1.20.0 milestone Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@shatzi shatzi shatzi approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

comp: debugger Dynamic Instrumentation

Projects

None yet

Milestone

1.20.0

Development

Successfully merging this pull request may close these issues.

2 participants

@jpbempel @shatzi