Skip to content

Capture app, job and databricks parameters in spark streaming spans#5796

Merged
paul-laffon-dd merged 2 commits into
masterfrom
paul.laffon/spark-streaming-config
Aug 30, 2023
Merged

Capture app, job and databricks parameters in spark streaming spans#5796
paul-laffon-dd merged 2 commits into
masterfrom
paul.laffon/spark-streaming-config

Conversation

@paul-laffon-dd

@paul-laffon-dd paul-laffon-dd commented Aug 28, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

What Does This Do

Capture spark application, job and databricks parameters in spark streaming spans

Those parameters are currently missing on streaming spans, preventing to link those back to databricks tasks

@paul-laffon-dd
paul-laffon-dd force-pushed the paul.laffon/spark-streaming-config branch from ae9d22c to af6172c Compare August 29, 2023 08:40
@paul-laffon-dd
paul-laffon-dd force-pushed the paul.laffon/spark-streaming-config branch from af6172c to 9f7c88f Compare August 29, 2023 09:23
@pr-commenter

pr-commenter Bot commented Aug 29, 2023 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~4818319bb7 1.20.0-SNAPSHOT~d1da713c06
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.20.0-SNAPSHOT~d1da713c06, baseline=1.20.0-SNAPSHOT~4818319bb7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (985.351 ms) : 0, 985351
Total [baseline] (8.62 s) : 0, 8620167
Agent [candidate] (986.526 ms) : 0, 986526
Total [candidate] (8.629 s) : 0, 8629173
section appsec
Agent [baseline] (1.071 s) : 0, 1070654
Total [baseline] (8.714 s) : 0, 8713647
Agent [candidate] (1.074 s) : 0, 1073678
Total [candidate] (8.749 s) : 0, 8749205
section iast
Agent [baseline] (1.097 s) : 0, 1097364
Total [baseline] (9.163 s) : 0, 9163370
Agent [candidate] (1.094 s) : 0, 1093736
Total [candidate] (9.217 s) : 0, 9217197
section profiling
Agent [baseline] (1.156 s) : 0, 1155624
Total [baseline] (8.831 s) : 0, 8830545
Agent [candidate] (1.168 s) : 0, 1168242
Total [candidate] (8.887 s) : 0, 8886872
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 985.351 ms -
Agent appsec 1.071 s 85.303 ms (8.7%)
Agent iast 1.097 s 112.013 ms (11.4%)
Agent profiling 1.156 s 170.273 ms (17.3%)
Total tracing 8.62 s -
Total appsec 8.714 s 93.48 ms (1.1%)
Total iast 9.163 s 543.204 ms (6.3%)
Total profiling 8.831 s 210.379 ms (2.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 986.526 ms -
Agent appsec 1.074 s 87.151 ms (8.8%)
Agent iast 1.094 s 107.209 ms (10.9%)
Agent profiling 1.168 s 181.716 ms (18.4%)
Total tracing 8.629 s -
Total appsec 8.749 s 120.032 ms (1.4%)
Total iast 9.217 s 588.024 ms (6.8%)
Total profiling 8.887 s 257.699 ms (3.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.20.0-SNAPSHOT~d1da713c06, baseline=1.20.0-SNAPSHOT~4818319bb7

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (611.377 ms) : 0, 611377
BytebuddyAgent [candidate] (611.971 ms) : 0, 611971
GlobalTracer [baseline] (283.128 ms) : 0, 283128
GlobalTracer [candidate] (283.443 ms) : 0, 283443
AppSec [baseline] (49.941 ms) : 0, 49941
AppSec [candidate] (50.21 ms) : 0, 50210
Remote Config [baseline] (663.41 µs) : 0, 663
Remote Config [candidate] (687.581 µs) : 0, 688
Telemetry [baseline] (5.966 ms) : 0, 5966
Telemetry [candidate] (5.978 ms) : 0, 5978
section appsec
BytebuddyAgent [baseline] (611.743 ms) : 0, 611743
BytebuddyAgent [candidate] (613.486 ms) : 0, 613486
GlobalTracer [baseline] (283.342 ms) : 0, 283342
GlobalTracer [candidate] (284.147 ms) : 0, 284147
AppSec [baseline] (134.907 ms) : 0, 134907
AppSec [candidate] (135.237 ms) : 0, 135237
Remote Config [baseline] (708.977 µs) : 0, 709
Remote Config [candidate] (743.933 µs) : 0, 744
Telemetry [baseline] (5.738 ms) : 0, 5738
Telemetry [candidate] (5.758 ms) : 0, 5758
section iast
BytebuddyAgent [baseline] (723.558 ms) : 0, 723558
BytebuddyAgent [candidate] (720.128 ms) : 0, 720128
GlobalTracer [baseline] (270.893 ms) : 0, 270893
GlobalTracer [candidate] (271.066 ms) : 0, 271066
AppSec [baseline] (47.209 ms) : 0, 47209
AppSec [candidate] (47.742 ms) : 0, 47742
Remote Config [baseline] (586.294 µs) : 0, 586
Remote Config [candidate] (552.952 µs) : 0, 553
Telemetry [baseline] (5.829 ms) : 0, 5829
Telemetry [candidate] (5.782 ms) : 0, 5782
IAST [baseline] (14.659 ms) : 0, 14659
IAST [candidate] (14.155 ms) : 0, 14155
section profiling
BytebuddyAgent [baseline] (624.373 ms) : 0, 624373
BytebuddyAgent [candidate] (630.59 ms) : 0, 630590
GlobalTracer [baseline] (346.081 ms) : 0, 346081
GlobalTracer [candidate] (350.503 ms) : 0, 350503
AppSec [baseline] (48.897 ms) : 0, 48897
AppSec [candidate] (49.367 ms) : 0, 49367
Remote Config [baseline] (639.482 µs) : 0, 639
Remote Config [candidate] (650.732 µs) : 0, 651
Telemetry [baseline] (5.918 ms) : 0, 5918
Telemetry [candidate] (6.008 ms) : 0, 6008
ProfilingAgent [baseline] (76.973 ms) : 0, 76973
ProfilingAgent [candidate] (78.07 ms) : 0, 78070
Profiling [baseline] (76.998 ms) : 0, 76998
Profiling [candidate] (78.095 ms) : 0, 78095
Loading
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.20.0-SNAPSHOT~d1da713c06, baseline=1.20.0-SNAPSHOT~4818319bb7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (987.808 ms) : 0, 987808
Total [baseline] (9.165 s) : 0, 9164566
Agent [candidate] (995.927 ms) : 0, 995927
Total [candidate] (9.216 s) : 0, 9215914
section appsec
Agent [baseline] (1.077 s) : 0, 1076530
Total [baseline] (9.273 s) : 0, 9273437
Agent [candidate] (1.08 s) : 0, 1080382
Total [candidate] (9.29 s) : 0, 9289914
section iast
Agent [baseline] (1.092 s) : 0, 1091748
Total [baseline] (9.429 s) : 0, 9429152
Agent [candidate] (1.105 s) : 0, 1104648
Total [candidate] (9.529 s) : 0, 9528972
section profiling
Agent [baseline] (1.158 s) : 0, 1158087
Total [baseline] (9.441 s) : 0, 9441480
Agent [candidate] (1.167 s) : 0, 1167497
Total [candidate] (9.581 s) : 0, 9581040
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 987.808 ms -
Agent appsec 1.077 s 88.721 ms (9.0%)
Agent iast 1.092 s 103.94 ms (10.5%)
Agent profiling 1.158 s 170.279 ms (17.2%)
Total tracing 9.165 s -
Total appsec 9.273 s 108.871 ms (1.2%)
Total iast 9.429 s 264.586 ms (2.9%)
Total profiling 9.441 s 276.914 ms (3.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 995.927 ms -
Agent appsec 1.08 s 84.455 ms (8.5%)
Agent iast 1.105 s 108.721 ms (10.9%)
Agent profiling 1.167 s 171.57 ms (17.2%)
Total tracing 9.216 s -
Total appsec 9.29 s 74.0 ms (0.8%)
Total iast 9.529 s 313.058 ms (3.4%)
Total profiling 9.581 s 365.127 ms (4.0%) 
gantt
    title petclinic - break down per module: candidate=1.20.0-SNAPSHOT~d1da713c06, baseline=1.20.0-SNAPSHOT~4818319bb7

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (612.818 ms) : 0, 612818
BytebuddyAgent [candidate] (617.202 ms) : 0, 617202
GlobalTracer [baseline] (283.824 ms) : 0, 283824
GlobalTracer [candidate] (287.449 ms) : 0, 287449
AppSec [baseline] (50.176 ms) : 0, 50176
AppSec [candidate] (50.045 ms) : 0, 50045
Remote Config [baseline] (667.748 µs) : 0, 668
Remote Config [candidate] (666.043 µs) : 0, 666
Telemetry [baseline] (6.074 ms) : 0, 6074
Telemetry [candidate] (6.027 ms) : 0, 6027
section appsec
BytebuddyAgent [baseline] (616.311 ms) : 0, 616311
BytebuddyAgent [candidate] (617.567 ms) : 0, 617567
GlobalTracer [baseline] (284.187 ms) : 0, 284187
GlobalTracer [candidate] (286.445 ms) : 0, 286445
AppSec [baseline] (134.992 ms) : 0, 134992
AppSec [candidate] (135.332 ms) : 0, 135332
Remote Config [baseline] (735.588 µs) : 0, 736
Remote Config [candidate] (736.363 µs) : 0, 736
Telemetry [baseline] (5.731 ms) : 0, 5731
Telemetry [candidate] (5.79 ms) : 0, 5790
section iast
BytebuddyAgent [baseline] (719.27 ms) : 0, 719270
BytebuddyAgent [candidate] (727.162 ms) : 0, 727162
GlobalTracer [baseline] (270.405 ms) : 0, 270405
GlobalTracer [candidate] (274.237 ms) : 0, 274237
AppSec [baseline] (46.822 ms) : 0, 46822
AppSec [candidate] (48.126 ms) : 0, 48126
Remote Config [baseline] (583.754 µs) : 0, 584
Remote Config [candidate] (556.618 µs) : 0, 557
Telemetry [baseline] (5.816 ms) : 0, 5816
Telemetry [candidate] (5.765 ms) : 0, 5765
IAST [baseline] (14.508 ms) : 0, 14508
IAST [candidate] (14.255 ms) : 0, 14255
section profiling
BytebuddyAgent [baseline] (625.035 ms) : 0, 625035
BytebuddyAgent [candidate] (630.711 ms) : 0, 630711
GlobalTracer [baseline] (347.007 ms) : 0, 347007
GlobalTracer [candidate] (350.104 ms) : 0, 350104
AppSec [baseline] (49.296 ms) : 0, 49296
AppSec [candidate] (49.321 ms) : 0, 49321
Remote Config [baseline] (649.408 µs) : 0, 649
Remote Config [candidate] (654.125 µs) : 0, 654
Telemetry [baseline] (6.0 ms) : 0, 6000
Telemetry [candidate] (6.049 ms) : 0, 6049
ProfilingAgent [baseline] (77.319 ms) : 0, 77319
ProfilingAgent [candidate] (77.451 ms) : 0, 77451
Profiling [baseline] (77.343 ms) : 0, 77343
Profiling [candidate] (77.475 ms) : 0, 77475
Loading

Load

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~4818319bb7 1.20.0-SNAPSHOT~d1da713c06
config baseline candidate
end_time 2023-08-30T08:06:59 2023-08-30T08:25:06
start_time 2023-08-30T08:06:42 2023-08-30T08:24:48
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~d1da713c06, baseline=1.20.0-SNAPSHOT~4818319bb7
    dateFormat X
    axisFormat %s
section baseline
no_agent (364.539 µs) : 344, 385
.   : milestone, 365,
appsec (663.246 µs) : 642, 685
.   : milestone, 663,
iast (456.511 µs) : 435, 478
.   : milestone, 457,
iast_FULL (514.517 µs) : 494, 535
.   : milestone, 515,
iast_INACTIVE (431.161 µs) : 410, 452
.   : milestone, 431,
profiling (431.192 µs) : 410, 452
.   : milestone, 431,
tracing (424.708 µs) : 404, 445
.   : milestone, 425,
section candidate
no_agent (359.487 µs) : 340, 379
.   : milestone, 359,
appsec (659.851 µs) : 639, 681
.   : milestone, 660,
iast (453.541 µs) : 433, 474
.   : milestone, 454,
iast_FULL (519.031 µs) : 499, 539
.   : milestone, 519,
iast_INACTIVE (429.163 µs) : 408, 451
.   : milestone, 429,
profiling (446.107 µs) : 425, 467
.   : milestone, 446,
tracing (429.639 µs) : 409, 450
.   : milestone, 430,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.539 µs [344.324 µs, 384.755 µs] -
appsec 663.246 µs [641.699 µs, 684.794 µs] 298.707 µs (81.9%)
iast 456.511 µs [435.466 µs, 477.555 µs] 91.971 µs (25.2%)
iast_FULL 514.517 µs [493.856 µs, 535.179 µs] 149.978 µs (41.1%)
iast_INACTIVE 431.161 µs [410.471 µs, 451.851 µs] 66.622 µs (18.3%)
profiling 431.192 µs [410.258 µs, 452.126 µs] 66.653 µs (18.3%)
tracing 424.708 µs [403.995 µs, 445.422 µs] 60.169 µs (16.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 359.487 µs [339.751 µs, 379.223 µs] -
appsec 659.851 µs [638.876 µs, 680.827 µs] 300.364 µs (83.6%)
iast 453.541 µs [432.645 µs, 474.437 µs] 94.054 µs (26.2%)
iast_FULL 519.031 µs [498.596 µs, 539.465 µs] 159.544 µs (44.4%)
iast_INACTIVE 429.163 µs [407.754 µs, 450.572 µs] 69.676 µs (19.4%)
profiling 446.107 µs [424.872 µs, 467.343 µs] 86.62 µs (24.1%)
tracing 429.639 µs [408.916 µs, 450.362 µs] 70.152 µs (19.5%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~d1da713c06, baseline=1.20.0-SNAPSHOT~4818319bb7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.336 ms) : 1317, 1356
.   : milestone, 1336,
appsec (1.687 ms) : 1663, 1712
.   : milestone, 1687,
iast (1.442 ms) : 1417, 1466
.   : milestone, 1442,
profiling (1.461 ms) : 1436, 1485
.   : milestone, 1461,
tracing (1.425 ms) : 1400, 1449
.   : milestone, 1425,
section candidate
no_agent (1.342 ms) : 1323, 1361
.   : milestone, 1342,
appsec (1.671 ms) : 1646, 1695
.   : milestone, 1671,
iast (1.442 ms) : 1418, 1466
.   : milestone, 1442,
profiling (1.473 ms) : 1446, 1499
.   : milestone, 1473,
tracing (1.45 ms) : 1426, 1474
.   : milestone, 1450,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.336 ms [1.317 ms, 1.356 ms] -
appsec 1.687 ms [1.663 ms, 1.712 ms] 350.988 µs (26.3%)
iast 1.442 ms [1.417 ms, 1.466 ms] 105.354 µs (7.9%)
profiling 1.461 ms [1.436 ms, 1.485 ms] 124.312 µs (9.3%)
tracing 1.425 ms [1.4 ms, 1.449 ms] 88.544 µs (6.6%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.342 ms [1.323 ms, 1.361 ms] -
appsec 1.671 ms [1.646 ms, 1.695 ms] 328.395 µs (24.5%)
iast 1.442 ms [1.418 ms, 1.466 ms] 99.954 µs (7.4%)
profiling 1.473 ms [1.446 ms, 1.499 ms] 130.524 µs (9.7%)
tracing 1.45 ms [1.426 ms, 1.474 ms] 108.189 µs (8.1%)

@paul-laffon-dd paul-laffon-dd added the inst: apache spark Apache Spark instrumentation label Aug 29, 2023
@paul-laffon-dd
paul-laffon-dd marked this pull request as ready for review August 29, 2023 11:37
@paul-laffon-dd
paul-laffon-dd requested a review from a team as a code owner August 29, 2023 11:37
raphaelgavache
raphaelgavache approved these changes Aug 29, 2023
View reviewed changes

@raphaelgavache raphaelgavache left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

PerfectSlayer
PerfectSlayer approved these changes Aug 30, 2023
View reviewed changes

@PerfectSlayer PerfectSlayer left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Just minors changes related to the Property API.

@paul-laffon-dd
paul-laffon-dd merged commit 3bd19ba into master Aug 30, 2023
@paul-laffon-dd
paul-laffon-dd deleted the paul.laffon/spark-streaming-config branch August 30, 2023 09:03
@github-actions github-actions Bot added this to the 1.20.0 milestone Aug 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@PerfectSlayer PerfectSlayer PerfectSlayer approved these changes
@raphaelgavache raphaelgavache raphaelgavache approved these changes
@ygree ygree Awaiting requested review from ygree ygree was automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

inst: apache spark Apache Spark instrumentation

Projects

None yet

Milestone

1.20.0

Development

Successfully merging this pull request may close these issues.

3 participants

@paul-laffon-dd @PerfectSlayer @raphaelgavache