Skip to content

Get databricks cluster name from spark conf, if absent in job properties#5775

Merged
paul-laffon-dd merged 2 commits into
masterfrom
paul.laffon/databricks-cluster-name
Aug 28, 2023
Merged

Get databricks cluster name from spark conf, if absent in job properties#5775
paul-laffon-dd merged 2 commits into
masterfrom
paul.laffon/databricks-cluster-name

Conversation

@paul-laffon-dd

@paul-laffon-dd paul-laffon-dd commented Aug 24, 2023 •
edited by am312
Loading

Copy link
Copy Markdown
Contributor

What Does This Do

Rely on the spark.databricks.clusterUsageTags.clusterName present in the SparkConf instead of the one present in the jobProperties

After some trial and errors, it turned out that the value in jobProperties can sometimes be missing, while it is always present in the SparkConf

@pr-commenter

pr-commenter Bot commented Aug 24, 2023 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Startup

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~72572d1d17 1.20.0-SNAPSHOT~77c9c5af5a
config baseline candidate
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
module Agent Agent
parent None None
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 62 cases.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.20.0-SNAPSHOT~77c9c5af5a, baseline=1.20.0-SNAPSHOT~72572d1d17

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (992.574 ms) : 0, 992574
Total [baseline] (9.151 s) : 0, 9150564
Agent [candidate] (994.72 ms) : 0, 994720
Total [candidate] (9.215 s) : 0, 9214613
section appsec
Agent [baseline] (1.079 s) : 0, 1078575
Total [baseline] (9.214 s) : 0, 9213836
Agent [candidate] (1.07 s) : 0, 1070404
Total [candidate] (9.244 s) : 0, 9243914
section iast
Agent [baseline] (1.096 s) : 0, 1096277
Total [baseline] (9.478 s) : 0, 9477756
Agent [candidate] (1.099 s) : 0, 1099189
Total [candidate] (9.44 s) : 0, 9439837
section profiling
Agent [baseline] (1.158 s) : 0, 1157827
Total [baseline] (9.459 s) : 0, 9458720
Agent [candidate] (1.155 s) : 0, 1155135
Total [candidate] (9.401 s) : 0, 9400676
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 992.574 ms -
Agent appsec 1.079 s 86.001 ms (8.7%)
Agent iast 1.096 s 103.703 ms (10.4%)
Agent profiling 1.158 s 165.253 ms (16.6%)
Total tracing 9.151 s -
Total appsec 9.214 s 63.272 ms (0.7%)
Total iast 9.478 s 327.192 ms (3.6%)
Total profiling 9.459 s 308.156 ms (3.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 994.72 ms -
Agent appsec 1.07 s 75.684 ms (7.6%)
Agent iast 1.099 s 104.469 ms (10.5%)
Agent profiling 1.155 s 160.415 ms (16.1%)
Total tracing 9.215 s -
Total appsec 9.244 s 29.3 ms (0.3%)
Total iast 9.44 s 225.223 ms (2.4%)
Total profiling 9.401 s 186.063 ms (2.0%) 
gantt
    title petclinic - break down per module: candidate=1.20.0-SNAPSHOT~77c9c5af5a, baseline=1.20.0-SNAPSHOT~72572d1d17

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (615.667 ms) : 0, 615667
BytebuddyAgent [candidate] (616.6 ms) : 0, 616600
GlobalTracer [baseline] (285.257 ms) : 0, 285257
GlobalTracer [candidate] (286.788 ms) : 0, 286788
AppSec [baseline] (50.462 ms) : 0, 50462
AppSec [candidate] (50.106 ms) : 0, 50106
Remote Config [baseline] (661.922 µs) : 0, 662
Remote Config [candidate] (670.516 µs) : 0, 671
Telemetry [baseline] (6.008 ms) : 0, 6008
Telemetry [candidate] (6.058 ms) : 0, 6058
section appsec
BytebuddyAgent [baseline] (616.268 ms) : 0, 616268
BytebuddyAgent [candidate] (610.826 ms) : 0, 610826
GlobalTracer [baseline] (286.179 ms) : 0, 286179
GlobalTracer [candidate] (283.77 ms) : 0, 283770
AppSec [baseline] (135.061 ms) : 0, 135061
AppSec [candidate] (135.097 ms) : 0, 135097
Remote Config [baseline] (729.424 µs) : 0, 729
Remote Config [candidate] (714.95 µs) : 0, 715
Telemetry [baseline] (5.789 ms) : 0, 5789
Telemetry [candidate] (5.76 ms) : 0, 5760
section iast
BytebuddyAgent [baseline] (722.67 ms) : 0, 722670
BytebuddyAgent [candidate] (724.211 ms) : 0, 724211
GlobalTracer [baseline] (271.332 ms) : 0, 271332
GlobalTracer [candidate] (272.607 ms) : 0, 272607
AppSec [baseline] (47.384 ms) : 0, 47384
AppSec [candidate] (47.47 ms) : 0, 47470
Remote Config [baseline] (550.079 µs) : 0, 550
Remote Config [candidate] (560.621 µs) : 0, 561
Telemetry [baseline] (5.715 ms) : 0, 5715
Telemetry [candidate] (5.775 ms) : 0, 5775
IAST [baseline] (14.042 ms) : 0, 14042
IAST [candidate] (14.048 ms) : 0, 14048
section profiling
ProfilingAgent [baseline] (76.472 ms) : 0, 76472
ProfilingAgent [candidate] (76.4 ms) : 0, 76400
BytebuddyAgent [baseline] (625.242 ms) : 0, 625242
BytebuddyAgent [candidate] (623.954 ms) : 0, 623954
GlobalTracer [baseline] (347.507 ms) : 0, 347507
GlobalTracer [candidate] (346.203 ms) : 0, 346203
AppSec [baseline] (49.134 ms) : 0, 49134
AppSec [candidate] (49.271 ms) : 0, 49271
Remote Config [baseline] (646.142 µs) : 0, 646
Remote Config [candidate] (644.871 µs) : 0, 645
Telemetry [baseline] (5.974 ms) : 0, 5974
Telemetry [candidate] (5.951 ms) : 0, 5951
Profiling [baseline] (76.496 ms) : 0, 76496
Profiling [candidate] (76.424 ms) : 0, 76424
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.20.0-SNAPSHOT~77c9c5af5a, baseline=1.20.0-SNAPSHOT~72572d1d17

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (985.598 ms) : 0, 985598
Total [baseline] (8.627 s) : 0, 8627019
Agent [candidate] (995.667 ms) : 0, 995667
Total [candidate] (8.655 s) : 0, 8654804
section appsec
Agent [baseline] (1.071 s) : 0, 1070899
Total [baseline] (8.708 s) : 0, 8708085
Agent [candidate] (1.074 s) : 0, 1074211
Total [candidate] (8.728 s) : 0, 8728193
section iast
Agent [baseline] (1.088 s) : 0, 1087792
Total [baseline] (9.175 s) : 0, 9175130
Agent [candidate] (1.091 s) : 0, 1091359
Total [candidate] (9.142 s) : 0, 9141560
section profiling
Agent [baseline] (1.162 s) : 0, 1162021
Total [baseline] (8.884 s) : 0, 8883672
Agent [candidate] (1.16 s) : 0, 1160058
Total [candidate] (8.87 s) : 0, 8870127
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 985.598 ms -
Agent appsec 1.071 s 85.301 ms (8.7%)
Agent iast 1.088 s 102.194 ms (10.4%)
Agent profiling 1.162 s 176.423 ms (17.9%)
Total tracing 8.627 s -
Total appsec 8.708 s 81.066 ms (0.9%)
Total iast 9.175 s 548.111 ms (6.4%)
Total profiling 8.884 s 256.653 ms (3.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 995.667 ms -
Agent appsec 1.074 s 78.544 ms (7.9%)
Agent iast 1.091 s 95.693 ms (9.6%)
Agent profiling 1.16 s 164.391 ms (16.5%)
Total tracing 8.655 s -
Total appsec 8.728 s 73.389 ms (0.8%)
Total iast 9.142 s 486.756 ms (5.6%)
Total profiling 8.87 s 215.323 ms (2.5%) 
gantt
    title insecure-bank - break down per module: candidate=1.20.0-SNAPSHOT~77c9c5af5a, baseline=1.20.0-SNAPSHOT~72572d1d17

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (610.977 ms) : 0, 610977
BytebuddyAgent [candidate] (617.495 ms) : 0, 617495
GlobalTracer [baseline] (283.589 ms) : 0, 283589
GlobalTracer [candidate] (286.945 ms) : 0, 286945
AppSec [baseline] (50.168 ms) : 0, 50168
AppSec [candidate] (49.946 ms) : 0, 49946
Remote Config [baseline] (656.468 µs) : 0, 656
Remote Config [candidate] (666.405 µs) : 0, 666
Telemetry [baseline] (5.937 ms) : 0, 5937
Telemetry [candidate] (6.054 ms) : 0, 6054
section appsec
BytebuddyAgent [baseline] (611.937 ms) : 0, 611937
BytebuddyAgent [candidate] (613.37 ms) : 0, 613370
GlobalTracer [baseline] (283.188 ms) : 0, 283188
GlobalTracer [candidate] (285.086 ms) : 0, 285086
AppSec [baseline] (134.901 ms) : 0, 134901
AppSec [candidate] (134.913 ms) : 0, 134913
Remote Config [baseline] (722.403 µs) : 0, 722
Remote Config [candidate] (728.489 µs) : 0, 728
Telemetry [baseline] (5.801 ms) : 0, 5801
Telemetry [candidate] (5.741 ms) : 0, 5741
section iast
BytebuddyAgent [baseline] (716.408 ms) : 0, 716408
BytebuddyAgent [candidate] (718.127 ms) : 0, 718127
GlobalTracer [baseline] (269.427 ms) : 0, 269427
GlobalTracer [candidate] (271.042 ms) : 0, 271042
AppSec [baseline] (47.431 ms) : 0, 47431
AppSec [candidate] (47.524 ms) : 0, 47524
Remote Config [baseline] (555.656 µs) : 0, 556
Remote Config [candidate] (565.096 µs) : 0, 565
Telemetry [baseline] (5.695 ms) : 0, 5695
Telemetry [candidate] (5.766 ms) : 0, 5766
IAST [baseline] (14.002 ms) : 0, 14002
IAST [candidate] (13.983 ms) : 0, 13983
section profiling
ProfilingAgent [baseline] (75.838 ms) : 0, 75838
ProfilingAgent [candidate] (77.801 ms) : 0, 77801
BytebuddyAgent [baseline] (628.394 ms) : 0, 628394
BytebuddyAgent [candidate] (626.333 ms) : 0, 626333
GlobalTracer [baseline] (348.841 ms) : 0, 348841
GlobalTracer [candidate] (347.189 ms) : 0, 347189
AppSec [baseline] (49.242 ms) : 0, 49242
AppSec [candidate] (49.129 ms) : 0, 49129
Remote Config [baseline] (647.951 µs) : 0, 648
Remote Config [candidate] (656.522 µs) : 0, 657
Telemetry [baseline] (6.007 ms) : 0, 6007
Telemetry [candidate] (5.977 ms) : 0, 5977
Profiling [baseline] (75.862 ms) : 0, 75862
Profiling [candidate] (77.825 ms) : 0, 77825
Loading

Load

Parameters

Baseline Candidate
commit 1.20.0-SNAPSHOT~72572d1d17 1.20.0-SNAPSHOT~77c9c5af5a
config baseline candidate
end_time 2023-08-28T09:14:30 2023-08-28T09:32:31
start_time 2023-08-28T09:14:13 2023-08-28T09:32:13
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 24 cases.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~77c9c5af5a, baseline=1.20.0-SNAPSHOT~72572d1d17
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.331 ms) : 1312, 1350
.   : milestone, 1331,
appsec (1.642 ms) : 1618, 1667
.   : milestone, 1642,
iast (1.45 ms) : 1425, 1475
.   : milestone, 1450,
profiling (1.465 ms) : 1440, 1490
.   : milestone, 1465,
tracing (1.438 ms) : 1414, 1462
.   : milestone, 1438,
section candidate
no_agent (1.315 ms) : 1296, 1334
.   : milestone, 1315,
appsec (1.654 ms) : 1630, 1678
.   : milestone, 1654,
iast (1.436 ms) : 1411, 1460
.   : milestone, 1436,
profiling (1.484 ms) : 1458, 1510
.   : milestone, 1484,
tracing (1.454 ms) : 1430, 1478
.   : milestone, 1454,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.331 ms [1.312 ms, 1.35 ms] -
appsec 1.642 ms [1.618 ms, 1.667 ms] 310.87 µs (23.3%)
iast 1.45 ms [1.425 ms, 1.475 ms] 118.566 µs (8.9%)
profiling 1.465 ms [1.44 ms, 1.49 ms] 133.7 µs (10.0%)
tracing 1.438 ms [1.414 ms, 1.462 ms] 106.232 µs (8.0%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.315 ms [1.296 ms, 1.334 ms] -
appsec 1.654 ms [1.63 ms, 1.678 ms] 339.492 µs (25.8%)
iast 1.436 ms [1.411 ms, 1.46 ms] 120.788 µs (9.2%)
profiling 1.484 ms [1.458 ms, 1.51 ms] 169.419 µs (12.9%)
tracing 1.454 ms [1.43 ms, 1.478 ms] 139.367 µs (10.6%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.20.0-SNAPSHOT~77c9c5af5a, baseline=1.20.0-SNAPSHOT~72572d1d17
    dateFormat X
    axisFormat %s
section baseline
no_agent (360.024 µs) : 340, 380
.   : milestone, 360,
appsec (649.545 µs) : 629, 670
.   : milestone, 650,
iast (446.09 µs) : 426, 467
.   : milestone, 446,
iast_FULL (514.471 µs) : 494, 535
.   : milestone, 514,
iast_INACTIVE (431.319 µs) : 410, 452
.   : milestone, 431,
profiling (431.778 µs) : 411, 453
.   : milestone, 432,
tracing (429.231 µs) : 409, 450
.   : milestone, 429,
section candidate
no_agent (358.859 µs) : 339, 379
.   : milestone, 359,
appsec (649.033 µs) : 629, 669
.   : milestone, 649,
iast (449.652 µs) : 429, 470
.   : milestone, 450,
iast_FULL (510.114 µs) : 490, 531
.   : milestone, 510,
iast_INACTIVE (431.335 µs) : 410, 453
.   : milestone, 431,
profiling (430.736 µs) : 410, 452
.   : milestone, 431,
tracing (429.535 µs) : 409, 451
.   : milestone, 430,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 360.024 µs [339.963 µs, 380.085 µs] -
appsec 649.545 µs [629.067 µs, 670.023 µs] 289.521 µs (80.4%)
iast 446.09 µs [425.514 µs, 466.666 µs] 86.066 µs (23.9%)
iast_FULL 514.471 µs [493.994 µs, 534.948 µs] 154.447 µs (42.9%)
iast_INACTIVE 431.319 µs [410.167 µs, 452.471 µs] 71.295 µs (19.8%)
profiling 431.778 µs [410.702 µs, 452.854 µs] 71.754 µs (19.9%)
tracing 429.231 µs [408.519 µs, 449.944 µs] 69.207 µs (19.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 358.859 µs [338.87 µs, 378.847 µs] -
appsec 649.033 µs [628.837 µs, 669.229 µs] 290.174 µs (80.9%)
iast 449.652 µs [429.154 µs, 470.15 µs] 90.793 µs (25.3%)
iast_FULL 510.114 µs [489.685 µs, 530.543 µs] 151.255 µs (42.1%)
iast_INACTIVE 431.335 µs [410.169 µs, 452.501 µs] 72.476 µs (20.2%)
profiling 430.736 µs [409.629 µs, 451.842 µs] 71.877 µs (20.0%)
tracing 429.535 µs [408.561 µs, 450.508 µs] 70.676 µs (19.7%)

@paul-laffon-dd paul-laffon-dd added the inst: apache spark Apache Spark instrumentation label Aug 24, 2023
@paul-laffon-dd
paul-laffon-dd marked this pull request as ready for review August 24, 2023 12:24
@paul-laffon-dd
paul-laffon-dd requested a review from a team as a code owner August 24, 2023 12:24
@am312

am312 commented Aug 24, 2023

Copy link
Copy Markdown
Contributor

In cases where both values are present, do we know that they're guaranteed to be equal? Should this new clusterName value be a fallback used only when the current value in jobProperties is missing? Just trying not to disrupt any expected behaviors here.

@paul-laffon-dd paul-laffon-dd changed the title Get databricks cluster name from spark conf, instead of job properties Get databricks cluster name from spark conf, if absent in job properties Aug 28, 2023
@paul-laffon-dd

paul-laffon-dd commented Aug 28, 2023

Copy link
Copy Markdown
Contributor Author

Yes, if both values are equals, they will have the same value. We can use the clusterName present in the spark conf as a fallback if the value is not present in the job properties. Changed the PR to reflect it

@paul-laffon-dd
paul-laffon-dd merged commit b603c6e into master Aug 28, 2023
@paul-laffon-dd
paul-laffon-dd deleted the paul.laffon/databricks-cluster-name branch August 28, 2023 16:41
@github-actions github-actions Bot added this to the 1.20.0 milestone Aug 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raphaelgavache raphaelgavache raphaelgavache approved these changes
+1 more reviewer
@am312 am312 am312 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

inst: apache spark Apache Spark instrumentation

Projects

None yet

Milestone

1.20.0

Development

Successfully merging this pull request may close these issues.

3 participants

@paul-laffon-dd @am312 @raphaelgavache