Skip to content

Add field parent_stage_ids to spark.stage spans#6309

Merged
paul-laffon-dd merged 1 commit into
masterfrom
paul.laffon/spark-parent-stage-ids
Dec 12, 2023
Merged

Add field parent_stage_ids to spark.stage spans#6309
paul-laffon-dd merged 1 commit into
masterfrom
paul.laffon/spark-parent-stage-ids

Conversation

@paul-laffon-dd

@paul-laffon-dd paul-laffon-dd commented Dec 4, 2023

Copy link
Copy Markdown
Contributor

What Does This Do

Add field parent_stage_ids to spark.stage spans

Motivation

In a spark job, input of a stage can be the output of a previous stage. This new fields parent_stage_ids will capture the dependency between stage to display it in the UI

Additional Notes

Specific implementation based on the scala version because the signature of scala.Seq changed in scala 2.13

@paul-laffon-dd
Add field parent_stage_ids to spark.stage spans
38c2568 
In a spark job, input of a stage can be the output of a previous stage. This new fields parent_stage_ids will capture the dependency between stage to display it in the UI

Specific implementation based on the scala version because the signature of scala.Seq is different
@pr-commenter

pr-commenter Bot commented Dec 4, 2023

Copy link
Copy Markdown

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master paul.laffon/spark-parent-stage-ids
git_commit_date 1701468764 1701681014
git_commit_sha 7fb8faa 38c2568
release_version 1.25.0-SNAPSHOT~7fb8faa37b 1.25.0-SNAPSHOT~38c2568e9d
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1701683752 1701683752
ci_job_id 379752424 379752424
ci_pipeline_id 24080570 24080570
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 44 metrics, 10 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.25.0-SNAPSHOT~38c2568e9d, baseline=1.25.0-SNAPSHOT~7fb8faa37b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.043 s) : 0, 1042503
Total [baseline] (9.314 s) : 0, 9314366
Agent [candidate] (1.05 s) : 0, 1050117
Total [candidate] (9.292 s) : 0, 9292357
section appsec
Agent [baseline] (1.136 s) : 0, 1136207
Total [baseline] (9.448 s) : 0, 9447932
Agent [candidate] (1.139 s) : 0, 1138562
Total [candidate] (9.406 s) : 0, 9405845
section iast
Agent [baseline] (1.16 s) : 0, 1160000
Total [baseline] (9.586 s) : 0, 9586174
Agent [candidate] (1.162 s) : 0, 1161533
Total [candidate] (9.588 s) : 0, 9588381
section profiling
Agent [baseline] (1.24 s) : 0, 1239742
Total [baseline] (9.615 s) : 0, 9614982
Agent [candidate] (1.255 s) : 0, 1255403
Total [candidate] (9.742 s) : 0, 9742122
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.043 s -
Agent appsec 1.136 s 93.704 ms (9.0%)
Agent iast 1.16 s 117.498 ms (11.3%)
Agent profiling 1.24 s 197.24 ms (18.9%)
Total tracing 9.314 s -
Total appsec 9.448 s 133.566 ms (1.4%)
Total iast 9.586 s 271.808 ms (2.9%)
Total profiling 9.615 s 300.616 ms (3.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.05 s -
Agent appsec 1.139 s 88.445 ms (8.4%)
Agent iast 1.162 s 111.416 ms (10.6%)
Agent profiling 1.255 s 205.287 ms (19.5%)
Total tracing 9.292 s -
Total appsec 9.406 s 113.487 ms (1.2%)
Total iast 9.588 s 296.024 ms (3.2%)
Total profiling 9.742 s 449.764 ms (4.8%) 
gantt
    title petclinic - break down per module: candidate=1.25.0-SNAPSHOT~38c2568e9d, baseline=1.25.0-SNAPSHOT~7fb8faa37b

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (645.021 ms) : 0, 645021
BytebuddyAgent [candidate] (649.794 ms) : 0, 649794
GlobalTracer [baseline] (306.834 ms) : 0, 306834
GlobalTracer [candidate] (309.085 ms) : 0, 309085
AppSec [baseline] (48.554 ms) : 0, 48554
AppSec [candidate] (48.811 ms) : 0, 48811
Remote Config [baseline] (669.826 µs) : 0, 670
Remote Config [candidate] (683.652 µs) : 0, 684
Telemetry [baseline] (7.094 ms) : 0, 7094
Telemetry [candidate] (7.215 ms) : 0, 7215
section appsec
BytebuddyAgent [baseline] (649.026 ms) : 0, 649026
BytebuddyAgent [candidate] (649.455 ms) : 0, 649455
GlobalTracer [baseline] (307.652 ms) : 0, 307652
GlobalTracer [candidate] (307.627 ms) : 0, 307627
AppSec [baseline] (137.027 ms) : 0, 137027
AppSec [candidate] (137.574 ms) : 0, 137574
Remote Config [baseline] (645.617 µs) : 0, 646
Remote Config [candidate] (652.0 µs) : 0, 652
Telemetry [baseline] (7.468 ms) : 0, 7468
Telemetry [candidate] (8.772 ms) : 0, 8772
section iast
BytebuddyAgent [baseline] (765.635 ms) : 0, 765635
BytebuddyAgent [candidate] (767.19 ms) : 0, 767190
GlobalTracer [baseline] (284.577 ms) : 0, 284577
GlobalTracer [candidate] (285.076 ms) : 0, 285076
AppSec [baseline] (46.857 ms) : 0, 46857
AppSec [candidate] (46.641 ms) : 0, 46641
Remote Config [baseline] (618.093 µs) : 0, 618
Remote Config [candidate] (624.808 µs) : 0, 625
Telemetry [baseline] (8.76 ms) : 0, 8760
Telemetry [candidate] (7.372 ms) : 0, 7372
IAST [baseline] (19.256 ms) : 0, 19256
IAST [candidate] (20.241 ms) : 0, 20241
section profiling
BytebuddyAgent [baseline] (660.712 ms) : 0, 660712
BytebuddyAgent [candidate] (669.654 ms) : 0, 669654
GlobalTracer [baseline] (378.75 ms) : 0, 378750
GlobalTracer [candidate] (382.718 ms) : 0, 382718
AppSec [baseline] (48.602 ms) : 0, 48602
AppSec [candidate] (49.53 ms) : 0, 49530
Remote Config [baseline] (698.027 µs) : 0, 698
Remote Config [candidate] (692.098 µs) : 0, 692
Telemetry [baseline] (8.111 ms) : 0, 8111
Telemetry [candidate] (7.533 ms) : 0, 7533
ProfilingAgent [baseline] (88.333 ms) : 0, 88333
ProfilingAgent [candidate] (90.025 ms) : 0, 90025
Profiling [baseline] (88.358 ms) : 0, 88358
Profiling [candidate] (90.05 ms) : 0, 90050
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.25.0-SNAPSHOT~38c2568e9d, baseline=1.25.0-SNAPSHOT~7fb8faa37b

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.042 s) : 0, 1042136
Total [baseline] (8.724 s) : 0, 8723904
Agent [candidate] (1.043 s) : 0, 1042684
Total [candidate] (8.736 s) : 0, 8736241
section iast
Agent [baseline] (1.158 s) : 0, 1158280
Total [baseline] (9.25 s) : 0, 9249675
Agent [candidate] (1.164 s) : 0, 1164438
Total [candidate] (9.263 s) : 0, 9263010
section iast_TELEMETRY_OFF
Agent [baseline] (1.149 s) : 0, 1149113
Total [baseline] (9.241 s) : 0, 9240808
Agent [candidate] (1.16 s) : 0, 1160285
Total [candidate] (9.27 s) : 0, 9269737
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.042 s -
Agent iast 1.158 s 116.144 ms (11.1%)
Agent iast_TELEMETRY_OFF 1.149 s 106.977 ms (10.3%)
Total tracing 8.724 s -
Total iast 9.25 s 525.771 ms (6.0%)
Total iast_TELEMETRY_OFF 9.241 s 516.904 ms (5.9%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.043 s -
Agent iast 1.164 s 121.755 ms (11.7%)
Agent iast_TELEMETRY_OFF 1.16 s 117.601 ms (11.3%)
Total tracing 8.736 s -
Total iast 9.263 s 526.769 ms (6.0%)
Total iast_TELEMETRY_OFF 9.27 s 533.496 ms (6.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.25.0-SNAPSHOT~38c2568e9d, baseline=1.25.0-SNAPSHOT~7fb8faa37b

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (645.723 ms) : 0, 645723
BytebuddyAgent [candidate] (645.441 ms) : 0, 645441
GlobalTracer [baseline] (305.711 ms) : 0, 305711
GlobalTracer [candidate] (307.064 ms) : 0, 307064
AppSec [baseline] (48.502 ms) : 0, 48502
AppSec [candidate] (48.21 ms) : 0, 48210
Remote Config [baseline] (668.571 µs) : 0, 669
Remote Config [candidate] (669.274 µs) : 0, 669
Telemetry [baseline] (7.089 ms) : 0, 7089
Telemetry [candidate] (7.104 ms) : 0, 7104
section iast
BytebuddyAgent [baseline] (765.623 ms) : 0, 765623
BytebuddyAgent [candidate] (770.62 ms) : 0, 770620
GlobalTracer [baseline] (284.454 ms) : 0, 284454
GlobalTracer [candidate] (284.491 ms) : 0, 284491
AppSec [baseline] (46.478 ms) : 0, 46478
AppSec [candidate] (46.404 ms) : 0, 46404
Remote Config [baseline] (602.007 µs) : 0, 602
Remote Config [candidate] (597.406 µs) : 0, 597
Telemetry [baseline] (6.594 ms) : 0, 6594
Telemetry [candidate] (8.086 ms) : 0, 8086
IAST [baseline] (20.162 ms) : 0, 20162
IAST [candidate] (19.618 ms) : 0, 19618
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (757.221 ms) : 0, 757221
BytebuddyAgent [candidate] (764.212 ms) : 0, 764212
GlobalTracer [baseline] (284.769 ms) : 0, 284769
GlobalTracer [candidate] (287.21 ms) : 0, 287210
AppSec [baseline] (48.439 ms) : 0, 48439
AppSec [candidate] (47.4 ms) : 0, 47400
Remote Config [baseline] (584.168 µs) : 0, 584
Remote Config [candidate] (594.167 µs) : 0, 594
Telemetry [baseline] (7.887 ms) : 0, 7887
Telemetry [candidate] (8.703 ms) : 0, 8703
IAST [baseline] (15.952 ms) : 0, 15952
IAST [candidate] (17.628 ms) : 0, 17628
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2023-12-04T09:35:06 2023-12-04T09:51:39
git_branch master paul.laffon/spark-parent-stage-ids
git_commit_date 1701468764 1701681014
git_commit_sha 7fb8faa 38c2568
release_version 1.25.0-SNAPSHOT~7fb8faa37b 1.25.0-SNAPSHOT~38c2568e9d
start_time 2023-12-04T09:34:53 2023-12-04T09:51:26
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1701683752 1701683752
ci_job_id 379752424 379752424
ci_pipeline_id 24080570 24080570
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 14 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~38c2568e9d, baseline=1.25.0-SNAPSHOT~7fb8faa37b
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.357 ms) : 1338, 1376
.   : milestone, 1357,
appsec (1.768 ms) : 1742, 1794
.   : milestone, 1768,
iast (1.498 ms) : 1474, 1523
.   : milestone, 1498,
profiling (1.503 ms) : 1478, 1528
.   : milestone, 1503,
tracing (1.488 ms) : 1463, 1513
.   : milestone, 1488,
section candidate
no_agent (1.375 ms) : 1356, 1395
.   : milestone, 1375,
appsec (1.761 ms) : 1736, 1786
.   : milestone, 1761,
iast (1.54 ms) : 1516, 1564
.   : milestone, 1540,
profiling (1.497 ms) : 1471, 1522
.   : milestone, 1497,
tracing (1.483 ms) : 1458, 1509
.   : milestone, 1483,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.357 ms [1.338 ms, 1.376 ms] -
appsec 1.768 ms [1.742 ms, 1.794 ms] 410.6 µs (30.3%)
iast 1.498 ms [1.474 ms, 1.523 ms] 140.861 µs (10.4%)
profiling 1.503 ms [1.478 ms, 1.528 ms] 145.613 µs (10.7%)
tracing 1.488 ms [1.463 ms, 1.513 ms] 131.1 µs (9.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.375 ms [1.356 ms, 1.395 ms] -
appsec 1.761 ms [1.736 ms, 1.786 ms] 385.362 µs (28.0%)
iast 1.54 ms [1.516 ms, 1.564 ms] 164.768 µs (12.0%)
profiling 1.497 ms [1.471 ms, 1.522 ms] 121.506 µs (8.8%)
tracing 1.483 ms [1.458 ms, 1.509 ms] 108.032 µs (7.9%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.25.0-SNAPSHOT~38c2568e9d, baseline=1.25.0-SNAPSHOT~7fb8faa37b
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.489 µs) : 347, 388
.   : milestone, 367,
iast (472.394 µs) : 452, 493
.   : milestone, 472,
iast_FULL (535.762 µs) : 515, 556
.   : milestone, 536,
iast_INACTIVE (445.898 µs) : 425, 467
.   : milestone, 446,
iast_TELEMETRY_OFF (469.255 µs) : 449, 490
.   : milestone, 469,
tracing (440.748 µs) : 420, 462
.   : milestone, 441,
section candidate
no_agent (364.621 µs) : 344, 385
.   : milestone, 365,
iast (471.189 µs) : 451, 492
.   : milestone, 471,
iast_FULL (534.744 µs) : 514, 555
.   : milestone, 535,
iast_INACTIVE (447.491 µs) : 427, 468
.   : milestone, 447,
iast_TELEMETRY_OFF (464.524 µs) : 444, 485
.   : milestone, 465,
tracing (447.961 µs) : 427, 469
.   : milestone, 448,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.489 µs [346.847 µs, 388.132 µs] -
iast 472.394 µs [451.571 µs, 493.217 µs] 104.905 µs (28.5%)
iast_FULL 535.762 µs [515.295 µs, 556.228 µs] 168.272 µs (45.8%)
iast_INACTIVE 445.898 µs [424.837 µs, 466.959 µs] 78.408 µs (21.3%)
iast_TELEMETRY_OFF 469.255 µs [448.6 µs, 489.909 µs] 101.765 µs (27.7%)
tracing 440.748 µs [419.613 µs, 461.883 µs] 73.258 µs (19.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.621 µs [343.772 µs, 385.47 µs] -
iast 471.189 µs [450.509 µs, 491.869 µs] 106.567 µs (29.2%)
iast_FULL 534.744 µs [514.454 µs, 555.033 µs] 170.122 µs (46.7%)
iast_INACTIVE 447.491 µs [426.579 µs, 468.402 µs] 82.869 µs (22.7%)
iast_TELEMETRY_OFF 464.524 µs [444.12 µs, 484.928 µs] 99.902 µs (27.4%)
tracing 447.961 µs [426.857 µs, 469.065 µs] 83.34 µs (22.9%)

@paul-laffon-dd paul-laffon-dd added the inst: apache spark Apache Spark instrumentation label Dec 4, 2023
raphaelgavache
raphaelgavache approved these changes Dec 4, 2023
View reviewed changes

@raphaelgavache raphaelgavache left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@paul-laffon-dd
paul-laffon-dd marked this pull request as ready for review December 4, 2023 13:08
@paul-laffon-dd
paul-laffon-dd requested a review from a team as a code owner December 4, 2023 13:08
@paul-laffon-dd
paul-laffon-dd requested review from am312 and ygree December 4, 2023 13:08
@paul-laffon-dd
paul-laffon-dd merged commit 2aa6cb3 into master Dec 12, 2023
@paul-laffon-dd
paul-laffon-dd deleted the paul.laffon/spark-parent-stage-ids branch December 12, 2023 09:21
@github-actions github-actions Bot added this to the 1.26.0 milestone Dec 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@raphaelgavache raphaelgavache raphaelgavache approved these changes
@ygree ygree Awaiting requested review from ygree ygree was automatically assigned from DataDog/apm-java
+1 more reviewer
@am312 am312 am312 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

inst: apache spark Apache Spark instrumentation

Projects

None yet

Milestone

1.26.0

Development

Successfully merging this pull request may close these issues.

3 participants

@paul-laffon-dd @am312 @raphaelgavache