Skip to content

modifying benchmarks to reduce non-determinism#3150

Merged
tlhunter merged 4 commits into
masterfrom
tlhunter/sirun-shenanigans
May 12, 2023
Merged

modifying benchmarks to reduce non-determinism#3150
tlhunter merged 4 commits into
masterfrom
tlhunter/sirun-shenanigans

Conversation

@tlhunter

@tlhunter tlhunter commented May 12, 2023 •
edited
Loading

Copy link
Copy Markdown
Member

What does this PR do?

  • increases number of iterations per benchmark run for logs
  • decreases benchmark runs for logs
  • test run time goes from ~7s to ~30s
  • note that the negative performance results is due to the test being updated in the PR but not in master

Motivation

  • a forthcoming PR adds some weirdness to the benchmark results
  • seems like we're hitting some sort of code file threshold

@github-actions

github-actions Bot commented May 12, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Overall package size

Self size: 4.12 MB
Deduped: 58.25 MB
No deduping: 58.3 MB

Dependency sizes

name version self size total size
@datadog/pprof 2.2.1 14.24 MB 15.12 MB
@datadog/native-iast-taint-tracking 1.4.1 14.85 MB 14.86 MB
@datadog/native-appsec 3.1.0 13.31 MB 13.32 MB
protobufjs 7.1.2 2.76 MB 6.55 MB
@datadog/native-iast-rewriter 2.0.1 2.09 MB 2.1 MB
@datadog/native-metrics 2.0.0 898.77 kB 1.3 MB
opentracing 0.14.7 194.81 kB 194.81 kB
semver 7.3.8 88.2 kB 118.6 kB
@datadog/sketches-js 2.1.0 109.9 kB 109.9 kB
lodash.sortby 4.7.0 75.76 kB 75.76 kB
lru-cache 7.14.0 74.95 kB 74.95 kB
ipaddr.js 2.0.1 59.52 kB 59.52 kB
ignore 5.2.0 48.87 kB 48.87 kB
import-in-the-middle 1.3.5 34.34 kB 38.81 kB
istanbul-lib-coverage 3.2.0 29.34 kB 29.34 kB
retry 0.10.1 27.44 kB 27.44 kB
lodash.uniq 4.5.0 25.01 kB 25.01 kB
limiter 1.1.5 23.17 kB 23.17 kB
lodash.kebabcase 4.1.1 17.75 kB 17.75 kB
lodash.pick 4.4.0 16.33 kB 16.33 kB
node-abort-controller 3.0.1 14.33 kB 14.33 kB
crypto-randomuuid 1.0.0 11.18 kB 11.18 kB
diagnostics_channel 1.1.0 7.07 kB 7.07 kB
path-to-regexp 0.1.7 6.78 kB 6.78 kB
koalas 1.0.2 6.47 kB 6.47 kB
methods 1.1.2 5.29 kB 5.29 kB
module-details-from-path 1.0.3 4.47 kB 4.47 kB

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@codecov

codecov Bot commented May 12, 2023 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

Merging #3150 (45a50e0) into master (563729a) will decrease coverage by 0.22%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3150      +/-   ##
==========================================
- Coverage   86.98%   86.76%   -0.22%     
==========================================
  Files         322      308      -14     
  Lines       11763    11380     -383     
  Branches       33       33              
==========================================
- Hits        10232     9874     -358     
+ Misses       1531     1506      -25

see 17 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@pr-commenter

pr-commenter Bot commented May 12, 2023 •
edited
Loading

Copy link
Copy Markdown

Benchmarks

Comparing candidate commit 45a50e0 in PR branch tlhunter/sirun-shenanigans with baseline commit 563729a in branch master.

Found 0 performance improvements and 20 performance regressions! Performance is the same for 420 metrics, 32 unstable metrics.

scenario:log-with-debug-16

  • 🟥 cpu_user_time [+8.364ms; +14.775ms] or [+6.081%; +10.742%]
  • 🟥 execution_time [+10.772ms; +11.748ms] or [+6.648%; +7.251%]
  • 🟥 instructions [+60; +60] or [+14.386%; +14.426%]

scenario:log-with-error-16

  • 🟥 cpu_user_time [+8.859ms; +13.209ms] or [+6.388%; +9.524%]
  • 🟥 execution_time [+11.062ms; +12.445ms] or [+6.827%; +7.681%]
  • 🟥 instructions [+60; +60] or [+14.389%; +14.428%]

scenario:log-skip-log-16

  • 🟥 cpu_user_time [+8.640ms; +14.559ms] or [+6.226%; +10.491%]
  • 🟥 execution_time [+10.303ms; +12.383ms] or [+6.331%; +7.609%]
  • 🟥 instructions [+60; +61] or [+14.334%; +14.447%]

scenario:log-without-log-16

  • 🟥 execution_time [+8.644ms; +10.177ms] or [+5.300%; +6.241%]
  • 🟥 instructions [+46; +46] or [+11.058%; +11.098%]

scenario:log-with-error-18

  • 🟥 execution_time [+10.693ms; +16.344ms] or [+6.417%; +9.808%]
  • 🟥 instructions [+59; +60] or [+14.243%; +14.304%]

scenario:log-without-log-18

  • 🟥 execution_time [+8.419ms; +9.116ms] or [+5.043%; +5.461%]
  • 🟥 instructions [+45; +46] or [+10.869%; +10.957%]

scenario:log-with-debug-18

  • 🟥 cpu_user_time [+9.034ms; +14.728ms] or [+6.433%; +10.488%]
  • 🟥 execution_time [+10.376ms; +11.943ms] or [+6.216%; +7.155%]
  • 🟥 instructions [+59; +60] or [+14.250%; +14.348%]

scenario:log-skip-log-18

  • 🟥 execution_time [+10.260ms; +11.123ms] or [+6.138%; +6.655%]
  • 🟥 instructions [+59; +60] or [+14.248%; +14.304%]

@tlhunter
tlhunter marked this pull request as ready for review May 12, 2023 18:12
@tlhunter
tlhunter requested a review from a team as a code owner May 12, 2023 18:12
tlhunter
tlhunter commented May 12, 2023
View reviewed changes
Comment thread benchmark/sirun/log/index.js

const {
DD_TRACE_DEBUG = 'true',
ITERATIONS = 1000,

@tlhunter tlhunter May 12, 2023

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really no reason to set this with an env var

@tlhunter tlhunter mentioned this pull request May 12, 2023
Closed
rochdev
rochdev previously approved these changes May 12, 2023
View reviewed changes
@tlhunter
tlhunter merged commit cad3d92 into master May 12, 2023
@tlhunter
tlhunter deleted the tlhunter/sirun-shenanigans branch May 12, 2023 18:53
thedavl pushed a commit that referenced this pull request May 30, 2023
@tlhunter @thedavl
modifying benchmarks to reduce non-determinism (#3150)
f61b239
uurien pushed a commit that referenced this pull request Jun 1, 2023
@tlhunter Ugaitz Urien
modifying benchmarks to reduce non-determinism (#3150)
724290f
uurien pushed a commit that referenced this pull request Jun 1, 2023
@tlhunter Ugaitz Urien
modifying benchmarks to reduce non-determinism (#3150)
cc23c29
uurien pushed a commit that referenced this pull request Jun 1, 2023
@tlhunter Ugaitz Urien
modifying benchmarks to reduce non-determinism (#3150)
f21de3d
uurien pushed a commit that referenced this pull request Jun 2, 2023
@tlhunter @uurien
modifying benchmarks to reduce non-determinism (#3150)
2936393
uurien pushed a commit that referenced this pull request Jun 2, 2023
@tlhunter @uurien
modifying benchmarks to reduce non-determinism (#3150)
34f033f
uurien pushed a commit that referenced this pull request Jun 2, 2023
@tlhunter @uurien
modifying benchmarks to reduce non-determinism (#3150)
1b95e65
@erikvanbrakel erikvanbrakel mentioned this pull request Jul 5, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rochdev rochdev rochdev approved these changes

Assignees

No one assigned

Labels

semver-patch

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@tlhunter @rochdev