Skip to content

Improve benchmarks#1002

Merged
lcobucci merged 3 commits into
5.1.xfrom
improve-benchmarks
Mar 8, 2023
Merged

Improve benchmarks#1002
lcobucci merged 3 commits into
5.1.xfrom
improve-benchmarks

Conversation

@lcobucci

@lcobucci lcobucci commented Mar 7, 2023

Copy link
Copy Markdown
Owner

Fix #525

This completely redesigns the benchmark suite and adds new items to analyse the performance impact of token issuing/parsing.

@lcobucci lcobucci added this to the 5.1.0 milestone Mar 7, 2023
@lcobucci
lcobucci requested review from Ocramius and Slamdunk March 7, 2023 22:59
@lcobucci lcobucci self-assigned this Mar 7, 2023
lcobucci added 3 commits March 8, 2023 00:00
This modifies the structure of the suite to allow us to more easily
identify and compare the algorithms' performance for each operation.

Transforming the `aggregate` report from:

```sh
+--------------+-------------------+-----+------+-----+----------+----------+--------+
| benchmark    | subject           | set | revs | its | mem_peak | mode     | rstdev |
+--------------+-------------------+-----+------+-----+----------+----------+--------+
| Sha256Bench  | benchSignature    |     | 100  | 5   | 4.543mb  | 2.655ms  | ±0.17% |
| Sha256Bench  | benchVerification |     | 100  | 5   | 4.543mb  | 1.329ms  | ±0.36% |
| Sha384Bench  | benchSignature    |     | 100  | 5   | 4.543mb  | 2.656ms  | ±0.24% |
| Sha384Bench  | benchVerification |     | 100  | 5   | 4.543mb  | 1.334ms  | ±0.21% |
| Sha512Bench  | benchSignature    |     | 100  | 5   | 4.543mb  | 2.662ms  | ±0.15% |
| Sha512Bench  | benchVerification |     | 100  | 5   | 4.543mb  | 1.335ms  | ±0.20% |
| Sha256Bench  | benchSignature    |     | 100  | 5   | 4.543mb  | 2.988μs  | ±1.78% |
| Sha256Bench  | benchVerification |     | 100  | 5   | 4.543mb  | 3.131μs  | ±6.67% |
| Sha384Bench  | benchSignature    |     | 100  | 5   | 4.543mb  | 3.345μs  | ±1.97% |
| Sha384Bench  | benchVerification |     | 100  | 5   | 4.543mb  | 3.485μs  | ±1.34% |
| Sha512Bench  | benchSignature    |     | 100  | 5   | 4.543mb  | 3.381μs  | ±7.22% |
| Sha512Bench  | benchVerification |     | 100  | 5   | 4.543mb  | 3.505μs  | ±0.46% |
| EddsaBench   | benchSignature    |     | 100  | 5   | 4.542mb  | 30.035μs | ±0.89% |
| EddsaBench   | benchVerification |     | 100  | 5   | 4.543mb  | 73.330μs | ±0.25% |
| Blake2bBench | benchSignature    |     | 100  | 5   | 4.542mb  | 0.770μs  | ±0.00% |
| Blake2bBench | benchVerification |     | 100  | 5   | 4.543mb  | 0.870μs  | ±0.56% |
| Sha256Bench  | benchSignature    |     | 100  | 5   | 4.543mb  | 1.387ms  | ±0.37% |
| Sha256Bench  | benchVerification |     | 100  | 5   | 4.543mb  | 1.407ms  | ±0.15% |
| Sha384Bench  | benchSignature    |     | 100  | 5   | 4.543mb  | 2.484ms  | ±0.24% |
| Sha384Bench  | benchVerification |     | 100  | 5   | 4.543mb  | 2.264ms  | ±0.30% |
| Sha512Bench  | benchSignature    |     | 100  | 5   | 4.543mb  | 1.772ms  | ±0.41% |
| Sha512Bench  | benchVerification |     | 100  | 5   | 4.543mb  | 2.083ms  | ±0.04% |
+--------------+-------------------+-----+------+-----+----------+----------+--------+
```

Into:

```sh
+----------------------+---------+---------+------+-----+----------+-----------+---------+
| benchmark            | subject | set     | revs | its | mem_peak | mode      | rstdev  |
+----------------------+---------+---------+------+-----+----------+-----------+---------+
| createsignaturebench | hmac    | hs256   | 100  | 5   | 1.522mb  | 1.762μs   | ±20.24% |
| createsignaturebench | hmac    | hs384   | 100  | 5   | 1.522mb  | 1.997μs   | ±20.96% |
| createsignaturebench | hmac    | hs512   | 100  | 5   | 1.522mb  | 1.986μs   | ±18.27% |
| createsignaturebench | rsa     | rs256   | 100  | 5   | 1.522mb  | 1.184ms   | ±1.33%  |
| createsignaturebench | rsa     | rs384   | 100  | 5   | 1.522mb  | 1.186ms   | ±1.01%  |
| createsignaturebench | rsa     | rs512   | 100  | 5   | 1.522mb  | 1.195ms   | ±0.73%  |
| createsignaturebench | ecdsa   | es256   | 100  | 5   | 1.522mb  | 434.401μs | ±1.95%  |
| createsignaturebench | ecdsa   | es384   | 100  | 5   | 1.522mb  | 1.110ms   | ±0.49%  |
| createsignaturebench | ecdsa   | es512   | 100  | 5   | 1.522mb  | 683.722μs | ±1.33%  |
| createsignaturebench | blake2b | blake2b | 100  | 5   | 1.522mb  | 0.451μs   | ±23.04% |
| verifysignaturebench | hmac    | hs256   | 100  | 5   | 1.522mb  | 1.829μs   | ±1.28%  |
| verifysignaturebench | hmac    | hs384   | 100  | 5   | 1.522mb  | 2.072μs   | ±10.81% |
| verifysignaturebench | hmac    | hs512   | 100  | 5   | 1.522mb  | 2.070μs   | ±1.78%  |
| verifysignaturebench | rsa     | rs256   | 100  | 5   | 1.522mb  | 407.624μs | ±0.62%  |
| verifysignaturebench | rsa     | rs384   | 100  | 5   | 1.522mb  | 407.269μs | ±1.75%  |
| verifysignaturebench | rsa     | rs512   | 100  | 5   | 1.522mb  | 408.983μs | ±1.20%  |
| verifysignaturebench | ecdsa   | es256   | 100  | 5   | 1.522mb  | 448.420μs | ±0.87%  |
| verifysignaturebench | ecdsa   | es384   | 100  | 5   | 1.522mb  | 957.291μs | ±2.92%  |
| verifysignaturebench | ecdsa   | es512   | 100  | 5   | 1.522mb  | 889.006μs | ±1.09%  |
| verifysignaturebench | blake2b | blake2b | 100  | 5   | 1.522mb  | 0.520μs   | ±2.43%  |
+----------------------+---------+---------+------+-----+----------+-----------+---------+
```

Also allowing us to filter for groups (`symmetric`, `asymmetric`,
`hmac`, `rsa`, `ecdsa`, and `blake2b`).

Signed-off-by: Luís Cobucci <[email protected]>
This introduces benchmarks for token issuing and parsing, making sure
can analyse performance impact beyond the signature
creation/verification.

Signed-off-by: Luís Cobucci <[email protected]>
This enables defining groups/filters and many other flags provided by
the tool.

Signed-off-by: Luís Cobucci <[email protected]>
@lcobucci
lcobucci force-pushed the improve-benchmarks branch from d471663 to fbf8cd5 Compare March 7, 2023 23:00

@Ocramius Ocramius left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall OK with redesigning this, just unsure whether we'd lose our ability to compare this to 4.x performance.

@lcobucci

lcobucci commented Mar 8, 2023

Copy link
Copy Markdown
Owner Author

Overall OK with redesigning this, just unsure whether we'd lose our ability to compare this to 4.x performance.

I don't think we lose it, it'd require tinkering with the output, though.

This can be done in 4.x, the bench suite hasn't changed for quite some time. I'm just not sure if it's worth it

@lcobucci
lcobucci merged commit 67f5054 into 5.1.x Mar 8, 2023
@lcobucci
lcobucci deleted the improve-benchmarks branch March 8, 2023 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create benchmarks for token parsing and issuing

3 participants