Collect and visualize benchmark results by ian-r-rose · Pull Request #208 · coiled/benchmarks

ian-r-rose · 2022-07-12T23:50:56Z

Fixes #186, fixes #187, fixes #188, fixes #189.

This is a proof-of-concept based on the discussion in #186 using the "custom fixtures" solution, for the reasons therein.

The stack involves:

Metrics are collected using pytest fixtures. Currently this just does wall clock time, but I've built in fields for others and I don't see many barriers to implementing them.
Historical benchmarking data is stored in sqlite databases in s3. This could also be CSV, JSON, Parquet, but I found that working with sqlalchemy+sqlite handles multiple connections nicely when running with pytest-xdist, and using alembic allows for not-too-horrible migrations should we want to change the schema. Eventually, this could be replaced with some notion of repeated "jobs" in coiled itself.
Historical benchmarking data is currently parameterized per-platform, per-coiled-runtime, per-python-version. This is subject to change.
Historical data are visualized using static sites on github pages. I've used panel to generate them, but other tools are possible.
Sample: https://coiled.github.io/coiled-runtime/benchmark-ubuntu-latest-0.0.4-py3.9.html

quicker Initial migration actually skip Upload to s3 debug

ncclementi

I know this is WIP but I left some comments.

ncclementi · 2022-07-13T14:40:38Z

+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.RUNTIME_CI_BOT_AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.RUNTIME_CI_BOT_AWS_SECRET_ACCESS_KEY }}
+          AWS_DEFAULT_REGION: us-east-2  # this is needed for boto for some reason


Could this be connected to #207

It might be, I was thinking along the same lines! Not sure what's going on there, but I also wonder whether it's related to the region thing that you ran into with the s3 pytest fixtures here

gjoseph92

Super cool!

ncclementi

Added some comments, mostly for my understanding

Co-authored-by: James Bourbeau <[email protected]>

ian-r-rose · 2022-07-21T20:31:08Z

Okay, this should be good to go

jrbourbeau

Thanks for all your work on this @ian-r-rose -- looking forward to building up a bunch of test suite runs : )

Ian Rose added 14 commits July 1, 2022 14:38

Proof of concept custom benchamrking WIP

be70464

Cleanup

a62e0d1

Handle missing data a bit better

8662c06

Style

26b5cf2

Make benchmarking conditional.

eec4b29

Run migrations in fixture

b86b629

xdist-safe init

dd1c0fa

Add fields for coiled runtime version, dask version, and CI run url

878183f

Add to ci env

5f3fb2b

Autouse time benchmark for parquet benchmarks

0504ef2

New benchmark workflow

412d096

quicker Initial migration actually skip Upload to s3 debug

Work on migrations

b88e7be

Minor cleanup

0cfab48

WIP deploying to gh-pages

0371f5a

ncclementi reviewed Jul 13, 2022

View reviewed changes

gjoseph92 reviewed Jul 13, 2022

View reviewed changes

Comment thread .github/workflows/benchmarks.yml Outdated

Comment thread .github/workflows/benchmarks.yml Outdated

More use of benchmark fixture

7020055

ian-r-rose force-pushed the custom-dashboarding branch from 1ac9d62 to 7020055 Compare July 13, 2022 21:45

Ian Rose added 4 commits July 13, 2022 16:13

Implement memory metrics for benchmarks

4f35c78

Add path to model.

d8b5488

Update dashboard

20dc421

Sort tests

f83904b

ian-r-rose force-pushed the custom-dashboarding branch from d7756be to 9f99cfb Compare July 15, 2022 18:44

WIP restructuring benchmarks to combine dbs later

c0ebf7d

ian-r-rose force-pushed the custom-dashboarding branch from 9f99cfb to c0ebf7d Compare July 15, 2022 19:07

Add python version and platform to schema

c122576

ian-r-rose force-pushed the custom-dashboarding branch 4 times, most recently from 66477d2 to 60b2317 Compare July 18, 2022 18:50

Ian Rose added 5 commits July 18, 2022 17:11

Add some docs

76d542e

Remove test tests

6b4f2bb

Fix env?

f637433

Distinguish between coiled runtime version and coiled software name

4887631

Update dashboard for new schema

be60409

ian-r-rose force-pushed the custom-dashboarding branch from bc68a61 to be60409 Compare July 19, 2022 01:51

de-walrus

32400da

ian-r-rose marked this pull request as ready for review July 19, 2022 02:25

ian-r-rose changed the title ~~[WIP] Custom dashboarding~~ Collect and visualize benchmark results Jul 19, 2022

Ian Rose added 5 commits July 18, 2022 19:58

Remove dedicated workflow

2359fc6

Benchmark everything

a76a7b7

Automatically apply wall clock measuring to small_client

0f112b4

Handle missing values a bit better

6455929

Organize by category

7fe5eb4

ncclementi reviewed Jul 20, 2022

View reviewed changes

Comment thread ci/environment-dashboard.yml Outdated

Comment thread dashboard.py

Comment thread dashboard.py Outdated

Comment thread tests/benchmarks/test_coiled.py Outdated

ncclementi mentioned this pull request Jul 20, 2022

Choose benchmarking tool #186

Closed

Ian Rose added 2 commits July 20, 2022 12:16

Use dropna instead of getitem

51b776c

Add some version constraints

da2969e

ian-r-rose mentioned this pull request Jul 20, 2022

Determine where historical benchmarking data should live #187

Closed

jrbourbeau reviewed Jul 21, 2022

View reviewed changes

Comment thread .github/workflows/tests.yml Outdated

Comment thread conftest.py Outdated

Comment thread conftest.py

Comment thread setup.cfg

Comment thread tests/benchmarks/test_coiled.py Outdated

Comment thread tests/benchmarks/test_parquet.py

Comment thread benchmark_schema.py Outdated

ian-r-rose and others added 3 commits July 21, 2022 08:49

Update conftest.py

142b15d

Co-authored-by: James Bourbeau <[email protected]>

Benchmark time as a top-level autouse fixture

c3a7b75

Add note to schema about not-yet-collected fields

486e7a0

jrbourbeau reviewed Jul 21, 2022

View reviewed changes

Comment thread .github/workflows/tests.yml

Ian Rose added 3 commits July 21, 2022 12:33

Add distributed version.

0bdf21d

Add index page for benchmarks

ab8a02a

Only run on main

19f9bbb

jrbourbeau approved these changes Jul 22, 2022

View reviewed changes

jrbourbeau merged commit 6ab59e1 into main Jul 22, 2022

Uh oh!

Conversation

ian-r-rose commented Jul 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ncclementi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ncclementi Jul 13, 2022

Choose a reason for hiding this comment

Uh oh!

ian-r-rose Jul 13, 2022

Choose a reason for hiding this comment

Uh oh!

gjoseph92 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ncclementi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ian-r-rose commented Jul 21, 2022

Uh oh!

jrbourbeau left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ian-r-rose commented Jul 12, 2022 •

edited

Loading