Torchbench nightly MPS runs #135386

skotapati · 2024-09-06T19:13:59Z

Add a workflow to run TorchBench with nightly mps builds & upload performance data to the HUD

cc @kulinseth @albanD @malfet @DenisVieriu97 @jhavukainen @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @rec

pytorch-bot · 2024-09-06T19:14:03Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/135386

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

[DomainsOnly] Jobs fail with GLIBC version not found

✅ No Failures

As of commit ba9c6f6 with merge base f4008a5 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

.github/workflows/torchbench-nightly-mps.yml

.github/workflows/_mac-build.yml

.ci/pytorch/macos-test.sh

skotapati · 2024-09-24T05:57:36Z

@pytorchbot rebase

pytorchmergebot · 2024-09-24T05:59:12Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-09-24T05:59:15Z

Successfully rebased dev/skotapati/nightly_perf_mps onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout dev/skotapati/nightly_perf_mps && git pull --rebase)

skotapati · 2024-11-06T00:55:18Z

@pytorchbot rebase

pytorchmergebot · 2024-11-06T00:56:47Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2024-11-06T00:56:51Z

Successfully rebased dev/skotapati/nightly_perf_mps onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout dev/skotapati/nightly_perf_mps && git pull --rebase)

pytorchmergebot · 2024-11-14T21:23:46Z

Successfully rebased dev/skotapati/nightly_perf_mps onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout dev/skotapati/nightly_perf_mps && git pull --rebase)

malfet · 2024-11-14T21:48:04Z

@pytorchbot merge -f "Lint + MPS tests are green"

pytorchmergebot · 2024-11-14T21:50:12Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Add a workflow to run TorchBench with nightly mps builds & upload performance data to the HUD Solves: pytorch#115201 Pull Request resolved: pytorch#135386 Approved by: https://github.com/DenisVieriu97, https://github.com/kulinseth, https://github.com/malfet Co-authored-by: Nikita Shulga <[email protected]> Co-authored-by: Huy Do <[email protected]>

After #135386 and #141999, MPS benchmark has been running for a while and the data has been uploaded correctly. However, the dashboard is still using the old schema that requires the output CSV files to be named in a certain way for its query to work https://github.com/pytorch/test-infra/blob/main/torchci/clickhouse_queries/compilers_benchmark_performance/query.sql#L32-L40. Specifically, the filename needs to be in the following format `inductor_${backend}_${suite}_${dtype}_${mode}_${device}_${target}.csv`. The new schema gets away with all this hacky setup, but the dashboard hasn't been migrated to the new schema yet. So, this is a quick way to just get the data to show up first. ### Testing https://github.com/pytorch/pytorch/actions/runs/12153886764 Pull Request resolved: #142034 Approved by: https://github.com/skotapati, https://github.com/malfet

After pytorch#135386 and pytorch#141999, MPS benchmark has been running for a while and the data has been uploaded correctly. However, the dashboard is still using the old schema that requires the output CSV files to be named in a certain way for its query to work https://github.com/pytorch/test-infra/blob/main/torchci/clickhouse_queries/compilers_benchmark_performance/query.sql#L32-L40. Specifically, the filename needs to be in the following format `inductor_${backend}_${suite}_${dtype}_${mode}_${device}_${target}.csv`. The new schema gets away with all this hacky setup, but the dashboard hasn't been migrated to the new schema yet. So, this is a quick way to just get the data to show up first. ### Testing https://github.com/pytorch/pytorch/actions/runs/12153886764 Pull Request resolved: pytorch#142034 Approved by: https://github.com/skotapati, https://github.com/malfet

The bash script modification in #135386 results in tests on mac in default shard not running. This PR is expected to cause test failures, but we need to start getting signal, so landing with known failures Pull Request resolved: #142270 Approved by: https://github.com/malfet, https://github.com/seemethere, https://github.com/atalman Co-authored-by: Nikita Shulga <[email protected]>

skotapati self-assigned this Sep 6, 2024

pytorch-bot bot added the topic: not user facing topic category label Sep 6, 2024

skotapati added ciflow/mps Run MPS tests (subset of trunk) module: mps Related to Apple Metal Performance Shaders framework labels Sep 6, 2024

pytorchbot added the open source label Sep 6, 2024

DenisVieriu97 requested changes Sep 10, 2024

View reviewed changes

.github/workflows/torchbench-nightly-mps.yml Outdated Show resolved Hide resolved

.github/workflows/torchbench-nightly-mps.yml Outdated Show resolved Hide resolved

skotapati force-pushed the dev/skotapati/nightly_perf_mps branch from d8a3973 to 3556e72 Compare September 12, 2024 18:12

skotapati marked this pull request as ready for review September 13, 2024 17:19

skotapati requested a review from a team as a code owner September 13, 2024 17:19

janeyx99 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Sep 16, 2024

huydhn reviewed Sep 17, 2024

View reviewed changes

.github/workflows/_mac-build.yml Outdated Show resolved Hide resolved

huydhn reviewed Sep 18, 2024

View reviewed changes

.ci/pytorch/macos-test.sh Outdated Show resolved Hide resolved

pytorch-bot bot added ciflow/inductor module: dynamo labels Sep 23, 2024

skotapati force-pushed the dev/skotapati/nightly_perf_mps branch from 35e2bc0 to 82f9804 Compare September 24, 2024 04:11

pytorchmergebot force-pushed the dev/skotapati/nightly_perf_mps branch from 82f9804 to 079e108 Compare September 24, 2024 05:59

skotapati force-pushed the dev/skotapati/nightly_perf_mps branch 2 times, most recently from 2ef9f7c to a9ad6b6 Compare September 26, 2024 18:24

skotapati requested review from albanD, kulinseth and malfet October 10, 2024 19:36

pytorchmergebot force-pushed the dev/skotapati/nightly_perf_mps branch from 8187142 to 5f4f5d0 Compare November 6, 2024 00:56

pytorch deleted a comment from pytorch-bot bot Nov 6, 2024

skotapati added 8 commits November 14, 2024 21:23

Try using different runner

61707be

Add a few additional models

93e4985

Remove unnecessary cd

798dba1

Switch to nightly dispatch now that runs are working

2550054

linter fixes

093ec06

Additional linter fixes

6431fde

Remove unnecessary option from build

10346e7

Fix typo

ba9c6f6

pytorchmergebot force-pushed the dev/skotapati/nightly_perf_mps branch from 8e2506d to ba9c6f6 Compare November 14, 2024 21:23

malfet approved these changes Nov 14, 2024

View reviewed changes

pytorchmergebot added the merging label Nov 14, 2024

pytorchmergebot closed this in d751b27 Nov 14, 2024

pytorchmergebot added Merged and removed merging labels Nov 14, 2024

huydhn mentioned this pull request Dec 4, 2024

Use the correct CSV filenames for MPS benchmark #142034

Closed

clee2000 mentioned this pull request Dec 9, 2024

[Do not revert] Re-enable Mac testing #142270

Closed

malfet mentioned this pull request Dec 9, 2024

MacOS tests has not been running for few weeks #142206

Closed

huydhn added the test-config/default label Feb 27, 2025

Torchbench nightly MPS runs #135386

Torchbench nightly MPS runs #135386

Uh oh!

Conversation

skotapati commented Sep 6, 2024 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/135386

❗ 1 Active SEVs

✅ No Failures

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

skotapati commented Sep 24, 2024

Uh oh!

pytorchmergebot commented Sep 24, 2024

Uh oh!

pytorchmergebot commented Sep 24, 2024

Uh oh!

skotapati commented Nov 6, 2024

Uh oh!

pytorchmergebot commented Nov 6, 2024

Uh oh!

pytorchmergebot commented Nov 6, 2024

Uh oh!

pytorchmergebot commented Nov 14, 2024

Uh oh!

malfet commented Nov 14, 2024

Uh oh!

pytorchmergebot commented Nov 14, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

skotapati commented Sep 6, 2024 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Sep 6, 2024 •

edited

Loading