add nvidia-smi to run_torchbench #83857

davidberard98 · 2022-08-22T17:38:25Z

Stack from ghstack (oldest at bottom):

-> add nvidia-smi to run_torchbench #83857

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test.

RUN_TORCHBENCH: nvfuser

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

facebook-github-bot · 2022-08-22T17:38:32Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/83857
✖️ Python docs build was skipped
✖️ C++ docs build was skipped
❓Need help or want to give feedback on the CI? Visit our office hours
↩️ [fb-only] Re-run with SSH instructions

❌ 1 New Failures

As of commit b60bfee (more details on the Dr. CI page):

Expand to see more

1/1 failures introduced in this PR

🕵️‍♀️ 1 failure not recognized by patterns:

The following CI failures may be due to changes from the PR

Job	Step
^build	^Unknown

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: c9f07d6 Pull Request resolved: #83857

davidberard98 · 2022-08-22T21:31:31Z

@xuzhao9 do you know why torchbench is OOMing here? I tried running this on the AWS cluster on A100s but I couldn't repro the OOM issue

xuzhao9 · 2022-08-22T23:29:58Z

@davidberard98 The runner has 8xNvidia T4 GPUs (each 16GB), not A100, maybe that's the difference?

davidberard98 · 2022-08-23T21:57:51Z

@pytorchbot rebase -s

pytorchmergebot · 2022-08-23T23:22:44Z

@pytorchbot successfully started a rebase job. Check the current status here

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

pytorchmergebot · 2022-08-23T23:23:01Z

Successfully rebased gh/davidberard98/141/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/83857)

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: 685b989 Pull Request resolved: #83857

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: 397272e Pull Request resolved: #83857

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: a57984d Pull Request resolved: #83857

xuzhao9 · 2022-08-27T02:59:32Z

.github/workflows/run_torchbench.yml

          . "${HOME}"/anaconda3/etc/profile.d/conda.sh
          conda activate pr-ci
-          python3 pytorch/.github/scripts/run_torchbench.py \
+          # python3 -c "import torch; torch.rand((4, 4), device='cuda')"


Looks like we can remove this? I am also okay to keep and uncomment it, just to make sure the hardware works.

thanks for catching that - I will remove it. It doesn't actually work because pytorch isn't built at this point.

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: 20d277a Pull Request resolved: #83857

davidberard98 · 2022-08-30T20:04:23Z

@pytorchbot rebase -s

pytorchmergebot · 2022-08-30T20:06:15Z

@pytorchbot successfully started a rebase job. Check the current status here

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

pytorchmergebot · 2022-08-30T20:06:28Z

Successfully rebased gh/davidberard98/141/orig onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via ghstack checkout https://github.com/pytorch/pytorch/pull/83857)

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: fc150ae Pull Request resolved: #83857

davidberard98 · 2022-08-31T02:09:53Z

@pytorchbot merge

pytorchmergebot · 2022-08-31T02:11:27Z

@pytorchbot successfully started a merge job. Check the current status here.
The merge job was triggered without a flag. This means that your change will be merged once all checks on your PR have passed (ETA: 0-4 Hours). If this is not the intended behavior, feel free to use some of the other merge options in the wiki.
Please reach out to the PyTorch DevX Team with feedback or questions!

github-actions · 2022-08-31T02:12:20Z

Hey @davidberard98.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

Summary: Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser Pull Request resolved: #83857 Approved by: https://github.com/xuzhao9 Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/71d99662a0d7f8a9ad68999c9a014b71591cbb68 Reviewed By: mehtanirav Differential Revision: D39172015 Pulled By: davidberard98 fbshipit-source-id: 208f7d8bf00937a459bb5abd5baf9461660d19c3

add nvidia-smi to run_torchbench

cb0f3dc

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

davidberard98 requested a review from a team as a code owner August 22, 2022 17:38

facebook-github-bot added the cla signed label Aug 22, 2022

davidberard98 requested review from xuzhao9 and removed request for a team and xuzhao9 August 22, 2022 17:46

davidberard98 marked this pull request as draft August 22, 2022 18:00

davidberard98 requested a review from xuzhao9 August 22, 2022 21:30

Update on "add nvidia-smi to run_torchbench"

c269399

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Update on "add nvidia-smi to run_torchbench"

468d477

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Update on "add nvidia-smi to run_torchbench"

cabc40b

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Update on "add nvidia-smi to run_torchbench"

23ee2a6

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Update on "add nvidia-smi to run_torchbench"

99d02db

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Update on "add nvidia-smi to run_torchbench"

f3c36bc

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

davidberard98 mentioned this pull request Aug 24, 2022

[test] sanity check in log_extract.py #84002

Closed

Update on "add nvidia-smi to run_torchbench"

2edea3a

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Update on "add nvidia-smi to run_torchbench"

da3b093

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

davidberard98 marked this pull request as ready for review August 26, 2022 18:04

davidberard98 requested a review from a team August 26, 2022 18:04

xuzhao9 approved these changes Aug 27, 2022

View reviewed changes

Update on "add nvidia-smi to run_torchbench"

3819dfe

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Update on "add nvidia-smi to run_torchbench"

b60bfee

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

pytorch-bot bot added the release notes: releng release notes category label Aug 30, 2022

pytorchmergebot added the Merged label Aug 31, 2022

pytorchmergebot closed this in 71d9966 Aug 31, 2022

facebook-github-bot deleted the gh/davidberard98/141/head branch September 3, 2022 14:20

add nvidia-smi to run_torchbench #83857

add nvidia-smi to run_torchbench #83857

Uh oh!

Conversation

davidberard98 commented Aug 22, 2022 • edited by pytorchmergebot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Aug 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

❌ 1 New Failures

🕵️‍♀️ 1 failure not recognized by patterns:

Uh oh!

davidberard98 commented Aug 22, 2022

Uh oh!

xuzhao9 commented Aug 22, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

davidberard98 commented Aug 23, 2022

Uh oh!

pytorchmergebot commented Aug 23, 2022

Uh oh!

pytorchmergebot commented Aug 23, 2022

Uh oh!

xuzhao9 Aug 27, 2022

Choose a reason for hiding this comment

Uh oh!

davidberard98 Aug 27, 2022

Choose a reason for hiding this comment

Uh oh!

davidberard98 commented Aug 30, 2022

Uh oh!

pytorchmergebot commented Aug 30, 2022

Uh oh!

pytorchmergebot commented Aug 30, 2022

Uh oh!

davidberard98 commented Aug 31, 2022

Uh oh!

pytorchmergebot commented Aug 31, 2022

Uh oh!

github-actions bot commented Aug 31, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

davidberard98 commented Aug 22, 2022 •

edited by pytorchmergebot

Loading

facebook-github-bot commented Aug 22, 2022 •

edited

Loading

xuzhao9 commented Aug 22, 2022 •

edited

Loading