[NVFuser] Upstream push 0811 #83239

jjsjann123 · 2022-08-11T08:23:17Z

Stack from ghstack (oldest at bottom):

-> [NVFuser] Upstream push 0811 #83239

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/

Code changes includes:

codegen improvements:
1. double support in expression evaluator
bug fixes:
1. dropout fix - rework RNG to support broadcasted dropout (Fixes NVFuser produces incorrect output for broadcasted dropout #82784)
2. expand fix - Patch expand+reduction, expand+view, rework view analysis and guard
scheduler:
1. manual transpose schedule example
2. WIP transpose scheduler

Commits that's in this PR from the devel branch:

b7435afcd22c917713c2f41a7237bc26e1183f14 Transpose scheduler, step 1 (#1854)
8a45dbf72034684eb8e18b1835b533e90b68f184 Add an example on how to manually schedule transpose (#1889)
83dbf56a9554b2efbd5416461d938fff477b0b27 Patch dropout fix (#1898)
69d3519a532250719b1aa8341b50e067b181b42d Expand+Reduction, Expand+View support, rework View analysis and guards (#1883)
15091c488e96343bdc49e3990acbf238a3b3da51 Rework RNG to correctly support broadcasted dropout (#1888)
aafe2d048aaac596e503596a41303423619f3954 Make ExpressionEvaluator support Double (#1885)

RUN_TORCHBENCH: nvfuser

Differential Revision: D38657074

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. double support in expression evaluator - bug fixes: 1. dropout fix - rework RNG to support broadcasted dropout (Fixes #82784) 2. expand fix - Patch expand+reduction, expand+view, rework view analysis and guard - scheduler: 1. manual transpose schedule example 2. WIP transpose scheduler Commits that's in this PR from the devel branch: ``` b7435af Transpose scheduler, step 1 (#1854) 8a45dbf Add an example on how to manually schedule transpose (#1889) 83dbf56 Patch dropout fix (#1898) 69d3519 Expand+Reduction, Expand+View support, rework View analysis and guards (#1883) 15091c4 Rework RNG to correctly support broadcasted dropout (#1888) aafe2d0 Make ExpressionEvaluator support Double (#1885) ``` RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

facebook-github-bot · 2022-08-11T08:23:24Z

🔗 Helpful links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/83239
📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓Need help or want to give feedback on the CI? Visit our office hours

❌ 1 New Failures, 8 Pending

As of commit 0e04196 (more details on the Dr. CI page):

Expand to see more

1/1 failures introduced in this PR

🕵️ 1 new failure recognized by patterns

The following CI failures do not appear to be due to upstream breakages

TorchBench CI (pytorch-linux-py3.7-cu102) / run-torchbench (1/1)

Step: "Run TorchBench" (full log | diagnosis details)

2022-08-23T23:43:13.9998350Z RuntimeError: CUDA error: out of memory

2022-08-23T23:43:13.9993176Z   File "/home/ec2-user/nvme/pytorch-org-runner/_work/pytorch/pytorch/benchmark/userbenchmark/nvfuser/__init__.py", line 88, in run
2022-08-23T23:43:13.9993668Z     run_nvfuser_microbenchmarks(extra_args=args)
2022-08-23T23:43:13.9994353Z   File "/home/ec2-user/nvme/pytorch-org-runner/_work/pytorch/pytorch/benchmark/userbenchmark/nvfuser/__init__.py", line 79, in run_nvfuser_microbenchmarks
2022-08-23T23:43:13.9994814Z     inputs = b.get_inputs()
2022-08-23T23:43:13.9995409Z   File "/home/ec2-user/nvme/pytorch-org-runner/_work/pytorch/pytorch/benchmark/userbenchmark/nvfuser/__init__.py", line 44, in get_inputs
2022-08-23T23:43:13.9995909Z     _, inputs = log_extract.load_graph_and_inputs(self.ir)
2022-08-23T23:43:13.9996476Z   File "/home/ec2-user/nvme/srcs/pytorch/torch/utils/jit/log_extract.py", line 52, in load_graph_and_inputs
2022-08-23T23:43:13.9996904Z     inputs.append(make_tensor_from_type(tensorType))
2022-08-23T23:43:13.9997469Z   File "/home/ec2-user/nvme/srcs/pytorch/torch/utils/jit/log_extract.py", line 39, in make_tensor_from_type
2022-08-23T23:43:13.9997959Z     return torch.empty_strided(size=size, stride=stride, device=device, dtype=dtype)
2022-08-23T23:43:13.9998350Z RuntimeError: CUDA error: out of memory
2022-08-23T23:43:13.9998792Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2022-08-23T23:43:13.9999263Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2022-08-23T23:43:14.3148866Z done
2022-08-23T23:43:14.3149854Z Failed to call userbenchmark ['/home/ec2-user/anaconda3/envs/pr-ci/bin/python', 'run_benchmark.py', 'nvfuser']. Error: Command '['/home/ec2-user/anaconda3/envs/pr-ci/bin/python', 'run_benchmark.py', 'nvfuser']' returned non-zero exit status 1.
2022-08-23T23:43:14.3304964Z Ready to run TorchBench with benchmark. Result will be saved in the directory: /home/ec2-user/.torchbench/bisection/pr83239.
2022-08-23T23:43:14.3308722Z Traceback (most recent call last):
2022-08-23T23:43:14.3309116Z   File "pytorch/.github/scripts/run_torchbench.py", line 176, in <module>
2022-08-23T23:43:14.3309616Z     run_userbenchmarks(pytorch_path=args.pytorch_path, torchbench_path=args.torchbench_path,
2022-08-23T23:43:14.3310088Z   File "pytorch/.github/scripts/run_torchbench.py", line 136, in run_userbenchmarks
2022-08-23T23:43:14.3310518Z     subprocess.check_call(command, cwd=torchbench_path, env=env)

This comment was automatically generated by Dr. CI (expand for details).

Please report bugs/suggestions to the (internal) Dr. CI Users group.

Click here to manually regenerate this comment.

davidberard98 · 2022-08-11T17:08:29Z

@jjsjann123 I see a few build errors, can you take a look?

jjsjann123 · 2022-08-11T17:09:48Z

@jjsjann123 I see a few build errors, can you take a look?

Already on them~ Will update you once I got them patched

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. double support in expression evaluator - bug fixes: 1. dropout fix - rework RNG to support broadcasted dropout (Fixes #82784) 2. expand fix - Patch expand+reduction, expand+view, rework view analysis and guard - scheduler: 1. manual transpose schedule example 2. WIP transpose scheduler Commits that's in this PR from the devel branch: ``` b7435af Transpose scheduler, step 1 (#1854) 8a45dbf Add an example on how to manually schedule transpose (#1889) 83dbf56 Patch dropout fix (#1898) 69d3519 Expand+Reduction, Expand+View support, rework View analysis and guards (#1883) 15091c4 Rework RNG to correctly support broadcasted dropout (#1888) aafe2d0 Make ExpressionEvaluator support Double (#1885) ``` RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

jjsjann123 · 2022-08-11T22:28:40Z

hmm. error here doesn't seem to be caused by our update: https://github.com/pytorch/pytorch/runs/7796124160?check_suite_focus=true

I'll try my luck with rebase...

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. double support in expression evaluator - bug fixes: 1. dropout fix - rework RNG to support broadcasted dropout (Fixes #82784) 2. expand fix - Patch expand+reduction, expand+view, rework view analysis and guard - scheduler: 1. manual transpose schedule example 2. WIP transpose scheduler Commits that's in this PR from the devel branch: ``` b7435af Transpose scheduler, step 1 (#1854) 8a45dbf Add an example on how to manually schedule transpose (#1889) 83dbf56 Patch dropout fix (#1898) 69d3519 Expand+Reduction, Expand+View support, rework View analysis and guards (#1883) 15091c4 Rework RNG to correctly support broadcasted dropout (#1888) aafe2d0 Make ExpressionEvaluator support Double (#1885) ``` RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

jjsjann123 · 2022-08-11T22:37:10Z

Don't see any obvious fix in the viable/strict history. Still rebased 🤞

davidberard98 · 2022-08-11T22:42:12Z

@jjsjann123

2022-08-11T21:58:12.3415340Z [ 98%] �[32mBuilding CUDA object test_jit/CMakeFiles/test_jit.dir/__/__/torch/csrc/jit/codegen/cuda/test/test_gpu_rng.cu.o�[0m
2022-08-11T21:58:12.3601782Z nvcc fatal   : Unknown option '-Wno-unused-variable'
2022-08-11T21:58:12.3609717Z test_jit/CMakeFiles/test_jit.dir/build.make:842: recipe for target 'test_jit/CMakeFiles/test_jit.dir/__/__/torch/csrc/jit/codegen/cuda/test/test_gpu_rng.cu.o' failed
2022-08-11T21:58:12.3610427Z make[2]: *** [test_jit/CMakeFiles/test_jit.dir/__/__/torch/csrc/jit/codegen/cuda/test/test_gpu_rng.cu.o] Error 1
2022-08-11T21:58:12.3651861Z make[2]: *** Waiting for unfinished jobs....

is this the failure?

jjsjann123 · 2022-08-11T23:02:04Z

@jjsjann123

2022-08-11T21:58:12.3415340Z [ 98%] �[32mBuilding CUDA object test_jit/CMakeFiles/test_jit.dir/__/__/torch/csrc/jit/codegen/cuda/test/test_gpu_rng.cu.o�[0m
2022-08-11T21:58:12.3601782Z nvcc fatal   : Unknown option '-Wno-unused-variable'
2022-08-11T21:58:12.3609717Z test_jit/CMakeFiles/test_jit.dir/build.make:842: recipe for target 'test_jit/CMakeFiles/test_jit.dir/__/__/torch/csrc/jit/codegen/cuda/test/test_gpu_rng.cu.o' failed
2022-08-11T21:58:12.3610427Z make[2]: *** [test_jit/CMakeFiles/test_jit.dir/__/__/torch/csrc/jit/codegen/cuda/test/test_gpu_rng.cu.o] Error 1
2022-08-11T21:58:12.3651861Z make[2]: *** Waiting for unfinished jobs....

is this the failure?

Ha, file extension... great catch 🙇

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. double support in expression evaluator - bug fixes: 1. dropout fix - rework RNG to support broadcasted dropout (Fixes #82784) 2. expand fix - Patch expand+reduction, expand+view, rework view analysis and guard - scheduler: 1. manual transpose schedule example 2. WIP transpose scheduler Commits that's in this PR from the devel branch: ``` b7435af Transpose scheduler, step 1 (#1854) 8a45dbf Add an example on how to manually schedule transpose (#1889) 83dbf56 Patch dropout fix (#1898) 69d3519 Expand+Reduction, Expand+View support, rework View analysis and guards (#1883) 15091c4 Rework RNG to correctly support broadcasted dropout (#1888) aafe2d0 Make ExpressionEvaluator support Double (#1885) ``` RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

jjsjann123 · 2022-08-11T23:13:46Z

errr... there's actually a kernel in that file and we do want nvcc.... we need to change the build... I'm patching it now.

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. double support in expression evaluator - bug fixes: 1. dropout fix - rework RNG to support broadcasted dropout (Fixes #82784) 2. expand fix - Patch expand+reduction, expand+view, rework view analysis and guard - scheduler: 1. manual transpose schedule example 2. WIP transpose scheduler Commits that's in this PR from the devel branch: ``` b7435af Transpose scheduler, step 1 (#1854) 8a45dbf Add an example on how to manually schedule transpose (#1889) 83dbf56 Patch dropout fix (#1898) 69d3519 Expand+Reduction, Expand+View support, rework View analysis and guards (#1883) 15091c4 Rework RNG to correctly support broadcasted dropout (#1888) aafe2d0 Make ExpressionEvaluator support Double (#1885) ``` RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

davidberard98 · 2022-08-24T17:50:51Z

@davidberard98 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: 397272e Pull Request resolved: #83857

facebook-github-bot · 2022-08-25T02:22:04Z

@pytorchbot merge

(Initiating merge automatically since Phabricator Diff has merged)

pytorchmergebot · 2022-08-25T02:23:18Z

@pytorchbot successfully started a merge job. Check the current status here.
The merge job was triggered without a flag. This means that your change will be merged once all checks on your PR have passed (ETA: 0-4 Hours). If this is not the intended behavior, feel free to use some of the other merge options in the wiki.
Please reach out to the PyTorch DevX Team with feedback or questions!

github-actions · 2022-08-25T02:24:22Z

Hey @jjsjann123.
You've committed this PR, but it does not have both a 'release notes: ...' and 'topics: ...' label. Please add one of each to the PR. The 'release notes: ...' label should represent the part of PyTorch that this PR changes (fx, autograd, distributed, etc) and the 'topics: ...' label should represent the kind of PR it is (not user facing, new feature, bug fix, perf improvement, etc). The list of valid labels can be found here for the 'release notes: ...' and here for the 'topics: ...'.
For changes that are 'topic: not user facing' there is no need for a release notes label.

Summary: Pull Request resolved: #83239 Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. double support in expression evaluator - bug fixes: 1. dropout fix - rework RNG to support broadcasted dropout (Fixes #82784) 2. expand fix - Patch expand+reduction, expand+view, rework view analysis and guard - scheduler: 1. manual transpose schedule example 2. WIP transpose scheduler Commits that's in this PR from the devel branch: ``` b7435af Transpose scheduler, step 1 (#1854) 8a45dbf Add an example on how to manually schedule transpose (#1889) 83dbf56 Patch dropout fix (#1898) 69d3519 Expand+Reduction, Expand+View support, rework View analysis and guards (#1883) 15091c4 Rework RNG to correctly support broadcasted dropout (#1888) aafe2d0 Make ExpressionEvaluator support Double (#1885) ``` RUN_TORCHBENCH: nvfuser Test Plan: Imported from OSS Reviewed By: qihqi Differential Revision: D38657074 Pulled By: davidberard98 fbshipit-source-id: b306eecb7df8e24c06b055fc9e1b11b8dcd1a0ea

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: a57984d Pull Request resolved: #83857

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

cherry-pick upstream CI fixes from pytorch#83067 & pytorch#83239

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: 20d277a Pull Request resolved: #83857

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser ghstack-source-id: fc150ae Pull Request resolved: #83857

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser Pull Request resolved: #83857 Approved by: https://github.com/xuzhao9

Summary: Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser Pull Request resolved: #83857 Approved by: https://github.com/xuzhao9 Test Plan: contbuild & OSS CI, see https://hud.pytorch.org/commit/pytorch/pytorch/71d99662a0d7f8a9ad68999c9a014b71591cbb68 Reviewed By: mehtanirav Differential Revision: D39172015 Pulled By: davidberard98 fbshipit-source-id: 208f7d8bf00937a459bb5abd5baf9461660d19c3

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. double support in expression evaluator - bug fixes: 1. dropout fix - rework RNG to support broadcasted dropout (Fixes #82784) 2. expand fix - Patch expand+reduction, expand+view, rework view analysis and guard - scheduler: 1. manual transpose schedule example 2. WIP transpose scheduler Commits that's in this PR from the devel branch: ``` b7435afcd22c917713c2f41a7237bc26e1183f14 Transpose scheduler, step 1 (#1854) 8a45dbf72034684eb8e18b1835b533e90b68f184 Add an example on how to manually schedule transpose (#1889) 83dbf56a9554b2efbd5416461d938fff477b0b27 Patch dropout fix (#1898) 69d3519a532250719b1aa8341b50e067b181b42d Expand+Reduction, Expand+View support, rework View analysis and guards (#1883) 15091c488e96343bdc49e3990acbf238a3b3da51 Rework RNG to correctly support broadcasted dropout (#1888) aafe2d048aaac596e503596a41303423619f3954 Make ExpressionEvaluator support Double (#1885) ``` RUN_TORCHBENCH: nvfuser Differential Revision: [D38657074](https://our.internmc.facebook.com/intern/diff/D38657074) Pull Request resolved: pytorch/pytorch#83239 Approved by: https://github.com/davidberard98

Syncing nvfuser devel branch to upstream master. https://github.com/csarofeen/pytorch/ Code changes includes: - codegen improvements: 1. double support in expression evaluator - bug fixes: 1. dropout fix - rework RNG to support broadcasted dropout (Fixes #82784) 2. expand fix - Patch expand+reduction, expand+view, rework view analysis and guard - scheduler: 1. manual transpose schedule example 2. WIP transpose scheduler Commits that's in this PR from the devel branch: ``` e31d69f Transpose scheduler, step 1 (#1854) 269b4af Add an example on how to manually schedule transpose (#1889) 6ecf4d8 Patch dropout fix (#1898) ecae77a Expand+Reduction, Expand+View support, rework View analysis and guards (#1883) 4e6c0c1 Rework RNG to correctly support broadcasted dropout (#1888) ae37ca1 Make ExpressionEvaluator support Double (#1885) ``` RUN_TORCHBENCH: nvfuser Differential Revision: [D38657074](https://our.internmc.facebook.com/intern/diff/D38657074) Pull Request resolved: pytorch/pytorch#83239 Approved by: https://github.com/davidberard98

facebook-github-bot added the cla signed label Aug 11, 2022

facebook-github-bot added the oncall: jit Add this issue/PR to JIT oncall triage queue label Aug 11, 2022

jjsjann123 mentioned this pull request Aug 11, 2022

[NVFuser] Upstream push 0811 #83237

Closed

jjsjann123 requested a review from davidberard98 August 11, 2022 08:24

pytorchbot added the open source label Aug 11, 2022

jjsjann123 mentioned this pull request Aug 11, 2022

expanded_extent_ == nullptr || isBroadcast() INTERNAL ASSERT FAILED #82858

Closed

jjsjann123 added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 11, 2022

davidberard98 added a commit that referenced this pull request Aug 24, 2022

Update base for Update on "add nvidia-smi to run_torchbench"

55a25de

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

davidberard98 added a commit that referenced this pull request Aug 24, 2022

Update on "add nvidia-smi to run_torchbench"

f3c36bc

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

davidberard98 added a commit that referenced this pull request Aug 24, 2022

Update base for Update on "add nvidia-smi to run_torchbench"

82d45de

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

davidberard98 added a commit that referenced this pull request Aug 24, 2022

Update on "add nvidia-smi to run_torchbench"

2edea3a

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

pytorchmergebot added the Merged label Aug 25, 2022

pytorchmergebot closed this in b21a6ff Aug 25, 2022

davidberard98 added a commit that referenced this pull request Aug 25, 2022

Update base for Update on "add nvidia-smi to run_torchbench"

d86912d

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

davidberard98 added a commit that referenced this pull request Aug 25, 2022

Update on "add nvidia-smi to run_torchbench"

da3b093

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

jjsjann123 deleted the gh/jjsjann123/3/head branch August 26, 2022 05:18

jjsjann123 added a commit to csarofeen/pytorch that referenced this pull request Aug 26, 2022

cherry-pick upstream fixes from pytorch#83239

3fe6175

jjsjann123 added a commit to csarofeen/pytorch that referenced this pull request Aug 26, 2022

Upstream cherry pick fixes 0811 (#1934)

88de85e

cherry-pick upstream CI fixes from pytorch#83067 & pytorch#83239

davidberard98 added a commit that referenced this pull request Aug 27, 2022

Update base for Update on "add nvidia-smi to run_torchbench"

9df3894

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

davidberard98 added a commit that referenced this pull request Aug 27, 2022

Update on "add nvidia-smi to run_torchbench"

3819dfe

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

pytorchmergebot pushed a commit that referenced this pull request Aug 30, 2022

Update on "add nvidia-smi to run_torchbench"

b60bfee

Seeing an OOM in #83239, this would help understand whether the issue is with the infra or with the test. RUN_TORCHBENCH: nvfuser [ghstack-poisoned]

[NVFuser] Upstream push 0811 #83239

[NVFuser] Upstream push 0811 #83239

Uh oh!

Conversation

jjsjann123 commented Aug 11, 2022 • edited by davidberard98 Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

facebook-github-bot commented Aug 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful links

❌ 1 New Failures, 8 Pending

🕵️ 1 new failure recognized by patterns

TorchBench CI (pytorch-linux-py3.7-cu102) / run-torchbench (1/1)

Uh oh!

davidberard98 commented Aug 11, 2022

Uh oh!

jjsjann123 commented Aug 11, 2022

Uh oh!

jjsjann123 commented Aug 11, 2022

Uh oh!

jjsjann123 commented Aug 11, 2022

Uh oh!

davidberard98 commented Aug 11, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjsjann123 commented Aug 11, 2022

Uh oh!

jjsjann123 commented Aug 11, 2022

Uh oh!

davidberard98 commented Aug 24, 2022

Uh oh!

facebook-github-bot commented Aug 25, 2022

Uh oh!

pytorchmergebot commented Aug 25, 2022

Uh oh!

github-actions bot commented Aug 25, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

jjsjann123 commented Aug 11, 2022 •

edited by davidberard98

Loading

facebook-github-bot commented Aug 11, 2022 •

edited

Loading

davidberard98 commented Aug 11, 2022 •

edited

Loading