Align CPU behavior with CUDA for `ConvTranspose` when `out_channels=0` #142859

chunyuan-w · 2024-12-11T09:16:05Z

Stack from ghstack (oldest at bottom):

-> Align CPU behavior with CUDA for ConvTranspose when out_channels=0 #142859

Fixes #142466.
Remove the weight.numel() != 0 check to align the behavior with CUDA for ConvTranspose when out_channels=0. After removing this check, the existing code is already able to give an empty output in such case.

Test plan:

python -u test/nn/test_convolution.py -k test_ConvTranspose_output_channels_0_cpu_float32
python -u test/nn/test_convolution.py -k test_ConvTranspose_output_channels_0_cuda_float32

[ghstack-poisoned]

pytorch-bot · 2024-12-11T09:16:09Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/142859

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (9 Unrelated Failures)

As of commit f090b3c with merge base f56c638 ():

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

pull / linux-focal-cuda12.4-py3.10-gcc9 / test (default, 4, 5, lf.linux.4xlarge.nvidia.gpu) (gh) (disabled by #131303 but the issue was closed recently and a rebase is needed to make it pass)
dynamo/test_exc.py::ExcTests::test_trigger_bisect_on_error
pull / linux-focal-cuda12.4-py3.10-gcc9-sm89 / test (default, 5, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (disabled by #131303 but the issue was closed recently and a rebase is needed to make it pass)
dynamo/test_exc.py::ExcTests::test_trigger_bisect_on_error
pull / linux-focal-py3.13-clang10 / test (default, 4, 5, lf.linux.4xlarge) (gh) (disabled by #131303 but the issue was closed recently and a rebase is needed to make it pass)
dynamo/test_exc.py::ExcTests::test_trigger_bisect_on_error
pull / linux-focal-py3.9-clang10 / test (default, 3, 5, lf.linux.4xlarge) (gh) (disabled by #131303 but the issue was closed recently and a rebase is needed to make it pass)
dynamo/test_exc.py::ExcTests::test_trigger_bisect_on_error
pull / linux-jammy-py3.10-clang15-asan / test (default, 2, 6, lf.linux.4xlarge) (gh) (disabled by #131303 but the issue was closed recently and a rebase is needed to make it pass)
dynamo/test_exc.py::ExcTests::test_trigger_bisect_on_error
pull / linux-jammy-py3.9-gcc11 / test (default, 3, 5, lf.linux.2xlarge) (gh) (disabled by #131303 but the issue was closed recently and a rebase is needed to make it pass)
dynamo/test_exc.py::ExcTests::test_trigger_bisect_on_error
rocm / linux-focal-rocm6.3-py3.10 / test (default, 1, 6, linux.rocm.gpu.2) (gh) (similar failure)
nn/test_convolution.py::TestConvolutionNNDeviceTypeCUDA::test_ConvTranspose_output_channels_0_cuda_float32
rocm / linux-focal-rocm6.3-py3.10 / test (default, 2, 6, linux.rocm.gpu.2) (gh) (disabled by #131303 but the issue was closed recently and a rebase is needed to make it pass)
dynamo/test_exc.py::ExcTests::test_trigger_bisect_on_error
trunk / macos-py3-arm64 / test (default, 1, 3, macos-m1-stable) (gh) (disabled by #131303 but the issue was closed recently and a rebase is needed to make it pass)
dynamo/test_exc.py::ExcTests::test_trigger_bisect_on_error

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ghstack-source-id: e5c5093 Pull Request resolved: #142859

chunyuan-w · 2024-12-20T08:48:39Z

Hi @mingfeima could you help review this PR when you get time?

shaoyuyoung · 2025-01-18T03:00:00Z

Hi @mingfeima, if you are very busy, can anyone else review this PR?

mingfeima · 2025-01-20T02:28:32Z

test/nn/test_convolution.py

                dtype,
            )

+    def test_ConvTranspose_output_channels_0(self):


is this test going to be launched for both CPU and CUDA device?

Updated the test to be launched for both CPU and CUDA device.

mingfeima · 2025-01-20T02:29:41Z

Generally LGTM:) just make sure that the test is enabled for both CUDA and CPU device.

[ghstack-poisoned]

ghstack-source-id: f9a8075 Pull Request resolved: #142859

chunyuan-w · 2025-01-20T05:59:53Z

Generally LGTM:) just make sure that the test is enabled for both CUDA and CPU device.

I moved the test to TestConvolutionNNDeviceType and it will be enabled for both CUDA and CPU:

python -u test/nn/test_convolution.py -k test_ConvTranspose_output_channels_0_cpu_float32
python -u test/nn/test_convolution.py -k test_ConvTranspose_output_channels_0_cuda_float32

One thing to note is that it fails on MPS in the assertion here. I guess similar fix is needed on MPS side to align with CUDA as well.

pytorch/aten/src/ATen/native/mps/OperationUtils.mm

Line 558 in 972d4a1

TORCH_INTERNAL_ASSERT([srcBuf length] > 0, "Placeholder tensor is empty!");

shaoyuyoung · 2025-01-20T06:08:18Z

One thing to note is that it fails on MPS in the assertion here. I guess similar fix is needed on MPS side to align with CUDA as well.

maybe @malfet is interested in this?

[ghstack-poisoned]

ghstack-source-id: ff3e207 Pull Request resolved: #142859

chunyuan-w · 2025-01-24T05:26:42Z

I verified that using the main branch (without the cpp code change in this PR), the case test_ConvTranspose_output_channels_0 will still fail on XLA with the same error message. Thus I added a @skipXLA decorator for this UT since the failure is not brought by this PR. Similar fix is needed on the XLA device as well.

The CI is all green now.

chunyuan-w · 2025-01-26T01:49:12Z

@pytorchbot merge

pytorchmergebot · 2025-01-26T01:50:59Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

malfet · 2025-01-26T17:47:32Z

@pytorchbot revert -m "It broke ROCM tests again, see https://hud.pytorch.org/hud/pytorch/pytorch/5cd2b34e821c293909fe3f6a958767b9c535c094/1?per_page=50&name_filter=rocm&mergeLF=true" -c ignoredsignal

pytorchmergebot · 2025-01-26T17:48:56Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot · 2025-01-26T17:49:07Z

@chunyuan-w your PR has been successfully reverted.

…annels=0` (#142859)" This reverts commit cb814c0. Reverted #142859 on behalf of https://github.com/malfet due to It broke ROCM tests again, see https://hud.pytorch.org/hud/pytorch/pytorch/5cd2b34e821c293909fe3f6a958767b9c535c094/1?per_page=50&name_filter=rocm&mergeLF=true ([comment](#142859 (comment)))

malfet

Rather than adding more and more "skipIfFoo", imo the right solution is to move the check up thru the stack (or turn the op into structured one)

pytorch#142859) Fixes pytorch#142466. Remove the `weight.numel() != 0` check to align the behavior with CUDA for `ConvTranspose` when `out_channels=0`. After removing this check, the existing code is already able to give an empty output in such case. Test plan: ``` python -u test/nn/test_convolution.py -k test_ConvTranspose_output_channels_0_cpu_float32 python -u test/nn/test_convolution.py -k test_ConvTranspose_output_channels_0_cuda_float32 ``` Pull Request resolved: pytorch#142859 Approved by: https://github.com/mingfeima, https://github.com/malfet

…annels=0` (pytorch#142859)" This reverts commit cb814c0. Reverted pytorch#142859 on behalf of https://github.com/malfet due to It broke ROCM tests again, see https://hud.pytorch.org/hud/pytorch/pytorch/5cd2b34e821c293909fe3f6a958767b9c535c094/1?per_page=50&name_filter=rocm&mergeLF=true ([comment](pytorch#142859 (comment)))

jeffdaily · 2025-01-28T18:48:27Z

test/nn/test_convolution.py

+    # aten/src/ATen/native/mps/OperationUtils.mm: TORCH_INTERNAL_ASSERT([srcBuf length] > 0, "Placeholder tensor is empty!"); on MPS
+    @expectedFailureMPS
+    @skipXLA
+    def test_ConvTranspose_output_channels_0(self, device, dtype):


Suggested change

def test_ConvTranspose_output_channels_0(self, device, dtype):

@skipCUDAIfRocm

def test_ConvTranspose_output_channels_0(self, device, dtype):

I tried a few things to get this to work on ROCm but it's going to take longer to dig into it. I updated the use_miopen check to avoid miopen when weight.size(1) == 0 but it falls through to hipblas and then fails there instead.

github-actions · 2025-03-29T19:34:03Z

Looks like this PR hasn't been updated in a while so we're going to go ahead and mark this as Stale.
Feel free to remove the Stale label if you feel this was a mistake.
If you are unable to remove the Stale label please contact a maintainer in order to do so.
If you want the bot to never mark this PR stale again, add the no-stale label.
Stale pull requests will automatically be closed after 30 days of inactivity.

Update

6260e5f

[ghstack-poisoned]

pytorchbot added the open source label Dec 11, 2024

chunyuan-w added a commit that referenced this pull request Dec 11, 2024

Align CPU behavior with CUDA for ConvTranspose when out_channels=0

20ec751

ghstack-source-id: e5c5093 Pull Request resolved: #142859

chunyuan-w marked this pull request as draft December 11, 2024 09:18

chunyuan-w added topic: not user facing topic category ciflow/trunk Trigger trunk jobs on your pull request labels Dec 11, 2024

chunyuan-w marked this pull request as ready for review December 12, 2024 05:17

chunyuan-w requested a review from mingfeima December 12, 2024 10:16

chunyuan-w mentioned this pull request Dec 16, 2024

[Device] ConvTranspose bahaves differently on CPU and CUDA when out_channels=0 #142466

Closed

mingfeima reviewed Jan 20, 2025

View reviewed changes

mingfeima approved these changes Jan 20, 2025

View reviewed changes

Update

4b94c78

[ghstack-poisoned]

pytorch-bot bot had a problem deploying to upload-benchmark-results January 20, 2025 03:32 Error

Update

b0f820b

[ghstack-poisoned]

chunyuan-w added a commit that referenced this pull request Jan 20, 2025

Align CPU behavior with CUDA for ConvTranspose when out_channels=0

8aa710a

ghstack-source-id: f9a8075 Pull Request resolved: #142859

pytorch-bot bot temporarily deployed to upload-benchmark-results January 20, 2025 04:18 Inactive

Update

fed8805

[ghstack-poisoned]

chunyuan-w added a commit that referenced this pull request Jan 21, 2025

Align CPU behavior with CUDA for ConvTranspose when out_channels=0

41a913a

ghstack-source-id: ff3e207 Pull Request resolved: #142859

pytorch-bot bot temporarily deployed to upload-benchmark-results January 21, 2025 03:12 Inactive

pytorchmergebot added the merging label Jan 26, 2025

pytorchmergebot closed this in cb814c0 Jan 26, 2025

pytorchmergebot removed the merging label Jan 26, 2025

pytorchmergebot reopened this Jan 26, 2025

pytorch-bot bot temporarily deployed to upload-benchmark-results January 26, 2025 18:20 Inactive

malfet requested changes Jan 26, 2025

View reviewed changes

jeffdaily added the ciflow/rocm Trigger "default" config CI on ROCm label Jan 27, 2025

pytorch-bot bot had a problem deploying to upload-benchmark-results January 27, 2025 22:22 Failure

pytorch-bot bot temporarily deployed to upload-benchmark-results January 27, 2025 22:22 Inactive

jeffdaily reviewed Jan 28, 2025

View reviewed changes

github-actions bot added the Stale label Mar 29, 2025

github-actions bot closed this Apr 28, 2025

github-actions bot deleted the gh/chunyuan-w/1/head branch June 7, 2025 02:16

	def test_ConvTranspose_output_channels_0(self, device, dtype):
	@skipCUDAIfRocm
	def test_ConvTranspose_output_channels_0(self, device, dtype):

Align CPU behavior with CUDA for ConvTranspose when out_channels=0 #142859

Align CPU behavior with CUDA for ConvTranspose when out_channels=0 #142859

Conversation

chunyuan-w commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Dec 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/142859

✅ You can merge normally! (9 Unrelated Failures)

Uh oh!

chunyuan-w commented Dec 20, 2024

Uh oh!

shaoyuyoung commented Jan 18, 2025

Uh oh!

mingfeima Jan 20, 2025

Choose a reason for hiding this comment

Uh oh!

chunyuan-w Jan 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mingfeima commented Jan 20, 2025

Uh oh!

chunyuan-w commented Jan 20, 2025

Uh oh!

shaoyuyoung commented Jan 20, 2025

Uh oh!

chunyuan-w commented Jan 24, 2025

Uh oh!

chunyuan-w commented Jan 26, 2025

Uh oh!

pytorchmergebot commented Jan 26, 2025

Merge started

Uh oh!

malfet commented Jan 26, 2025

Uh oh!

pytorchmergebot commented Jan 26, 2025

Uh oh!

pytorchmergebot commented Jan 26, 2025

Uh oh!

malfet left a comment

Choose a reason for hiding this comment

Uh oh!

jeffdaily Jan 28, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Mar 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

Align CPU behavior with CUDA for `ConvTranspose` when `out_channels=0` #142859

Align CPU behavior with CUDA for `ConvTranspose` when `out_channels=0` #142859

chunyuan-w commented Dec 11, 2024 •

edited

Loading

pytorch-bot bot commented Dec 11, 2024 •

edited

Loading

chunyuan-w Jan 20, 2025 •

edited

Loading