Move non inductor workflows cuda 12.6->cuda 12.8 #155234

atalman · 2025-06-05T16:28:46Z

Move non inductor workflows cuda 12.6->cuda 12.8

pytorch-bot · 2025-06-05T16:28:50Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155234

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 4 New Failures, 1 Cancelled Job, 7 Unrelated Failures

As of commit bd33e48 with merge base 8892b78 ():

NEW FAILURES - The following jobs have failed:

Build libtorch docker images / libtorch-cxx11-builder:cuda12.9 (gh)
Final attempt failed. Child_process exited with error code 1
pull / linux-jammy-py3.9-clang12 / test (dynamo_wrapped, 2, 3, lf.linux.2xlarge) (gh)
'test/test_reductions.py::TestReductionsCPU::test_sum_all_cpu_float64'
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 3, 10, linux.s390x) (gh)
dynamo/test_modules 1/1 failed!
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 5, 10, linux.s390x) (gh)
test_testing.py::TestImports::test_circular_dependencies

CANCELLED JOB - The following job was cancelled. Please retry:

Limited CI on H100 / linux-jammy-cuda12.8-py3.10-gcc11-sm90 / build (gh)

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

periodic / linux-jammy-cuda12.8-py3-gcc11-slow-gradcheck / test (default, 4, 8, lf.linux.g5.4xlarge.nvidia.gpu, module:slowgradcheck) (gh) (similar failure)
inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda
pull / cuda12.8-py3.10-gcc9-sm75 / test (pr_time_benchmarks, 1, 1, linux.g4dn.metal.nvidia.gpu) (gh) (similar failure)
MISSING REGRESSION TEST
pull / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, lf.linux.2xlarge) (gh) (similar failure)
exir/backend/test/test_to_backend_multi_method.py::TestToBackendMultiMethod::test_multi_method_end_to_end
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 2, 10, linux.s390x) (gh) (similar failure)
test_dynamo_pystub_suggestion
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 6, 10, linux.s390x) (gh) (similar failure)
test_public_bindings.py::TestPublicBindings::test_modules_can_be_imported
s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / test (default, 9, 10, linux.s390x) (gh) (similar failure)
inductor/test_config.py::TestInductorConfig::test_invalid_backend
slow / linux-jammy-cuda12.8-py3.10-gcc11-sm86 / test (slow, 2, 3, linux.g5.4xlarge.nvidia.gpu) (gh) (similar failure)
test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_conv1d_cuda_float32

This comment was automatically generated by Dr. CI and updates every 15 minutes.

atalman · 2025-06-09T20:45:47Z

Observing these failures on trunk:
test_nn.py::TestNNDeviceTypeCUDA::test_variable_sequence_cuda_float32 GH job link HUD commit link

inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda GH job link HUD commit link

test_expanded_weights.py::TestExpandedWeightModuleCUDA::test_module_nn_GRU_eval_mode_cuda_float32 GH job link HUD commit link

test_ops.py::TestCommonCUDA::test_compare_cpu_nn_functional_conv1d_cuda_float32 GH job link HUD commit link

Looks like errors on this issue are pre-existing on trunk this Friday

atalman · 2025-06-09T22:02:32Z

@pytorchmergebot merge -f "lint is green all other tests have been already run"

pytorchmergebot · 2025-06-09T22:04:09Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

clee2000 · 2025-06-10T03:32:52Z

@pytorchbot revert -m "causing a bunch of tests to fail? ex test_nn.py::TestNNDeviceTypeCUDA::test_variable_sequence_cuda_float32 GH job link HUD commit link, some of the failures attributed to broken trunk on friday seem real?" -c ignoredsignal

pytorchmergebot · 2025-06-10T03:34:29Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

This reverts commit ede6ead. Reverted #155234 on behalf of https://github.com/clee2000 due to causing a bunch of tests to fail? ex test_nn.py::TestNNDeviceTypeCUDA::test_variable_sequence_cuda_float32 [GH job link](https://github.com/pytorch/pytorch/actions/runs/15545607752/job/43773157441) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/ede6ead8cd8e925cb093f2b3016342e645bd728d), some of the failures attributed to broken trunk on friday seem real? ([comment](#155234 (comment)))

pytorchmergebot · 2025-06-10T03:34:39Z

@atalman your PR has been successfully reverted.

atalman · 2025-06-11T14:53:17Z

This are existing issues:

test_foreach.py::TestForeachCUDA::test_pointwise_op_with_tensor_of_scalarlist_overload__foreach_addcdiv_is_fastpath_True_cuda_complex128 GH job link HUD commit link

atalman · 2025-06-12T12:39:49Z

@pytorchmergebot merge -f "all previously failed workflows are passing now"

pytorchmergebot · 2025-06-12T12:41:41Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda: Added in #150059 Fails in debug mode [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44706020831) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda: [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44707119967) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) started failing after moving to new cuda version #155234 I'll ping people if this gets merged Pull Request resolved: #156731 Approved by: https://github.com/huydhn

test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda: Added in pytorch#150059 Fails in debug mode [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44706020831) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda: [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44707119967) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) started failing after moving to new cuda version pytorch#155234 I'll ping people if this gets merged Pull Request resolved: pytorch#156731 Approved by: https://github.com/huydhn

test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda: Added in #150059 Fails in debug mode [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44706020831) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda: [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44707119967) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) started failing after moving to new cuda version #155234 I'll ping people if this gets merged Pull Request resolved: #156731 Approved by: https://github.com/huydhn (cherry picked from commit 2ff3280)

[ez] Disable some failing periodic tests (#156731) test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda: Added in #150059 Fails in debug mode [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44706020831) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda: [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44707119967) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) started failing after moving to new cuda version #155234 I'll ping people if this gets merged Pull Request resolved: #156731 Approved by: https://github.com/huydhn (cherry picked from commit 2ff3280) Co-authored-by: Catherine Lee <[email protected]>

pytorch-bot bot added the topic: not user facing topic category label Jun 5, 2025

atalman added ci-no-td Do not run TD on this PR ciflow/pull ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR ciflow/slow labels Jun 5, 2025

atalman marked this pull request as ready for review June 5, 2025 16:36

atalman requested review from a team and jeffdaily as code owners June 5, 2025 16:36

Skylion007 approved these changes Jun 5, 2025

View reviewed changes

zxiiro approved these changes Jun 5, 2025

View reviewed changes

cyyever approved these changes Jun 6, 2025

View reviewed changes

atalman force-pushed the cuda12_8_migration branch from 3e6a325 to 875345a Compare June 6, 2025 18:40

malfet approved these changes Jun 6, 2025

View reviewed changes

atalman force-pushed the cuda12_8_migration branch from 875345a to 24d5cb3 Compare June 9, 2025 20:55

pytorchmergebot added the merging label Jun 9, 2025

pytorchmergebot closed this in ede6ead Jun 9, 2025

pytorchmergebot added Merged and removed merging labels Jun 9, 2025

pytorchmergebot added the Reverted label Jun 10, 2025

pytorchmergebot reopened this Jun 10, 2025

atalman mentioned this pull request Jun 10, 2025

CUDA 12.6->12.8 slow and periodic failures #155607

Open

atalman force-pushed the cuda12_8_migration branch from dd43110 to 39272f2 Compare June 10, 2025 22:39

atalman mentioned this pull request Jun 11, 2025

foreach CUDA tests flaky on CUDA 12.6+ due to flaky profiler results #148681

Open

atalman added 8 commits June 11, 2025 09:20

fixes

384d330

fix

7ad86d7

fixes

cbb1b92

lint

586d74c

fixes

7621422

lint

eb11dbd

more

0f33c17

for_each

14021a7

atalman mentioned this pull request Jun 11, 2025

[BE]: Update cudnn to 9.10.2.21 #155576

Closed

atalman added the keep-going Don't stop on first failure, keep running tests until the end label Jun 11, 2025

fix

bd33e48

pytorchmergebot added the merging label Jun 12, 2025

pytorchmergebot closed this in c199a4d Jun 12, 2025

pytorchmergebot removed the merging label Jun 12, 2025

atalman mentioned this pull request Jun 12, 2025

DISABLED test_compare_cpu_nn_functional_conv1d_cuda_float32 (__main__.TestCommonCUDA) #155822

Open

clee2000 mentioned this pull request Jun 24, 2025

[ez] Disable some failing periodic tests #156731

Closed

pytorchbot mentioned this pull request Jul 3, 2025

[ez] Disable some failing periodic tests #157560

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Move non inductor workflows cuda 12.6->cuda 12.8 #155234

Move non inductor workflows cuda 12.6->cuda 12.8 #155234

Uh oh!

atalman commented Jun 5, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Jun 5, 2025 •

edited

Loading

Uh oh!

atalman commented Jun 9, 2025 •

edited

Loading

Uh oh!

atalman commented Jun 9, 2025

Uh oh!

pytorchmergebot commented Jun 9, 2025

Uh oh!

clee2000 commented Jun 10, 2025

Uh oh!

pytorchmergebot commented Jun 10, 2025

Uh oh!

pytorchmergebot commented Jun 10, 2025

Uh oh!

atalman commented Jun 11, 2025

Uh oh!

atalman commented Jun 12, 2025

Uh oh!

pytorchmergebot commented Jun 12, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

Move non inductor workflows cuda 12.6->cuda 12.8 #155234

Move non inductor workflows cuda 12.6->cuda 12.8 #155234

Uh oh!

Conversation

atalman commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/155234

❌ 4 New Failures, 1 Cancelled Job, 7 Unrelated Failures

Uh oh!

atalman commented Jun 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

atalman commented Jun 9, 2025

Uh oh!

pytorchmergebot commented Jun 9, 2025

Merge started

Uh oh!

clee2000 commented Jun 10, 2025

Uh oh!

pytorchmergebot commented Jun 10, 2025

Uh oh!

pytorchmergebot commented Jun 10, 2025

Uh oh!

atalman commented Jun 11, 2025

Uh oh!

atalman commented Jun 12, 2025

Uh oh!

pytorchmergebot commented Jun 12, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

atalman commented Jun 5, 2025 •

edited

Loading

pytorch-bot bot commented Jun 5, 2025 •

edited

Loading

atalman commented Jun 9, 2025 •

edited

Loading