[CI] Disable some tests that are failing in periodic #150059

clee2000 · 2025-03-26T22:29:59Z

Disabling some tests to restore periodic

nogpu avx512 timeout:
https://hud.pytorch.org/pytorch/pytorch/commit/59f14d19aea4091c65cca2417c509e3dbf60c0ed#38492953496-box

profiler failure: https://hud.pytorch.org/pytorch/pytorch/commit/7ae0ce6360b6e4f944906502d20da24c04debee5#38461255009-box

test_accelerator failure:
https://hud.pytorch.org/pytorch/pytorch/commit/87bfd66c3c7061db6d36d8daa62f08f507f90e39#39476723746-box
origin: 146098

test_overrides failure:
https://hud.pytorch.org/pytorch/pytorch/commit/bf752c36da08871d76a66fd52ad09f87e66fc770#39484562957-box
origin: 146098

inductor cpu repro:
https://hud.pytorch.org/pytorch/pytorch/commit/bb9c4260249ea0c57e87395eff5271fb479efb6a#38447525659-box

functorch eager transforms:
https://hud.pytorch.org/pytorch/pytorch/commit/8f858e226ba81fde41d39aa34f1fd4cb4a4ecc51#39488068620-box
https://hud.pytorch.org/pytorch/pytorch/commit/f2cea01f7195e59abd154b5551213ee3e38fa40d#39555064878
https://hud.pytorch.org/pytorch/pytorch/commit/b5281a4a1806c978e34c5cfa0befd298e469b7fd#39599355600
either 148288 or 148261?

https://hud.pytorch.org/hud/pytorch/pytorch/2ec9aceaeb77176c4bdeb2d008a34cba0cd57e3c/1?per_page=100&name_filter=periodic&mergeLF=true

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov

pytorch-bot · 2025-03-26T22:30:03Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/150059

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 76 Pending

As of commit b069100 with merge base 85e4e51 ():

NEW FAILURE - The following job has failed:

s390x-periodic / linux-manylinux-2_28-py3-cpu-s390x / build (gh)
sccache: error: couldn't connect to server

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

ZainRizvi

Prob best to share the full list of failures on treehuggers as well (in addition to merging this PR). Someone might decide they actually care about fixing some of these test.

ZainRizvi · 2025-03-28T16:42:32Z

test/test_overrides.py

+if os.getenv("ATEN_CPU_CAPABILITY") in ("default", "avx2"):
+    # This test is not supported on ARM
+    print("Skipping due to failing when cuda build runs on non cuda machine, see #150059 for example")
+    sys.exit()


Have you tried using pytest.skip(allow_module_level=True) (docs)? That might let you properly skip all tests (and have them be marked as skipped) instead of this silent skip

I just tried this on a dummy test file, unfortunately it just says 1 skipped for the entire file and I can't seem to get it to print the skip reason either

ZainRizvi · 2025-03-28T16:45:45Z

test/functorch/test_eager_transforms.py


    @dtypes(torch.float)
+    @unittest.skipIf(
+        TEST_CUDA_MEM_LEAK_CHECK, "Leaking memory, see #150059 for example"


Could you include the full link here for the benefit of future devs?

atalman

lgtm. Thank you very much for this

clee2000 · 2025-03-28T20:29:48Z

@pytorchbot merge -f "previous run passed, most recent commit mostly just comment changes + 1 more test"

pytorchmergebot · 2025-03-28T20:31:18Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

atalman · 2025-03-28T20:44:26Z

@pytorchbot cherry-pick --onto release/2.7 --c regression

pytorchbot · 2025-03-28T20:48:44Z

Cherry picking #150059

Command git -C /home/runner/work/pytorch/pytorch cherry-pick -x 9092dd2e828c4af67620e36b6d985b8d4ab2997b returned non-zero exit code 1

Auto-merging .github/workflows/periodic.yml
Auto-merging test/inductor/test_cpu_repro.py
CONFLICT (content): Merge conflict in test/inductor/test_cpu_repro.py
Auto-merging test/profiler/test_profiler.py
error: could not apply 9092dd2e828... [CI] Disable some tests that are failing in periodic (#150059)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git cherry-pick --continue".
hint: You can instead skip this commit with "git cherry-pick --skip".
hint: To abort and get back to the state before "git cherry-pick",
hint: run "git cherry-pick --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Details for Dev Infra team

Raised by workflow job

test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda: Added in #150059 Fails in debug mode [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44706020831) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda: [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44707119967) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) started failing after moving to new cuda version #155234 I'll ping people if this gets merged Pull Request resolved: #156731 Approved by: https://github.com/huydhn

test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda: Added in pytorch#150059 Fails in debug mode [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44706020831) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda: [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44707119967) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) started failing after moving to new cuda version pytorch#155234 I'll ping people if this gets merged Pull Request resolved: pytorch#156731 Approved by: https://github.com/huydhn

test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda: Added in #150059 Fails in debug mode [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44706020831) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda: [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44707119967) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) started failing after moving to new cuda version #155234 I'll ping people if this gets merged Pull Request resolved: #156731 Approved by: https://github.com/huydhn (cherry picked from commit 2ff3280)

[ez] Disable some failing periodic tests (#156731) test_torch.py::TestTorchDeviceTypeCUDA::test_storage_use_count_cuda: Added in #150059 Fails in debug mode [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44706020831) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) inductor/test_inductor_freezing.py::FreezingGpuTests::test_cpp_wrapper_cuda: [GH job link](https://github.com/pytorch/pytorch/actions/runs/15856606665/job/44707119967) [HUD commit link](https://hud.pytorch.org/pytorch/pytorch/commit/4491326fb0c0e67eca1598ae33c41cdfced2cd33) started failing after moving to new cuda version #155234 I'll ping people if this gets merged Pull Request resolved: #156731 Approved by: https://github.com/huydhn (cherry picked from commit 2ff3280) Co-authored-by: Catherine Lee <[email protected]>

Flamefire · 2025-10-30T09:07:57Z

test/test_overrides.py

 Tensor = torch.Tensor

+if os.getenv("ATEN_CPU_CAPABILITY") in ("default", "avx2"):
+    # This test is not supported on ARM


@clee2000 Just came across this: The comment seems wrongly pasted from somewhere else.

I assume this test is supposed to be skipped when no GPU is available. So wouldn't checking TEST_CUDA be better?

pytorch-bot bot added ciflow/inductor module: inductor topic: not user facing topic category labels Mar 26, 2025

clee2000 added the ciflow/periodic Trigger jobs ran periodically on master (periodic.yml) on the PR label Mar 26, 2025

clee2000 changed the title ~~[CI] Disable some tests that are failing in periodic, add another shard for nogpu_AVX512~~ [CI] Disable some tests that are failing in periodic Mar 26, 2025

clee2000 closed this Mar 27, 2025

clee2000 reopened this Mar 27, 2025

clee2000 added the keep-going Don't stop on first failure, keep running tests until the end label Mar 27, 2025

tc

aa32624

clee2000 force-pushed the csl/disable_some_periodic_tests branch from 4095f65 to aa32624 Compare March 27, 2025 17:41

clee2000 added 3 commits March 27, 2025 11:29

tc

f2cea01

tc

445cb38

tc

b5281a4

clee2000 marked this pull request as ready for review March 28, 2025 16:08

clee2000 requested a review from a team as a code owner March 28, 2025 16:08

ZainRizvi approved these changes Mar 28, 2025

View reviewed changes

atalman approved these changes Mar 28, 2025

View reviewed changes

malfet approved these changes Mar 28, 2025

View reviewed changes

tc

b069100

pytorchmergebot added the merging label Mar 28, 2025

pytorchmergebot closed this in 9092dd2 Mar 28, 2025

pytorchmergebot added Merged and removed merging labels Mar 28, 2025

This was referenced Mar 31, 2025

[cherry-pick] [CI] Disable some tests that are failing in periodic #150059 #150327

Merged

[v.2.7.0] Release Tracker #149044

Closed

This was referenced Apr 2, 2025

Revert "[fx] Move Node._prepend/Node._remove_from_list to C++ (#148261)" #150498

Closed

Revert "[fx] Move map_aggregate to C++ (#148243)" #150500

Closed

github-actions bot deleted the csl/disable_some_periodic_tests branch May 2, 2025 02:17

clee2000 mentioned this pull request Jun 24, 2025

[ez] Disable some failing periodic tests #156731

Closed

Flamefire reviewed Oct 30, 2025

View reviewed changes

Flamefire mentioned this pull request Oct 30, 2025

Fix skip condition in test_overrides #166625

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CI] Disable some tests that are failing in periodic #150059

[CI] Disable some tests that are failing in periodic #150059

Uh oh!

clee2000 commented Mar 26, 2025 •

edited

Loading

Uh oh!

pytorch-bot bot commented Mar 26, 2025 •

edited

Loading

Uh oh!

ZainRizvi left a comment

Uh oh!

ZainRizvi Mar 28, 2025

Uh oh!

clee2000 Mar 28, 2025

Uh oh!

ZainRizvi Mar 28, 2025

Uh oh!

atalman left a comment

Uh oh!

clee2000 commented Mar 28, 2025

Uh oh!

pytorchmergebot commented Mar 28, 2025

Uh oh!

atalman commented Mar 28, 2025

Uh oh!

pytorchbot commented Mar 28, 2025

Uh oh!

Flamefire Oct 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

[CI] Disable some tests that are failing in periodic #150059

[CI] Disable some tests that are failing in periodic #150059

Uh oh!

Conversation

clee2000 commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/150059

❌ 1 New Failure, 76 Pending

Uh oh!

ZainRizvi left a comment

Choose a reason for hiding this comment

Uh oh!

ZainRizvi Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

clee2000 Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

ZainRizvi Mar 28, 2025

Choose a reason for hiding this comment

Uh oh!

atalman left a comment

Choose a reason for hiding this comment

Uh oh!

clee2000 commented Mar 28, 2025

Uh oh!

pytorchmergebot commented Mar 28, 2025

Merge started

Uh oh!

atalman commented Mar 28, 2025

Uh oh!

pytorchbot commented Mar 28, 2025

Cherry picking #150059

Uh oh!

Flamefire Oct 30, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

clee2000 commented Mar 26, 2025 •

edited

Loading

pytorch-bot bot commented Mar 26, 2025 •

edited

Loading