[CI][Distributed] Not to test distributed_test.py with UCC #137932
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/137932
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (2 Unrelated Failures)As of commit 7e08190 with merge base c37bb49 ( BROKEN TRUNK - The following jobs failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
| "UCC_TL_UCP_TUNE": "cuda:0", # don't use UCP TL on CUDA as it is not well supported | ||
| "UCC_EC_CUDA_USE_COOPERATIVE_LAUNCH": "n", # CI nodes (M60) fail if it is on | ||
| } | ||
| # Test with UCC backend is deprecated. |
There was a problem hiding this comment.
Can we remove it completely?
There was a problem hiding this comment.
Not for now, as it is not decided.
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 1 jobs have failed, first few of them are: trunk / linux-focal-cuda11.8-py3.10-gcc9-experimental-split-build-test / test (distributed, 1, 3, lf.linux.g4dn.12xlarge.nvidia.gpu) Details for Dev Infra teamRaised by workflow job |
malfet
left a comment
malfet
left a comment
There was a problem hiding this comment.
Do you want to post a release note somewhere saying that UCC is kind of in maintenance mode?
|
@pytorchbot merge -f "TestRuntimeEstimator.test_transformer_runtime should be already disabled; not sure why it is blocking the merge; also not sure why the trunk job is run on g4dn instance which impacts the runtime estimation." |
Merge startedYour change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Stack from ghstack (oldest at bottom):
Some UCC tests became unstable recently, with or without the M60 to T4 upgrade.
See for example: #137855 (without upgrade), #137161 (with upgrade).
So I am extracting the disablement from #137161 here.
Failure signature:
Earlier discussed here:
https://github.com/pytorch/pytorch/pull/137161/files#r1797353294
Cc: @Aidyn-A @eqy