🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/162351

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit f8b1b0c with merge base 991e3d0 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

need to update it more places

Change looks good, would need to upload the NCCL packages to download.pytorch.org (eg https://download.pytorch.org/whl/nightly/nvidia-nccl-cu13/) and get signals on ciflow/binaries before we merge
cc @atalman

need to update it more places

@ezyang, ah you meant the workflows. Fixed those.

@pytorchbot merge

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

This update breaks all nccl ops on H100 with "no kernel image available" on cuda 12.9. Note we cannot use 12.8 for reasons, and cannot use 13.0 because our driver version is insufficient, so 12.9 is the only option

@pytorchbot revert -m "Broke H100 on 12.9" -c nosignal

Reverting out of caution as H100 is very widely used across our userbase.
@ptrblck can someone on your end take a look?

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

Reverting PR 162351 failed

Reason: Command git -C /home/runner/work/pytorch/pytorch revert --no-edit 5504a06e014d96e5d226f08d403ae1117edf343e returned non-zero exit code 1

CONFLICT (modify/delete): .github/workflows/generated-linux-binary-manywheel-main.yml deleted in HEAD and modified in parent of 5504a06e014 ([BE]: Update NCCL to 2.28.3 (#162351)).  Version parent of 5504a06e014 ([BE]: Update NCCL to 2.28.3 (#162351)) of .github/workflows/generated-linux-binary-manywheel-main.yml left in tree.
error: could not revert 5504a06e014... [BE]: Update NCCL to 2.28.3 (#162351)
hint: After resolving the conflicts, mark them with
hint: "git add/rm <pathspec>", then run
hint: "git revert --continue".
hint: You can instead skip this commit with "git revert --skip".
hint: To abort and get back to the state before "git revert",
hint: run "git revert --abort".
hint: Disable this message with "git config set advice.mergeConflict false"

Hi @albanD @ngimel, would you mind sharing some links to the H100 failures so that we could take a look? Thanks.

@albanD A quick check does not show missing architectures. We'll follow up on Slack to get a repro

Discussed offline, it's a nccl back where it incorrectly handles error propagation for static linking situation. Static linking is used for local source builds.