[triton] Update pin for PyTorch 2.6/Triton 3.2 #139206

bertmaher · 2024-10-29T17:15:09Z

Bump the Triton pin to the release candidate commit for Triton 3.2.

A few changes beyond the pin bump itself are needed:

Remove the script that adds a git version hash suffix to the Triton wheel, since as of Add git commit to the version as a suffix triton-lang/triton#4812 Triton adds that itself
Add pybind11 to the Triton build setup, since Triton now depends on it
Use manylinux-2.28 for the Triton wheel builder, and use clang+lld for building to pick up the right glibc

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @aakhundov

pytorch-bot · 2024-10-29T17:15:14Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139206

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 New Failures, 1 Unrelated Failure

As of commit 53f4666 with merge base 740d1eb ():

NEW FAILURES - The following jobs have failed:

inductor / cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
convnext_base
inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) (gh)
convnext_base
linux-binary-manywheel / manywheel-py3_9-cuda11_8-test / test (gh)
ERROR: Could not find a version that satisfies the requirement pytorch-triton==3.2.0+35c6c7c628; platform_system == "Linux" and platform_machine == "x86_64" and python_version < "3.13" (from torch) (from versions: 2.0.0+0d7e753227, 2.0.0+3aa3d7024e, 2.0.0+af76c989eb, 2.0.0+b8b470bc59, 2.0.0+c8bfe3f548, 2.0.0+d54c04abe2, 2.1.0, 2.1.0+2c32f43999, 2.1.0+3c400e7818, 2.1.0+440fd1bf20, 2.1.0+46672772b4, 2.1.0+6e4932cda8, 2.1.0+7d1a95b046, 2.1.0+9820899b38, 2.1.0+9e3e10c5ed, 2.1.0+bcad9dabe1, 2.1.0+e6216047b8, 2.1.0+e650d3708b, 2.2.0+e28a256d71, 3.0.0+45fff310c8, 3.0.0+757b6a61e7, 3.0.0+901819d2b6, 3.0.0+989adb9a29, 3.0.0+a9bc1a3647, 3.0.0+dedb7bdf33, 3.1.0+5fe38ffd73, 3.1.0+cf34004b8a, 3.2.0+git35c6c7c6)
linux-binary-manywheel / manywheel-py3_9-cuda12_4-test / test (gh)
ERROR: Could not find a version that satisfies the requirement pytorch-triton==3.2.0+35c6c7c628; platform_system == "Linux" and platform_machine == "x86_64" and python_version < "3.13" (from torch) (from versions: 2.0.0+0d7e753227, 2.0.0+3aa3d7024e, 2.0.0+af76c989eb, 2.0.0+b8b470bc59, 2.0.0+c8bfe3f548, 2.0.0+d54c04abe2, 2.1.0, 2.1.0+2c32f43999, 2.1.0+3c400e7818, 2.1.0+440fd1bf20, 2.1.0+46672772b4, 2.1.0+6e4932cda8, 2.1.0+7d1a95b046, 2.1.0+9820899b38, 2.1.0+9e3e10c5ed, 2.1.0+bcad9dabe1, 2.1.0+e6216047b8, 2.1.0+e650d3708b, 2.2.0+e28a256d71, 3.0.0+45fff310c8, 3.0.0+757b6a61e7, 3.0.0+901819d2b6, 3.0.0+989adb9a29, 3.0.0+a9bc1a3647, 3.0.0+dedb7bdf33, 3.1.0+5fe38ffd73, 3.1.0+cf34004b8a, 3.2.0+git35c6c7c6)
linux-binary-manywheel / manywheel-py3_9-cuda12_6-test / test (gh)
ERROR: Could not find a version that satisfies the requirement pytorch-triton==3.2.0+35c6c7c628; platform_system == "Linux" and platform_machine == "x86_64" and python_version < "3.13" (from torch) (from versions: 2.0.0+0d7e753227, 2.0.0+3aa3d7024e, 2.0.0+af76c989eb, 2.0.0+b8b470bc59, 2.0.0+c8bfe3f548, 2.0.0+d54c04abe2, 2.1.0, 2.1.0+2c32f43999, 2.1.0+3c400e7818, 2.1.0+440fd1bf20, 2.1.0+46672772b4, 2.1.0+6e4932cda8, 2.1.0+7d1a95b046, 2.1.0+9820899b38, 2.1.0+9e3e10c5ed, 2.1.0+bcad9dabe1, 2.1.0+e6216047b8, 2.1.0+e650d3708b, 2.2.0+e28a256d71, 3.0.0+45fff310c8, 3.0.0+757b6a61e7, 3.0.0+901819d2b6, 3.0.0+989adb9a29, 3.0.0+a9bc1a3647, 3.0.0+dedb7bdf33, 3.1.0+5fe38ffd73, 3.1.0+cf34004b8a, 3.2.0+git35c6c7c6)

FLAKY - The following job failed but was likely due to flakiness present on trunk:

trunk / win-vs2019-cpu-py3 / test (default, 1, 3, lf.windows.4xlarge.nonephemeral) (gh) (similar failure)
'Test'

This comment was automatically generated by Dr. CI and updates every 15 minutes.

davidberard98 · 2024-10-30T23:38:52Z

I'm going to look at the cumsum failure

Edit: issue here #139348

Binary build is failing in trunk after #139206 lands, for example, https://github.com/pytorch/pytorch/actions/runs/11981181986/job/33410250461#step:17:539. It's a bit tricky to spot the issue but the difference is between `3.2.0+35c6c7c628` set by PyTorch and `3.2.0+git35c6c7c6` from triton (look closely one has the length of 10, the other of 8 characters) Triton now has its own nightly build logic in triton-lang/triton#4812 that takes only 8 characters by default while the original logic from PT took 10. So, PT nightly couldn't find the dependency. Pull Request resolved: #141410 Approved by: https://github.com/seemethere, https://github.com/malfet

guangyey · 2024-11-25T03:20:52Z

Thanks @bertmaher, we will rebase our PR #137886 to update XPU triton.

This reverts commit 74cde3c. # Motivation PyTorch community has uplifted triton to 3.2.0 in pytorch/pytorch#139206. We should follow it. # Solution Revert this commit introduced in #2716

Triton xpu build was stopped by #139206 temporally to wait triton xpu upgrade PR #137886 landed. Works for #139722 and #114850 Pull Request resolved: #141775 Approved by: https://github.com/atalman

Bump the Triton pin to the release candidate commit for Triton 3.2. A few changes beyond the pin bump itself are needed: * Remove the script that adds a git version hash suffix to the Triton wheel, since as of triton-lang/triton#4812 Triton adds that itself * Add `pybind11` to the Triton build setup, since Triton now depends on it * Use manylinux-2.28 for the Triton wheel builder, and use clang+lld for building to pick up the right glibc Pull Request resolved: pytorch#139206 Approved by: https://github.com/malfet, https://github.com/atalman Co-authored-by: Andrey Talman <[email protected]>

This reverts commit c93e57e. Reverted pytorch#139206 on behalf of https://github.com/atalman due to Will revert and reland skipping xpu builds ([comment](pytorch#139206 (comment)))

Bump the Triton pin to the release candidate commit for Triton 3.2. A few changes beyond the pin bump itself are needed: * Remove the script that adds a git version hash suffix to the Triton wheel, since as of triton-lang/triton#4812 Triton adds that itself * Add `pybind11` to the Triton build setup, since Triton now depends on it * Use manylinux-2.28 for the Triton wheel builder, and use clang+lld for building to pick up the right glibc Pull Request resolved: pytorch#139206 Approved by: https://github.com/malfet, https://github.com/atalman Co-authored-by: Andrey Talman <[email protected]>

Binary build is failing in trunk after pytorch#139206 lands, for example, https://github.com/pytorch/pytorch/actions/runs/11981181986/job/33410250461#step:17:539. It's a bit tricky to spot the issue but the difference is between `3.2.0+35c6c7c628` set by PyTorch and `3.2.0+git35c6c7c6` from triton (look closely one has the length of 10, the other of 8 characters) Triton now has its own nightly build logic in triton-lang/triton#4812 that takes only 8 characters by default while the original logic from PT took 10. So, PT nightly couldn't find the dependency. Pull Request resolved: pytorch#141410 Approved by: https://github.com/seemethere, https://github.com/malfet

Triton xpu build was stopped by pytorch#139206 temporally to wait triton xpu upgrade PR pytorch#137886 landed. Works for pytorch#139722 and pytorch#114850 Pull Request resolved: pytorch#141775 Approved by: https://github.com/atalman

Summary: Following upstream Triton pin update (pytorch/pytorch#139206), we can now enable more kernels in the CI. Pull Request resolved: #96 Reviewed By: adamomainz Differential Revision: D66763281 Pulled By: xuzhao9 fbshipit-source-id: 943472995c2da0f4cd0d01665b53786614f122ae

Update triton version to align with PyTorch: pytorch/pytorch#139206

…4a (#3497) Summary: X-link: facebookresearch/FBGEMM#577 Update triton version to align with PyTorch: pytorch/pytorch#139206 Pull Request resolved: #3497 Reviewed By: q10 Differential Revision: D67075647 Pulled By: brad-mengchi fbshipit-source-id: 6adc76484d1f6b827d62615c4311cf4b9fffd6b6

cpuhrsch · 2025-01-13T20:27:54Z

This seems to break int_mm #144705 . I'l try to produce a standalone Triton kernel to reproduce this error.

…4a (pytorch#577) Summary: Pull Request resolved: facebookresearch/FBGEMM#577 Update triton version to align with PyTorch: pytorch/pytorch#139206 X-link: pytorch#3497 Reviewed By: q10 Differential Revision: D67075647 Pulled By: brad-mengchi fbshipit-source-id: 6adc76484d1f6b827d62615c4311cf4b9fffd6b6

bertmaher requested review from Chillee, desertfire, eellison, jataylo, jeffdaily, jithunnair-amd, pruthvistony and shunting314 as code owners October 29, 2024 17:15

pytorch-bot bot added ciflow/inductor ciflow/trunk Trigger trunk jobs on your pull request topic: not user facing topic category labels Oct 29, 2024

bertmaher mentioned this pull request Oct 29, 2024

[v2.6] Inductor issue tracker for Triton release/3.2 #139175

Closed

jataylo added the ciflow/rocm Trigger "default" config CI on ROCm label Oct 29, 2024

pytorch-bot bot added the module: inductor label Oct 29, 2024

bertmaher force-pushed the pinbump26 branch 2 times, most recently from c1ad437 to 5b030ef Compare November 2, 2024 01:55

jataylo mentioned this pull request Nov 4, 2024

[ROCm] [Upstream Triton] Flex attention Assertion idx < size()' failed. #139621

Closed

davidberard98 mentioned this pull request Nov 4, 2024

[inductor][triton 3.2] fix test_codegen_config_option_dont_assume_alignment for triton 3.2 #139640

Closed

bertmaher added the keep-going Don't stop on first failure, keep running tests until the end label Nov 4, 2024

huydhn mentioned this pull request Nov 23, 2024

Fix pytorch-triton nightly checksum shorthash #141410

Closed

guangyey mentioned this pull request Nov 25, 2024

Revert "bump triton verion to 3.1.0 (#2716)" intel/intel-xpu-backend-for-triton#2812

Merged

benjaminglass1 mentioned this pull request Nov 25, 2024

UNSTABLE inductor / cuda12.4-py3.10-gcc9-sm86 / test (inductor_timm) #141498

Closed

guangyey mentioned this pull request Nov 26, 2024

Update triton xpu commit pin #137886

Closed

chuanqi129 mentioned this pull request Nov 28, 2024

[CD] Add triton xpu build back #141775

Closed

jataylo mentioned this pull request Dec 3, 2024

Nightly torch wheel is not picking up correct pytorch-triton/pytorch-triton-rocm package #141956

Closed

xuzhao9 mentioned this pull request Dec 4, 2024

Enable more kernels in CI after pytorch triton pin update meta-pytorch/tritonbench#96

Closed

brad-mengchi added a commit to pytorch/FBGEMM that referenced this pull request Dec 11, 2024

Update triton version

8a0debc

Update triton version to align with PyTorch: pytorch/pytorch#139206

brad-mengchi mentioned this pull request Dec 11, 2024

Update triton version to 3.2.0+35c6c7c6284582b3f41c71c150e11b517acf074a pytorch/FBGEMM#3497

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[triton] Update pin for PyTorch 2.6/Triton 3.2 #139206

[triton] Update pin for PyTorch 2.6/Triton 3.2 #139206

Uh oh!

bertmaher commented Oct 29, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Oct 29, 2024 •

edited

Loading

Uh oh!

davidberard98 commented Oct 30, 2024 •

edited

Loading

Uh oh!

guangyey commented Nov 25, 2024

Uh oh!

cpuhrsch commented Jan 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

[triton] Update pin for PyTorch 2.6/Triton 3.2 #139206

[triton] Update pin for PyTorch 2.6/Triton 3.2 #139206

Uh oh!

Conversation

bertmaher commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Oct 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/139206

❌ 5 New Failures, 1 Unrelated Failure

Uh oh!

davidberard98 commented Oct 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guangyey commented Nov 25, 2024

Uh oh!

cpuhrsch commented Jan 13, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

bertmaher commented Oct 29, 2024 •

edited

Loading

pytorch-bot bot commented Oct 29, 2024 •

edited

Loading

davidberard98 commented Oct 30, 2024 •

edited

Loading