[ROCm] Bump FBGEMM commit to avoid CK errors #162590

pragupta · 2025-09-10T13:38:09Z

Trying to solve the following CK build errors:

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang @naromero77amd

pytorch-bot · 2025-09-10T13:38:12Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/162590

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

❌ 15 New Failures, 2 Cancelled Jobs, 20 Unrelated Failures

As of commit e4e727c with merge base c238820 ():

NEW FAILURES - The following jobs have failed:

linux-binary-libtorch / libtorch-cuda12_6-shared-with-deps-release-test / test (gh)
Final attempt failed. Child_process exited with error code 6
linux-binary-manywheel / manywheel-py3_10-cuda12_6-test / test (gh)
Final attempt failed. Child_process exited with error code 6
linux-binary-manywheel / manywheel-py3_11-cuda12_6-test / test (gh)
Final attempt failed. Child_process exited with error code 6
linux-binary-manywheel / manywheel-py3_12-cuda12_6-test / test (gh)
Final attempt failed. Child_process exited with error code 6
linux-binary-manywheel / manywheel-py3_13-cuda12_6-test / test (gh)
Final attempt failed. Child_process exited with error code 6
linux-binary-manywheel / manywheel-py3_13t-cuda12_6-test / test (gh)
Final attempt failed. Child_process exited with error code 6
linux-binary-manywheel / manywheel-py3_14-cuda12_6-test / test (gh)
Final attempt failed. Child_process exited with error code 6
linux-binary-manywheel / manywheel-py3_14t-cuda12_6-test / test (gh)
Final attempt failed. Child_process exited with error code 6
linux-s390x-binary-manywheel / manywheel-py3_10-cpu-s390x-build / build (gh)
undefined reference to 'dlclose'
linux-s390x-binary-manywheel / manywheel-py3_11-cpu-s390x-build / build (gh)
undefined reference to 'dlclose'
linux-s390x-binary-manywheel / manywheel-py3_12-cpu-s390x-build / build (gh)
undefined reference to 'dlclose'
linux-s390x-binary-manywheel / manywheel-py3_13-cpu-s390x-build / build (gh)
undefined reference to 'dlclose'
linux-s390x-binary-manywheel / manywheel-py3_13t-cpu-s390x-build / build (gh)
undefined reference to 'dlclose'
linux-s390x-binary-manywheel / manywheel-py3_14-cpu-s390x-build / build (gh)
undefined reference to 'dlclose'
linux-s390x-binary-manywheel / manywheel-py3_14t-cpu-s390x-build / build (gh)
undefined reference to 'dlclose'

CANCELLED JOBS - The following jobs were cancelled. Please retry:

linux-binary-libtorch / libtorch-rocm6_3-shared-with-deps-release-build / build (gh)
##[error]The operation was canceled.
linux-binary-libtorch / libtorch-rocm6_4-shared-with-deps-release-build / build (gh)
##[error]The operation was canceled.

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

linux-binary-manywheel / manywheel-py3_10-xpu-test (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 1
linux-binary-manywheel / manywheel-py3_11-xpu-test (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 1
linux-binary-manywheel / manywheel-py3_12-xpu-test (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
linux-binary-manywheel / manywheel-py3_13-xpu-test (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 1
linux-binary-manywheel / manywheel-py3_13t-xpu-test (gh) (matched linux rule in flaky-rules.json)
The process '/usr/bin/git' failed with exit code 128
rocm / linux-jammy-rocm-py3.10 / test (default, 2, 6, linux.rocm.gpu.2) (gh) (similar failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
rocm / linux-jammy-rocm-py3.10 / test (default, 3, 6, linux.rocm.gpu.2) (gh) (disabled by #121530 but the issue was closed recently and a rebase is needed to make it pass)
functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad__batch_norm_with_update_cuda_float64
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 3, 6, linux.rocm.gpu.gfx942.1) (gh) (disabled by #121530 but the issue was closed recently and a rebase is needed to make it pass)
functorch/test_ops.py::TestOperatorsCUDA::test_vmap_autograd_grad__batch_norm_with_update_cuda_float64
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 6, 6, linux.rocm.gpu.gfx942.1) (gh) (disabled by #157900, #157901)
inductor/test_cudagraph_trees_expandable_segments.py::CudaGraphTreeTests::test_graph_partition_reorder_custom_op_with_no_dependency1
windows-binary-wheel / wheel-py3_11-xpu-build (gh) (matched win rule in flaky-rules.json)
##[error]The operation was canceled.

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

pull / linux-jammy-py3.10-clang12 / test (crossref, 2, 2, linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.10-clang12 / test (default, 3, 5, linux.4xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.10-clang12 / test (dynamo_wrapped, 3, 3, linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.10-clang18-asan / test (default, 6, 7, linux.4xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.10-gcc11 / test (default, 5, 5, linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.13-clang12 / test (crossref, 1, 2, linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.13-clang12 / test (default, 1, 5, linux.4xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
pull / linux-jammy-py3.13-clang12 / test (dynamo_wrapped, 2, 3, linux.2xlarge) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops
rocm-mi300 / linux-noble-rocm-py3.12-mi300 / test (default, 2, 6, linux.rocm.gpu.gfx942.1) (gh) (trunk failure)
'Test'
trunk / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 4, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu) (gh) (trunk failure)
test_quantization.py::TestQuantizeFxOps::test_general_shape_ops

This comment was automatically generated by Dr. CI and updates every 15 minutes.

pytorch-bot · 2025-09-12T11:33:56Z

To add the ciflow label ciflow/binaries_libtorch please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

jithunnair-amd · 2025-09-12T11:40:24Z

@pragupta Can you please include the changes from my PR #162648 to see if narrowing down the compilation targets for FBGEMM GENAI to gfx942 helps avoid these errors?

linux-foundation-easycla · 2025-09-17T12:06:30Z

The committers listed above are authorized under a signed CLA.

✅ login: pragupta / name: Prachi Gupta (fa9701f, e4e727c)

pytorch-bot · 2025-09-17T12:40:40Z

To add the ciflow label ciflow/rocm please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

pytorch-bot · 2025-09-17T12:40:40Z

To add the ciflow label ciflow/rocm-mi300 please first approve the workflows that are awaiting approval (scroll to the bottom of this page).

This helps ensure we don't trigger CI on this PR until it is actually authorized to do so. Please ping one of the reviewers if you do not have access to approve and run workflows.

q10 · 2025-09-18T17:49:48Z

@pragupta is this PR ready for review? We would like to land this to unblock the CI.

sampathvic · 2025-09-18T22:29:04Z

Sorry I meant to say I saw you imported. I wasn't sure if that prevented me from force merging.

My bad, I didn't realize that. Sorry about that. So we have to wait for @pragupta to merge then is it?

jeffdaily · 2025-09-18T22:42:10Z

I can pytorchbot merge -f. No need to wait. But does that mess up your Meta-internal import?

sampathvic · 2025-09-18T23:21:44Z

I can pytorchbot merge -f. No need to wait. But does that mess up your Meta-internal import?

@jeffdaily I think it should be fine.

jeffdaily · 2025-09-19T01:12:24Z

@pytorchbot merge -f "CI passing except known unrelated binary build failure in ROCm"

pytorchmergebot · 2025-09-19T01:13:58Z

Merge started

Your change will be merged immediately since you used the force (-f) flag, bypassing any CI checks (ETA: 1-5 minutes). Please use -f as last resort and instead consider -i/--ignore-current to continue the merge ignoring current failures. This will allow currently pending tests to finish and report signal before the merge.

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

malfet · 2025-09-19T18:11:23Z

@pytorchbot revert -m "This breaks CUDA 13 builds" -c nosignal

pytorchmergebot · 2025-09-19T18:12:49Z

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

This reverts commit c9b80c4. Reverted #162590 on behalf of https://github.com/malfet due to This breaks CUDA 13 builds ([comment](#162590 (comment)))

pytorchmergebot · 2025-09-19T18:13:04Z

@pragupta your PR has been successfully reverted.

This PR was reopened (likely due to being reverted), so your approval was removed. Please request another review.

Fixes #ISSUE_NUMBER Pull Request resolved: pytorch#162590 Approved by: https://github.com/jeffdaily

This reverts commit c9b80c4. Reverted pytorch#162590 on behalf of https://github.com/malfet due to This breaks CUDA 13 builds ([comment](pytorch#162590 (comment)))

Fixes #ISSUE_NUMBER Pull Request resolved: pytorch#162590 Approved by: https://github.com/jeffdaily

This reverts commit c9b80c4. Reverted pytorch#162590 on behalf of https://github.com/malfet due to This breaks CUDA 13 builds ([comment](pytorch#162590 (comment)))

cthi · 2025-09-22T15:50:27Z

@pragupta I've bumped fbgemm in #163411, which should fix the cuda13 build issue, and also includes the CK fix you added in fbgemm.

pragupta · 2025-09-22T16:01:18Z

@cthi thank you! Since #163411 brings in newer fbgemm, this PR is no longer needed. Closing

Fixes #ISSUE_NUMBER Pull Request resolved: pytorch#162590 Approved by: https://github.com/jeffdaily

This reverts commit c9b80c4. Reverted pytorch#162590 on behalf of https://github.com/malfet due to This breaks CUDA 13 builds ([comment](pytorch#162590 (comment)))

Bump FBGEMM commit to avoid CK errors

fa9701f

pytorch-bot bot added the topic: not user facing topic category label Sep 10, 2025

jithunnair-amd added ciflow/trunk Trigger trunk jobs on your pull request ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 labels Sep 10, 2025

pytorchbot added the open source label Sep 10, 2025

jithunnair-amd changed the title ~~Bump FBGEMM commit to avoid CK errors~~ [ROCm] Bump FBGEMM commit to avoid CK errors Sep 12, 2025

pytorch-bot bot added the module: rocm AMD GPU support for Pytorch label Sep 12, 2025

jithunnair-amd added the ciflow/binaries_libtorch Trigger binary build and upload jobs for libtorch on the PR label Sep 12, 2025

pytorch-bot bot removed the ciflow/binaries_libtorch Trigger binary build and upload jobs for libtorch on the PR label Sep 12, 2025

jithunnair-amd added the ciflow/binaries_libtorch Trigger binary build and upload jobs for libtorch on the PR label Sep 12, 2025

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Sep 17, 2025

pytorch-bot bot removed ciflow/binaries_libtorch Trigger binary build and upload jobs for libtorch on the PR ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 labels Sep 17, 2025

Bump FBGEMM commit to include ck compile flags

e4e727c

pragupta force-pushed the pg-bump-fbgemm branch from 313bcdc to e4e727c Compare September 17, 2025 12:39

jithunnair-amd added ciflow/binaries_libtorch Trigger binary build and upload jobs for libtorch on the PR ciflow/rocm Trigger "default" config CI on ROCm ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 labels Sep 17, 2025

pytorch-bot bot removed ciflow/rocm Trigger "default" config CI on ROCm ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 labels Sep 17, 2025

jithunnair-amd added ciflow/rocm Trigger "default" config CI on ROCm ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 labels Sep 17, 2025

jeffdaily previously approved these changes Sep 18, 2025

View reviewed changes

pytorchmergebot added the merging label Sep 19, 2025

pytorchmergebot closed this in c9b80c4 Sep 19, 2025

pytorchmergebot added Merged and removed merging labels Sep 19, 2025

robert-hardwick mentioned this pull request Sep 19, 2025

[CD] - Manywheel CUDA builds failing since Sept 18 #163342

Closed

pytorchmergebot added Reverted ci-no-td Do not run TD on this PR labels Sep 19, 2025

pytorchmergebot reopened this Sep 19, 2025

malfet added the ciflow/binaries_wheel Trigger binary build and upload jobs for wheel on the PR label Sep 19, 2025

malfet mentioned this pull request Sep 19, 2025

[CD] Simplify NVIDIA driver installation step #163349

Closed

pragupta closed this Sep 22, 2025

[ROCm] Bump FBGEMM commit to avoid CK errors #162590

[ROCm] Bump FBGEMM commit to avoid CK errors #162590

Uh oh!

Conversation

pragupta commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/162590

❌ 15 New Failures, 2 Cancelled Jobs, 20 Unrelated Failures

Uh oh!

pytorch-bot bot commented Sep 12, 2025

Uh oh!

jithunnair-amd commented Sep 12, 2025

Uh oh!

linux-foundation-easycla bot commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Sep 17, 2025

Uh oh!

pytorch-bot bot commented Sep 17, 2025

Uh oh!

q10 commented Sep 18, 2025

Uh oh!

sampathvic commented Sep 18, 2025

Uh oh!

jeffdaily commented Sep 18, 2025

Uh oh!

sampathvic commented Sep 18, 2025

Uh oh!

jeffdaily commented Sep 19, 2025

Uh oh!

pytorchmergebot commented Sep 19, 2025

Merge started

Uh oh!

malfet commented Sep 19, 2025

Uh oh!

pytorchmergebot commented Sep 19, 2025

Uh oh!

pytorchmergebot commented Sep 19, 2025

Uh oh!

cthi commented Sep 22, 2025

Uh oh!

pragupta commented Sep 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

pragupta commented Sep 10, 2025 •

edited

Loading

pytorch-bot bot commented Sep 10, 2025 •

edited

Loading

linux-foundation-easycla bot commented Sep 17, 2025 •

edited

Loading