[ROCm][TunableOp] Stricter unit tests for online and offline tuning #150142

naromero77amd · 2025-03-27T22:48:42Z

Improvements to unit tests and warnings for unsupported cases in offline tuning. Here are more details:

Previously we only compared the OpSig for the untuned vs. tuned entries. This was not strict enough so we now compare OpSig+ParamSig.
The main offline and online UTs are now stricter to make sure we exercise the code paths for the four combinations of transA and transB.
Offline tuning does not support some tensor shapes. Emit warning and skip tuning.

cc @jeffdaily @sunway513 @jithunnair-amd @pruthvistony @ROCmSupport @dllehr-amd @jataylo @hongxiayang

pytorch-bot · 2025-03-27T22:48:46Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/150142

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit b637fde with merge base 965784e ():

UNSTABLE - The following jobs are marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

jeffdaily

approved, with minor nit

test/test_linalg.py

naromero77amd · 2025-03-29T01:06:08Z

@pytorchbot rebase

pytorchmergebot · 2025-03-29T01:07:40Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-03-29T01:07:42Z

Successfully rebased tunableop_ut_improvements onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout tunableop_ut_improvements && git pull --rebase)

naromero77amd · 2025-03-29T01:08:31Z

@pytorchbot merge

pytorchmergebot · 2025-03-29T01:10:39Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-03-29T01:26:33Z

Merge failed

Reason: 3 jobs have failed, first few of them are: linux-binary-libtorch-release / libtorch-cpu-shared-with-deps-release-build / build, linux-binary-manywheel / manywheel-py3_9-cuda11_8-build / build, linux-binary-manywheel / manywheel-py3_9-cuda12_6-build / build

Details for Dev Infra team

Raised by workflow job

naromero77amd · 2025-03-29T19:59:10Z

@pytorchbot rebase

pytorchmergebot · 2025-03-29T20:00:39Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-03-29T20:00:41Z

Tried to rebase and push PR #150142, but it was already up to date. Try rebasing against main by issuing:
@pytorchbot rebase -b main

naromero77amd · 2025-03-29T20:03:08Z

@pytorchbot rebase -b main

pytorchmergebot · 2025-03-29T20:04:38Z

@pytorchbot started a rebase job onto refs/remotes/origin/main. Check the current status here

pytorchmergebot · 2025-03-29T20:04:41Z

Successfully rebased tunableop_ut_improvements onto refs/remotes/origin/main, please pull locally before adding more changes (for example, via git checkout tunableop_ut_improvements && git pull --rebase)

naromero77amd · 2025-03-29T20:06:19Z

rocm tests are passing but getting weird build failures for cuda, will try rebasing on main and then merging.

naromero77amd · 2025-03-31T04:05:01Z

@pytorchbot merge

pytorchmergebot · 2025-03-31T04:06:37Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

…ytorch#150142) Improvements to unit tests and warnings for unsupported cases in offline tuning. Here are more details: - Previously we only compared the OpSig for the untuned vs. tuned entries. This was not strict enough so we now compare OpSig+ParamSig. - The main offline and online UTs are now stricter to make sure we exercise the code paths for the four combinations of transA and transB. - Offline tuning does not support some tensor shapes. Emit warning and skip tuning. Pull Request resolved: pytorch#150142 Approved by: https://github.com/jeffdaily Co-authored-by: Jeff Daily <[email protected]>

…ytorch#150142) Improvements to unit tests and warnings for unsupported cases in offline tuning. Here are more details: - Previously we only compared the OpSig for the untuned vs. tuned entries. This was not strict enough so we now compare OpSig+ParamSig. - The main offline and online UTs are now stricter to make sure we exercise the code paths for the four combinations of transA and transB. - Offline tuning does not support some tensor shapes. Emit warning and skip tuning. Pull Request resolved: pytorch#150142 Approved by: https://github.com/jeffdaily Co-authored-by: Jeff Daily <[email protected]> (cherry picked from commit ca2ffc2)

…ledGEMM rowwise fix (#2106) Align TunableOp UTs, features, and bug fixes with upstream PyTorch main UTs: pytorch#148982 pytorch#149930 pytorch#150142 pytorch#150463 Feature: offline tuning for submatrices: pytorch#151138 Bug Fix: ScaledGEMM rowwise pytorch#152403 --------- Co-authored-by: Jeff Daily <[email protected]>

naromero77amd requested review from IvanYashchuk, eqy, lezcano, nikitaved and syed-ahmed as code owners March 27, 2025 22:48

pytorch-bot bot added module: rocm AMD GPU support for Pytorch release notes: linalg_frontend release notes category labels Mar 27, 2025

naromero77amd requested a review from jeffdaily March 27, 2025 22:48

naromero77amd added the topic: not user facing topic category label Mar 27, 2025

pytorchbot added the open source label Mar 27, 2025

naromero77amd added the ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 label Mar 28, 2025

jeffdaily approved these changes Mar 28, 2025

View reviewed changes

test/test_linalg.py Outdated Show resolved Hide resolved

pytorch-bot bot removed the ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 label Mar 28, 2025

jeffdaily added the ciflow/rocm Trigger "default" config CI on ROCm label Mar 28, 2025

pytorchmergebot force-pushed the tunableop_ut_improvements branch from e62566f to 895ed5a Compare March 29, 2025 01:07

pytorch-bot bot removed the ciflow/rocm Trigger "default" config CI on ROCm label Mar 29, 2025

naromero77amd added the ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 label Mar 29, 2025

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Mar 29, 2025

pytorchmergebot added the merging label Mar 29, 2025

pytorchmergebot removed the merging label Mar 29, 2025

naromero77amd and others added 5 commits March 29, 2025 20:04

Modify helper function to also take into account op signature.

83e20d6

Add warning message for unsupported cases.

1442c69

Stricter UT for online tuning.

f9f032f

Stricter UT for offline tuning.

8d5c4df

Update test/test_linalg.py

b637fde

pytorchmergebot force-pushed the tunableop_ut_improvements branch from 895ed5a to b637fde Compare March 29, 2025 20:04

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Mar 29, 2025

naromero77amd added ciflow/trunk Trigger trunk jobs on your pull request and removed ciflow/rocm-mi300 Trigger "default" config CI on ROCm MI300 labels Mar 29, 2025

pytorchmergebot added the merging label Mar 31, 2025

pytorchmergebot closed this in ca2ffc2 Mar 31, 2025

pytorchmergebot added Merged and removed merging labels Mar 31, 2025

naromero77amd deleted the tunableop_ut_improvements branch March 31, 2025 04:14

naromero77amd mentioned this pull request May 8, 2025

[release/2.7][ROCm][TunableOp] UTs, submatrix offline tuning, and ScaledGEMM rowwise fix ROCm/pytorch#2106

Merged

[ROCm][TunableOp] Stricter unit tests for online and offline tuning #150142

[ROCm][TunableOp] Stricter unit tests for online and offline tuning #150142

Uh oh!

Conversation

naromero77amd commented Mar 27, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Mar 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/150142

✅ You can merge normally! (2 Unrelated Failures)

Uh oh!

jeffdaily left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

naromero77amd commented Mar 29, 2025

Uh oh!

pytorchmergebot commented Mar 29, 2025

Uh oh!

pytorchmergebot commented Mar 29, 2025

Uh oh!

naromero77amd commented Mar 29, 2025

Uh oh!

pytorchmergebot commented Mar 29, 2025

Merge started

Uh oh!

pytorchmergebot commented Mar 29, 2025

Merge failed

Uh oh!

naromero77amd commented Mar 29, 2025

Uh oh!

pytorchmergebot commented Mar 29, 2025

Uh oh!

pytorchmergebot commented Mar 29, 2025

Uh oh!

naromero77amd commented Mar 29, 2025

Uh oh!

pytorchmergebot commented Mar 29, 2025

Uh oh!

pytorchmergebot commented Mar 29, 2025

Uh oh!

naromero77amd commented Mar 29, 2025

Uh oh!

naromero77amd commented Mar 31, 2025

Uh oh!

pytorchmergebot commented Mar 31, 2025

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

naromero77amd commented Mar 27, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Mar 27, 2025 •

edited

Loading