Skip to content

[test][easy] Add debug utils for cpu select algorithm test #135038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
henrylhtsang wants to merge 1 commit into from
Closed

[test][easy] Add debug utils for cpu select algorithm test #135038

henrylhtsang wants to merge 1 commit into from

Conversation

@henrylhtsang
Copy link
Contributor

@henrylhtsang henrylhtsang commented Sep 3, 2024 •
edited
Loading

Summary: Add debug utils to debug a flaky test in fbcode ci.

Some context: #126545

Test Plan: ci

Differential Revision: D62005445

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang

@pytorch-bot
Copy link

pytorch-bot bot commented Sep 3, 2024 •
edited
Loading

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/135038

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 2fa2703 with merge base 04118d8 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Sep 3, 2024

This pull request was exported from Phabricator. Differential Revision: D62005445

@henrylhtsang
Copy link
Contributor Author

henrylhtsang commented Sep 5, 2024

@jgong5 can you take a look?

jgong5
jgong5 approved these changes Sep 6, 2024
View reviewed changes
Copy link
Collaborator

@jgong5 jgong5 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Better to wrap it with is_fbcode() check though.

test/inductor/test_cpu_select_algorithm.py Outdated
Comment on lines 242 to 250
Copy link
Collaborator

@jgong5 jgong5 Sep 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it is only used for debugging fbcode thing, can we wrap it with is_fbcode() check?

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 6, 2024
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Sep 6, 2024

This pull request was exported from Phabricator. Differential Revision: D62005445

henrylhtsang added a commit to henrylhtsang/pytorch that referenced this pull request Sep 6, 2024
@henrylhtsang @facebook-github-bot
[test][easy] Add debug utils for cpu select algorithm test (pytorch#1…
40b6195 
…35038)

Summary:
Pull Request resolved: pytorch#135038

Add debug utils to debug a flaky test in fbcode ci.

Test Plan: ci

Differential Revision: D62005445
@henrylhtsang @facebook-github-bot
[test][easy] Add debug utils for cpu select algorithm test (pytorch#1…
2fa2703 
…35038)

Summary:
Pull Request resolved: pytorch#135038

Add debug utils to debug a flaky test in fbcode ci.

Test Plan:
ci

```
buck2 test -j 18 'fbcode//mode/opt' fbcode//caffe2/test/inductor:cpu_select_algorithm_cpu -- --exact 'caffe2/test/inductor:cpu_select_algorithm_cpu - test_linear_with_pointwise_batch_size_384_in_features_196_out_features_384_bias_False_epilogue_sigmoid_cpu_bfloat16 (caffe2.test.inductor.test_cpu_select_algorithm.TestSelectAlgorithmCPU)' --run-disabled --stress-runs 10 --record-results --print-passing-details
```

Differential Revision: D62005445
@facebook-github-bot
Copy link
Contributor

facebook-github-bot commented Sep 6, 2024

This pull request was exported from Phabricator. Differential Revision: D62005445

@henrylhtsang
Copy link
Contributor Author

henrylhtsang commented Sep 6, 2024

@pytorchbot merge -i

@pytorchmergebot
Copy link
Collaborator

pytorchmergebot commented Sep 6, 2024

Merge started

Your change will be merged while ignoring the following 0 checks:

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@henrylhtsang
Copy link
Contributor Author

henrylhtsang commented Sep 17, 2024

It looks like in some cases we have torch.ops.mkldnn._is_mkldnn_bf16_supported() = False. I don't know why that would happen occasionally, considering they even use the same hardware.

Quickest way forward might be to add that check in the if conditions.

@henrylhtsang
Copy link
Contributor Author

henrylhtsang commented Sep 17, 2024

It looks like in some cases we have torch.ops.mkldnn._is_mkldnn_bf16_supported() = False. I don't know why that would happen occasionally, considering they even use the same hardware.

Quickest way forward might be to add that check in the if conditions.

Correction:
The is_mkldnn_bf16_supported = False one is using Intel(R) Xeon(R) CPU E5-2680 v4, while the is_mkldnn_bf16_supported = True one is using Intel(R) Xeon(R) Platinum 8321HC. I figured this might be something obvious, so @jgong5 do let me know if this is expected.

@henrylhtsang henrylhtsang mentioned this pull request Sep 18, 2024
Closed
Chao1Han pushed a commit to Chao1Han/pytorch that referenced this pull request Sep 20, 2024
@henrylhtsang @Chao1Han
[test][easy] Add debug utils for cpu select algorithm test (pytorch#1…
09e429a 
…35038)

Summary: Add debug utils to debug a flaky test in fbcode ci.

Some context: pytorch#126545

Test Plan: ci

Differential Revision: D62005445

Pull Request resolved: pytorch#135038
Approved by: https://github.com/jgong5, https://github.com/XuehaiPan
pytorchmergebot pushed a commit that referenced this pull request Sep 24, 2024
@henrylhtsang @pytorchmergebot
[inductor][test] Check if mkl dnn bf16 is supported when using bf16 (#…
02ef5dd 
…136290)

Sometimes the test is run with older cpu, e.g. Intel(R) Xeon(R) CPU E5-2680 v4. If we inspect its `lscpu`, in the flags, we don't see a `avx512_bf16`. So that probably means bf16 is not supported for those hardwares, and hence the unit test can fail. So we add the check in the code.

Context: #135038

Differential Revision: D62984129

Pull Request resolved: #136290
Approved by: https://github.com/XuehaiPan, https://github.com/chenyang78
PaliC pushed a commit to PaliC/pytorch that referenced this pull request Sep 25, 2024
@henrylhtsang @PaliC
[inductor][test] Check if mkl dnn bf16 is supported when using bf16 (p…
0fd4c9d 
…ytorch#136290)

Sometimes the test is run with older cpu, e.g. Intel(R) Xeon(R) CPU E5-2680 v4. If we inspect its `lscpu`, in the flags, we don't see a `avx512_bf16`. So that probably means bf16 is not supported for those hardwares, and hence the unit test can fail. So we add the check in the code.

Context: pytorch#135038

Differential Revision: D62984129

Pull Request resolved: pytorch#136290
Approved by: https://github.com/XuehaiPan, https://github.com/chenyang78
BoyuanFeng pushed a commit to BoyuanFeng/pytorch that referenced this pull request Sep 25, 2024
@henrylhtsang @BoyuanFeng
[inductor][test] Check if mkl dnn bf16 is supported when using bf16 (p…
a0a4f3b 
…ytorch#136290)

Sometimes the test is run with older cpu, e.g. Intel(R) Xeon(R) CPU E5-2680 v4. If we inspect its `lscpu`, in the flags, we don't see a `avx512_bf16`. So that probably means bf16 is not supported for those hardwares, and hence the unit test can fail. So we add the check in the code.

Context: pytorch#135038

Differential Revision: D62984129

Pull Request resolved: pytorch#136290
Approved by: https://github.com/XuehaiPan, https://github.com/chenyang78
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@jgong5 jgong5 jgong5 approved these changes

@XuehaiPan XuehaiPan XuehaiPan approved these changes

Assignees

No one assigned

Labels

ciflow/trunk Trigger trunk jobs on your pull request fb-exported Merged module: inductor topic: not user facing topic category

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@henrylhtsang @facebook-github-bot @pytorchmergebot @jgong5 @XuehaiPan