[inductor] [cpp] use non-temporal tile load for A #129455
Conversation
[ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/129455
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 415b882 with merge base cb2bce9 ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Use non-temporal tile load `_tile_stream_loadd` for A to keep B in L1. **TODOs:** - [ ] Collect benchmark data before and after this change cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang [ghstack-poisoned]
Use non-temporal tile load `_tile_stream_loadd` for A to keep B in L1. **TODOs:** - [ ] Collect benchmark data before and after this change cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang [ghstack-poisoned]
Use non-temporal tile load `_tile_stream_loadd` for A to keep B in L1. ## Performance data Performance speedups with >=5% on BF16 AMP, with this PR vs. without this PR, measured on CPU with AMX support: - Static shapes Single-threaded | Model Family | Model Name | Speedup | |--------------|------------|---------| torchbench | timm_vision_transformer | 5% huggingface | MT5ForConditionalGeneration | 5% huggingface | MobileBertForMaskedLM | 5% timm_models | gmixer_24_224 | 5% No perf regressions. TODO: collect benchmark for - Static shapes Multi-threaded - Dynamic shapes Single-threaded - Dynamic shapes Multi-threaded cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang [ghstack-poisoned]
Updated the performance status in the PR description: |
Use non-temporal tile load `_tile_stream_loadd` for A to keep B in L1. Verified AMP static shapes and dynamic shapes on CPU with AMX support and no obvious performance boost (no regression either) at end-to-end level. We're expecting to get performance gain when adding #129348 (also in this ghstack) on top of this PR. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx peterbell10 ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang [ghstack-poisoned]
|
@pytorchbot merge |
Merge failedReason: This PR needs a If not, please add the To add a label, you can comment to pytorchbot, for example For more information, see Details for Dev Infra teamRaised by workflow job |
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
ghstack-source-id: 65ff95d Pull Request resolved: pytorch/pytorch#129455
Use non-temporal tile load `_tile_stream_loadd` for A to keep B in L1. Verified AMP static shapes and dynamic shapes on CPU with AMX support and no obvious performance boost (no regression either) at end-to-end level. We're expecting to get performance gain when adding pytorch#129348 (also in this ghstack) on top of this PR. Pull Request resolved: pytorch#129455 Approved by: https://github.com/jgong5
Stack from ghstack (oldest at bottom):
Use non-temporal tile load
_tile_stream_loaddfor A to keep B in L1.Verified AMP static shapes and dynamic shapes on CPU with AMX support and no obvious performance boost (no regression either) at end-to-end level. We're expecting to get performance gain when adding #129348 (also in this ghstack) on top of this PR.
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @peterbell10 @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang