[AOTI] Relax input alignment assertion #143236
Conversation
Summary: #142136 added a runtime alignment assertion. But the assumption is probably too strict for more flexible use cases of AOTI, e.g. python deployment, see a recent error torchchat ran into for more details, https://github.com/pytorch/torchchat/actions/runs/12322072267/job/34394851280 . This PR relaxes the runtime check and implements copy_misaligned_inputs in cpp instead. [ghstack-poisoned]
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/143236
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 75b5782 with merge base 9706ada ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
Summary: #142136 added a runtime alignment assertion. But the assumption is probably too strict for more flexible use cases of AOTI, e.g. python deployment, see a recent error torchchat ran into for more details, https://github.com/pytorch/torchchat/actions/runs/12322072267/job/34394851280 . This PR relaxes the runtime check and implements copy_misaligned_inputs in cpp instead. ghstack-source-id: 76513df Pull Request resolved: #143236
Summary: #142136 added a runtime alignment assertion. But the assumption is probably too strict for more flexible use cases of AOTI, e.g. python deployment, see a recent error torchchat ran into for more details, https://github.com/pytorch/torchchat/actions/runs/12322072267/job/34394851280 . This PR relaxes the runtime check and implements copy_misaligned_inputs in cpp instead. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames chauhang aakhundov [ghstack-poisoned]
|
@pytorchbot merge |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
Merge failedReason: 23 jobs have failed, first few of them are: inductor / unit-test / cuda12.4-py3.13-gcc9-sm86 / build, inductor / unit-test / linux-jammy-cpu-py3.12-gcc11-inductor-triton-cpu / build, inductor / unit-test / cuda12.4-py3.12-gcc9-sm86 / build, inductor / unit-test / linux-jammy-cpu-py3.12-gcc11-inductor-halide / build, inductor / unit-test / cuda12.4-py3.10-gcc9-sm86 / build Details for Dev Infra teamRaised by workflow job |
Summary: #142136 added a runtime alignment assertion. But the assumption is probably too strict for more flexible use cases of AOTI, e.g. python deployment, see a recent error torchchat ran into for more details, https://github.com/pytorch/torchchat/actions/runs/12322072267/job/34394851280 . This PR relaxes the runtime check and implements copy_misaligned_inputs in cpp instead. ghstack-source-id: 989e60a Pull Request resolved: #143236
|
@desertfire has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@pytorchbot merge (Initiating merge automatically since Phabricator Diff has merged) |
Merge startedYour change will be merged once all checks pass (ETA 0-4 Hours). Learn more about merging in the wiki. Questions? Feedback? Please reach out to the PyTorch DevX Team |
|
I think most tensors are allocated aligned, but slicing and views may break that in Python/Pytorch. One solution might be to literally clone every input in flat_inputs. And run a few benchmarks? How are views and slices dealt with in Inductor? Unless there's a copy to make these aligned, the same issue would happen with slices generated with inductor, or when calling from byte code into JIT-compiled Inductor code? Did adding assertions actually improve code quality/speed? The right place may be just around the actual control transfer to C/C++ -- you'd want to do this in the Python caller (or a C/C++ service function that clones all Pytorch arrays as arguments, optionally only if and only if they need a copy ). Also, If we push it in the callee, then every caller, even those that already provide aligned arguments will pay. (Or at least have to evaluate an alignment check and conditional...) Either way, the payoff may be significant for large matrices -- copy is O(n^2), matmul O(n^3) for a [n,n] *[n,n] GEMM. |
Stack from ghstack (oldest at bottom):
Summary: #142136 added a runtime alignment assertion. But the assumption is probably too strict for more flexible use cases of AOTI, e.g. python deployment, see a recent error torchchat ran into for more details, https://github.com/pytorch/torchchat/actions/runs/12322072267/job/34394851280 . This PR relaxes the runtime check and implements copy_misaligned_inputs in cpp instead.
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @chauhang @aakhundov
Differential Revision: D67287922