Redesign custom op functionlaization for better re-inplace #134409

laithsakka · 2024-08-25T06:07:56Z

Stack from ghstack (oldest at bottom):

-> Redesign custom op functionlaization for better re-inplace #134409

The new implementation (auto_functionalized_v2) is enabled by default but can be disable
using an inductor flag.
In export mode the old implementation is used.

Motiviation
Previous functionalization fails to re-inplace arguments when they are view over other tensors.
see issue #131192
The new functionalization is easier to re-inplace for views.

A) Functionalizations pass
consider a program:


func(t)
    x = t[0]
    y = t[1]
    foo(x, y) # custom operator with x, y mutable
    return (x, y, t)

To functionalize foo we generate a function that operates on the base tensors of the inputs; (x.base() and y.base())
and record how to regenerates the views out of the base for argument x by recording ViewInfo=(x.base(), x.size(), x.stride, x,storage_offset())

Due to some limitations on the torch.export arguments format, we have to generate alot of arguments, but this is something we can simplify in the future, for the example above we get the following function.

auto_functionalized = torch.ops.higher_order.auto_functionalized(torch.ops.mylib.foo.default,
  _x_base_index = 0, _x_size = (), _x_stride = (), _x_storage_offset = 0 ,
  _y_base_index = 0,_y_size = (), _y_stride = (), _y_storage_offset = 1   ,
  _all_bases = [arg0_1])

In the code above:
- _all_bases[t]: refers to a unique set of bases for all foo arguments.
- for each argument x we have _x_base_index, _x_size, _x_stride, _x_storage_offset that can be used to (1) regenerate x from _all_bases[_x_base_index] or a copy of a the base.
the output of auto_functionalized is foo output , followed by x tensors one for each base in _all_bases, that is a copy of the base tensor after observing the mutations of the all the arguments that are views of that base.
for each use of a base in _all_bases or a view of it , that are after the call to foo, replace it with a view of the new output

for the function above after functionalization we get :

   def forward(self, arg0_1: "f32[2][1]cpu"):
       auto_functionalized = torch.ops.higher_order.auto_functionalized(torch.ops.mylib.foo.default, _x_base_index = 0, _x_size = (), _x_stride = (), _x_storage_offset = 0, _y_base_index = 0, _y_size = (), _y_stride = (), _y_storage_offset = 1, _all_bases = [arg0_1])
       getitem_1: "f32[2][1]cpu" = auto_functionalized[1];  auto_functionalized = None
       copy_: "f32[2][1]cpu" = torch.ops.aten.copy_.default(arg0_1, getitem_1);  arg0_1 = copy_ = None
       
       # No stacktrace found for following nodes
       select_2: "f32[][]cpu" = torch.ops.aten.select.int(getitem_1, 0, 0)
       select_3: "f32[][]cpu" = torch.ops.aten.select.int(getitem_1, 0, 1);  getitem_1 = None
       return (select_2, select_3)

B) Semantics of auto_functionalize
The new semantics of auto_functionalize is as the following:

For each base in all_bases, copy the base and create all_bases copies. (if a base is inplaced we do not need to copy it)
For each arg, regenerate the arg from the copy of its base using the view information above.
return the original foo output followed by the new bases.

C) Re-inplace pass
since auto_functionalize not copy the bases, what we actually inplace is the bases.
(run just like before but on the beses instead of args).

For each base b in _all_bases check if there is any use of base (or its aliases/views) after auto_functionalize (before its overwritten with a copy) if there is not any, then inplace it (avoid copying it in step 1 above).

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang @rec

[ghstack-poisoned]

pytorch-bot · 2024-08-25T06:07:58Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/134409

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (9 Unrelated Failures)

As of commit a0367e5 with merge base c818ecd ():

BROKEN TRUNK - The following jobs failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

This comment was automatically generated by Dr. CI and updates every 15 minutes.

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

ghstack-source-id: 7b86295 Pull Request resolved: #134409

test/dynamo/test_misc.py

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

ghstack-source-id: 9ec1119 Pull Request resolved: #134409

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

laithsakka · 2024-08-26T19:56:18Z

add prints to invesitaget tests failing only in oss.

- The new implementation (auto_functionalized_v2) is enabled by default but can be disable using an inductor flag. - In export mode the old implementation is used. **Motiviation** Previous functionalization fails to re-inplace arguments when they are view over other tensors. see issue #131192 The new functionalization is easier to re-inplace for views. **A) Functionalizations pass** consider a program: ``` func(t) x = t[0] y = t[1] foo(x, y) # custom operator with x, y mutable return (x, y, t) ``` - To functionalize `foo` we generate a function that operates on the base tensors of the inputs; (x.base() and y.base()) and record how to regenerates the views out of the base for argument x by recording ```ViewInfo=(x.base(), x.size(), x.stride, x,storage_offset())``` - Due to some limitations on the torch.export arguments format, we have to generate alot of arguments, but this is something we can simplify in the future, for the example above we get the following function. ``` auto_functionalized = torch.ops.higher_order.auto_functionalized(torch.ops.mylib.foo.default, _x_base_index = 0, _x_size = (), _x_stride = (), _x_storage_offset = 0 , _y_base_index = 0,_y_size = (), _y_stride = (), _y_storage_offset = 1 , _all_bases = [arg0_1]) ``` - In the code above: - _all_bases[t]: refers to a unique set of bases for all foo arguments. - for each argument x we have _x_base_index, _x_size, _x_stride, _x_storage_offset that can be used to (1) regenerate x from _all_bases[_x_base_index] or a copy of a the base. - the output of auto_functionalized is foo output , followed by x tensors one for each base in _all_bases, that is a copy of the base tensor after observing the mutations of the all the arguments that are views of that base. - for each use of a base in _all_bases or a view of it , that are after the call to foo, replace it with a view of the new output for the function above after functionalization we get : ``` def forward(self, arg0_1: "f32[2][1]cpu"): auto_functionalized = torch.ops.higher_order.auto_functionalized(torch.ops.mylib.foo.default, _x_base_index = 0, _x_size = (), _x_stride = (), _x_storage_offset = 0, _y_base_index = 0, _y_size = (), _y_stride = (), _y_storage_offset = 1, _all_bases = [arg0_1]) getitem_1: "f32[2][1]cpu" = auto_functionalized[1]; auto_functionalized = None copy_: "f32[2][1]cpu" = torch.ops.aten.copy_.default(arg0_1, getitem_1); arg0_1 = copy_ = None # No stacktrace found for following nodes select_2: "f32[][]cpu" = torch.ops.aten.select.int(getitem_1, 0, 0) select_3: "f32[][]cpu" = torch.ops.aten.select.int(getitem_1, 0, 1); getitem_1 = None return (select_2, select_3) ``` **B) Semantics of auto_functionalize** The new semantics of auto_functionalize is as the following: 1. For each base in all_bases, copy the base and create all_bases copies. (if a base is inplaced we do not need to copy it) 2. For each arg, regenerate the arg from the copy of its base using the view information above. 3. return the original foo output followed by the new bases. **C) Re-inplace pass** since auto_functionalize not copy the bases, what we actually inplace is the bases. (run just like before but on the beses instead of args). 1. For each base b in _all_bases check if there is any use of base (or its aliases/views) after auto_functionalize (before its overwritten with a copy) if there is not any, then inplace it (avoid copying it in step 1 above). cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

ghstack-source-id: 66b4b77 Pull Request resolved: #134409

laithsakka · 2024-09-04T00:51:16Z

@pytorchbot merge

pytorchmergebot · 2024-09-04T00:53:25Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2024-09-04T02:11:14Z

Merge failed

Reason: 1 jobs have failed, first few of them are: linux-binary-libtorch-cxx11-abi / libtorch-cpu-shared-with-deps-cxx11-abi-test / test

Details for Dev Infra team

Raised by workflow job

- The new implementation (auto_functionalized_v2) is enabled by default but can be disable using an inductor flag. - In export mode the old implementation is used. **Motiviation** Previous functionalization fails to re-inplace arguments when they are view over other tensors. see issue #131192 The new functionalization is easier to re-inplace for views. **A) Functionalizations pass** consider a program: ``` func(t) x = t[0] y = t[1] foo(x, y) # custom operator with x, y mutable return (x, y, t) ``` - To functionalize `foo` we generate a function that operates on the base tensors of the inputs; (x.base() and y.base()) and record how to regenerates the views out of the base for argument x by recording ```ViewInfo=(x.base(), x.size(), x.stride, x,storage_offset())``` - Due to some limitations on the torch.export arguments format, we have to generate alot of arguments, but this is something we can simplify in the future, for the example above we get the following function. ``` auto_functionalized = torch.ops.higher_order.auto_functionalized(torch.ops.mylib.foo.default, _x_base_index = 0, _x_size = (), _x_stride = (), _x_storage_offset = 0 , _y_base_index = 0,_y_size = (), _y_stride = (), _y_storage_offset = 1 , _all_bases = [arg0_1]) ``` - In the code above: - _all_bases[t]: refers to a unique set of bases for all foo arguments. - for each argument x we have _x_base_index, _x_size, _x_stride, _x_storage_offset that can be used to (1) regenerate x from _all_bases[_x_base_index] or a copy of a the base. - the output of auto_functionalized is foo output , followed by x tensors one for each base in _all_bases, that is a copy of the base tensor after observing the mutations of the all the arguments that are views of that base. - for each use of a base in _all_bases or a view of it , that are after the call to foo, replace it with a view of the new output for the function above after functionalization we get : ``` def forward(self, arg0_1: "f32[2][1]cpu"): auto_functionalized = torch.ops.higher_order.auto_functionalized(torch.ops.mylib.foo.default, _x_base_index = 0, _x_size = (), _x_stride = (), _x_storage_offset = 0, _y_base_index = 0, _y_size = (), _y_stride = (), _y_storage_offset = 1, _all_bases = [arg0_1]) getitem_1: "f32[2][1]cpu" = auto_functionalized[1]; auto_functionalized = None copy_: "f32[2][1]cpu" = torch.ops.aten.copy_.default(arg0_1, getitem_1); arg0_1 = copy_ = None # No stacktrace found for following nodes select_2: "f32[][]cpu" = torch.ops.aten.select.int(getitem_1, 0, 0) select_3: "f32[][]cpu" = torch.ops.aten.select.int(getitem_1, 0, 1); getitem_1 = None return (select_2, select_3) ``` **B) Semantics of auto_functionalize** The new semantics of auto_functionalize is as the following: 1. For each base in all_bases, copy the base and create all_bases copies. (if a base is inplaced we do not need to copy it) 2. For each arg, regenerate the arg from the copy of its base using the view information above. 3. return the original foo output followed by the new bases. **C) Re-inplace pass** since auto_functionalize not copy the bases, what we actually inplace is the bases. (run just like before but on the beses instead of args). 1. For each base b in _all_bases check if there is any use of base (or its aliases/views) after auto_functionalize (before its overwritten with a copy) if there is not any, then inplace it (avoid copying it in step 1 above). cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

ghstack-source-id: a273c31 Pull Request resolved: #134409

laithsakka · 2024-09-04T17:01:45Z

@pytorchbot merge

pytorchmergebot · 2024-09-04T17:03:33Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Pull Request resolved: #135142 Approved by: https://github.com/zou3519 ghstack dependencies: #134409

…o_functionalized` (pytorch#134831) `unsafe_remove_auto_functionalized_pass` can be written as using `decompose_auto_functionalized`, this way we do not have to update it each time we do a change to `auto_functionalize` (Ex pytorch#134409) , and we avoid duplicate logics implemented in two different ways. Pull Request resolved: pytorch#134831 Approved by: https://github.com/zou3519

…34409) - The new implementation (auto_functionalized_v2) is enabled by default but can be disable using an inductor flag. - In export mode the old implementation is used. **Motiviation** Previous functionalization fails to re-inplace arguments when they are view over other tensors. see issue pytorch#131192 The new functionalization is easier to re-inplace for views. **A) Functionalizations pass** consider a program: ``` func(t) x = t[0] y = t[1] foo(x, y) # custom operator with x, y mutable return (x, y, t) ``` - To functionalize `foo` we generate a function that operates on the base tensors of the inputs; (x.base() and y.base()) and record how to regenerates the views out of the base for argument x by recording ```ViewInfo=(x.base(), x.size(), x.stride, x,storage_offset())``` - Due to some limitations on the torch.export arguments format, we have to generate alot of arguments, but this is something we can simplify in the future, for the example above we get the following function. ``` auto_functionalized = torch.ops.higher_order.auto_functionalized(torch.ops.mylib.foo.default, _x_base_index = 0, _x_size = (), _x_stride = (), _x_storage_offset = 0 , _y_base_index = 0,_y_size = (), _y_stride = (), _y_storage_offset = 1 , _all_bases = [arg0_1]) ``` - In the code above: - _all_bases[t]: refers to a unique set of bases for all foo arguments. - for each argument x we have _x_base_index, _x_size, _x_stride, _x_storage_offset that can be used to (1) regenerate x from _all_bases[_x_base_index] or a copy of a the base. - the output of auto_functionalized is foo output , followed by x tensors one for each base in _all_bases, that is a copy of the base tensor after observing the mutations of the all the arguments that are views of that base. - for each use of a base in _all_bases or a view of it , that are after the call to foo, replace it with a view of the new output for the function above after functionalization we get : ``` def forward(self, arg0_1: "f32[2][1]cpu"): auto_functionalized = torch.ops.higher_order.auto_functionalized(torch.ops.mylib.foo.default, _x_base_index = 0, _x_size = (), _x_stride = (), _x_storage_offset = 0, _y_base_index = 0, _y_size = (), _y_stride = (), _y_storage_offset = 1, _all_bases = [arg0_1]) getitem_1: "f32[2][1]cpu" = auto_functionalized[1]; auto_functionalized = None copy_: "f32[2][1]cpu" = torch.ops.aten.copy_.default(arg0_1, getitem_1); arg0_1 = copy_ = None # No stacktrace found for following nodes select_2: "f32[][]cpu" = torch.ops.aten.select.int(getitem_1, 0, 0) select_3: "f32[][]cpu" = torch.ops.aten.select.int(getitem_1, 0, 1); getitem_1 = None return (select_2, select_3) ``` **B) Semantics of auto_functionalize** The new semantics of auto_functionalize is as the following: 1. For each base in all_bases, copy the base and create all_bases copies. (if a base is inplaced we do not need to copy it) 2. For each arg, regenerate the arg from the copy of its base using the view information above. 3. return the original foo output followed by the new bases. **C) Re-inplace pass** since auto_functionalize not copy the bases, what we actually inplace is the bases. (run just like before but on the beses instead of args). 1. For each base b in _all_bases check if there is any use of base (or its aliases/views) after auto_functionalize (before its overwritten with a copy) if there is not any, then inplace it (avoid copying it in step 1 above). Pull Request resolved: pytorch#134409 Approved by: https://github.com/zou3519

…5142) Pull Request resolved: pytorch#135142 Approved by: https://github.com/zou3519 ghstack dependencies: pytorch#134409

the new design

b4d435d

[ghstack-poisoned]

laithsakka mentioned this pull request Aug 25, 2024

Remove ansi escape chars in assertExpectedInline and add options to skip comments and to skip empty lines #134248

Closed

pytorch-bot bot added ciflow/inductor module: dynamo module: inductor labels Aug 25, 2024

laithsakka changed the title ~~the new design~~ Redesign functionlaization for better re-inplace Aug 25, 2024

This was referenced Aug 25, 2024

Make relationship between outputs and inputs of auto_functionalize static and not depending on addresses of new inputs passed #134315

Closed

Re-design functionalization to minimize miss in-placing when args are views. #133045

Closed

Update on "Redesign functionlaization for better re-inplace "

17c00e0

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

Update on "Redesign functionlaization for better re-inplace "

efbe75a

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

Update on "Redesign functionlaization for better re-inplace "

5f07565

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

laithsakka requested review from angelayi, avikchaudhuri, tugsbayasgalan, ydwu4 and zhxchen17 as code owners August 25, 2024 23:19

laithsakka marked this pull request as draft August 26, 2024 02:47

Update on "Redesign functionlaization for better re-inplace "

925d715

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

laithsakka added a commit that referenced this pull request Aug 26, 2024

the new design

8331022

ghstack-source-id: 7b86295 Pull Request resolved: #134409

laithsakka commented Aug 26, 2024

View reviewed changes

test/dynamo/test_misc.py Outdated Show resolved Hide resolved

Update on "Redesign functionlaization for better re-inplace "

e64cabd

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

laithsakka added a commit that referenced this pull request Aug 26, 2024

the new design

0fb217a

ghstack-source-id: 9ec1119 Pull Request resolved: #134409

Update on "Redesign functionlaization for better re-inplace "

6e6c8a5

cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy yf225 chenyang78 kadeng muchulee8 ColinPeppler amjames desertfire chauhang rec [ghstack-poisoned]

laithsakka added the release notes: inductor label Sep 3, 2024

laithsakka added a commit that referenced this pull request Sep 3, 2024

the new design

9403a8e

ghstack-source-id: 66b4b77 Pull Request resolved: #134409

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 4, 2024

pytorchmergebot added the merging label Sep 4, 2024

pytorchmergebot removed the merging label Sep 4, 2024

laithsakka added a commit that referenced this pull request Sep 4, 2024

the new design

1988256

ghstack-source-id: a273c31 Pull Request resolved: #134409

pytorchmergebot added the merging label Sep 4, 2024

pytorchmergebot closed this in c8ab9b0 Sep 4, 2024

pytorchmergebot added Merged and removed merging labels Sep 4, 2024

This was referenced Sep 4, 2024

Track base of FunctionalTensor in inference mode. #135141

Closed

Fix: use clone_preserve_strides in auto_functionalized_v2 #135142

Closed

laithsakka mentioned this pull request Sep 5, 2024

make torch.compile work with vLLM (facebook/opt-125m , meta-llama/Llama-2-7b-hf, meta-llama/Llama-3-8b-hf) models #130174

Closed

laithsakka mentioned this pull request Sep 30, 2024

Inductor max used memory perform worse with as_strided vs split_with_sizes + subscripts #137057

Closed

github-actions bot deleted the gh/laithsakka/50/head branch October 5, 2024 02:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Redesign custom op functionlaization for better re-inplace #134409

Redesign custom op functionlaization for better re-inplace #134409

Uh oh!

laithsakka commented Aug 25, 2024 •

edited

Loading

Uh oh!

pytorch-bot bot commented Aug 25, 2024 •

edited

Loading

Uh oh!

Uh oh!

laithsakka commented Aug 26, 2024

Uh oh!

laithsakka commented Sep 4, 2024

Uh oh!

pytorchmergebot commented Sep 4, 2024

Uh oh!

pytorchmergebot commented Sep 4, 2024

Uh oh!

laithsakka commented Sep 4, 2024

Uh oh!

pytorchmergebot commented Sep 4, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Redesign custom op functionlaization for better re-inplace #134409

Redesign custom op functionlaization for better re-inplace #134409

Uh oh!

Conversation

laithsakka commented Aug 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 25, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/134409

✅ You can merge normally! (9 Unrelated Failures)

Uh oh!

Uh oh!

laithsakka commented Aug 26, 2024

Uh oh!

laithsakka commented Sep 4, 2024

Uh oh!

pytorchmergebot commented Sep 4, 2024

Merge started

Uh oh!

pytorchmergebot commented Sep 4, 2024

Merge failed

Uh oh!

laithsakka commented Sep 4, 2024

Uh oh!

pytorchmergebot commented Sep 4, 2024

Merge started

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

laithsakka commented Aug 25, 2024 •

edited

Loading

pytorch-bot bot commented Aug 25, 2024 •

edited

Loading