Skip to content

Conversation

@chunyuan-w
Copy link
Collaborator

@chunyuan-w chunyuan-w commented Sep 23, 2024

Stack from ghstack (oldest at bottom):

Fixes the max-autotune failure of soft_actor_critic of Torchbench in FP32 single thread dynamic shape case:

  File "/home/user/inductor/pytorch/torch/_inductor/codegen/cpp_micro_gemm.py", line 136, in codegen_call
    C_ptr = f"&({kernel.index(C, [0, 0])})"
  File "/home/user/inductor/pytorch/torch/_inductor/codegen/cpp_template_kernel.py", line 135, in index
    else self.args.input(node.get_name())
  File "/home/user/inductor/pytorch/torch/_inductor/codegen/common.py", line 1251, in input
    assert name not in V.graph.removed_buffers, name
AssertionError: buf_GemmOut

The 1st and 2nd linear does not need to use local buffer while the 3rd linear needs to use local buffer.
The 3rd linear which uses local buffer will add its global buffer (named as buf_GemmOut) into V.graph.removed_buffers.

When scheduling the nodes, the 1st linear (won't use local buffer) will get its output buffer (also named as buf_GemmOut) from the input and found that it's in the V.graph.removed_buffers and raise AssertionError. The issue is that the output buffer of all these linears are all names with buf_GemmOut, which have a conflict.

Rename these buffers by adding the name of the template_buffer as the prefix.

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @yf225 @chenyang78 @kadeng @muchulee8 @ColinPeppler @amjames @desertfire @chauhang

@pytorch-bot
Copy link

pytorch-bot bot commented Sep 23, 2024

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/136419

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit e31d24e with merge base e9bfbf7 (image):
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[ghstack-poisoned]
[ghstack-poisoned]
@chunyuan-w chunyuan-w marked this pull request as ready for review September 24, 2024 02:09
chunyuan-w added a commit that referenced this pull request Sep 24, 2024
ghstack-source-id: 2a0254b
Pull Request resolved: #136419
@chunyuan-w chunyuan-w requested a review from jgong5 September 24, 2024 06:32
[ghstack-poisoned]
[ghstack-poisoned]
chunyuan-w added a commit that referenced this pull request Sep 25, 2024
ghstack-source-id: f15a75d
Pull Request resolved: #136419
[ghstack-poisoned]
[ghstack-poisoned]
@chunyuan-w chunyuan-w added the ciflow/trunk Trigger trunk jobs on your pull request label Sep 27, 2024
@chunyuan-w
Copy link
Collaborator Author

@pytorchbot merge

@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

@github-actions github-actions bot deleted the gh/chunyuan-w/32/head branch October 28, 2024 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants