[Inductor CUTLASS backend] Step 5: CUTLASS gemm kernels #107933
Conversation
[ghstack-poisoned]
| ): | ||
| Y_compiled = torch.compile(mm, dynamic=dynamic)(a, b) | ||
| Y = mm(a, b) | ||
| torch.testing.assert_close(Y_compiled, Y) |
There was a problem hiding this comment.
I think we should somehow make sure here that the function under test (e.g. a generated CUDA/Cutlass matmul implementation) is actually being called and not, for example, a Triton variant.
There was a problem hiding this comment.
Agreed. run_and_get_code may be helpful (example from other test).
There was a problem hiding this comment.
Makes sense, updated max_autotune_gemm_backends to ensure that there is a test case for "CUTLASS" only.
| extern "C" { | ||
| {{kernel.def_kernel(inputs=[X, W, Bias], outputs=[Y], names_str="X, W, Bias, Y", input_reorder=input_reorder)}} { | ||
| try { | ||
| // printf("X: %p\n", (void*)(X)); |
There was a problem hiding this comment.
We could conditionally generate these in if (in an uncommented manner) if some debug flag is set, otherwise I think they should not become part of the generated code.
There was a problem hiding this comment.
Yes didn't fix it in draft code ^^.
| num_3x_ops += 1 | ||
| else: | ||
| num_2x_ops += 1 | ||
| print(f"Got cutlass configs: {len(res)=}, {num_3x_ops=}, {num_2x_ops=}") |
There was a problem hiding this comment.
Leftover print statement from development?
There was a problem hiding this comment.
Yes didn't fix it in draft code ^^.
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
| ): | ||
| Y_compiled = torch.compile(mm, dynamic=dynamic)(a, b) | ||
| Y = mm(a, b) | ||
| torch.testing.assert_close(Y_compiled, Y) |
There was a problem hiding this comment.
Makes sense, updated max_autotune_gemm_backends to ensure that there is a test case for "CUTLASS" only.
| num_3x_ops += 1 | ||
| else: | ||
| num_2x_ops += 1 | ||
| print(f"Got cutlass configs: {len(res)=}, {num_3x_ops=}, {num_2x_ops=}") |
There was a problem hiding this comment.
Yes didn't fix it in draft code ^^.
| extern "C" { | ||
| {{kernel.def_kernel(inputs=[X, W, Bias], outputs=[Y], names_str="X, W, Bias, Y", input_reorder=input_reorder)}} { | ||
| try { | ||
| // printf("X: %p\n", (void*)(X)); |
There was a problem hiding this comment.
Yes didn't fix it in draft code ^^.
This is the step 5 to add cutlass as an alternative inductor backend. Feature request: #106991. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
This is the step 5 to add cutlass as an alternative inductor backend. Feature request: #106991. cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng Xia-Weiwen wenzhe-nrv jiayisunx peterbell10 ngimel yf225 chenyang78 kadeng muchulee8 aakhundov [ghstack-poisoned]
|
@kadeng @aakhundov I'm sorry I messed up the PR stack and somehow ghstack recreated a new PR for this: #108015. Let's move to this PR for discussions. |
This is the step 5 to add cutlass as an alternative inductor backend.
Feature request: #106991.
Stack from ghstack (oldest at bottom):
cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @peterbell10 @ngimel @yf225 @chenyang78 @kadeng @muchulee8 @aakhundov