check error status of CUDA launch after Magma kernels #29003
Conversation
There was a problem hiding this comment.
I think the error checks are sufficient in magma<function_name> definitions, unless you think otherwise.
For the inverse failure, the source of failure could have been the sswap call in magma_sgetri_gpu. Curiously, I grepped to see if there was any error checking in place, and there isn't (weirdly enough).
| infos[i] = info_array[i]; | ||
| } | ||
| } | ||
| AT_CUDA_CHECK(cudaGetLastError()); |
There was a problem hiding this comment.
Why would it be required to do it here?
| for (int64_t i = 0; i < batch_size; i++) { | ||
| infos[i] = info_array[i]; | ||
| } | ||
| AT_CUDA_CHECK(cudaGetLastError()); |
| magmaGetri<scalar_t>( | ||
| n, self_data, n, ipiv.data_ptr<magma_int_t>(), dwork.data_ptr<scalar_t>(), lwork, &info_tmp); | ||
| info = info_tmp; | ||
| AT_CUDA_CHECK(cudaGetLastError()); |
|
thanks for the review, made changes. sorry for the delay. |
There was a problem hiding this comment.
Thank you for adding these @soumith, no problems about the delay.
Is there a way to test this?
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
@soumith is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Summary: as part of pytorch/hub#62 I found that the stack-trace of a failed kernel launch was being recorded elsewhere, even with CUDA_LAUNCH_BLOCKING=1. So, I started debugging, and found that magma launches don't do error checking. I eventually found the issue to be that I didn't compile-in sm37 SASS into the magma binary and the failure was on `x.inverse()`, and that's somehow a problem for magma 2.5.1 (but not 2.5.0). Pull Request resolved: pytorch/pytorch#29003 Differential Revision: D18397358 Pulled By: soumith fbshipit-source-id: 04baca68eac209d7af773daddd0193697d4ab0d9
as part of pytorch/hub#62 I found that the stack-trace of a failed kernel launch was being recorded elsewhere, even with CUDA_LAUNCH_BLOCKING=1.
So, I started debugging, and found that magma launches don't do error checking.
I eventually found the issue to be that I didn't compile-in sm37 SASS into the magma binary and the failure was on
x.inverse(), and that's somehow a problem for magma 2.5.1 (but not 2.5.0).