Extends type and shape tracing with device #9796
Conversation
mruberry
requested review from
apaszke,
colesbury,
ezyang,
gchanan,
soumith and
zdevito
as code owners
|
@pytorchbot retest this please |
|
CI failure looks real, I guess? |
|
@pytorchbot retest this please |
1 similar comment
|
@pytorchbot retest this please |
|
The CI failure was real; I had to merge with master on a file not presented in the web UX. My mistake submitting that earlier commit. The current CI failures appear unrelated. |
|
@pytorchbot retest this please. |
|
The failure of all 3 rocm builds is worrying; running one last retest to see if it persists. @pytorchbot retest this please. |
|
5 failures: pr/pytorch-linux-trust-pynightly 17:24:33 FATAL: command execution failed Seems unrelated. pr/py2-clang3.8-rocmnightly-ubuntu16.04 17:42:06 CMake Error at caffe2/CMakeLists.txt:273 (set_target_properties): Seems unrelated. pr/caffe2-py2-gcc5-ubuntu16.04-test 18:01:04 lib/python2.7/dist-packages/caffe2/python/operator_test/fc_operator_test.py::TestFcOperator::test_fc_transposed FAILED [ 80%] Seems unrelated. pr/caffe2-py2-cuda9.1-cudnn7-ubuntu16.04-test 18:07:46 Build timed out (after 45 minutes). Marking the build as failed. Seems unrelated. pr/caffe2-py2-clang3.8-rocmnightly-ubuntu16.04-build 17:43:31 CMake Error at caffe2/CMakeLists.txt:273 (set_target_properties): Same issue as prior ROCm build. @ezyang any idea on these set_target_properties issues? |
|
@pytorchbot retest this please |
|
No it's very puzzling. Even more puzzling because you don't have any cmake changes. |
|
oh I know! the rocmnightly job is "stale": it is failing but it's not a real failure, it's just that our CI is stupid and doesn't know to clear the old failures. So I think this PR is good to go. |
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
ezyang is landing this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
|
Sorry, this merge conflicted before it could land. I fixed the merge conflict, rerunning tests... |
facebook-github-bot
left a comment
facebook-github-bot
left a comment
There was a problem hiding this comment.
ezyang has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Summary: This PR extends the existing type and shape metadata tracing and verification done in autograd with device information. This expansion of tracing is required for #8354, is likely useful in other scenarios, and is a healthy sanity check, just like type and shape tracing. The precise changes are: - TypeAndShape -> InputMetadata, now includes device() - Creating InputMetadata is simplified to just require a tensor, and callers were updated to use this simpler invocation wherever possible - The gradient accumulator of a variable is now reset when set_data() is called if either the type or device changes, and this reset now locks to avoid contention with acquiring the gradient accumulator - Mismatched devices during backward() will throw a runtime error, just like mismatched type and shape - (Bonus!) Two uninitialized pointers in THCReduce are now initialized (to nullptr) to prevent build warnings fyi colesbury Pull Request resolved: pytorch/pytorch#9796 Reviewed By: goldsborough Differential Revision: D9119325 Pulled By: ezyang fbshipit-source-id: 76d1861b8d4f74db0575ff1f3bd965e18f9463de
Summary: This PR extends the existing type and shape metadata tracing and verification done in autograd with device information. This expansion of tracing is required for pytorch#8354, is likely useful in other scenarios, and is a healthy sanity check, just like type and shape tracing. The precise changes are: - TypeAndShape -> InputMetadata, now includes device() - Creating InputMetadata is simplified to just require a tensor, and callers were updated to use this simpler invocation wherever possible - The gradient accumulator of a variable is now reset when set_data() is called if either the type or device changes, and this reset now locks to avoid contention with acquiring the gradient accumulator - Mismatched devices during backward() will throw a runtime error, just like mismatched type and shape - (Bonus!) Two uninitialized pointers in THCReduce are now initialized (to nullptr) to prevent build warnings fyi colesbury Pull Request resolved: pytorch#9796 Reviewed By: goldsborough Differential Revision: D9119325 Pulled By: ezyang fbshipit-source-id: 76d1861b8d4f74db0575ff1f3bd965e18f9463de
Summary: This PR extends the existing type and shape metadata tracing and verification done in autograd with device information. This expansion of tracing is required for pytorch#8354, is likely useful in other scenarios, and is a healthy sanity check, just like type and shape tracing. The precise changes are: - TypeAndShape -> InputMetadata, now includes device() - Creating InputMetadata is simplified to just require a tensor, and callers were updated to use this simpler invocation wherever possible - The gradient accumulator of a variable is now reset when set_data() is called if either the type or device changes, and this reset now locks to avoid contention with acquiring the gradient accumulator - Mismatched devices during backward() will throw a runtime error, just like mismatched type and shape - (Bonus!) Two uninitialized pointers in THCReduce are now initialized (to nullptr) to prevent build warnings fyi colesbury Pull Request resolved: pytorch#9796 Reviewed By: goldsborough Differential Revision: D9119325 Pulled By: ezyang fbshipit-source-id: 76d1861b8d4f74db0575ff1f3bd965e18f9463de
4 participants
This PR extends the existing type and shape metadata tracing and verification done in autograd with device information. This expansion of tracing is required for #8354, is likely useful in other scenarios, and is a healthy sanity check, just like type and shape tracing.
The precise changes are:
fyi @colesbury