Show full call stack in TorchScript exception even when calls were inlined. #29911
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…lined.
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
1. at fail.py:23:11
@torch.jit.script
def bar(c, b):
return foo(c, b)
~~~ <--- HERE
2. at fail.py:19:11
@torch.jit.script
def foo(c, b):
return baz(c, b)
~~~ <--- HERE
3. at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
[ghstack-poisoned]
This was referenced Nov 15, 2019
facebook-github-bot
added
the
oncall: jit
suo
approved these changes
Nov 15, 2019
zdevito
reviewed
Nov 15, 2019
torch/csrc/jit/interpreter.cpp
Outdated
| if (i > 0) { | ||
| out << "during call "; | ||
| } | ||
| int idx = 1; |
Contributor
There was a problem hiding this comment.
Can you format the error trace to more closely match python format. e.g. call it a Traceback have the string "(most recent call last)".
I think it should also mention the function name being called in addition to the source range.
torch/csrc/jit/interpreter.cpp
Outdated
| 1; // make sure we report the call node, not the node after it | ||
| Node* node = frame.function->instructions_source_[pc]; | ||
| if (i > 0) { | ||
| out << "during call "; |
Contributor
There was a problem hiding this comment.
"during call" formatting here is going to cause the formatting to be different depending of if things have been inlined. I think we should have a test that runs the model with/without inlining and checks the stack trace appears byte-for-byte the same.
| with self.assertRaises(RuntimeError) as cm: | ||
| bar(torch.rand(10), torch.rand(9)) | ||
| FileCheck().check("The above operation failed in interpreter, with the following stack trace").check("bar").check("foo").check("baz").run(str(cm.exception)) | ||
|
|
Contributor
There was a problem hiding this comment.
We should have a test that includes an interface call (see test_interface) to exercise the combined inlined/not-inlined case.
Member
CircleCI build failures summaryAs of commit a7a5854:
Here are the reasons each build failed:
This comment was automatically generated by Dr. CI. Please report bugs/suggestions on the GitHub issue tracker. This comment has been revised 5 time(s). |
…lls were inlined."
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
1. at fail.py:23:11
@torch.jit.script
def bar(c, b):
return foo(c, b)
~~~ <--- HERE
2. at fail.py:19:11
@torch.jit.script
def foo(c, b):
return baz(c, b)
~~~ <--- HERE
3. at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Differential Revision: [D18532812](https://our.internmc.facebook.com/intern/diff/D18532812)
[ghstack-poisoned]
…lls were inlined."
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
This change also makes torchscript stacktraces look like python ones.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 41, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter.
Traceback (most recent call last):
File "fail.py", line 33
return foo(c, b)
File "fail.py", line 29, in foo
return baz(c, b)
File "fail.py", line 25, in baz
return c + b
```
Output of non-scripted python code:
```
Traceback (most recent call last):
File "fail.py", line 36, in <module>
bar(torch.rand(10), torch.rand(9))
File "fail.py", line 21, in bar
return foo(c, b)
File "fail.py", line 18, in foo
return baz(c, b)
File "fail.py", line 15, in baz
return c + b
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
```
Make torchscript stack traces look like python ones.
Differential Revision: [D18532812](https://our.internmc.facebook.com/intern/diff/D18532812)
[ghstack-poisoned]
Author
|
@zdevito could you please take another look? |
added 3 commits
November 15, 2019 19:48
…lls were inlined."
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
This change also makes torchscript stacktraces look like python ones.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 41, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter.
Traceback (most recent call last):
File "fail.py", line 33
return foo(c, b)
File "fail.py", line 29, in foo
return baz(c, b)
File "fail.py", line 25, in baz
return c + b
```
Output of non-scripted python code:
```
Traceback (most recent call last):
File "fail.py", line 36, in <module>
bar(torch.rand(10), torch.rand(9))
File "fail.py", line 21, in bar
return foo(c, b)
File "fail.py", line 18, in foo
return baz(c, b)
File "fail.py", line 15, in baz
return c + b
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
```
Make torchscript stack traces look like python ones.
Differential Revision: [D18532812](https://our.internmc.facebook.com/intern/diff/D18532812)
[ghstack-poisoned]
…lls were inlined."
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
This change also makes torchscript stacktraces look like python ones.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 41, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter.
Traceback (most recent call last):
File "fail.py", line 33
return foo(c, b)
File "fail.py", line 29, in foo
return baz(c, b)
File "fail.py", line 25, in baz
return c + b
```
Output of non-scripted python code:
```
Traceback (most recent call last):
File "fail.py", line 36, in <module>
bar(torch.rand(10), torch.rand(9))
File "fail.py", line 21, in bar
return foo(c, b)
File "fail.py", line 18, in foo
return baz(c, b)
File "fail.py", line 15, in baz
return c + b
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
```
Make torchscript stack traces look like python ones.
Differential Revision: [D18532812](https://our.internmc.facebook.com/intern/diff/D18532812)
[ghstack-poisoned]
…lls were inlined."
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
This change also makes torchscript stacktraces look like python ones.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 41, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter.
Traceback (most recent call last):
File "fail.py", line 33
return foo(c, b)
File "fail.py", line 29, in foo
return baz(c, b)
File "fail.py", line 25, in baz
return c + b
```
Output of non-scripted python code:
```
Traceback (most recent call last):
File "fail.py", line 36, in <module>
bar(torch.rand(10), torch.rand(9))
File "fail.py", line 21, in bar
return foo(c, b)
File "fail.py", line 18, in foo
return baz(c, b)
File "fail.py", line 15, in baz
return c + b
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
```
Make torchscript stack traces look like python ones.
Differential Revision: [D18532812](https://our.internmc.facebook.com/intern/diff/D18532812)
[ghstack-poisoned]
zdevito
reviewed
Nov 19, 2019
| out << "during call "; | ||
| if (node->callstack()) { | ||
| for (const auto& p : (*node->callstack())->vec()) { | ||
| p.second.print_with_context( |
Contributor
There was a problem hiding this comment.
I know that Python doesn't add more context by default, but I think we should do it anyway. For instance, python notebooks will print more of the context in stack traces. My previous comments were mostly about the wording differences for the stuff around the highlighting. I also think we should keep the highlighting on, since it makes it easier to read which part of the line we are talking about.
added 3 commits
November 18, 2019 20:46
…lls were inlined."
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
This change also makes torchscript stacktraces look like python ones.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 41, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter.
Traceback (most recent call last):
File "fail.py", line 33
@torch.jit.script
def bar(c, b):
return foo(c, b)
~~~ <--- HERE
File "fail.py", line 29, in foo
@torch.jit.script
def foo(c, b):
return baz(c, b)
~~~ <--- HERE
File "fail.py", line 25, in baz
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output of non-scripted python code:
```
Traceback (most recent call last):
File "fail.py", line 36, in <module>
bar(torch.rand(10), torch.rand(9))
File "fail.py", line 21, in bar
return foo(c, b)
File "fail.py", line 18, in foo
return baz(c, b)
File "fail.py", line 15, in baz
return c + b
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
```
Differential Revision: [D18532812](https://our.internmc.facebook.com/intern/diff/D18532812)
[ghstack-poisoned]
…lls were inlined."
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
This change also makes torchscript stacktraces look like python ones.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 41, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter.
Traceback (most recent call last):
File "fail.py", line 33
@torch.jit.script
def bar(c, b):
return foo(c, b)
~~~ <--- HERE
File "fail.py", line 29, in foo
@torch.jit.script
def foo(c, b):
return baz(c, b)
~~~ <--- HERE
File "fail.py", line 25, in baz
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output of non-scripted python code:
```
Traceback (most recent call last):
File "fail.py", line 36, in <module>
bar(torch.rand(10), torch.rand(9))
File "fail.py", line 21, in bar
return foo(c, b)
File "fail.py", line 18, in foo
return baz(c, b)
File "fail.py", line 15, in baz
return c + b
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
```
Differential Revision: [D18532812](https://our.internmc.facebook.com/intern/diff/D18532812)
[ghstack-poisoned]
…lls were inlined."
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
This change also makes torchscript stacktraces look like python ones.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 41, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter.
Traceback (most recent call last):
File "fail.py", line 33
@torch.jit.script
def bar(c, b):
return foo(c, b)
~~~ <--- HERE
File "fail.py", line 29, in foo
@torch.jit.script
def foo(c, b):
return baz(c, b)
~~~ <--- HERE
File "fail.py", line 25, in baz
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output of non-scripted python code:
```
Traceback (most recent call last):
File "fail.py", line 36, in <module>
bar(torch.rand(10), torch.rand(9))
File "fail.py", line 21, in bar
return foo(c, b)
File "fail.py", line 18, in foo
return baz(c, b)
File "fail.py", line 15, in baz
return c + b
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
```
Differential Revision: [D18532812](https://our.internmc.facebook.com/intern/diff/D18532812)
[ghstack-poisoned]
Author
|
It's already committed. |
xxtEchjovs44
pushed a commit
to xxtEchjovs44/pytorch
that referenced
this pull request
Jan 29, 2020
…lined.
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
This change also makes torchscript stacktraces look like python ones.
Example:
```
@torch.jit.script
def baz(c, b):
return c + b
@torch.jit.script
def foo(c, b):
return baz(c, b)
@torch.jit.script
def bar(c, b):
return foo(c, b)
bar(torch.rand(10), torch.rand(9))
```
Output before:
```
Traceback (most recent call last):
File "fail.py", line 25, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter, with the following stack trace:
at fail.py:15:11
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output after:
```
Traceback (most recent call last):
File "fail.py", line 41, in <module>
bar(torch.rand(10), torch.rand(9))
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
The above operation failed in interpreter.
Traceback (most recent call last):
File "fail.py", line 33
@torch.jit.script
def bar(c, b):
return foo(c, b)
~~~ <--- HERE
File "fail.py", line 29, in foo
@torch.jit.script
def foo(c, b):
return baz(c, b)
~~~ <--- HERE
File "fail.py", line 25, in baz
@torch.jit.script
def baz(c, b):
return c + b
~~~~~ <--- HERE
```
Output of non-scripted python code:
```
Traceback (most recent call last):
File "fail.py", line 36, in <module>
bar(torch.rand(10), torch.rand(9))
File "fail.py", line 21, in bar
return foo(c, b)
File "fail.py", line 18, in foo
return baz(c, b)
File "fail.py", line 15, in baz
return c + b
RuntimeError: The size of tensor a (10) must match the size of tensor b (9) at non-singleton dimension 0
```
ghstack-source-id: ba40b19
Pull Request resolved: pytorch/pytorch#29911
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Stack from ghstack:
Summary:
This uses newly added InlinedCallStack to print the original call stack
even if the real call stack is shallower because of inlining.
This change also makes torchscript stacktraces look like python ones.
Example:
Output before:
Output after:
Output of non-scripted python code:
Differential Revision: D18532812