Skip to content

TestScript. test_tracing_multiple_methods is flaky in CI #29367

@mruberry

Description

@mruberry

Two back-to-back failures:

https://circleci.com/gh/pytorch/pytorch/3507116?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link/console

https://circleci.com/gh/pytorch/pytorch/3507660?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-build-link/console

Nov 07 03:54:07 ======================================================================
Nov 07 03:54:07 ERROR: test_tracing_multiple_methods (__main__.TestScript)
Nov 07 03:54:07 ----------------------------------------------------------------------
Nov 07 03:54:07 Traceback (most recent call last):
Nov 07 03:54:07   File "test_jit.py", line 3988, in test_tracing_multiple_methods
Nov 07 03:54:07     module = torch.jit.trace_module(n, inputs)
Nov 07 03:54:07   File "/opt/python/3.5/lib/python3.5/site-packages/torch/jit/__init__.py", line 1031, in trace_module
Nov 07 03:54:07     check_tolerance, _force_outplace, True, _module_class)
Nov 07 03:54:07   File "/opt/python/3.5/lib/python3.5/site-packages/torch/autograd/grad_mode.py", line 49, in decorate_no_grad
Nov 07 03:54:07     return func(*args, **kwargs)
Nov 07 03:54:07   File "/opt/python/3.5/lib/python3.5/site-packages/torch/jit/__init__.py", line 674, in _check_trace
Nov 07 03:54:07     raise TracingCheckError(*diag_info)
Nov 07 03:54:07 torch.jit.TracingCheckError: Tracing failed sanity checks!
Nov 07 03:54:07 ERROR: Graphs differed across invocations!
Nov 07 03:54:07 	Graph diff:
Nov 07 03:54:07 		- graph(%self : ClassType<Module>,
Nov 07 03:54:07 		+ graph(%self.1 : ClassType<Module>,
Nov 07 03:54:07 		?            ++
Nov 07 03:54:07 		        %input : Tensor):
Nov 07 03:54:07 		-   %2 : ClassType<Module> = prim::GetAttr[name="conv"](%self)
Nov 07 03:54:07 		+   %2 : ClassType<Module> = prim::GetAttr[name="conv"](%self.1)
Nov 07 03:54:07 		?                                                            ++
Nov 07 03:54:07 		+   %3 : Tensor = prim::CallMethod[name="forward"](%2, %input)
Nov 07 03:54:07 		-   %3 : Tensor = prim::GetAttr[name="bias"](%2)
Nov 07 03:54:07 		-   %4 : ClassType<Module> = prim::GetAttr[name="conv"](%self)
Nov 07 03:54:07 		-   %5 : Tensor = prim::GetAttr[name="weight"](%4)
Nov 07 03:54:07 		-   %6 : int = prim::Constant[value=1]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %7 : int = prim::Constant[value=1]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %8 : int[] = prim::ListConstruct(%6, %7)
Nov 07 03:54:07 		-   %9 : int = prim::Constant[value=0]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %10 : int = prim::Constant[value=0]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %11 : int[] = prim::ListConstruct(%9, %10)
Nov 07 03:54:07 		-   %12 : int = prim::Constant[value=1]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %13 : int = prim::Constant[value=1]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %14 : int[] = prim::ListConstruct(%12, %13)
Nov 07 03:54:07 		-   %15 : bool = prim::Constant[value=0]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %16 : int = prim::Constant[value=0]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %17 : int = prim::Constant[value=0]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %18 : int[] = prim::ListConstruct(%16, %17)
Nov 07 03:54:07 		-   %19 : int = prim::Constant[value=1]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %20 : bool = prim::Constant[value=0]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %21 : bool = prim::Constant[value=0]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %22 : bool = prim::Constant[value=1]() # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   %23 : Tensor = aten::_convolution(%input, %5, %3, %8, %11, %14, %15, %18, %19, %20, %21, %22) # /opt/python/3.5/lib/python3.5/site-packages/torch/nn/modules/conv.py:342:0
Nov 07 03:54:07 		-   return (%23)
Nov 07 03:54:07 		?            -
Nov 07 03:54:07 		+   return (%3)
Nov 07 03:54:07 	First diverging operator:
Nov 07 03:54:07 	Node diff:
Nov 07 03:54:07 		- %2 : ClassType<Module> = prim::GetAttr[name="conv"](%self)
Nov 07 03:54:07 		+ %2 : ClassType<Module> = prim::GetAttr[name="conv"](%self.1)
Nov 07 03:54:07 		?                                                          ++
Nov 07 03:54:07 
Nov 07 03:54:07 
Nov 07 03:54:07 ----------------------------------------------------------------------
Nov 07 03:54:07 Ran 2252 tests in 169.006s
Nov 07 03:54:07 
Nov 07 03:54:07 FAILED (errors=1, skipped=61, expected failures=1)
Nov 07 03:54:07 Traceback (most recent call last):
Nov 07 03:54:07   File "test/run_test.py", line 454, in <module>
Nov 07 03:54:07     main()
Nov 07 03:54:07   File "test/run_test.py", line 446, in main
Nov 07 03:54:07     raise RuntimeError(message)
Nov 07 03:54:07 RuntimeError: test_jit failed!
Nov 07 03:54:07 + cleanup
Nov 07 03:54:07 + retcode=1
Nov 07 03:54:07 + set +x
Nov 07 03:54:07 =================== sccache compilation log ===================

@suo do you know who should take a look?

cc @ezyang @gchanan @zou3519 @jerryzh168 @suo

Metadata

Metadata

Assignees

Labels

high prioritymodule: flaky-testsProblem is a flaky test in CIoncall: jitAdd this issue/PR to JIT oncall triage queuetriage reviewtriagedThis issue has been looked at a team member, and triaged and prioritized into an appropriate module

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions