💊 CI failures summary and remediations

As of commit 1b242ef (more details on the Dr. CI page):

• 3/3 failures possibly* introduced in this PR
  • 1/3 non-CircleCI failure(s)

🕵️ 2 new failures recognized by patterns

The following CI failures do not appear to be due to upstream breakages:

binary_linux_libtorch_3_7m_cpu_devtoolset7_shared-with-deps_build (1/2)

Step: "Checkout pytorch/builder repo" (full log | diagnosis details | 🔁 rerun)

+ git submodule update --init --recursive 
Warning: Permanently added the RSA host key for IP address '140.82.112.3' to the list of known hosts.  
fatal: reference is not a tree: cd5a9b73c3028d2496666201588111a8c8d84878 
Unable to checkout 'cd5a9b73c3028d2496666201588111a8c8d84878' in submodule path 'third_party/nccl/nccl' 
+ sleep 4 
+ git submodule update --init --recursive 
fatal: reference is not a tree: cd5a9b73c3028d2496666201588111a8c8d84878 
Unable to checkout 'cd5a9b73c3028d2496666201588111a8c8d84878' in submodule path 'third_party/nccl/nccl' 
+ sleep 8 
+ git submodule update --init --recursive 
fatal: reference is not a tree: cd5a9b73c3028d2496666201588111a8c8d84878 
Unable to checkout 'cd5a9b73c3028d2496666201588111a8c8d84878' in submodule path 'third_party/nccl/nccl' 

pytorch_xla_linux_bionic_py3_6_clang9_test (2/2)

Step: "Run tests" (full log | diagnosis details | 🔁 rerun)

Nov 10 11:14:45     return DeviceTypeTestBase.assertEqual(self, x, y, *args, **kwargs) 
Nov 10 11:14:45   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1037, in assertEqual 
Nov 10 11:14:45     exact_dtype=exact_dtype, exact_device=exact_device) 
Nov 10 11:14:45   File "/var/lib/jenkins/workspace/xla/test/pytorch_test_base.py", line 552, in assertEqual 
Nov 10 11:14:45     return DeviceTypeTestBase.assertEqual(self, x, y, *args, **kwargs) 
Nov 10 11:14:45   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 1151, in assertEqual 
Nov 10 11:14:45     super().assertEqual(x, y, msg=msg) 
Nov 10 11:14:45 AssertionError: True != 46 
Nov 10 11:14:45  
Nov 10 11:14:45 ====================================================================== 
Nov 10 11:14:45 FAIL [0.175s]: test_all_any_vs_numpy_xla_uint8 (__main__.TestTorchDeviceTypeXLA) 
Nov 10 11:14:45 ---------------------------------------------------------------------- 
Nov 10 11:14:45 Traceback (most recent call last): 
Nov 10 11:14:45   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_device_type.py", line 272, in instantiated_test 
Nov 10 11:14:45     result = test_fn(self, *args) 
Nov 10 11:14:45   File "/var/lib/jenkins/workspace/xla/test/../../test/test_torch.py", line 19633, in test_all_any_vs_numpy 
Nov 10 11:14:45     _test_all_any(x) 
Nov 10 11:14:45   File "/var/lib/jenkins/workspace/xla/test/../../test/test_torch.py", line 19618, in _test_all_any 
Nov 10 11:14:45     self.compare_with_numpy(torch.all, np.all, x) 
Nov 10 11:14:45   File "/opt/conda/lib/python3.6/site-packages/torch/testing/_internal/common_utils.py", line 913, in compare_with_numpy 
Nov 10 11:14:45     self.assertEqual(np_result, torch_result, **kwargs) 

ci.pytorch.org: 1 failed

• Failed: pr/pytorch-linux-bionic-rocm3.8-py3.6

@heitorschueroff Thanks for looking at it.

As for XLA, I m not really sure what is happening.
Maybe @JackCaoG can have a look.

Thanks!

@kshitij12345 XLA change is ready, I will merge it when this pr is merged.

@JackCaoG Thanks for updating XLA. @kshitij12345 Could you rebase please, I'll merge it then.

@heitorschueroff Have fixed the conflict. ROCm failure looks irrelevant.

@heitorschueroff Have fixed the conflict. ROCm failure looks irrelevant.

Thank you for this contribution, I'm important your changes now.

@heitorschueroff Done.

@heitorschueroff merged this pull request in 6575e67.

@JackCaoG Please merge the XLA fix. Thanks!

@kshitij12345 there are a few problems with this PR

1. as noted already, it breaks XLA CI, @heitorschueroff please avoid landing things before XLA fix is merged
2. it produces results of the same dtype as input, this is different from numpy:

In [9]: a=torch.randn(3,3)                                                                                                                                                                    
In [10]: anp=a.numpy()                                                                                                                                                                        
In [11]: np.all(anp, 0)                                                                                                                                                                       
Out[11]: array([ True,  True,  True])
In [12]: a.all(0)                                                                                                                                                                             
Out[12]: tensor([1., 1., 1.])

3. it introduces performance cliff for non-bool input types, because it does not use binary_kernel_reduce_vec, and instead uses binary_kernel_reduce which is much (5-10x) slower.

Can you please work on fixing 2) and 3) ?

@kshitij12345 there are a few problems with this PR

1. as noted already, it breaks XLA CI, @heitorschueroff please avoid landing things before XLA fix is merged
2. it produces results of the same dtype as input, this is different from numpy:

In [9]: a=torch.randn(3,3)                                                                                                                                                                    
In [10]: anp=a.numpy()                                                                                                                                                                        
In [11]: np.all(anp, 0)                                                                                                                                                                       
Out[11]: array([ True,  True,  True])
In [12]: a.all(0)                                                                                                                                                                             
Out[12]: tensor([1., 1., 1.])

1. it introduces performance cliff for non-bool input types, because it does not use binary_kernel_reduce_vec, and instead uses binary_kernel_reduce which is much (5-10x) slower.

Can you please work on fixing 2) and 3) ?

I merged XLA change. If this pr will be reverted I will revert the xla pr as well. Otherwise I will work on a companion pr to fix the result type.

I'd look at the logical_or and logical_and kernels for how to get the dtypes correct; note that it's a little different because of the performance issue @ngimel mentioned above.

Also please don't forget to update documentation.

@ngimel
For 2, it would be a BC-Breaking to change the result-type to bool,
I think it would be better to add a warning first otherwise it may break code for users. Let me know what you think about that.

Behaviour in previous version (1.5.1)

>>> import torch
>>> torch.__version__
'1.5.1+cu101'
>>> x = torch.zeros(3,3)
>>> x.to(torch.uint8).all()
tensor(0, dtype=torch.uint8)
>>> x.to(torch.bool).all()
tensor(False)

cc @mruberry for deprecating return type for uint8. In any case, for all other types there are no bc breaking concerns, so we should implement correct behavior.

I would try to update the uint8 behavior to be consistent and document the change as BC-breaking. If a scripted network relies on the current behavior (extremely unlikely) we can write an upgrader.