-
Notifications
You must be signed in to change notification settings - Fork 1.5k
tests.test_spatial_resample error #4432
Copy link
Copy link
Closed
Description
Describe the bug
randomly run into this issue:
[2022-06-01T14:46:36.804Z] ======================================================================
[2022-06-01T14:46:36.804Z] ERROR: test_ill_affine_1 (tests.test_spatial_resample.TestSpatialResample)
[2022-06-01T14:46:36.804Z] ----------------------------------------------------------------------
[2022-06-01T14:46:36.804Z] Traceback (most recent call last):
[2022-06-01T14:46:36.804Z] File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/transforms/spatial/array.py", line 274, in __call__
[2022-06-01T14:46:36.804Z] if pytorch_after(1, 8, 0)
[2022-06-01T14:46:36.804Z] RuntimeError: CUDA error: an illegal memory access was encountered
[2022-06-01T14:46:36.804Z] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
[2022-06-01T14:46:36.804Z] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[2022-06-01T14:46:36.804Z]
[2022-06-01T14:46:36.804Z] During handling of the above exception, another exception occurred:
[2022-06-01T14:46:36.804Z]
[2022-06-01T14:46:36.804Z] Traceback (most recent call last):
[2022-06-01T14:46:36.804Z] File "/usr/local/lib/python3.7/dist-packages/parameterized/parameterized.py", line 533, in standalone_func
[2022-06-01T14:46:36.804Z] return func(*(a + p.args), **p.kwargs)
[2022-06-01T14:46:36.804Z] File "/home/jenkins/agent/workspace/MONAI-premerge/monai/tests/test_spatial_resample.py", line 161, in test_ill_affine
[2022-06-01T14:46:36.804Z] SpatialResample()(img=img, dst_affine=dst_affine)
[2022-06-01T14:46:36.804Z] File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/utils/deprecate_utils.py", line 217, in _wrapper
[2022-06-01T14:46:36.804Z] return func(*args, **kwargs)
[2022-06-01T14:46:36.804Z] File "/home/jenkins/agent/workspace/MONAI-premerge/monai/monai/transforms/spatial/array.py", line 278, in __call__
[2022-06-01T14:46:36.804Z] raise ValueError(f"src affine is not invertible: {src_affine_}") from e
[2022-06-01T14:46:36.804Z] File "/usr/local/lib/python3.7/dist-packages/torch/_tensor.py", line 628, in __format__
[2022-06-01T14:46:36.804Z] return object.__format__(self, format_spec)
[2022-06-01T14:46:36.804Z] File "/usr/local/lib/python3.7/dist-packages/torch/_tensor.py", line 305, in __repr__
[2022-06-01T14:46:36.804Z] return torch._tensor_str._str(self)
[2022-06-01T14:46:36.804Z] File "/usr/local/lib/python3.7/dist-packages/torch/_tensor_str.py", line 434, in _str
[2022-06-01T14:46:36.804Z] return _str_intern(self)
[2022-06-01T14:46:36.804Z] File "/usr/local/lib/python3.7/dist-packages/torch/_tensor_str.py", line 409, in _str_intern
[2022-06-01T14:46:36.804Z] tensor_str = _tensor_str(self, indent)
[2022-06-01T14:46:36.804Z] File "/usr/local/lib/python3.7/dist-packages/torch/_tensor_str.py", line 264, in _tensor_str
[2022-06-01T14:46:36.804Z] formatter = _Formatter(get_summarized_data(self) if summarize else self)
[2022-06-01T14:46:36.804Z] File "/usr/local/lib/python3.7/dist-packages/torch/_tensor_str.py", line 100, in __init__
[2022-06-01T14:46:36.804Z] nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))
[2022-06-01T14:46:36.804Z] RuntimeError: CUDA error: an illegal memory access was encountered
[2022-06-01T14:46:36.804Z] CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
[2022-06-01T14:46:36.804Z] For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
[2022-06-01T14:46:36.804Z]
MONAI/monai/transforms/spatial/array.py
Lines 272 to 276 in 8ed38fd
| xform = ( | |
| torch.linalg.solve(src_affine_, dst_affine) | |
| if pytorch_after(1, 8, 0) | |
| else torch.solve(dst_affine, src_affine_).solution # type: ignore | |
| ) |
torch 1.10.2, cuda 10.2
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels