🐛 Describe the bug

torch.inverse and the related modules won't work with multi-threading:

import concurrent.futures
import torch

def run_task(req):
    out = torch.inverse(torch.eye(4, device="cuda:0"))
    return req, out


infer_tasks = [1, 2, 3, 4, 5, 6, 7, 8]

with concurrent.futures.ThreadPoolExecutor(4, "BatchTask") as executor:
    futures = {executor.submit(run_task, t) for t in infer_tasks}
    for fut in concurrent.futures.as_completed(futures):
        print("outcome", fut.result())

run this script will raise a runtime error:

Traceback (most recent call last):
  File "my_test.py", line 15, in <module>
    print("outcome", fut.result())
  File "/home/anaconda3/envs/py38/lib/python3.8/concurrent/futures/_base.py", line 437, in result
    return self.__get_result()
  File "/home/anaconda3/envs/py38/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/home/anaconda3/envs/py38/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "my_test.py", line 5, in run_task
    out = torch.inverse(torch.eye(4, device="cuda:0"))
RuntimeError: lazy wrapper should be called at most once

Versions

Collecting environment information...
PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A

OS: Ubuntu 18.04.6 LTS (x86_64)
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
Clang version: Could not collect
CMake version: version 3.10.2
Libc version: glibc-2.27

Python version: 3.8.13 (default, Mar 28 2022, 11:38:47) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.4.0-135-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.4.48
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA TITAN Xp COLLECTORS EDITION
GPU 1: NVIDIA TITAN Xp COLLECTORS EDITION

(monai ref: Project-MONAI/MONAI#5696)

cc @jianyuh @nikitaved @pearu @mruberry @walterddr @IvanYashchuk @xwang233 @lezcano