[2022-10-12T13:26:23.622Z] Running ./3d_classification/densenet_training_array.ipynb
[2022-10-12T13:26:23.622Z] Checking PEP8 compliance...
[2022-10-12T13:26:24.554Z] Running notebook...
[2022-10-12T13:26:24.554Z] Before:
[2022-10-12T13:26:24.554Z]     "max_epochs = 5\n",
[2022-10-12T13:26:24.554Z] After:
[2022-10-12T13:26:24.554Z]     "max_epochs = 1\n",
[2022-10-12T13:26:24.554Z] Before:
[2022-10-12T13:26:24.554Z]     "val_interval = 2\n",
[2022-10-12T13:26:24.554Z] After:
[2022-10-12T13:26:24.554Z]     "val_interval = 1\n",
[2022-10-12T13:26:28.729Z] MONAI version: 1.0.0+37.g9731823d
[2022-10-12T13:26:28.729Z] Numpy version: 1.22.2
[2022-10-12T13:26:28.729Z] Pytorch version: 1.10.2+cu102
[2022-10-12T13:26:28.729Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2022-10-12T13:26:28.729Z] MONAI rev id: 9731823da40ae7835afad601d7d2b601534c80d3
[2022-10-12T13:26:28.729Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2022-10-12T13:26:28.729Z] 
[2022-10-12T13:26:28.729Z] Optional dependencies:
[2022-10-12T13:26:28.729Z] Pytorch Ignite version: 0.4.10
[2022-10-12T13:26:28.729Z] Nibabel version: 4.0.2
[2022-10-12T13:26:28.729Z] scikit-image version: 0.19.3
[2022-10-12T13:26:28.729Z] Pillow version: 7.0.0
[2022-10-12T13:26:28.729Z] Tensorboard version: 2.10.0
[2022-10-12T13:26:28.729Z] gdown version: 4.5.1
[2022-10-12T13:26:28.729Z] TorchVision version: 0.11.3+cu102
[2022-10-12T13:26:28.729Z] tqdm version: 4.64.1
[2022-10-12T13:26:28.729Z] lmdb version: 1.3.0
[2022-10-12T13:26:28.729Z] psutil version: 5.9.2
[2022-10-12T13:26:28.729Z] pandas version: 1.1.5
[2022-10-12T13:26:28.729Z] einops version: 0.5.0
[2022-10-12T13:26:28.729Z] transformers version: 4.21.3
[2022-10-12T13:26:28.729Z] mlflow version: 1.29.0
[2022-10-12T13:26:28.729Z] pynrrd version: 1.0.0
[2022-10-12T13:26:28.729Z] 
[2022-10-12T13:26:28.729Z] For details about installing the optional dependencies, please visit:
[2022-10-12T13:26:28.729Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2022-10-12T13:26:28.729Z] 
[2022-10-12T13:26:29.661Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-10-12T13:26:29.661Z]   warnings.warn(
[2022-10-12T13:32:20.557Z] 
Executing:   0%|          | 0/16 [00:00<?, ?cell/s]
Executing:   6%|▋         | 1/16 [00:01<00:23,  1.59s/cell]
Executing:  19%|█▉        | 3/16 [00:06<00:27,  2.15s/cell]
Executing:  31%|███▏      | 5/16 [00:09<00:21,  1.99s/cell]
Executing:  50%|█████     | 8/16 [05:45<07:55, 59.50s/cell]
Executing:  56%|█████▋    | 9/16 [05:50<05:41, 48.72s/cell]2022-10-12 13:32:20.155755: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512F AVX512_VNNI FMA
[2022-10-12T13:32:20.557Z] To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
[2022-10-12T13:32:20.557Z] 2022-10-12 13:32:20.296520: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[2022-10-12T13:32:20.557Z] 2022-10-12 13:32:20.328749: E tensorflow/stream_executor/cuda/cuda_blas.cc:2981] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
[2022-10-12T13:32:21.120Z] 2022-10-12 13:32:20.984730: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-10-12T13:32:21.121Z] 2022-10-12 13:32:20.984811: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /opt/conda/lib/python3.8/site-packages/torch/lib:/opt/conda/lib/python3.8/site-packages/torch_tensorrt/lib:/usr/local/cuda/compat/lib:/usr/local/nvidia/lib:/usr/local/nvidia/lib64
[2022-10-12T13:32:21.121Z] 2022-10-12 13:32:20.984818: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
[2022-10-12T13:32:31.508Z] 
Executing:  62%|██████▎   | 10/16 [05:57<03:56, 39.42s/cell]
Executing:  81%|████████▏ | 13/16 [05:57<01:00, 20.29s/cell]
Executing:  88%|████████▊ | 14/16 [05:58<00:33, 16.64s/cell]
Executing:  88%|████████▊ | 14/16 [06:01<00:51, 25.84s/cell]
[2022-10-12T13:32:31.508Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-10-12T13:32:31.508Z]   warnings.warn(
[2022-10-12T13:32:31.508Z] Traceback (most recent call last):
[2022-10-12T13:32:31.508Z]   File "/opt/conda/bin/papermill", line 8, in <module>
[2022-10-12T13:32:31.508Z]     sys.exit(papermill())
[2022-10-12T13:32:31.508Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1130, in __call__
[2022-10-12T13:32:31.508Z]     return self.main(*args, **kwargs)
[2022-10-12T13:32:31.508Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1055, in main
[2022-10-12T13:32:31.508Z]     rv = self.invoke(ctx)
[2022-10-12T13:32:31.508Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
[2022-10-12T13:32:31.508Z]     return ctx.invoke(self.callback, **ctx.params)
[2022-10-12T13:32:31.508Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 760, in invoke
[2022-10-12T13:32:31.508Z]     return __callback(*args, **kwargs)
[2022-10-12T13:32:31.508Z]   File "/opt/conda/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
[2022-10-12T13:32:31.508Z]     return f(get_current_context(), *args, **kwargs)
[2022-10-12T13:32:31.508Z]   File "/opt/conda/lib/python3.8/site-packages/papermill/cli.py", line 250, in papermill
[2022-10-12T13:32:31.508Z]     execute_notebook(
[2022-10-12T13:32:31.508Z]   File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 128, in execute_notebook
[2022-10-12T13:32:31.508Z]     raise_for_execution_errors(nb, output_path)
[2022-10-12T13:32:31.508Z]   File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2022-10-12T13:32:31.508Z]     raise error
[2022-10-12T13:32:31.508Z] papermill.exceptions.PapermillExecutionError: 
[2022-10-12T13:32:31.508Z] ---------------------------------------------------------------------------
[2022-10-12T13:32:31.508Z] Exception encountered at "In [10]":
[2022-10-12T13:32:31.508Z] ---------------------------------------------------------------------------
[2022-10-12T13:32:31.508Z] ValueError                                Traceback (most recent call last)
[2022-10-12T13:32:31.508Z] Cell In [10], line 9
[2022-10-12T13:32:31.508Z]       6 depth_slice = img.shape[2] // 2
[2022-10-12T13:32:31.508Z]       7 occ_sens_b_box = [-1, -1, depth_slice, depth_slice, -1, -1, -1, -1]
[2022-10-12T13:32:31.508Z] ----> 9 occ_result, _ = occ_sens(x=img, b_box=occ_sens_b_box)
[2022-10-12T13:32:31.508Z]      10 occ_result = occ_result[..., label.argmax().item()]
[2022-10-12T13:32:31.508Z]      12 fig, axes = plt.subplots(1, 2, figsize=(25, 15), facecolor="white")
[2022-10-12T13:32:31.508Z] 
[2022-10-12T13:32:31.508Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/visualize/occlusion_sensitivity.py:305, in OcclusionSensitivity.__call__(self, x, b_box, **kwargs)
[2022-10-12T13:32:31.508Z]     303 # check that the grid is bigger than the mask size
[2022-10-12T13:32:31.508Z]     304 if any(m > g for g, m in zip(grid.shape[2:], mask_size)):
[2022-10-12T13:32:31.508Z] --> 305     raise ValueError("Image (after cropping with bounding box) should be bigger than mask.")
[2022-10-12T13:32:31.508Z]     307 # get additive and multiplicative factors if they are unchanged for all patches (i.e., not mean_patch)
[2022-10-12T13:32:31.508Z]     308 add: Optional[Union[float, torch.Tensor]]
[2022-10-12T13:32:31.508Z] 
[2022-10-12T13:32:31.508Z] ValueError: Image (after cropping with bounding box) should be bigger than mask.