[2022-11-15T01:42:04.924Z] Running ./3d_segmentation/spleen_segmentation_3d.ipynb
[2022-11-15T01:42:04.924Z] Checking PEP8 compliance...
[2022-11-15T01:42:05.487Z] Running notebook...
[2022-11-15T01:42:05.487Z] Before:
[2022-11-15T01:42:05.487Z] "max_epochs = 600\n",
[2022-11-15T01:42:05.487Z] After:
[2022-11-15T01:42:05.487Z] "max_epochs = 1\n",
[2022-11-15T01:42:05.487Z] Before:
[2022-11-15T01:42:05.487Z] "val_interval = 2\n",
[2022-11-15T01:42:05.487Z] After:
[2022-11-15T01:42:05.487Z] "val_interval = 1\n",
[2022-11-15T01:42:10.743Z] MONAI version: 1.0.0+110.g53321873
[2022-11-15T01:42:10.743Z] Numpy version: 1.22.2
[2022-11-15T01:42:10.743Z] Pytorch version: 1.13.0a0+d0d6b1f
[2022-11-15T01:42:10.743Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2022-11-15T01:42:10.743Z] MONAI rev id: 5332187324bc2471476581e729d685e30d03d40a
[2022-11-15T01:42:10.743Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2022-11-15T01:42:10.743Z]
[2022-11-15T01:42:10.743Z] Optional dependencies:
[2022-11-15T01:42:10.743Z] Pytorch Ignite version: 0.4.10
[2022-11-15T01:42:10.743Z] Nibabel version: 4.0.2
[2022-11-15T01:42:10.743Z] scikit-image version: 0.19.3
[2022-11-15T01:42:10.743Z] Pillow version: 9.0.1
[2022-11-15T01:42:10.743Z] Tensorboard version: 2.10.1
[2022-11-15T01:42:10.743Z] gdown version: 4.5.3
[2022-11-15T01:42:10.743Z] TorchVision version: 0.14.0a0
[2022-11-15T01:42:10.743Z] tqdm version: 4.64.1
[2022-11-15T01:42:10.743Z] lmdb version: 1.3.0
[2022-11-15T01:42:10.743Z] psutil version: 5.9.2
[2022-11-15T01:42:10.743Z] pandas version: 1.4.4
[2022-11-15T01:42:10.743Z] einops version: 0.6.0
[2022-11-15T01:42:10.743Z] transformers version: 4.21.3
[2022-11-15T01:42:10.743Z] mlflow version: 1.30.0
[2022-11-15T01:42:10.743Z] pynrrd version: 1.0.0
[2022-11-15T01:42:10.743Z]
[2022-11-15T01:42:10.743Z] For details about installing the optional dependencies, please visit:
[2022-11-15T01:42:10.743Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2022-11-15T01:42:10.743Z]
[2022-11-15T01:42:12.112Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-15T01:42:12.112Z] warnings.warn(
[2022-11-15T01:45:07.901Z]
Executing: 0%| | 0/38 [00:00<?, ?cell/s]
Executing: 3%|▎ | 1/38 [00:01<00:58, 1.58s/cell]
Executing: 8%|▊ | 3/38 [00:08<01:40, 2.86s/cell]
Executing: 11%|█ | 4/38 [00:11<01:47, 3.17s/cell]
Executing: 26%|██▋ | 10/38 [01:21<04:23, 9.41s/cell]
Executing: 47%|████▋ | 18/38 [01:22<01:19, 3.99s/cell]
Executing: 53%|█████▎ | 20/38 [02:25<02:38, 8.78s/cell]
Executing: 58%|█████▊ | 22/38 [02:27<01:54, 7.16s/cell]
Executing: 63%|██████▎ | 24/38 [02:41<01:40, 7.16s/cell]
Executing: 71%|███████ | 27/38 [02:41<00:53, 4.90s/cell]
Executing: 76%|███████▋ | 29/38 [02:45<00:37, 4.18s/cell]
Executing: 84%|████████▍ | 32/38 [02:52<00:21, 3.62s/cell]
Executing: 84%|████████▍ | 32/38 [02:55<00:32, 5.49s/cell]
[2022-11-15T01:45:07.901Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-15T01:45:07.901Z] warnings.warn(
[2022-11-15T01:45:07.901Z] Traceback (most recent call last):
[2022-11-15T01:45:07.901Z] File "/opt/conda/bin/papermill", line 8, in <module>
[2022-11-15T01:45:07.901Z] sys.exit(papermill())
[2022-11-15T01:45:07.901Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
[2022-11-15T01:45:07.901Z] return self.main(*args, **kwargs)
[2022-11-15T01:45:07.901Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1053, in main
[2022-11-15T01:45:07.901Z] rv = self.invoke(ctx)
[2022-11-15T01:45:07.901Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
[2022-11-15T01:45:07.901Z] return ctx.invoke(self.callback, **ctx.params)
[2022-11-15T01:45:07.901Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 754, in invoke
[2022-11-15T01:45:07.901Z] return __callback(*args, **kwargs)
[2022-11-15T01:45:07.901Z] File "/opt/conda/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
[2022-11-15T01:45:07.901Z] return f(get_current_context(), *args, **kwargs)
[2022-11-15T01:45:07.901Z] File "/opt/conda/lib/python3.8/site-packages/papermill/cli.py", line 250, in papermill
[2022-11-15T01:45:07.901Z] execute_notebook(
[2022-11-15T01:45:07.901Z] File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 128, in execute_notebook
[2022-11-15T01:45:07.901Z] raise_for_execution_errors(nb, output_path)
[2022-11-15T01:45:07.901Z] File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2022-11-15T01:45:07.901Z] raise error
[2022-11-15T01:45:07.901Z] papermill.exceptions.PapermillExecutionError:
[2022-11-15T01:45:07.901Z] ---------------------------------------------------------------------------
[2022-11-15T01:45:07.901Z] Exception encountered at "In [17]":
[2022-11-15T01:45:07.901Z] ---------------------------------------------------------------------------
[2022-11-15T01:45:07.901Z] RuntimeError Traceback (most recent call last)
[2022-11-15T01:45:07.901Z] Cell In [17], line 15
[2022-11-15T01:45:07.901Z] 13 val_outputs, val_labels = from_engine(["pred", "label"])(val_data)
[2022-11-15T01:45:07.901Z] 14 # compute metric for current iteration
[2022-11-15T01:45:07.901Z] ---> 15 dice_metric(y_pred=val_outputs, y=val_labels)
[2022-11-15T01:45:07.901Z] 17 # aggregate the final mean dice result
[2022-11-15T01:45:07.901Z] 18 metric_org = dice_metric.aggregate().item()
[2022-11-15T01:45:07.901Z]
[2022-11-15T01:45:07.901Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/metrics/metric.py:329, in CumulativeIterationMetric.__call__(self, y_pred, y)
[2022-11-15T01:45:07.901Z] 313 def __call__(self, y_pred: TensorOrList, y: Optional[TensorOrList] = None):
[2022-11-15T01:45:07.901Z] 314 """
[2022-11-15T01:45:07.901Z] 315 Execute basic computation for model prediction and ground truth.
[2022-11-15T01:45:07.901Z] 316 It can support both `list of channel-first Tensor` and `batch-first Tensor`.
[2022-11-15T01:45:07.901Z] (...)
[2022-11-15T01:45:07.901Z] 327 The computed metric values at the iteration level.
[2022-11-15T01:45:07.901Z] 328 """
[2022-11-15T01:45:07.901Z] --> 329 ret = super().__call__(y_pred=y_pred, y=y)
[2022-11-15T01:45:07.901Z] 330 if isinstance(ret, (tuple, list)):
[2022-11-15T01:45:07.901Z] 331 self.extend(*ret)
[2022-11-15T01:45:07.901Z]
[2022-11-15T01:45:07.901Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/metrics/metric.py:68, in IterationMetric.__call__(self, y_pred, y)
[2022-11-15T01:45:07.901Z] 66 # handling a list of channel-first data
[2022-11-15T01:45:07.901Z] 67 if isinstance(y_pred, (list, tuple)) or isinstance(y, (list, tuple)):
[2022-11-15T01:45:07.901Z] ---> 68 return self._compute_list(y_pred, y)
[2022-11-15T01:45:07.901Z] 69 # handling a single batch-first data
[2022-11-15T01:45:07.901Z] 70 if isinstance(y_pred, torch.Tensor):
[2022-11-15T01:45:07.901Z]
[2022-11-15T01:45:07.901Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/metrics/metric.py:90, in IterationMetric._compute_list(self, y_pred, y)
[2022-11-15T01:45:07.901Z] 76 """
[2022-11-15T01:45:07.901Z] 77 Execute the metric computation for `y_pred` and `y` in a list of "channel-first" tensors.
[2022-11-15T01:45:07.901Z] 78
[2022-11-15T01:45:07.902Z] (...)
[2022-11-15T01:45:07.902Z] 87 Note: subclass may enhance the operation to have multi-thread support.
[2022-11-15T01:45:07.902Z] 88 """
[2022-11-15T01:45:07.902Z] 89 if y is not None:
[2022-11-15T01:45:07.902Z] ---> 90 ret = [self._compute_tensor(p.detach().unsqueeze(0), y_.detach().unsqueeze(0)) for p, y_ in zip(y_pred, y)]
[2022-11-15T01:45:07.902Z] 91 else:
[2022-11-15T01:45:07.902Z] 92 ret = [self._compute_tensor(p_.detach().unsqueeze(0), None) for p_ in y_pred]
[2022-11-15T01:45:07.902Z]
[2022-11-15T01:45:07.902Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/metrics/metric.py:90, in <listcomp>(.0)
[2022-11-15T01:45:07.902Z] 76 """
[2022-11-15T01:45:07.902Z] 77 Execute the metric computation for `y_pred` and `y` in a list of "channel-first" tensors.
[2022-11-15T01:45:07.902Z] 78
[2022-11-15T01:45:07.902Z] (...)
[2022-11-15T01:45:07.902Z] 87 Note: subclass may enhance the operation to have multi-thread support.
[2022-11-15T01:45:07.902Z] 88 """
[2022-11-15T01:45:07.902Z] 89 if y is not None:
[2022-11-15T01:45:07.902Z] ---> 90 ret = [self._compute_tensor(p.detach().unsqueeze(0), y_.detach().unsqueeze(0)) for p, y_ in zip(y_pred, y)]
[2022-11-15T01:45:07.902Z] 91 else:
[2022-11-15T01:45:07.902Z] 92 ret = [self._compute_tensor(p_.detach().unsqueeze(0), None) for p_ in y_pred]
[2022-11-15T01:45:07.902Z]
[2022-11-15T01:45:07.902Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/metrics/meandice.py:83, in DiceMetric._compute_tensor(self, y_pred, y)
[2022-11-15T01:45:07.902Z] 81 raise ValueError(f"y_pred should have at least 3 dimensions (batch, channel, spatial), got {dims}.")
[2022-11-15T01:45:07.902Z] 82 # compute dice (BxC) for each channel for each batch
[2022-11-15T01:45:07.902Z] ---> 83 return compute_dice(
[2022-11-15T01:45:07.902Z] 84 y_pred=y_pred, y=y, include_background=self.include_background, ignore_empty=self.ignore_empty
[2022-11-15T01:45:07.902Z] 85 )
[2022-11-15T01:45:07.902Z]
[2022-11-15T01:45:07.902Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/metrics/meandice.py:143, in compute_dice(y_pred, y, include_background, ignore_empty)
[2022-11-15T01:45:07.902Z] 141 n_len = len(y_pred.shape)
[2022-11-15T01:45:07.902Z] 142 reduce_axis = list(range(2, n_len))
[2022-11-15T01:45:07.902Z] --> 143 intersection = torch.sum(y * y_pred, dim=reduce_axis)
[2022-11-15T01:45:07.902Z] 145 y_o = torch.sum(y, reduce_axis)
[2022-11-15T01:45:07.902Z] 146 y_pred_o = torch.sum(y_pred, dim=reduce_axis)
[2022-11-15T01:45:07.902Z]
[2022-11-15T01:45:07.902Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/data/meta_tensor.py:249, in MetaTensor.__torch_function__(cls, func, types, args, kwargs)
[2022-11-15T01:45:07.902Z] 247 if kwargs is None:
[2022-11-15T01:45:07.902Z] 248 kwargs = {}
[2022-11-15T01:45:07.902Z] --> 249 ret = super().__torch_function__(func, types, args, kwargs)
[2022-11-15T01:45:07.902Z] 250 # if `out` has been used as argument, metadata is not copied, nothing to do.
[2022-11-15T01:45:07.902Z] 251 # if "out" in kwargs:
[2022-11-15T01:45:07.902Z] 252 # return ret
[2022-11-15T01:45:07.902Z] 253 # we might have 1 or multiple outputs. Might be MetaTensor, might be something
[2022-11-15T01:45:07.902Z] 254 # else (e.g., `__repr__` returns a string).
[2022-11-15T01:45:07.902Z] 255 # Convert to list (if necessary), process, and at end remove list if one was added.
[2022-11-15T01:45:07.902Z] 256 if (
[2022-11-15T01:45:07.902Z] 257 hasattr(torch, "return_types")
[2022-11-15T01:45:07.902Z] 258 and hasattr(func, "__name__")
[2022-11-15T01:45:07.902Z] (...)
[2022-11-15T01:45:07.902Z] 262 ):
[2022-11-15T01:45:07.902Z] 263 # for torch.max(torch.tensor(1.0), dim=0), the return type is named-tuple like
[2022-11-15T01:45:07.902Z]
[2022-11-15T01:45:07.902Z] File /opt/conda/lib/python3.8/site-packages/torch/_tensor.py:1265, in Tensor.__torch_function__(cls, func, types, args, kwargs)
[2022-11-15T01:45:07.902Z] 1262 return NotImplemented
[2022-11-15T01:45:07.902Z] 1264 with _C.DisableTorchFunction():
[2022-11-15T01:45:07.902Z] -> 1265 ret = func(*args, **kwargs)
[2022-11-15T01:45:07.902Z] 1266 if func in get_default_nowrap_functions():
[2022-11-15T01:45:07.902Z] 1267 return ret
[2022-11-15T01:45:07.902Z]
[2022-11-15T01:45:07.902Z] RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
