[2023-03-15T07:02:35.257Z] Running ./acceleration/fast_training_tutorial.ipynb
[2023-03-15T07:02:35.257Z] Checking PEP8 compliance...
[2023-03-15T07:02:36.183Z] Running notebook...
[2023-03-15T07:02:36.183Z] Before:
[2023-03-15T07:02:36.183Z]     "    max_epochs = 6\n",
[2023-03-15T07:02:36.183Z]     "    max_epochs = 300\n",
[2023-03-15T07:02:36.183Z] After:
[2023-03-15T07:02:36.183Z]     "    max_epochs = 1\n",
[2023-03-15T07:02:36.183Z]     "    max_epochs = 1\n",
[2023-03-15T07:02:36.183Z] Before:
[2023-03-15T07:02:36.183Z]     "    val_interval = 5  # do validation for every epoch\n",
[2023-03-15T07:02:36.183Z] After:
[2023-03-15T07:02:36.183Z]     "    val_interval = 1  # do validation for every epoch\n",
[2023-03-15T07:02:40.354Z] MONAI version: 1.1.0+114.g9fd6d4ce
[2023-03-15T07:02:40.354Z] Numpy version: 1.22.2
[2023-03-15T07:02:40.354Z] Pytorch version: 1.10.2+cu102
[2023-03-15T07:02:40.354Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2023-03-15T07:02:40.354Z] MONAI rev id: 9fd6d4ced3efb344cf3b24c51d899d8fafc882d5
[2023-03-15T07:02:40.354Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2023-03-15T07:02:40.354Z] 
[2023-03-15T07:02:40.354Z] Optional dependencies:
[2023-03-15T07:02:40.354Z] Pytorch Ignite version: 0.4.11
[2023-03-15T07:02:40.354Z] ITK version: 5.3.0
[2023-03-15T07:02:40.354Z] Nibabel version: 5.0.1
[2023-03-15T07:02:40.354Z] scikit-image version: 0.19.3
[2023-03-15T07:02:40.354Z] Pillow version: 9.4.0
[2023-03-15T07:02:40.354Z] Tensorboard version: 2.9.0
[2023-03-15T07:02:40.354Z] gdown version: 4.6.4
[2023-03-15T07:02:40.354Z] TorchVision version: 0.11.3+cu102
[2023-03-15T07:02:40.354Z] tqdm version: 4.64.1
[2023-03-15T07:02:40.354Z] lmdb version: 1.4.0
[2023-03-15T07:02:40.354Z] psutil version: 5.9.4
[2023-03-15T07:02:40.354Z] pandas version: 1.1.5
[2023-03-15T07:02:40.354Z] einops version: 0.6.0
[2023-03-15T07:02:40.354Z] transformers version: 4.21.3
[2023-03-15T07:02:40.354Z] mlflow version: 2.2.2
[2023-03-15T07:02:40.354Z] pynrrd version: 1.0.0
[2023-03-15T07:02:40.354Z] 
[2023-03-15T07:02:40.354Z] For details about installing the optional dependencies, please visit:
[2023-03-15T07:02:40.354Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2023-03-15T07:02:40.354Z] 
[2023-03-15T07:02:42.242Z] papermill  --progress-bar -k python3
[2023-03-15T07:02:42.497Z] /usr/local/lib/python3.8/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-03-15T07:02:42.497Z]   warnings.warn(
[2023-03-15T07:05:06.273Z] 
Executing:   0%|          | 0/36 [00:00<?, ?cell/s]
Executing:   3%|▎         | 1/36 [00:01<00:55,  1.57s/cell]
Executing:  11%|█         | 4/36 [00:07<00:58,  1.83s/cell]
Executing:  17%|█▋        | 6/36 [00:10<00:51,  1.72s/cell]
Executing:  42%|████▏     | 15/36 [00:47<01:13,  3.48s/cell]
Executing:  64%|██████▍   | 23/36 [01:32<00:58,  4.50s/cell]
Executing:  69%|██████▉   | 25/36 [02:21<01:22,  7.46s/cell]
Executing:  69%|██████▉   | 25/36 [02:23<01:03,  5.74s/cell]
[2023-03-15T07:05:06.273Z] /usr/local/lib/python3.8/dist-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2023-03-15T07:05:06.273Z]   warnings.warn(
[2023-03-15T07:05:06.273Z] Traceback (most recent call last):
[2023-03-15T07:05:06.273Z]   File "/usr/local/bin/papermill", line 8, in <module>
[2023-03-15T07:05:06.273Z]     sys.exit(papermill())
[2023-03-15T07:05:06.273Z]   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1130, in __call__
[2023-03-15T07:05:06.273Z]     return self.main(*args, **kwargs)
[2023-03-15T07:05:06.273Z]   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1055, in main
[2023-03-15T07:05:06.273Z]     rv = self.invoke(ctx)
[2023-03-15T07:05:06.273Z]   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1404, in invoke
[2023-03-15T07:05:06.273Z]     return ctx.invoke(self.callback, **ctx.params)
[2023-03-15T07:05:06.273Z]   File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 760, in invoke
[2023-03-15T07:05:06.273Z]     return __callback(*args, **kwargs)
[2023-03-15T07:05:06.273Z]   File "/usr/local/lib/python3.8/dist-packages/click/decorators.py", line 26, in new_func
[2023-03-15T07:05:06.273Z]     return f(get_current_context(), *args, **kwargs)
[2023-03-15T07:05:06.273Z]   File "/usr/local/lib/python3.8/dist-packages/papermill/cli.py", line 250, in papermill
[2023-03-15T07:05:06.273Z]     execute_notebook(
[2023-03-15T07:05:06.273Z]   File "/usr/local/lib/python3.8/dist-packages/papermill/execute.py", line 128, in execute_notebook
[2023-03-15T07:05:06.273Z]     raise_for_execution_errors(nb, output_path)
[2023-03-15T07:05:06.273Z]   File "/usr/local/lib/python3.8/dist-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2023-03-15T07:05:06.273Z]     raise error
[2023-03-15T07:05:06.273Z] papermill.exceptions.PapermillExecutionError: 
[2023-03-15T07:05:06.273Z] ---------------------------------------------------------------------------
[2023-03-15T07:05:06.273Z] Exception encountered at "In [11]":
[2023-03-15T07:05:06.273Z] ---------------------------------------------------------------------------
[2023-03-15T07:05:06.273Z] TypeError                                 Traceback (most recent call last)
[2023-03-15T07:05:06.273Z] Cell In[11], line 10
[2023-03-15T07:05:06.273Z]       1 set_determinism(seed=0)
[2023-03-15T07:05:06.273Z]       2 monai_start = time.time()
[2023-03-15T07:05:06.273Z]       3 (
[2023-03-15T07:05:06.273Z]       4     epoch_num,
[2023-03-15T07:05:06.273Z]       5     m_epoch_loss_values,
[2023-03-15T07:05:06.273Z]       6     m_metric_values,
[2023-03-15T07:05:06.273Z]       7     m_epoch_times,
[2023-03-15T07:05:06.273Z]       8     m_best,
[2023-03-15T07:05:06.273Z]       9     m_train_time,
[2023-03-15T07:05:06.273Z] ---> 10 ) = train_process(fast=True)
[2023-03-15T07:05:06.273Z]      11 m_total_time = time.time() - monai_start
[2023-03-15T07:05:06.273Z]      12 print(
[2023-03-15T07:05:06.273Z]      13     f"total time of {epoch_num} epochs with MONAI fast training: {m_train_time:.4f},"
[2023-03-15T07:05:06.273Z]      14     f" time of preparing cache: {(m_total_time - m_train_time):.4f}"
[2023-03-15T07:05:06.273Z]      15 )
[2023-03-15T07:05:06.273Z] 
[2023-03-15T07:05:06.273Z] Cell In[9], line 184, in train_process(fast)
[2023-03-15T07:05:06.273Z]     182 if fast:
[2023-03-15T07:05:06.273Z]     183     with torch.cuda.amp.autocast():
[2023-03-15T07:05:06.273Z] --> 184         val_outputs = sliding_window_inference(val_inputs, roi_size, sw_batch_size, model)
[2023-03-15T07:05:06.273Z]     185 else:
[2023-03-15T07:05:06.273Z]     186     val_outputs = sliding_window_inference(val_inputs, roi_size, sw_batch_size, model)
[2023-03-15T07:05:06.273Z] 
[2023-03-15T07:05:06.273Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/inferers/utils.py:258, in sliding_window_inference(inputs, roi_size, sw_batch_size, predictor, overlap, mode, sigma_scale, padding_mode, cval, sw_device, device, progress, roi_weight_map, process_fn, *args, **kwargs)
[2023-03-15T07:05:06.273Z]     249         warnings.warn(
[2023-03-15T07:05:06.273Z]     250             f"For axis-{axis-2} of output[{ss}], the output roi range is not int. "
[2023-03-15T07:05:06.273Z]     251             f"Input roi range is ({original_idx[axis].start}, {original_idx[axis].stop}). "
[2023-03-15T07:05:06.273Z]    (...)
[2023-03-15T07:05:06.273Z]     255             "Tips: if overlap*roi_size*zoom_scale is an integer, it usually works."
[2023-03-15T07:05:06.273Z]     256         )
[2023-03-15T07:05:06.273Z]     257     original_idx_zoom[axis] = slice(int(zoomed_start), int(zoomed_end), None)
[2023-03-15T07:05:06.273Z] --> 258 importance_map_zoom = resizer(importance_map.unsqueeze(0))[0].to(compute_dtype)
[2023-03-15T07:05:06.273Z]     259 # store results and weights
[2023-03-15T07:05:06.273Z]     260 output_image_list[ss][original_idx_zoom] += importance_map_zoom * seg_prob[idx - slice_g]
[2023-03-15T07:05:06.273Z] 
[2023-03-15T07:05:06.273Z] TypeError: 'MetaObj' object is not subscriptable