-
Notifications
You must be signed in to change notification settings - Fork 781
MLFlowHandler issue #1040
Copy link
Copy link
Closed
Project-MONAI/MONAI
#5527Description
[2022-11-15T01:37:10.470Z] Running ./3d_segmentation/unet_segmentation_3d_ignite.ipynb
[2022-11-15T01:37:10.470Z] Checking PEP8 compliance...
[2022-11-15T01:37:11.401Z] Running notebook...
[2022-11-15T01:37:11.401Z] Before:
[2022-11-15T01:37:11.401Z] "max_epochs = 10\n",
[2022-11-15T01:37:11.401Z] After:
[2022-11-15T01:37:11.401Z] "max_epochs = 1\n",
[2022-11-15T01:37:16.656Z] MONAI version: 1.0.0+110.g53321873
[2022-11-15T01:37:16.656Z] Numpy version: 1.22.2
[2022-11-15T01:37:16.656Z] Pytorch version: 1.13.0a0+d0d6b1f
[2022-11-15T01:37:16.656Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2022-11-15T01:37:16.656Z] MONAI rev id: 5332187324bc2471476581e729d685e30d03d40a
[2022-11-15T01:37:16.656Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2022-11-15T01:37:16.656Z]
[2022-11-15T01:37:16.656Z] Optional dependencies:
[2022-11-15T01:37:16.656Z] Pytorch Ignite version: 0.4.10
[2022-11-15T01:37:16.656Z] Nibabel version: 4.0.2
[2022-11-15T01:37:16.656Z] scikit-image version: 0.19.3
[2022-11-15T01:37:16.656Z] Pillow version: 9.0.1
[2022-11-15T01:37:16.656Z] Tensorboard version: 2.10.1
[2022-11-15T01:37:16.656Z] gdown version: 4.5.3
[2022-11-15T01:37:16.656Z] TorchVision version: 0.14.0a0
[2022-11-15T01:37:16.656Z] tqdm version: 4.64.1
[2022-11-15T01:37:16.656Z] lmdb version: 1.3.0
[2022-11-15T01:37:16.656Z] psutil version: 5.9.2
[2022-11-15T01:37:16.656Z] pandas version: 1.4.4
[2022-11-15T01:37:16.656Z] einops version: 0.6.0
[2022-11-15T01:37:16.656Z] transformers version: 4.21.3
[2022-11-15T01:37:16.656Z] mlflow version: 1.30.0
[2022-11-15T01:37:16.656Z] pynrrd version: 1.0.0
[2022-11-15T01:37:16.656Z]
[2022-11-15T01:37:16.656Z] For details about installing the optional dependencies, please visit:
[2022-11-15T01:37:16.656Z] https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2022-11-15T01:37:16.656Z]
[2022-11-15T01:37:17.220Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-15T01:37:17.220Z] warnings.warn(
[2022-11-15T01:37:41.289Z]
Executing: 0%| | 0/29 [00:00<?, ?cell/s]
Executing: 3%|▎ | 1/29 [00:01<00:44, 1.58s/cell]
Executing: 10%|█ | 3/29 [00:07<01:03, 2.44s/cell]
Executing: 17%|█▋ | 5/29 [00:11<00:55, 2.29s/cell]
Executing: 38%|███▊ | 11/29 [00:18<00:28, 1.59s/cell]
Executing: 45%|████▍ | 13/29 [00:21<00:23, 1.48s/cell]
Executing: 59%|█████▊ | 17/29 [00:21<00:10, 1.14cell/s]
Executing: 66%|██████▌ | 19/29 [00:22<00:08, 1.23cell/s]
Executing: 66%|██████▌ | 19/29 [00:24<00:12, 1.27s/cell]
[2022-11-15T01:37:41.289Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-15T01:37:41.289Z] warnings.warn(
[2022-11-15T01:37:41.289Z] Traceback (most recent call last):
[2022-11-15T01:37:41.289Z] File "/opt/conda/bin/papermill", line 8, in <module>
[2022-11-15T01:37:41.289Z] sys.exit(papermill())
[2022-11-15T01:37:41.289Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
[2022-11-15T01:37:41.289Z] return self.main(*args, **kwargs)
[2022-11-15T01:37:41.289Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1053, in main
[2022-11-15T01:37:41.289Z] rv = self.invoke(ctx)
[2022-11-15T01:37:41.289Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
[2022-11-15T01:37:41.289Z] return ctx.invoke(self.callback, **ctx.params)
[2022-11-15T01:37:41.289Z] File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 754, in invoke
[2022-11-15T01:37:41.289Z] return __callback(*args, **kwargs)
[2022-11-15T01:37:41.289Z] File "/opt/conda/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
[2022-11-15T01:37:41.289Z] return f(get_current_context(), *args, **kwargs)
[2022-11-15T01:37:41.289Z] File "/opt/conda/lib/python3.8/site-packages/papermill/cli.py", line 250, in papermill
[2022-11-15T01:37:41.289Z] execute_notebook(
[2022-11-15T01:37:41.289Z] File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 128, in execute_notebook
[2022-11-15T01:37:41.289Z] raise_for_execution_errors(nb, output_path)
[2022-11-15T01:37:41.289Z] File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2022-11-15T01:37:41.289Z] raise error
[2022-11-15T01:37:41.289Z] papermill.exceptions.PapermillExecutionError:
[2022-11-15T01:37:41.289Z] ---------------------------------------------------------------------------
[2022-11-15T01:37:41.289Z] Exception encountered at "In [9]":
[2022-11-15T01:37:41.289Z] ---------------------------------------------------------------------------
[2022-11-15T01:37:41.289Z] ValueError Traceback (most recent call last)
[2022-11-15T01:37:41.289Z] Cell In [9], line 26
[2022-11-15T01:37:41.289Z] 24 # MLFlowHandler plots loss at every iteration on MLFlow web UI
[2022-11-15T01:37:41.289Z] 25 mlflow_dir = os.path.join(log_dir, "mlruns")
[2022-11-15T01:37:41.289Z] ---> 26 train_mlflow_handler = MLFlowHandler(tracking_uri=Path(mlflow_dir).as_uri(), output_transform=lambda x: x)
[2022-11-15T01:37:41.289Z] 27 train_mlflow_handler.attach(trainer)
[2022-11-15T01:37:41.289Z]
[2022-11-15T01:37:41.289Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/handlers/mlflow_handler.py:95, in MLFlowHandler.__init__(self, tracking_uri, iteration_log, epoch_log, epoch_logger, iteration_logger, output_transform, global_epoch_transform, state_attributes, tag_name)
[2022-11-15T01:37:41.289Z] 93 if tracking_uri is not None:
[2022-11-15T01:37:41.289Z] 94 if isinstance(tracking_uri, str):
[2022-11-15T01:37:41.289Z] ---> 95 tracking_uri = Path(tracking_uri).as_uri()
[2022-11-15T01:37:41.289Z] 96 mlflow.set_tracking_uri(tracking_uri)
[2022-11-15T01:37:41.289Z] 98 self.iteration_log = iteration_log
[2022-11-15T01:37:41.289Z]
[2022-11-15T01:37:41.289Z] File /opt/conda/lib/python3.8/pathlib.py:748, in PurePath.as_uri(self)
[2022-11-15T01:37:41.289Z] 746 """Return the path as a 'file' URI."""
[2022-11-15T01:37:41.289Z] 747 if not self.is_absolute():
[2022-11-15T01:37:41.289Z] --> 748 raise ValueError("relative path can't be expressed as a file URI")
[2022-11-15T01:37:41.289Z] 749 return self._flavour.make_uri(self)
[2022-11-15T01:37:41.289Z]
[2022-11-15T01:37:41.289Z] ValueError: relative path can't be expressed as a file URI
[2022-11-15T01:37:41.289Z]
cc @Nic-Ma @binliunls could you please have a look?
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels