Skip to content

MLFlowHandler issue #1040

@wyli

Description

@wyli
[2022-11-15T01:37:10.470Z] Running ./3d_segmentation/unet_segmentation_3d_ignite.ipynb
[2022-11-15T01:37:10.470Z] Checking PEP8 compliance...
[2022-11-15T01:37:11.401Z] Running notebook...
[2022-11-15T01:37:11.401Z] Before:
[2022-11-15T01:37:11.401Z]     "max_epochs = 10\n",
[2022-11-15T01:37:11.401Z] After:
[2022-11-15T01:37:11.401Z]     "max_epochs = 1\n",
[2022-11-15T01:37:16.656Z] MONAI version: 1.0.0+110.g53321873
[2022-11-15T01:37:16.656Z] Numpy version: 1.22.2
[2022-11-15T01:37:16.656Z] Pytorch version: 1.13.0a0+d0d6b1f
[2022-11-15T01:37:16.656Z] MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
[2022-11-15T01:37:16.656Z] MONAI rev id: 5332187324bc2471476581e729d685e30d03d40a
[2022-11-15T01:37:16.656Z] MONAI __file__: /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/__init__.py
[2022-11-15T01:37:16.656Z] 
[2022-11-15T01:37:16.656Z] Optional dependencies:
[2022-11-15T01:37:16.656Z] Pytorch Ignite version: 0.4.10
[2022-11-15T01:37:16.656Z] Nibabel version: 4.0.2
[2022-11-15T01:37:16.656Z] scikit-image version: 0.19.3
[2022-11-15T01:37:16.656Z] Pillow version: 9.0.1
[2022-11-15T01:37:16.656Z] Tensorboard version: 2.10.1
[2022-11-15T01:37:16.656Z] gdown version: 4.5.3
[2022-11-15T01:37:16.656Z] TorchVision version: 0.14.0a0
[2022-11-15T01:37:16.656Z] tqdm version: 4.64.1
[2022-11-15T01:37:16.656Z] lmdb version: 1.3.0
[2022-11-15T01:37:16.656Z] psutil version: 5.9.2
[2022-11-15T01:37:16.656Z] pandas version: 1.4.4
[2022-11-15T01:37:16.656Z] einops version: 0.6.0
[2022-11-15T01:37:16.656Z] transformers version: 4.21.3
[2022-11-15T01:37:16.656Z] mlflow version: 1.30.0
[2022-11-15T01:37:16.656Z] pynrrd version: 1.0.0
[2022-11-15T01:37:16.656Z] 
[2022-11-15T01:37:16.656Z] For details about installing the optional dependencies, please visit:
[2022-11-15T01:37:16.656Z]     https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies
[2022-11-15T01:37:16.656Z] 
[2022-11-15T01:37:17.220Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-15T01:37:17.220Z]   warnings.warn(
[2022-11-15T01:37:41.289Z] 
Executing:   0%|          | 0/29 [00:00<?, ?cell/s]
Executing:   3%|▎         | 1/29 [00:01<00:44,  1.58s/cell]
Executing:  10%|█         | 3/29 [00:07<01:03,  2.44s/cell]
Executing:  17%|█▋        | 5/29 [00:11<00:55,  2.29s/cell]
Executing:  38%|███▊      | 11/29 [00:18<00:28,  1.59s/cell]
Executing:  45%|████▍     | 13/29 [00:21<00:23,  1.48s/cell]
Executing:  59%|█████▊    | 17/29 [00:21<00:10,  1.14cell/s]
Executing:  66%|██████▌   | 19/29 [00:22<00:08,  1.23cell/s]
Executing:  66%|██████▌   | 19/29 [00:24<00:12,  1.27s/cell]
[2022-11-15T01:37:41.289Z] /opt/conda/lib/python3.8/site-packages/papermill/iorw.py:153: UserWarning: the file is not specified with any extension : -
[2022-11-15T01:37:41.289Z]   warnings.warn(
[2022-11-15T01:37:41.289Z] Traceback (most recent call last):
[2022-11-15T01:37:41.289Z]   File "/opt/conda/bin/papermill", line 8, in <module>
[2022-11-15T01:37:41.289Z]     sys.exit(papermill())
[2022-11-15T01:37:41.289Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1128, in __call__
[2022-11-15T01:37:41.289Z]     return self.main(*args, **kwargs)
[2022-11-15T01:37:41.289Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1053, in main
[2022-11-15T01:37:41.289Z]     rv = self.invoke(ctx)
[2022-11-15T01:37:41.289Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 1395, in invoke
[2022-11-15T01:37:41.289Z]     return ctx.invoke(self.callback, **ctx.params)
[2022-11-15T01:37:41.289Z]   File "/opt/conda/lib/python3.8/site-packages/click/core.py", line 754, in invoke
[2022-11-15T01:37:41.289Z]     return __callback(*args, **kwargs)
[2022-11-15T01:37:41.289Z]   File "/opt/conda/lib/python3.8/site-packages/click/decorators.py", line 26, in new_func
[2022-11-15T01:37:41.289Z]     return f(get_current_context(), *args, **kwargs)
[2022-11-15T01:37:41.289Z]   File "/opt/conda/lib/python3.8/site-packages/papermill/cli.py", line 250, in papermill
[2022-11-15T01:37:41.289Z]     execute_notebook(
[2022-11-15T01:37:41.289Z]   File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 128, in execute_notebook
[2022-11-15T01:37:41.289Z]     raise_for_execution_errors(nb, output_path)
[2022-11-15T01:37:41.289Z]   File "/opt/conda/lib/python3.8/site-packages/papermill/execute.py", line 232, in raise_for_execution_errors
[2022-11-15T01:37:41.289Z]     raise error
[2022-11-15T01:37:41.289Z] papermill.exceptions.PapermillExecutionError: 
[2022-11-15T01:37:41.289Z] ---------------------------------------------------------------------------
[2022-11-15T01:37:41.289Z] Exception encountered at "In [9]":
[2022-11-15T01:37:41.289Z] ---------------------------------------------------------------------------
[2022-11-15T01:37:41.289Z] ValueError                                Traceback (most recent call last)
[2022-11-15T01:37:41.289Z] Cell In [9], line 26
[2022-11-15T01:37:41.289Z]      24 # MLFlowHandler plots loss at every iteration on MLFlow web UI
[2022-11-15T01:37:41.289Z]      25 mlflow_dir = os.path.join(log_dir, "mlruns")
[2022-11-15T01:37:41.289Z] ---> 26 train_mlflow_handler = MLFlowHandler(tracking_uri=Path(mlflow_dir).as_uri(), output_transform=lambda x: x)
[2022-11-15T01:37:41.289Z]      27 train_mlflow_handler.attach(trainer)
[2022-11-15T01:37:41.289Z] 
[2022-11-15T01:37:41.289Z] File /home/jenkins/agent/workspace/Monai-notebooks/MONAI/monai/handlers/mlflow_handler.py:95, in MLFlowHandler.__init__(self, tracking_uri, iteration_log, epoch_log, epoch_logger, iteration_logger, output_transform, global_epoch_transform, state_attributes, tag_name)
[2022-11-15T01:37:41.289Z]      93 if tracking_uri is not None:
[2022-11-15T01:37:41.289Z]      94     if isinstance(tracking_uri, str):
[2022-11-15T01:37:41.289Z] ---> 95         tracking_uri = Path(tracking_uri).as_uri()
[2022-11-15T01:37:41.289Z]      96     mlflow.set_tracking_uri(tracking_uri)
[2022-11-15T01:37:41.289Z]      98 self.iteration_log = iteration_log
[2022-11-15T01:37:41.289Z] 
[2022-11-15T01:37:41.289Z] File /opt/conda/lib/python3.8/pathlib.py:748, in PurePath.as_uri(self)
[2022-11-15T01:37:41.289Z]     746 """Return the path as a 'file' URI."""
[2022-11-15T01:37:41.289Z]     747 if not self.is_absolute():
[2022-11-15T01:37:41.289Z] --> 748     raise ValueError("relative path can't be expressed as a file URI")
[2022-11-15T01:37:41.289Z]     749 return self._flavour.make_uri(self)
[2022-11-15T01:37:41.289Z] 
[2022-11-15T01:37:41.289Z] ValueError: relative path can't be expressed as a file URI
[2022-11-15T01:37:41.289Z] 

cc @Nic-Ma @binliunls could you please have a look?

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions