Skip to content

Close MLFlow run automatically when workflow is end #5649

@binliunls

Description

@binliunls

The MLFlow will start a run in the backend to record experiments' details like parameters, metrics and images. Currently, the mlflow handler works fine if just run one round of several epoch training like an 100 epochs segmentation task, since the run will be killed as the process end. However, tasks like active learning, network architecture search and automl try to run a few rounds of several epochs training at one time. For exampe, an active learning may run 10 rounds of 100 epochs training in one process. This may lead to a problem that experiment details of all rounds would be recorded to the same run due to the check logic here. From the users' experience point of view, these rounds of experiments should be recorded to different runs (one run for one round). Therefore, it would be better that we could choose to shut down the current run after one round of training.

Describe the solution you'd like

  • Add an auto_close:bool parameter which decides whether to close current run at the complete phase of the workflow, default to False.
  • Make the default auto_close parameter of trainer and evaluator to True to close the run when training and inference is completed.

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

Relationships

None yet

Development

No branches or pull requests

Issue actions