-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Close MLFlow run automatically when workflow is end #5649
Description
The MLFlow will start a run in the backend to record experiments' details like parameters, metrics and images. Currently, the mlflow handler works fine if just run one round of several epoch training like an 100 epochs segmentation task, since the run will be killed as the process end. However, tasks like active learning, network architecture search and automl try to run a few rounds of several epochs training at one time. For exampe, an active learning may run 10 rounds of 100 epochs training in one process. This may lead to a problem that experiment details of all rounds would be recorded to the same run due to the check logic here. From the users' experience point of view, these rounds of experiments should be recorded to different runs (one run for one round). Therefore, it would be better that we could choose to shut down the current run after one round of training.
Describe the solution you'd like
- Add an
auto_close:boolparameter which decides whether to close currentrunat thecompletephase of the workflow, default to False. - Make the default
auto_closeparameter of trainer and evaluator to True to close the run when training and inference is completed.