-
Notifications
You must be signed in to change notification settings - Fork 1.5k
Inflexible score function of CheckpointSaver #2467
Copy link
Copy link
Closed
Labels
questionFurther information is requestedFurther information is requested
Description
MONAI/monai/handlers/checkpoint_saver.py
Lines 148 to 160 in 9f9b63a
| def _score_func(engine: Engine): | |
| if isinstance(key_metric_name, str): | |
| metric_name = key_metric_name | |
| elif hasattr(engine.state, "key_metric_name") and isinstance(engine.state.key_metric_name, str): | |
| metric_name = engine.state.key_metric_name | |
| else: | |
| raise ValueError( | |
| f"Incompatible values: save_key_metric=True and key_metric_name={key_metric_name}." | |
| ) | |
| return round(engine.state.metrics[metric_name], 4) | |
| if key_metric_filename is not None and key_metric_n_saved > 1: | |
| raise ValueError("if using fixed filename to save the best metric model, we should only save 1 model.") |
Currently the CheckpointSaver saves the check point that has the largest score value. However, this is not desirable when a lower score indicates better performance, e.g. validation loss. We can either allow the user to pass a custom score function (as in the ignite.handlers), or add a flag to control returning round(engine.state.metrics[metric_name], 4) or - round(engine.state.metrics[metric_name], 4)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
questionFurther information is requestedFurther information is requested