Skip to content

Inflexible score function of CheckpointSaver #2467

@sandylaker

Description

@sandylaker

def _score_func(engine: Engine):
if isinstance(key_metric_name, str):
metric_name = key_metric_name
elif hasattr(engine.state, "key_metric_name") and isinstance(engine.state.key_metric_name, str):
metric_name = engine.state.key_metric_name
else:
raise ValueError(
f"Incompatible values: save_key_metric=True and key_metric_name={key_metric_name}."
)
return round(engine.state.metrics[metric_name], 4)
if key_metric_filename is not None and key_metric_n_saved > 1:
raise ValueError("if using fixed filename to save the best metric model, we should only save 1 model.")

Currently the CheckpointSaver saves the check point that has the largest score value. However, this is not desirable when a lower score indicates better performance, e.g. validation loss. We can either allow the user to pass a custom score function (as in the ignite.handlers), or add a flag to control returning round(engine.state.metrics[metric_name], 4) or - round(engine.state.metrics[metric_name], 4)

Metadata

Metadata

Assignees

Labels

questionFurther information is requested

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions