Skip to content

save_metadata panics without supporting arrays #238

@Lun4m

Description

@Lun4m

What happened?

I'm using an anemoi-training fork that does not use supporting arrays when calling save_metadata and I get the following error:

[rank0]:   File ".../anemoi-training-downscaling/src/anemoi/training/diagnostics/callbacks/callbacks.py", line 1202, in _save_checkpoint
[rank0]:     save_metadata(inference_checkpoint_filepath, metadata)
[rank0]:   File ".../anemoi-utils/src/anemoi/utils/checkpoints.py", line 235, in save_metadata
[rank0]:     _write_array_to_bytes(supporting_arrays, "", metadata["supporting_arrays_paths"], zipf)
[rank0]:   File ".../anemoi-utils/src/anemoi/utils/checkpoints.py", line 179, in _write_array_to_bytes
[rank0]:     entry["path"],
[rank0]:     ~~~~~^^^^^^^^
[rank0]: KeyError: 'path'

This bug was introduced in #229, in src/anemoi/utils/checkpoints.py:

-       for name, entry in metadata["supporting_arrays_paths"].items():
-           value = supporting_arrays[name]
-           LOG.info(
-               "Saving supporting array `%s` to %s (shape=%s, dtype=%s)",
-               name,
-               entry["path"],
-               entry["shape"],
-               entry["dtype"],
-           )
-           zipf.writestr(entry["path"], value.tobytes())
+       _write_array_to_bytes(supporting_arrays, "", metadata["supporting_arrays_paths"], zipf)

In the old version, if metadata["supporting_arrays_paths"] is empty, entry["path"] is not called.
That's not true anymore in the updated version.

What are the steps to reproduce the bug?

Calling save_metadata with the default supporting_arrays = None argument

Version

commit 9dacd62

Platform (OS and architecture)

Linux uan01 5.14.21-150500.55.49_13.0.56-cray_shasta_c #1 SMP Mon Mar 4 14:19:49 UTC 2024 (9d8355b) x86_64 x86_64 x86_64 GNU/Linux

Relevant log output

Accompanying data

No response

Organisation

No response

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions