Wow, nice catch. Also should we follow up with taking an exclusive lock on a file when writing a checkpoint?

It would be good if there were an official, well known, non-footgun way of saving checkpoints in this context.

I think it's hard to completely eliminate "bad" behavior. For example, people could checkpoint a model in some local file, and then move the checkpoint to a network file system (https://github.com/stanford-futuredata/pytorch-distributed/blob/master/training/train_imagenet_nv.py#L408).

But locks on files might be a good thing to do anyway.