torch.load() - OSError on a file larger than 2GB #26998
Description
🐛 Bug
In version 1.2, torch.load() does not work with files larger than 2GB.
import torch
import pathlib
torch.save(torch.rand(int((2 * 1024**3) / 4), dtype=torch.float32, device="cpu"),
pathlib.Path("pytorch_test_load.temp"))
torch.load(pathlib.Path("pytorch_test_load.temp")) File "torch\serialization.py", line 581, in _load
deserialized_objects[key]._set_from_file(f, offset, f_should_read_directly)
OSError: [Errno 22] Invalid argument
This is probably due to #20270 or #20900
Windows 10
Python 3.7
Pytorch 1.2.0