Skip to content

Commit 85b0e03

Browse files
anibalipytorchmergebot
authored andcommitted
Default permissions for torch.hub downloads (#82869)
### Description The `download_url_to_file` function in torch.hub uses a temporary file to prevent overriding a local working checkpoint with a broken download.This temporary file is created using `NamedTemporaryFile`. However, since `NamedTemporaryFile` creates files with overly restrictive permissions (0600), the resulting download will not have default permissions and will not respect umask on Linux (since moving the file will retain the restrictive permissions of the temporary file). This is especially problematic when trying to share model checkpoints between multiple users as other users will not even have read access to the file. The change in this PR fixes the issue by using custom code to create the temporary file without changing the permissions to 0600 (unfortunately there is no way to override the permissions behaviour of existing Python standard library code). This ensures that the downloaded checkpoint file correctly have the default permissions applied. If a user wants to apply more restrictive permissions, they can do so via usual means (i.e. by setting umask). See these similar issues in other projects for even more context: * borgbackup/borg#6400 * borgbackup/borg#6933 * zarr-developers/zarr-python#325 ### Issue #81297 ### Testing Extended the unit test `test_download_url_to_file` to also check permissions. Pull Request resolved: #82869 Approved by: https://github.com/vmoens
1 parent 64d5851 commit 85b0e03

2 files changed

Lines changed: 18 additions & 2 deletions

File tree

‎test/test_hub.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -96,6 +96,12 @@ def test_download_url_to_file(self):
9696
hub.download_url_to_file(TORCHHUB_EXAMPLE_RELEASE_URL, f, progress=False)
9797
loaded_state = torch.load(f)
9898
self.assertEqual(sum_of_state_dict(loaded_state), SUM_OF_HUB_EXAMPLE)
99+
# Check that the downloaded file has default file permissions
100+
f_ref = os.path.join(tmpdir, 'reference')
101+
open(f_ref, 'w').close()
102+
expected_permissions = oct(os.stat(f_ref).st_mode & 0o777)
103+
actual_permissions = oct(os.stat(f).st_mode & 0o777)
104+
assert actual_permissions == expected_permissions
99105

100106
@retry(Exception, tries=3)
101107
def test_load_state_dict_from_url(self):

‎torch/hub.py‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,7 @@
88
import sys
99
import tempfile
1010
import torch
11+
import uuid
1112
import warnings
1213
import zipfile
1314
from pathlib import Path
@@ -628,9 +629,18 @@ def download_url_to_file(url: str, dst: str, hash_prefix: Optional[str] = None,
628629
# We deliberately save it in a temp file and move it after
629630
# download is complete. This prevents a local working checkpoint
630631
# being overridden by a broken download.
632+
# We deliberately do not use NamedTemporaryFile to avoid restrictive
633+
# file permissions being applied to the downloaded file.
631634
dst = os.path.expanduser(dst)
632-
dst_dir = os.path.dirname(dst)
633-
f = tempfile.NamedTemporaryFile(delete=False, dir=dst_dir)
635+
for seq in range(tempfile.TMP_MAX):
636+
tmp_dst = dst + '.' + uuid.uuid4().hex + '.partial'
637+
try:
638+
f = open(tmp_dst, 'w+b')
639+
except FileExistsError:
640+
continue
641+
break
642+
else:
643+
raise FileExistsError(errno.EEXIST, 'No usable temporary file name found')
634644

635645
try:
636646
if hash_prefix is not None:

0 commit comments

Comments
 (0)