Skip to content

Commit 7858680

Browse files
[HfFileSystem] List large repos in parallel (#5153)
* [HfFileSystem] List large repos in parallel The Hub `tree` endpoint returns at most 1000 entries per request (50 with `expand=True`) and paginates with a cursor: a page can only be fetched once the previous one is received. Listing `datasets/allenai/objaverse` (~800k files) with a single recursive request therefore requires ~800 *sequential* requests. Recursive listings are now split between sub-directories and listed concurrently: while a sub-tree is being paginated, it gets dispatched directory by directory as soon as those sub-directories hold enough entries to pay for their own request. A sub-tree that fits in a single page is never split, so the number of requests stays close to the theoretical minimum (one request per 1000 entries) while independent directories are fetched in parallel. `hffs.glob("datasets/allenai/objaverse/**/*")` goes from 960s to 20s, for the exact same output (807 requests against ~800 for the sequential listing). On top of that: - `maxdepth` is now taken into account while listing: a sub-tree deeper than `maxdepth` is listed one level deep instead of being fetched entirely and then filtered out. - New `HfFileSystem(max_threads=...)` argument and `HF_HUB_FS_MAX_THREADS` env variable, set to 1 to disable parallel listings. - New internals to paginate page by page: `paginate_pages()` and `HfApi._iter_repo_tree_pages()`. * fix test * remove flaky timign test * cursor's comment: don't split on last page * style * update test * fix listing threads not stopping on error or ctrl-c * fix splitting folders smaller than a page * fix max_threads naming to max_workers and drop the env var * fix redundant find test * fix listing timings in docs --------- Co-authored-by: lhoestq <[email protected]> Co-authored-by: Celina Hanouti <[email protected]>
1 parent 75fbd6d commit 7858680

6 files changed

Lines changed: 548 additions & 44 deletions

File tree

‎docs/source/en/guides/hf_file_system.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,27 @@ The optional `revision` argument can be passed to run an operation from a specif
4545

4646
Unlike Python's built-in `open`, `fsspec`'s `open` defaults to binary mode, `"rb"`. This means you must explicitly set mode as `"r"` for reading and `"w"` for writing in text mode. Appending to a file (modes `"a"` and `"ab"`) is not supported yet.
4747

48+
## Listing large repositories
49+
50+
Operations listing files recursively, such as `find` and `glob`, need to fetch the whole
51+
content of the repository. The Hub returns those entries page by page, and a page can only be requested once the
52+
previous one has been received: listing a repository holding 800k files requires ~800 sequential requests, and
53+
therefore about 30 minutes.
54+
55+
To avoid that, [`HfFileSystem`] lists large repositories directory by directory, in parallel:
56+
57+
```python
58+
>>> from huggingface_hub import hffs
59+
60+
>>> # Takes ~30s on a repository holding 800k files
61+
>>> files = hffs.glob("datasets/allenai/objaverse/**/*.glb", maxdepth=3)
62+
```
63+
64+
The number of concurrent requests is controlled by the `max_workers` argument of [`HfFileSystem`]. It defaults to 16
65+
and can be set to 1 to disable parallel listings.
66+
Note that passing a `maxdepth` also significantly speeds listings up, since the Hub then only returns the entries
67+
that are actually requested.
68+
4869
## Keep listings fresh
4970

5071
Directories are listed once and their content is cached, so a directory that someone else just changed is not

‎src/huggingface_hub/hf_api.py‎

Lines changed: 40 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -107,6 +107,7 @@
107107
SafetensorsParsingError,
108108
SafetensorsRepoMetadata,
109109
TensorInfo,
110+
_paginate_pages,
110111
build_hf_headers,
111112
chunk_iterable,
112113
experimental,
@@ -3980,14 +3981,51 @@ def list_repo_tree(
39803981
]
39813982
```
39823983
"""
3984+
for page, _ in self._iter_repo_tree_pages(
3985+
repo_id,
3986+
path_in_repo,
3987+
recursive=recursive,
3988+
expand=expand,
3989+
revision=revision,
3990+
repo_type=repo_type,
3991+
token=token,
3992+
):
3993+
yield from page
3994+
3995+
def _iter_repo_tree_pages(
3996+
self,
3997+
repo_id: str,
3998+
path_in_repo: str | None = None,
3999+
*,
4000+
recursive: bool = False,
4001+
expand: bool = False,
4002+
revision: str | None = None,
4003+
repo_type: str | None = None,
4004+
token: str | bool | None = None,
4005+
) -> Iterable[tuple[list[RepoFile | RepoFolder], bool]]:
4006+
"""Same as [`HfApi.list_repo_tree`] but yields the results page by page and with a `finished` boolean.
4007+
4008+
Internal method: pagination is capped by the Hub (1000 entries per page, 50 when `expand=True`)
4009+
and is cursor-based, so listing a large tree requires to follow the pages one by one. Yielding
4010+
page per page allows a caller to stop in the middle of the pagination and re-dispatch the
4011+
remaining of a sub-tree to (cheaper to list) sub-directories. See [`HfFileSystem`].
4012+
"""
39834013
repo_type = repo_type or constants.REPO_TYPE_MODEL
39844014
revision = quote(revision, safe="") if revision is not None else constants.DEFAULT_REVISION
39854015
headers = self._build_hf_headers(token=token)
39864016

39874017
encoded_path_in_repo = "/" + quote(path_in_repo, safe="") if path_in_repo else ""
39884018
tree_url = f"{self.endpoint}/api/{repo_type}s/{repo_id}/tree/{revision}{encoded_path_in_repo}"
3989-
for path_info in paginate(path=tree_url, headers=headers, params={"recursive": recursive, "expand": expand}):
3990-
yield (RepoFile(**path_info) if path_info["type"] == "file" else RepoFolder(**path_info))
4019+
for page, finished in _paginate_pages(
4020+
path=tree_url, headers=headers, params={"recursive": recursive, "expand": expand}
4021+
):
4022+
yield (
4023+
[
4024+
RepoFile(**path_info) if path_info["type"] == "file" else RepoFolder(**path_info)
4025+
for path_info in page
4026+
],
4027+
finished,
4028+
)
39914029

39924030
@validate_hf_hub_args
39934031
def verify_repo_checksums(

0 commit comments

Comments
 (0)