Repository navigation
Commit 7858680
[HfFileSystem] List large repos in parallel (#5153)
* [HfFileSystem] List large repos in parallel
The Hub `tree` endpoint returns at most 1000 entries per request (50 with
`expand=True`) and paginates with a cursor: a page can only be fetched once the
previous one is received. Listing `datasets/allenai/objaverse` (~800k files) with a
single recursive request therefore requires ~800 *sequential* requests.
Recursive listings are now split between sub-directories and listed concurrently:
while a sub-tree is being paginated, it gets dispatched directory by directory as
soon as those sub-directories hold enough entries to pay for their own request. A
sub-tree that fits in a single page is never split, so the number of requests stays
close to the theoretical minimum (one request per 1000 entries) while independent
directories are fetched in parallel. `hffs.glob("datasets/allenai/objaverse/**/*")`
goes from 960s to 20s, for the exact same output (807 requests against ~800 for the
sequential listing).
On top of that:
- `maxdepth` is now taken into account while listing: a sub-tree deeper than
`maxdepth` is listed one level deep instead of being fetched entirely and then
filtered out.
- New `HfFileSystem(max_threads=...)` argument and `HF_HUB_FS_MAX_THREADS` env
variable, set to 1 to disable parallel listings.
- New internals to paginate page by page: `paginate_pages()` and
`HfApi._iter_repo_tree_pages()`.
* fix test
* remove flaky timign test
* cursor's comment: don't split on last page
* style
* update test
* fix listing threads not stopping on error or ctrl-c
* fix splitting folders smaller than a page
* fix max_threads naming to max_workers and drop the env var
* fix redundant find test
* fix listing timings in docs
---------
Co-authored-by: lhoestq <[email protected]>
Co-authored-by: Celina Hanouti <[email protected]>1 parent 75fbd6d commit 7858680
6 files changed
Lines changed: 548 additions & 44 deletions
File tree
- docs/source/en/guides
- src/huggingface_hub
- utils
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
45 | 45 | | |
46 | 46 | | |
47 | 47 | | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
48 | 69 | | |
49 | 70 | | |
50 | 71 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
107 | 107 | | |
108 | 108 | | |
109 | 109 | | |
| 110 | + | |
110 | 111 | | |
111 | 112 | | |
112 | 113 | | |
| |||
3980 | 3981 | | |
3981 | 3982 | | |
3982 | 3983 | | |
| 3984 | + | |
| 3985 | + | |
| 3986 | + | |
| 3987 | + | |
| 3988 | + | |
| 3989 | + | |
| 3990 | + | |
| 3991 | + | |
| 3992 | + | |
| 3993 | + | |
| 3994 | + | |
| 3995 | + | |
| 3996 | + | |
| 3997 | + | |
| 3998 | + | |
| 3999 | + | |
| 4000 | + | |
| 4001 | + | |
| 4002 | + | |
| 4003 | + | |
| 4004 | + | |
| 4005 | + | |
| 4006 | + | |
| 4007 | + | |
| 4008 | + | |
| 4009 | + | |
| 4010 | + | |
| 4011 | + | |
| 4012 | + | |
3983 | 4013 | | |
3984 | 4014 | | |
3985 | 4015 | | |
3986 | 4016 | | |
3987 | 4017 | | |
3988 | 4018 | | |
3989 | | - | |
3990 | | - | |
| 4019 | + | |
| 4020 | + | |
| 4021 | + | |
| 4022 | + | |
| 4023 | + | |
| 4024 | + | |
| 4025 | + | |
| 4026 | + | |
| 4027 | + | |
| 4028 | + | |
3991 | 4029 | | |
3992 | 4030 | | |
3993 | 4031 | | |
| |||
0 commit comments