[CLI] [API] Add `HfApi.copy_files` method to copy files remotely and update 'hf buckets cp' by Wauplin · Pull Request #3874 · huggingface/huggingface_hub

Wauplin · 2026-03-02T15:12:16Z

Note: requires https://github.com/huggingface-internal/moon-landing/pull/17593 to be merged first. EDIT: merged!

This PR adds a new HfApi.copy_files API and extends hf buckets cp to support remote HF-handle copy workflows.

Copy from bucket to bucket (same bucket or different bucket)
Copy from repo (model/dataset/space) to bucket
Reject bucket->repo and repo->repo destinations (not supported yet)

If source is a file, copies it. If a directory, recursively copy files under source folder.

Repo source file with xet_hash: copied directly by hash
Repo source file without xet_hash (regular small file): download then re-upload
Bucket to bucket: always copied by hash

See https://github.com/huggingface-internal/moon-landing/pull/17593#issue-4201288199 PR description for working test.

Tested on https://huggingface.co/buckets/Wauplin/bucket-raw

hf buckets cp hf://models/openai-community/gpt2 hf://buckets/Wauplin/bucket-raw/models/gpt2
hf buckets cp hf://models/google/gemma-4-31B-it hf://buckets/Wauplin/bucket-raw/models/gemma4
hf buckets cp hf://models/zai-org/GLM-5.1 hf://buckets/Wauplin/bucket-raw/models/glm5.1

hf buckets cp hf://datasets/wikimedia/wikipedia hf://buckets/Wauplin/bucket-raw/datasets/wikipedia 
hf buckets cp hf://datasets/badlogicgames/pi-mono hf://buckets/Wauplin/bucket-raw/datasets/pi-mono-traces

hf buckets cp hf://buckets/julien-c/my-training-bucket/art hf://buckets/Wauplin/bucket-raw/buckets/art

Note

Medium Risk
Introduces new path parsing and copy semantics (including revision handling and mixed copy/download code paths) plus changes bucket batch operation payloads/chunking, which could impact correctness and performance of bucket file operations.

Overview
Adds HfApi.copy_files (exported as copy_files) to copy a file or folder from an hf:// bucket or repo (model/dataset/space, with optional @revision) into a bucket destination, using server-side hash copies when possible and falling back to download+reupload for non-Xet repo files.

Extends batch_bucket_files with a new copy operation type (NDJSON copyFile) and updates internal batching/chunk sizing and upload logic so copy-by-hash operations can be sent without uploading data.

Updates hf buckets cp to accept generic hf://... handles and enable remote-to-remote copies via api.copy_files, plus adds docs and tests covering bucket↔bucket, repo→bucket, and rejected bucket→repo cases.

^{Reviewed by Cursor Bugbot for commit 5508120. Bugbot is set up for automated code reviews on this repo. Configure here.}

bot-ci-comment · 2026-03-02T15:19:01Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Wauplin · 2026-03-03T14:04:07Z

Current status: x-bucket copies do not work. Need an extra call to CAS to tell about the xethash being registered in destination bucket

Wauplin · 2026-04-07T13:34:47Z

+            else:
+                all_adds.append((_download_from_repo(file.path), target_path))


sub-optimal: could be parallelize but that's not something we want to optimize for now

Wauplin · 2026-04-07T14:46:16Z



+def _parse_hf_copy_handle(hf_handle: str) -> _BucketCopyHandle | _RepoCopyHandle:
+    # TODO: Harmonize hf:// parsing. See https://github.com/huggingface/huggingface_hub/issues/3971


yes, #3971 is getting high in my priorities 🙈

hanouticelina

Made a first pass!

Co-authored-by: célina <[email protected]>

…gface_hub into feat/hfapi-copy-files

Co-authored-by: célina <[email protected]>

…gface_hub into feat/hfapi-copy-files t pu#

hanouticelina

Thank you!

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 9db2cec. Configure here.}

julien-c · 2026-04-10T10:44:22Z

+
+Notes:
+
+- Bucket-to-repo copy is not supported.


Wauplin added 3 commits March 2, 2026 15:51

Add HfApi copy_files and remote hf buckets cp support

1490f6c

Adjust copy_files return type and cp output message

bca3c0a

remove useless

0ad2305

Wauplin added 4 commits March 2, 2026 16:32

docs

122c4ae

do not catch

641470a

comment

5c5b5ea

much better

d97522f

Wauplin added 15 commits April 3, 2026 16:50

Merge branch 'main' into feat/hfapi-copy-files

b067a9e

Server-side copies

d99a877

Merge branch 'main' into feat/hfapi-copy-files

5c1279f

check remote path

1dee89a

simpler

da4efab

docs

c685a7a

type

b58c13f

no dummy check

d1115fa

fix imports and types

9f82d81

add todo

e6711cb

simplified

de374cf

useless

5e76d93

type

d601517

all good

47345a0

review tests

b46f914

Wauplin changed the title ~~[API] Add HfApi.copy_files method to copy files remotely~~ [CLI] [API] Add HfApi.copy_files method to copy files remotely and update 'hf buckets cp' Apr 7, 2026

Wauplin requested a review from hanouticelina April 7, 2026 14:44

Wauplin marked this pull request as ready for review April 7, 2026 14:44

Wauplin commented Apr 7, 2026

View reviewed changes

cursor Bot reviewed Apr 7, 2026

View reviewed changes

Comment thread src/huggingface_hub/hf_api.py

Comment thread src/huggingface_hub/_buckets.py Outdated

Comment thread tests/test_buckets_cli.py Outdated

mypy happy

a996a0f

hanouticelina reviewed Apr 7, 2026

View reviewed changes

Wauplin and others added 8 commits April 8, 2026 14:53

aze

2395834

Update src/huggingface_hub/hf_api.py

8f14491

Co-authored-by: célina <[email protected]>

rename to _build_copy_op

b134e2f

Merge branch 'feat/hfapi-copy-files' of github.com:huggingface/huggin…

a9d4237

…gface_hub into feat/hfapi-copy-files

Update src/huggingface_hub/hf_api.py

96987a5

Co-authored-by: célina <[email protected]>

type ignore

9d4ed1d

Merge branch 'feat/hfapi-copy-files' of github.com:huggingface/huggin…

80fb70f

…gface_hub into feat/hfapi-copy-files t pu#

type

f0c69ea

Wauplin requested a review from hanouticelina April 8, 2026 13:08

hanouticelina approved these changes Apr 8, 2026

View reviewed changes

parallel download of small files

9db2cec

cursor Bot reviewed Apr 9, 2026

View reviewed changes

Comment thread src/huggingface_hub/hf_api.py

Comment thread src/huggingface_hub/hf_api.py

fix

5508120

Wauplin merged commit d82a7f7 into main Apr 9, 2026
13 of 21 checks passed

Wauplin deleted the feat/hfapi-copy-files branch April 9, 2026 09:52

Wauplin mentioned this pull request Apr 9, 2026

Fix copy file to folder #4075

Merged

julien-c reviewed Apr 10, 2026

View reviewed changes

Wauplin mentioned this pull request Apr 24, 2026

feat(hub): add copyFiles API for remote file copy between buckets and repositories huggingface/huggingface.js#2121

Draft

		else:
		all_adds.append((_download_from_repo(file.path), target_path))



		def _parse_hf_copy_handle(hf_handle: str) -> _BucketCopyHandle \| _RepoCopyHandle:
		# TODO: Harmonize hf:// parsing. See https://github.com/huggingface/huggingface_hub/issues/3971

Conversation

Wauplin commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Note: requires https://github.com/huggingface-internal/moon-landing/pull/17593 to be merged first. EDIT: merged!

Uh oh!

bot-ci-comment Bot commented Mar 2, 2026

Uh oh!

Wauplin commented Mar 3, 2026

Uh oh!

Wauplin Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Wauplin Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hanouticelina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hanouticelina left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

julien-c Apr 10, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Wauplin commented Mar 2, 2026 •

edited

Loading