Skip to content

Releases

[v2.4.0] Copy files into Buckets with `HfFileSystem`, outdated CLI extension hints

Choose a tag to compare

@github-actions github-actions released this 29 Oct 14:56

🪣 Copy files into Buckets with HfFileSystem

HfFileSystem raised NotImplementedError on any copy involving a Bucket. Copying into a Bucket now works from the same Bucket, from another Bucket, or from a model, dataset or Space repo. Files stored with Xet are copied on the Hub by hash, so nothing is downloaded. Other repo files are downloaded and then uploaded to the Bucket. fsspec moves a file by copying it and deleting the source, so mv and rename now work inside a Bucket too.

>>> from huggingface_hub import hffs
>>> hffs.cp_file("buckets/my-username/my-bucket/data/train.csv", "buckets/my-username/my-bucket/backup/train.csv")
>>> hffs.mv("buckets/my-username/my-bucket/data/tmp.csv", "buckets/my-username/my-bucket/data/final.csv")
>>> hffs.copy("buckets/my-username/my-bucket/data", "buckets/my-username/other-bucket/data", recursive=True)
>>> hffs.cp_file("datasets/my-username/my-dataset/train.parquet", "buckets/my-username/my-bucket/train.parquet")

Copying from a Bucket to a repo is not supported yet.

🧩 See when a CLI extension is outdated

hf extensions list only showed when each extension was installed, so there was no way to know that an update was available. It now has a VERSION column (the commit the extension was installed from) and a LATEST column, plus a hint naming the outdated extensions. Running an outdated extension also prints a hint, at most once a day. To stay within GitHub's rate limits, hf looks up the latest commit of each extension at most once a day. Set HF_HUB_DISABLE_UPDATE_CHECK=1 to turn these checks off.

hf extensions list showing the VERSION and LATEST columns and a hint for outdated extensions

hf extensions update also repairs Python extensions that no longer start. After a Homebrew upgrade removed the Python an extension was installed with, running it failed with dyld: Library not loaded, and hf extensions update said it was already up to date. It now recreates the extension's virtual environment when its Python doesn't start, even if no newer commit exists.

$ hf extensions update speedtest
Creating virtualenv at: ~/.local/share/hf/extensions/hf-speedtest/venv
✓ Extension updated

🔧 Other QoL improvements

  • Add auth hint to HfFileSystem "repository not found" error by @davanstrien in #5258. A private repo accessed without a token gave the same (repository not found) error as a typo in the repo name. The error now also tells you to check that you are authenticated.

🐛 Bug and typo fixes

  • [Webhooks] Use "discussion" domain and default to all domains by @hanouticelina in #5263. hf webhooks create failed unless you passed --domain repo, and watching discussions was not possible because the accepted value was discussions instead of discussion. create_webhook now watches both domains when domains is omitted.
  • Validate filename against path traversal in try_to_load_from_cache by @bodapatisaikrishna in #5249. Filenames with ../ or absolute paths could resolve outside the cache folder. They are now rejected, as in hf_hub_download.
  • [HfFileSystem] Return file info from find(detail=True) on a file path by @ontdepart in #5246. It returned an empty dict, which broke datatrove readers pointing at a single hf:// file.
  • [Cache] Skip local-dir metadata when verifying checksums by @alphasafal in #5257. hf cache verify --local-dir reported the .cache/huggingface/ metadata as extra files, so --fail-on-extra-files failed on every clean download.
  • [CLI] Don't crash hf cache verify on Windows redirected stdout by @Sonica-B in #5255. When output was piped or redirected, hf cache verify crashed with UnicodeEncodeError instead of listing the files that failed verification.

📖 Documentation

  • [Docs] Fix docstring examples that fail when copy-pasted by @tkr109 in #5238
  • [Docs] Await endpoint batch inference before pausing by @emrehangorgec in #5247
  • Docs: build English only, other languages are machine-translated by @mishig25 in #5276. The docs in other languages are now machine-translated from the English docs, in 11 languages.

🏗️ Internal

  • Post-release: bump version to 2.4.0.dev0 by @huggingface-hub-bot[bot] in #5240
  • [Tests] Don't pin the avatar hash in OAuth test by @hanouticelina in #5241
  • [CI] Drop urllib3<2.0 test pin broken by vcrpy 9 by @hanouticelina in #5262
  • Bump the actions group with 7 updates by @dependabot[bot] in #5268

[v2.3.0] Bucket live-follow, faster `HfFileSystem` listings and more

Choose a tag to compare

@github-actions github-actions released this 22 Oct 12:38

🔄 Keep Bucket listings fresh with live_follow=True

HfFileSystem caches directory listings, so files added to a Bucket by another process did not show up until you called invalidate_cache(). With live_follow=True, HfFileSystem opens one background connection per listed Bucket to receive its file changes from the Hub. When a change arrives, it removes the cached listings it affects.

>>> from huggingface_hub import HfFileSystem
>>> hffs = HfFileSystem(live_follow=True)
>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # starts following that bucket
>>> # ... another process uploads a file to the bucket ...
>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # refreshed as changes arrive
['buckets/my-username/my-bucket/data/train.csv', 'buckets/my-username/my-bucket/data/new-file.csv']

It also works through fsspec with fsspec.filesystem("hf", live_follow=True), and through the storage_options argument of libraries such as pandas and pyarrow. The option is off by default because it keeps a connection open for each followed Bucket. It only affects listings and metadata (ls, info, exists, glob, ...), because reading a file always fetches its content from the Hub. It is not available for model, dataset and Space repos: use refresh=True to see a new commit.

  • [HfFileSystem] Add live-follow for Buckets by @lhoestq in #5096

🚀 HfFileSystem lists large repos in parallel

Recursive listings such as find and glob used to be slow on large repos. The Hub returns a repo's file tree 1000 entries at a time, and each page can only be requested after the previous one has arrived. Listing datasets/allenai/objaverse (~800k files) took ~800 requests in a row, about 13 minutes.

HfFileSystem now splits recursive listings by sub-directory and lists them in parallel, with almost the same number of requests. A sub-tree that fits in a single page is never split. maxdepth is now applied during the listing, so the Hub only returns the levels you ask for.

datasets/allenai/objaverse before after
find(), 798,924 files 801 requests, 776 s 807 requests, 19.7 s
find(maxdepth=2), 328 entries 801 requests, 776 s 6 requests, 1.5 s
>>> from huggingface_hub import hffs
>>> files = hffs.glob("datasets/allenai/objaverse/**/*.glb", maxdepth=3)

The new max_workers argument of HfFileSystem sets the number of concurrent requests. It defaults to 16. Set it to 1 to list sequentially like before.

  • [HfFileSystem] List large repos in parallel by @lhoestq in #5153

📎 Attach images and videos to discussions and PRs

The new HfApi.upload_attachment(path) uploads an image, video or audio file and returns its URL, which you can embed in a discussion, PR description or comment. Before, this required calling an undocumented Hub endpoint by hand.

>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> url = api.upload_attachment("screenshot.png")
>>> api.comment_discussion("username/repo-name", 5, f"Here is the fix:\n\n![screenshot]({url})")

On the CLI, hf discussions create (including --pr), comment and edit take a repeatable --attach FILE[#alt text] option. If the body already references the file as ![alt](./file), that reference is replaced with the uploaded URL. Other files are added at the end of the body, the same way the Hub web editor does it.

$ hf discussions create celinah/hfh-attach-test --title "Fix rendering" --pr \
    --body "Before: ![Before the fix](./before.png)" --attach ./before.png --attach "./after.png#After the fix"
num=1 url=https://huggingface.co/celinah/hfh-attach-test/discussions/1 ref=refs/pr/1

$ hf discussions info celinah/hfh-attach-test 1 --format json | jq -r '.events[0].content'
Before: ![Before the fix](https://cdn-uploads.huggingface.co/production/uploads/6192895f3b8aa351a46fadfd/QdFSiYU0Plm7GM3kGSipw.png)

![After the fix](https://cdn-uploads.huggingface.co/production/uploads/6192895f3b8aa351a46fadfd/efw80nbXnJzUL5-rJXLo0.png)

🧩 Read every safetensors shard header in one request

get_safetensors_metadata reads the header of each shard of a sharded model separately. The new get_sharded_safetensors_metadata gets all of them from the Hub in a single request and returns the same SafetensorsRepoMetadata. It only reads the main revision. get_safetensors_metadata is unchanged.

>>> from huggingface_hub import get_sharded_safetensors_metadata
>>> metadata = get_sharded_safetensors_metadata("HuggingFaceH4/zephyr-7b-beta")
>>> len(metadata.files_metadata)
8
  • [Safetensors] Add get_sharded_safetensors_metadata to get every shard header in one request by @mishig25 in #5159

🔒 Access requests on gated repos that hide requester identity

Owners of gated repos can now choose not to share the username and/or email of the people requesting access. On those repos, list_pending_access_requests (and the accepted and rejected variants) failed with a KeyError. Each AccessRequest now has a request_id, and accept_access_request, reject_access_request and cancel_access_request accept request_id= instead of user.

>>> from huggingface_hub import list_pending_access_requests, accept_access_request
>>> for request in list_pending_access_requests("my-org/gated-model"):
...     accept_access_request("my-org/gated-model", request_id=request.request_id)

Note

AccessRequest.username and AccessRequest.fullname are now typed str | None. They are only None on repos that hide usernames, where listing requests used to fail anyway, but type checkers may flag code that assumes a str.

  • [HfApi] Support gated repos hiding requester username/email by @Wauplin in #5047

🛡️ create_commit no longer deletes files you re-add in the same commit

create_commit skips adds and copies whose content is already on the Hub. When the same commit also deleted that path or its parent folder, only the delete was sent, and the file was removed from the repo without any error. This affected deleting a folder and pushing the same files back in one commit, and had been happening since v0.24. Those adds are now kept.

api.create_commit(repo_id, operations=[
    CommitOperationDelete("x/", is_folder=True),
    CommitOperationAdd("x/a.txt", b"keep me\n"),  # unchanged on the Hub
], commit_message="Reset x/")
# before: x/a.txt is deleted
# after:  x/a.txt is kept
  • [HfApi] Keep unchanged files re-added after a delete in create_commit by @hanouticelina in #5196

💼 hf jobs run keeps streaming logs and shows queue status

hf jobs run stopped printing logs when the log stream ended before the Job did, and crashed after a few dropped connections. It now reconnects until the Job reaches a final stage. Ctrl+C used to stop the logs while the CLI kept waiting silently for the Job to end. It now exits right away and tells you that the Job is still running and how to follow or cancel it.

hf jobs run before and after: logs keep streaming and Ctrl+C exits with a hint

While a Job waits for hardware, hf jobs run, hf jobs uv run and hf jobs rerun printed nothing, so a queued Job looked like a stuck CLI. They now show the Hub's status message and how long the Job has been waiting, in interactive terminals only:

hf jobs run showing the scheduling status while the Job waits for hardware

🐍 Python 3.15 support

huggingface_hub now officially supports Python 3.15 and is tested on it in CI. No code changes were needed. Python 3.10 is still the minimum version.

🤖 Inference

  • [Inference] Don't keep non-streaming responses alive in InferenceClient by @hanouticelina in #5197. A reused InferenceClient kept every non-streaming response in memory until close(). With 30 calls and a 1 MB prompt, that was 30 MB.
  • [Inference] Run provider get_response in a thread in AsyncInferenceClient by @hanouticelina in #5200. Queue-based providers (fal-ai, wavespeed, together video, zai-org) wait for results with blocking calls, which froze the event loop for the whole job.
  • [Inference] Raise InferenceTimeoutError on httpx timeouts by @hanouticelina in #5150. timeout now raises InferenceTimeoutError as documented, instead of the raw httpx timeout error.

🖥️ CLI

  • [Cache] Show unreferenced shared blobs in hf cache ls by @hanouticelina in #5148. Files left behind when an older huggingface_hub version deletes a repo now show up as a hint pointing to hf cache prune.
  • [CLI] Fix hf skills add when .claude/skills links to .agents/skills by @hanouticelina in #5204

🔧 Other QoL improvements

  • [Jobs] Export list_scheduled_jobs from the top-level package by @guptaparin in #5140
  • [Bot] Update hardware flavor enums and docs by @huggingface-hub-bot[bot] in #5175

🐛 Bug and typo fixes

  • fix(kernels): allow uploads and commits to kernel repos by @tengomucho in #5233
  • [HfApi] Respect custom endpoint in webhooks, access requests and user APIs by @hanouticelina in #5149. With HfApi(endpoint=...), 20 methods (webhooks, access requests, user and org info, followers, members) were sent to huggingface.co with the instance's token. Setups using HF_ENDPOINT were not affect...
Read more

[v2.2.0] `hf jobs stats` snapshots, download and `HfFileSystem` fixes

Choose a tag to compare

@github-actions github-actions released this 08 Oct 14:54

A small release this week, mostly bug fixes and quality-of-life improvements. hf jobs stats now returns right away instead of streaming forever, concurrent downloads reliably use hf_xet, and HfFileSystem caching is more predictable. Two behavior changes are listed under Breaking changes.

📊 hf jobs stats prints a snapshot by default

hf jobs stats used to stream metrics until the job finished, so scripts and agents calling it on a running job hung until their own timeout. It now prints the latest sample of each job once and exits, like hf jobs logs and hf spaces logs. Use -f/--follow for the previous live view. Snapshot output also supports --format json|agent|quiet, which outputs the raw API metrics.

On the Python side, HfApi.fetch_job_metrics gets the same follow parameter as fetch_job_logs. It defaults to False and returns only the current sample. Pass follow=True to stream until the job completes.

>>> hf jobs stats 6ac3bd9afbc85ba6823a9ad4
JOB ID                CPU % NUM CPU MEM % MEM USAGE      NET I/O         GPU UTIL % GPU MEM % GPU MEM USAGE
--------------------- ----- ------- ----- -------------- --------------- ---------- --------- -------------
6ac3bd9afbc85ba682... 0%    2.0     0.01% 1.0MB / 16.0GB 0.0bps / 0.0bps N/A        N/A       N/A
Hint: Use `hf jobs stats -f celinah/6ac3bd9afbc85ba6823a9ad4` to follow live stats.

# Stream live stats until the job completes
>>> hf jobs stats -f 6ac3bd9afbc85ba6823a9ad4
>>> from huggingface_hub import fetch_job_metrics
>>> for metrics in fetch_job_metrics(job_id=job.id, follow=True):
...     print(metrics)

Other Jobs fixes in this release: hf jobs logs and hf spaces logs without -f no longer hang while the job keeps printing, hf jobs stats no longer fails when one of the requested jobs has finished, and hf jobs stats --json prints [] when no job is running.

📚 Documentation: Jobs guide

⚡ Concurrent downloads always use hf_xet

In a fresh process, the first concurrent hf_hub_download calls could read a placeholder version for hf_xet while another thread was still looking it up. Those calls treated hf_xet as missing and downloaded through the plain HTTP path. This mostly affected parallel downloads such as snapshot_download, where some of the files skipped Xet. The version lookup is now cached only once it is resolved, so every download sees the real value.

  • Fix race in _get_version that makes concurrent downloads skip hf_xet by @BramVanroy in #5102

📂 HfFileSystem cache fixes

Three fixes to how HfFileSystem caches listings and missing paths:

  • ls() on a newly created empty bucket returns an empty list instead of raising FileNotFoundError.
  • Directory listings are now replaced in the cache instead of appended to. Before, ls() after find() or ls(refresh=True) could return duplicate entries, refresh=True kept deleted files, and listing a single bucket file cached it as the whole folder's listing.
  • exists(path, refresh=True) and invalidate_cache(path) now clear cached "not found" results. Before, a repo, branch or bucket created in the same process stayed missing until a full invalidate_cache().
>>> fs.exists(repo_id)
False
>>> api.create_repo(repo_id)
>>> fs.exists(repo_id, refresh=True)  # was False before this release
True

💔 Breaking changes

  • hf jobs stats and HfApi.fetch_job_metrics return a single snapshot by default instead of streaming until the job completes. Use -f/--follow or follow=True to keep the previous behavior. See the section above (#5090).

  • load_torch_model now raises FileNotFoundError instead of ValueError when the checkpoint path does not exist, and model_index_to_eval_results raises ValueError on an empty list (#5063).

  • [Bugfix] Fix wrong exception type and unbound variable guard by @vector15-05 in #5063

🖥️ CLI

  • [CLI] Don't glob-expand arguments on Windows by @hanouticelina in #5117. On Windows, quoted patterns such as --exclude ".venv/**" or --include "*.json" were expanded into local file names before reaching the command. hf download <repo> --include "*.json" in a folder containing a .json file silently downloaded nothing.
  • [CLI] Read --env-file / --secrets-file as UTF-8 and strip the BOM by @hanouticelina in #5066
  • [CLI] Upload existing paths with glob characters as-is in hf upload by @Ganesh2325 in #5059

🔧 Other QoL improvements

  • [Download] Warn when allow/ignore patterns match no files in snapshot_download by @Ad1th in #5067
  • [Cache] Speed up delete_revisions with a set lookup by @hanouticelina in #5118. Planning hf cache prune / hf cache rm no longer scales quadratically with the number of files per revision.
  • [Setup] Install hf-xet on Windows ARM64 by @angt in #5074

🐛 Bug and typo fixes

📖 Documentation

  • [Docs] Fix HTTP migration and async session references by @Yuilona in #5079
  • [Docs] Fix from_pretrained integration example by @Ayushdevo in #5105

🏗️ Internal

  • Post-release: bump version to 2.2.0.dev0 by @huggingface-hub-bot[bot] in #5054
  • [CI] Split tests into api / transfer / inference suites by @Wauplin in #5042
  • [Tests] Always clean up staging repos and buckets created by tests by @Wauplin in #5057
  • [CI] Reduce flaky tests (jedi cache, read-after-write, fewer reruns) by @Wauplin in #5058
  • Bump doc-builder workflow pin by @paulinebm in #5087
  • Bump doc-builder main docs workflow pin by @paulinebm in #5093
  • [Tests] Wait for bucket listings after writes by @hanouticelina in #5095

v2.1.1

Choose a tag to compare

@github-actions github-actions released this 01 Oct 12:46

[CLI] Rename 'hf spaces zero-gpu-quota' to 'hf spaces zero-gpu quota'- #5055

Full Changelog: v2.1.0...v2.1.1

[v2.1.0] Better Jobs tooling, ZeroGPU quota tracking, and revamped Model Catalog

Choose a tag to compare

@github-actions github-actions released this 01 Oct 12:12

📊 Jobs: retries, rescheduling, and port exposure

Hugging Face Jobs get a batch of quality-of-life improvements this release. Jobs can now retry automatically on failure with the new attempts parameter, and you can start a fresh Job from an existing Job's saved spec (including secrets and hardware settings) with rerun_job / hf jobs rerun. Scheduled Jobs can be rescheduled — change the cron expression or preset without recreating the Job — via update_scheduled_job_schedule / hf jobs scheduled reschedule. Finally, you can now expose ports on Jobs: use expose for token-protected access and expose_public for unauthenticated access, both at creation time and on a running Job with update_job_expose / hf jobs expose.

# Retry a job up to 2 times on failure, and rerun it later from its saved spec
>>> hf jobs run --attempts 3 --detach python:3.12 python train.py
>>> hf jobs rerun <job_id>

# Reschedule a scheduled job to run every day at 9:00
>>> hf jobs scheduled reschedule <scheduled_job_id> "0 9 * * 1"

# Expose ports on a running Job (8000 token-protected, 9000 public)
>>> hf jobs expose <job_id> 8000 --public 9000

📚 Documentation: Jobs guide

🤗 Track your ZeroGPU quota

The Hub now exposes a ZeroGPU quota endpoint, and huggingface_hub wraps it with a new HfApi.get_zero_gpu_quota() method returning a ZeroGpuQuota dataclass (values in GPU-seconds). This is particularly useful for apps, MCP servers and agents built on top of ZeroGPU Spaces: check how much quota is left and when it resets, and warn your users before they hit the daily limit. The same information is available from the CLI with hf spaces zero-gpu-quota, which also prints a hint to purchase credits when the quota is running low.

>>> from huggingface_hub import get_zero_gpu_quota
>>> quota = get_zero_gpu_quota()
ZeroGpuQuota(base=2400, remaining=1810, resets_at=datetime.datetime(2026, 9, 30, 9, 12, 3, tzinfo=datetime.timezone.utc), overquota_used=0)
>>> hf spaces zero-gpu quota
✓ ZeroGPU quota (in GPU-seconds)
  remaining: 1810
  base: 2400
  resets_at: 2026-09-30T18:12:03+00:00
  overquota_used: 0
  • [Spaces] Add get_zero_gpu_quota and hf spaces zero-gpu-quota by @Wauplin in #5040
  • [CLI] Rename 'hf spaces zero-gpu-quota' to 'hf spaces zero-gpu quota'- #5055 [Shipped as part of v2.1.1]

📚 Documentation: Manage your Space guide

⚡ HfFileSystem.get_file downloads Xet files with hf_xet

HfFileSystem.get_file now downloads Xet-backed files through hf_xet (the same code path as hf_hub_download) when writing to a local path, letting hf_xet write directly to disk instead of going through Python. It reuses the Xet hash already returned by info() (part of the tree listing), so no extra HTTP call is needed. This mostly benefits non-streaming datasets loads from hf://buckets/... paths: benchmarks on HF Jobs show get_file dropping from ~120s to ~7s on a 2 GB parquet file, and load_dataset("hf://buckets/...") from ~130s to ~17s. The classic HTTP path is still used for non-Xet files, when hf_xet is not installed, or when writing to a file-like object. As a side effect, a missing remote file no longer leaves an empty local file behind.

🧠 Inference Endpoints: Model Catalog migrates to the new v1 API

The catalog methods now use the new /api/v1/catalog API. list_inference_catalog returns InferenceCatalogModel dataclasses with their tested deployment recipes (hardware + engine combinations), and takes server-side filters (accelerator, engine, license, task, search, limit). create_inference_endpoint_from_catalog can deploy an exact recipe with the new recipe_id and gguf_file arguments. The CLI follows suit: hf endpoints catalog ls prints one row per recipe and supports the filters, while hf endpoints catalog deploy gains --recipe and --gguf-file.

>>> hf endpoints catalog ls --engine vllm --task text-generation --search llama --limit 10
REPO_ID                             TASK               LICENSE    ACCELERATOR ENGINE   GGUF_FILE                     RECIPE_ID
----------------------------------- ------------------ ---------- ----------- -------- ----------------------------- -------------------------
meta-llama/Llama-3.1-8B-Instruct    text-generation    Llama 3.1  gpu         vllm                                   sizzling-biryani-g4xsi1ac
bartowski/QwQ-32B-Preview-GGUF      text-generation    Apache 2.0 gpu         llamacpp QwQ-32B-Preview-Q8_0.gguf     baked-orange-m863gx7d
  • [Endpoints] Migrate catalog to the new /api/v1 API by @Wauplin in #4928

📚 Documentation: Inference Endpoints guide

🔒 Sandboxes require scoped credentials

Sandbox security is tightened in this release. Pooled sandboxes with missing, empty or malformed capability tokens are now refused instead of silently substituting the host-management credential, and the client is pinned to sbx-server 0.7.0, which removes host-token acceptance on per-sandbox routes entirely. Older pool hosts that do not provide scoped tokens must be upgraded/recycled: new clients cannot reach legacy hosts and legacy clients cannot reach new sandboxes (documented, accepted breaking change). The sandbox concepts and guides have been updated to describe the strict behavior, including how to safely use proxy_headers with external HTTP clients (disable redirects!).

  • [Sandbox] Require scoped credentials for pooled sandboxes by @Wauplin in #4782
  • [Sandbox] Bump sbx-server to 0.7.0 by @Wauplin in #5028

💔 Breaking Change

  • [Buckets] Make include/exclude patterns case-sensitive on all platforms by @Wauplin in #5007

Note

The Inference Endpoints catalog migration (#4928) and the Sandbox credential changes (#4782, #5028) are breaking changes detailed in their highlight sections above.

🖥️ CLI

  • [CLI] Escape control characters in listings and table output by @Wauplin in #5029
  • [CLI] Suggest hf auth login when hf auth whoami fails by @moon-bot-app[bot] in #5033
  • [CLI] Align hf auth whoami not-logged-in message with hf auth token by @Wauplin in #5034
  • [CLI] Raise if a bare --secrets NAME is not set locally in hf jobs by @Wauplin in #5044
  • [CLI] Hint to use --global after local 'hf skills add/update' by @Wauplin in #5024
  • [CLI] nit: update hf upload help text by @davanstrien in #5046

🔧 Other QoL Improvements

  • [HfApi] Support resource_group_id in duplicate_repo by @Wauplin in #5038
  • [Repos] Warn when duplicated repo files are still being copied by @Wauplin in #5041
  • Support secrets on Job-triggered webhooks by @moon-bot-app[bot] in #5009
  • [HfApi] Add visibility to create_bucket by @hanouticelina in #5053

🐛 Bug and typo fixes

  • [Buckets] Fix copy_files treating lexical siblings as an existing destination directory by @AnishPatel526 in #5001
  • [HTTP] Fix repo_id in errors for query strings and download URLs by @Wauplin in #5008
  • Fix scalar tags splitting into single-character list in card front matter by @zeenat28-ui in #5017
  • [datasets sql] Run the DuckDB CLI subprocess with UTF-8 encoding by @825pranav in #5020
  • [Auth] Store token names as UTF-8 instead of locale encoding by @Evolian-o in #5021
  • [Dataclasses] Accept bare typing.List/Dict/Set in @strict validation by @anishmehta24 in #5032
  • [Utils] Retry the first pagination page as well by @Wauplin in #5035

📖 Documentation

  • [docs] xet caching and concurrency updates by @cbensimon in #4883
  • [Spaces] Update docs for cpu-basic gating and dev mode by @Wauplin in #5037
  • [docs] Fix hf.cos typo in repository guide link by @pratikgx in #5016
  • [Docs] Fix links in i18n issue template by @pratikgx in #5051
  • [Docs] Fix duplicated words in docstring and comments by @Ad1th in #5052

🏗️ Internal

  • Post-release: bump version to 2.1.0.dev0 by @huggingface-hub-bot[bot] in #4999
  • [CI] Bump doc-builder workflows pin to 63ec6f1 by @Wauplin in #5010
  • [Tests] Reduce redundant Hub calls in repo settings and move tests by @Wauplin in #5036

[v2.0.0] The httpx2 release

Choose a tag to compare

@github-actions github-actions released this 24 Sep 11:35

See MIGRATION GUIDE in docs.

💔 Breaking Change: HTTP stack moves to httpx2

See migration plan in #4802 .

huggingface_hub v2.0 replaces its HTTP dependency with httpx2: clients and transport exceptions now come from httpx2/httpcore2 instead of httpx/httpcore. Custom client factories (via set_client_factory / set_async_client_factory) and any except clauses catching transport errors must use httpx2 types. For code that must support both v1.x (starting with v1.30.0) and v2.x, import the HTTP module from huggingface_hub.utils:

from huggingface_hub.utils import get_session, httpx

try:
    response = get_session().get("https://huggingface.co/api/models/gpt2")
    response.raise_for_status()
except httpx.HTTPError:
    ...

Note that httpx2 uses the operating system's certificate trust store by default (custom CA bundles via SSL_CERT_FILE / SSL_CERT_DIR still work), logging configuration should target the httpx2 and httpcore2 loggers, and the oauth extra now requires authlib>=1.8.0.

💔 Breaking Change: deprecated APIs are removed

This release also removes all APIs deprecated in 1.x:

  • upload_large_folder / HfApi.upload_large_folder → use upload_folder
  • duplicate_space → use duplicate_repo
  • request_space_storage / delete_space_storage → use the volume APIs (set_space_volumes, delete_space_volumes)
  • repo_type_and_id_from_hf_id → use parse_hf_uri
  • InferenceEndpointType.PROTECTED → use AUTHENTICATED (create_inference_endpoint(type="protected") now raises ValueError)
  • list_models(model_name=...) → use list_models(search=...)
  • create_repo / duplicate_repo(space_storage=...) → use space_volumes
  • text_generation(stop_sequences=...) → use stop
  • Boolean InferenceClient(token=...) → pass a token string or None

On the CLI side, the following entry points and options are gone: huggingface-cli (use hf), hf repo and hf repo-files delete (use hf repos and hf repos delete-files), hf upload-large-folder (use hf upload), hf repos create/duplicate --storage (use --volume), hf jobs ps and hf jobs scheduled ps -f/--filter (use --status/--label), and hf skills add/update --claude (Claude Code installation is now automatic).

A migration guide covering all of this is available at https://github.com/huggingface/huggingface_hub/blob/main/docs/source/en/concepts/migration_v2.md.

🏗️ Internal

  • [CI] Support major (pre-)releases in release workflow by @Wauplin in #4991
  • Post-release: bump version to 1.34.0.dev0 by @huggingface-hub-bot[bot] in #4993

[v1.33.0] Better UX on large uploads and simpler skills installs

Choose a tag to compare

@github-actions github-actions released this 24 Sep 09:49

📤 See progress while large Xet uploads validate

Very large Xet uploads used to sit silently between "Uploading" and "Committing" while shards were being finalized, which could take a long time with no feedback. The upload live display now shows a dedicated Validating bar — reported as a percentage instead of opaque internal validation-entry counts — both in the upload_folder display and in XetUploadProgressReporter (used by upload_file, create_commit and bucket uploads). This requires hf-xet>=1.6.0.

  • [Xet] Show Validating progress during shard finalization by @seanses in #4478

🧩 Skills are now installed for every agent by default

hf skills add no longer requires a special flag for Claude Code. The skill is now always installed to .agents/skills (or ~/.agents/skills with -g) and symlinked into .claude/skills (honoring CLAUDE_CONFIG_DIR when set), so a single command covers Claude Code, Codex, Cursor, OpenCode, Pi and any other agent that loads skills from .agents/skills. The --claude flag is deprecated: it still works as a no-op but prints a warning, and --dest keeps installing only to the directory you provide. hf skills update refreshes both roots the same way. Along the way, CLAUDE_CONFIG_DIR is now properly honored everywhere (it was previously ignored, which left skills invisible to relocated Claude Code installs).

# works with Claude Code, Codex, Cursor, OpenCode, Pi and any agent that loads skills from `.agents/skills`
hf skills add

📚 Documentation: CLI guide

💔 Breaking Change

  • Send PrivateLink config under privateService in create_inference_endpoint by @co42 in #4966
    • AWS PrivateLink settings are now sent as privateService.accountId and privateService.region. Use the new private_link_account_id and private_link_region parameters (the region is required when the account ID is set, and is independent of the compute region). The legacy account_id keyword still exists but is ignored and emits a FutureWarning; it is no longer included as a top-level payload field.

🖥️ CLI

  • [CLI] Honor --format / --json / -q on jobs run commands by @Wauplin in #4936 — docs
    • These flags previously leaked into the container command on hf jobs run, hf jobs uv run and their scheduled variants (--format became the script path, -q became the image name). They are now consumed wherever they appear; use -- when your script needs them.
  • [CLI] Allow bucket as a watched item type in hf webhooks create --watch by @davanstrien in #4958 — docs

📖 Documentation

  • [Docs] Sync provider task table with the PROVIDERS registry by @rishabhsai in #4930 — docs
  • Remove dots from allowed characters in job labels (docstring update) by @evalstate in #4962

🐛 Bug and typo fixes

  • [Download] Publish symlink support only after probing by @Wauplin in #4923
  • [Core] Skip invalid eval results entries instead of raising by @davanstrien in #4924
  • [Cache] Use posix paths for scan_cache_dir ref names on Windows by @gokay-ai in #4925
  • [Download] Measure http_get progress from the initial file position by @Wauplin in #4937
  • Fix format_timesince showing "60 seconds ago" / "60 minutes ago" / "24 hours ago" instead of rolling over by @Sreekant13 in #4943
  • [HfFileSystem] Make lpath optional in get_file (fsspec>=2026.9.0) by @assafvayner in #4951
  • [Buckets] Raise BucketBatchError when a batch reports failed operations by @hanouticelina in #4954 — docs
  • Align the hf package's python_requires with huggingface_hub by @sharziki in #4959
  • [MCP] Follow nextCursor when listing tools in MCPClient by @Kayvan-Zahiri in #4960
  • [Download] Fix Windows MAX_PATH error on tmp .incomplete file by @Wauplin in #4978
  • [HfFileSystem] Fix get_file with a bare local filename by @davanstrien in #4981
  • Apply the repo_id length limit to the namespace as well by @anishmehta24 in #4988

🏗️ Internal

  • Post-release: bump version to 1.33.0.dev0 by @huggingface-hub-bot[bot] in #4926
  • [Serialization] Fix device type when map_location is None by @Wauplin in #4938
  • [Internal] Soft deprecate re.match by @Wauplin in #4939
  • Fix Windows conftest erroring test subsets on lazy-loaded file_download submodule by @dltsum in #4945
  • Scope GITHUB_TOKEN permissions per job by @hf-security-analysis[bot] in #4982

[v1.32.0] Shared blob store, sandbox security hardening and faster imports

Choose a tag to compare

@github-actions github-actions released this 17 Sep 10:21

📂 [Cache] Shared blob store: deduplicate Xet files across repos

The cache now deduplicates Xet files across repos. A Xet file downloaded through hf_xet is stored once at <CACHE_DIR>/blobs/<prefix>/<xet_hash> and every repo that needs it gets a relative symlink instead of a download: no bytes are transferred and no extra space is used, even across different repos — or after the repo that first downloaded the file was deleted. The per-repo snapshot layout is unchanged, older clients keep reading and downloading normally, and any failure to share silently falls back to regular repo-local storage. Set HF_HUB_DISABLE_SHARED_BLOBS=1 to opt out entirely. Shared files carry a <xet_hash>.refs manifest listing the repo blobs referencing them, which hf cache rm consults on deletion and hf cache prune sweeps to reclaim payloads that no cached repo uses anymore.

📚 Documentation: Manage your cache

💻 [Jobs] Ship config inside UV scripts

A UV script that only runs correctly on a specific runtime can now carry that runtime with it. An optional [tool.hf-jobs] table in the script's PEP 723 header accepts image, flavor, python, timeout, name, namespace, env, secrets, labels, volumes, network_group and network_aliases, and hf jobs uv run reads it at submit time. CLI flags always win, and env/secrets/labels/volumes merge entry by entry instead of being replaced, so -e/-v add to what the script declares. Typos and unknown keys are rejected with the list of valid options, secrets are passed by name only (values come from your environment), and every run prints a config summary with script-sourced values marked and secrets redacted. Note that the table is read by the CLI only: run_uv_job() and create_scheduled_uv_job() ignore it.

# /// script
# requires-python = ">=3.11"
# dependencies = ["vllm", "datasets"]
#
# [tool.hf-jobs]
# image   = "vllm/vllm-openai:unlimited-ocr"
# flavor  = "l4x1"
# python  = "/usr/bin/python3"
# secrets = ["HF_TOKEN"]
# ///

📚 Documentation: Run and manage Jobs

  • [CLI] Read a UV script's [tool.hf-jobs] launch config (opus-generated) by @Wauplin in #4598

🛡️ [Sandbox] security hardening

An internal security review of the Sandbox API resulted in a 9-PR hardening series. Pooled sandboxes now use their own per-sandbox capability token instead of the host-wide credential, pool hosts are fully validated (initiator, namespace, image, flavor, command, URL) before any credential is sent to them, the sbx-server binary is pinned by digest and verified before being run as PID 1, and the local pool cache is bound to the endpoint, credential and namespace that wrote it. Secret values no longer end up in argv when using hf sandbox exec --secrets, background processes are addressed by their server-assigned id (so kill() actually stops them and reports honestly), transfers and command output are bounded to avoid unbounded memory usage in the client, and pool ownership is decided per host so a with SandboxPool(...) block never again tears down a colleague's discovered host. The security documentation was also rewritten to state precisely what the sandbox contract is — and what it is not — including a new "Known limitations" section.

📚 Documentation: Sandboxes

  • [sandbox audit] Make the sandbox security contract match the implementation by @Wauplin in #4831
  • [sandbox audit] Use each pooled sandbox's own capability token by @Wauplin in #4832
  • [sandbox audit] Validate a pool host before sending it a credential by @Wauplin in #4834
  • [sandbox audit] Bind the sandbox pool cache to the endpoint, credential and namespace that wrote it by @Wauplin in #4838
  • [sandbox audit] Bound what a transfer or a command's output costs the client by @Wauplin in #4839
  • [sandbox audit] Decide host teardown per host, and report it honestly by @Wauplin in #4840
  • [sandbox audit] Address background processes by their server-assigned id by @Wauplin in #4836
  • [sandbox audit] Pin the sandbox server binary by digest and verify it before running it by @Wauplin in #4837
  • [sandbox audit] Keep secret values out of argv in the CLI and fix the env/secrets docs by @Wauplin in #4835

🔒 Security hardening for serialization and validation

Three changes make the library safer against malicious or malformed inputs, with a stricter behavior worth noting. Torch checkpoint deserialization was hardened across 11 reported vulnerabilities: load_state_dict_from_file now defaults to safe=True (always using the safetensors loader), the pickle path defaults to weights_only=True, and unsafe combinations raise explicit errors instead of silently falling through. Path validation now rejects .. segments anywhere in path_in_repo (previously only a leading one was caught), so uploads like "a/../../etc/passwd" are refused. Finally, repo_id validation is restricted to ASCII word characters as documented, so non-ASCII ids like café are rejected client-side instead of failing later on the Hub.

🧰 Smarter cache management

hf cache rm accepts hf:// file URIs, so you can remove a single cached file — typically one GGUF quantization — without deleting the whole repo. The file is removed from every cached revision, and its blob is deleted only if no other cached file still references it. hf cache commands also support kernel repos: scan_cache_dir() used to treat kernels--* folders as corrupted and skip them, but kernel repos now show up in hf cache ls, rm and prune, and --repo-type kernel works for hf download and hf cache verify.

hf cache rm hf://models/unsloth/gemma-3-27b-it-GGUF/gemma-3-27b-it-Q4_K_M.gguf --dry-run
hf download kernels-community/activation README.md --repo-type kernel

📚 Documentation: Manage your cache

🚀 Faster imports

huggingface_hub now imports much faster: huggingface_hub.utils is lazy like the root package, heavy HfApi implementation imports and optional Torch/TensorBoard/NumPy/FastAPI/Starlette dependencies are deferred until used, and CLI commands are registered lazily while hf --help still renders the same complete output. Benchmark medians on Python 3.10: from huggingface_hub import * drops from 2.36s to 0.39s (626.8 → 49.6 MiB peak RSS), hf version from 0.37s to 0.17s, and hf env from 0.40s to 0.24s.

🖥️ CLI

  • [CLI] Make --format human override agent auto-detection by @hanouticelina in #4878
  • [CLI] Add hf --skills as an alias for hf skills preview by @Wauplin in #4906

🤖 Inference

📊 Jobs

  • [Jobs] Scope network groups to the namespace and resource group by @co42 in #4855
  • [Jobs] Expose the network group on JobInfo by @co42 in #4862

🔧 Other QoL Improvements

  • [Download] Send X-HF-Download-Counter header on download calls by @Wauplin in #4613
  • [Buckets] Send mtime when copying files by @Wauplin in #4920

🐛 Bug and typo fixes

  • fix(serialization): save_torch_state_dict cleanup regex silently matches/deletes unrelated files by @g0rdonL in #4857
  • [Sandbox] Preserve terminal command results by @evalstate in #4851
  • [Http] Fix race in get_session() creating multiple clients by @hanouticelina in #4868
  • [RepoCard] Fix AttributeError when model-index has source: null by @Saniyagupte in #4870
  • [Spaces] Fix AttributeError in SpaceRuntime when hardware is null by @Saniyagupte in #4873
  • [Buckets] Fix file metadata size on redirects by @davanstrien in #4874
  • [CLI] Fix card data serialization by @davanstrien in #4882
  • Fix doubled colon in snapshot download transfer bar description by @sandeepb2013 in #4890
  • [Download] Fix silently disabled tree cache on long Windows paths by @askalf in #4896
  • [Safetensors] Recognize proposed U3 metadata by @7dsolv in #4898
  • [Serialization] Sync _get_dtype_size with safetensors by @Wauplin in #4902

📖 Documentation

  • [Community] Fix docstring for DiscussionComment.last_edited_by by @Saniyagupte in #4864
  • [Docs] Fix hf discussions info options that do not exist by @rishabhsai in #4886

🏗️ Internal

  • [CI] Make CI less flaky and stop 6h hung jobs by @Wauplin in #4830
  • Bump the actions group with 4 updates by @dependabot[bot] in #4848
  • Post-release: bump version to 1.32.0.dev0 by @huggingface-hub-bot[bot] in #4856
  • Key the doc build concurrency group on the pull request number by @albertvillanova in #4876

[v1.31.0] Custom labels for Sandboxes, More resilient downloads and more

Choose a tag to compare

@github-actions github-actions released this 10 Sep 09:36

🏷️ Custom labels for Sandboxes

Dedicated sandboxes now accept custom labels, attached to the underlying Job. This is useful for cost attribution, bookkeeping, or finding and reconnecting to the sandboxes created by a given controller run. Labels use the same -l / --label KEY=VALUE syntax as hf jobs run. They are merged with the labels the SDK uses internally, and invalid or reserved labels are rejected before a billable Job is started. Pool-based sandboxes are unchanged: custom labels are only accepted for dedicated sandboxes.

>>> from huggingface_hub import Sandbox
>>> sandbox = Sandbox.create(image="python:3.12", labels={"controller-run": "run-42"})
hf sandbox create --label controller-run=run-42 --label team=data-infra

📥 More resilient downloads

A batch of fixes makes downloads more robust to unusual server responses, network hiccups and concurrent usage:

  • A timeout while waiting for the response headers of a streamed download is now retried and resumed like a body read failure, instead of escaping http_get() while retries remain.

  • Regular HTTP downloads no longer fail when the HEAD response has no Content-Length. The file size is validated against the GET response when available.

  • snapshot_download now writes the refs/ cache file atomically, reusing the logic already used by hf_hub_download. This fixes a long-standing race when many concurrent snapshot_download calls target the same repo (seen in vLLM / llm-compressor).

  • hf download --dry-run --local-dir ... no longer copies files from the Hub cache into the destination. On large files and slow disks this looked like a hang and could leave an incomplete file behind.

  • [Download] Share retry handling for stream entry and body failures by @Wauplin in #4826

  • [Download] Tolerate missing HEAD Content-Length by @Wauplin in #4805

  • [Download] Write cache ref file atomically in snapshot_download by @Wauplin in #4829

  • [Download] Prevent cached file copies during local-dir dry runs by @wakamex in #4817

🔌 httpx re-exported for library integrators

huggingface_hub now re-exports the HTTP library it uses as huggingface_hub.utils.httpx. Libraries built on top of huggingface_hub that need httpx types or exceptions (typically to catch errors) should import it from there rather than importing httpx directly. v1.x is built on httpx, and v2.x will move to its successor httpx2, so importing through huggingface_hub.utils keeps your code compatible with both. This is only for types and exceptions: to make requests to the Hub, keep using get_session().

from huggingface_hub.utils import httpx

try:
    ...
except httpx.HTTPError:
    ...

📚 Documentation: Utilities — The httpx module

  • [httpx migration] Expose httpx as a huggingface_hub submodule by @Wauplin in #4803

🔒 Security

HfFileSystem.get() now validates remote filenames before writing anything locally. A server-side filename such as folder/..\..\outside.txt could previously escape the destination directory on Windows during a recursive download. The same check already protected hf_hub_download, snapshot_download and bucket sync. Unsafe filenames now raise ValueError on all platforms, including when downloading to an explicitly named file or a file object.

  • [HfFileSystem] Reject unsafe paths before local downloads by @Wauplin in #4828

🖥️ CLI

  • [CLI] Fix hf buckets rm --recursive deleting lexical siblings of the prefix: removing logs/ no longer touches logs_root.txt by @Wauplin in #4804
  • [CLI] Don't let hf update self-upgrade a pip install on Windows: it now prints the pip install -U command to run instead of leaving a broken install by @Wauplin in #4823
  • [CLI] Raise explicit error for shell-script extensions on Windows by @hanouticelina in #4846
  • [CLI] Fix truncated command descriptions in the CLI reference by @hanouticelina in #4849

🐛 Bug and typo fixes

  • [hf_api] Make CommitInfo copy-able and pickle-able by @Wauplin in #4822
  • [LFS] Fix SliceFileObj.__iter__ yielding only the first 4MB chunk by @BenYang12 in #4844
  • Fix dotenv parser truncating unquoted values containing # (used by hf jobs --env-file) by @BenYang12 in #4842
  • Keep single-quoted values verbatim in load_dotenv by @AnishPatel526 in #4827

📖 Documentation

  • [Docs] Clarify Jobs argument forwarding with -- and align examples by @davanstrien in #4809

🏗️ Internal

  • Bump the actions group across 1 directory with 5 updates by @dependabot[bot] in #4798
  • Post-release: bump version to 1.31.0.dev0 by @huggingface-hub-bot[bot] in #4806

[v1.30.0] Scheduled job filters, repo-aware revision pinning and more

Choose a tag to compare

@github-actions github-actions released this 03 Sep 09:42

🖥️ hf jobs scheduled ls can now filter by status, label and name

You can now filter scheduled jobs with --status (active/suspended), repeatable -l/--label key=value and --name (a shortcut for --label name=NAME), matching the flags hf jobs ls already had. On the Python side, HfApi.list_scheduled_jobs gains a labels argument.

Warning

-f/--filter on hf jobs scheduled ls is deprecated and no longer applied: it is accepted but ignored, with a warning pointing to the new flags. Migrate as shown below. The option itself will be removed in a future release.

hf jobs scheduled ls --all
hf jobs scheduled ls --status suspended
hf jobs scheduled ls --name hourly-task --label env=prod
before after
-f name=daily-report --name daily-report
-f env=prod --label env=prod
-f suspend!=true (default, or --status active)
-f suspend=true --status suspended

📚 Documentation: Jobs guide

  • Align hf jobs scheduled ls filtering with hf jobs ls (--status/--label/--name) by @moon-bot-app[bot] in #4773

🔖 ResolvedRevision now pins the repo it was resolved for

ResolvedRevision didn't record which repo a commit hash was resolved against, so resolving "main" on two different repos returned the same commit oid. It now remembers its repo_id and repo_type: passed back for the same repo it is returned as is, for a different repo the requested revision is resolved again.

>>> from huggingface_hub import resolve_revision
>>> revision = resolve_revision("openai-community/gpt2")

# Same repo => returned as is
>>> resolve_revision("openai-community/gpt2", revision=revision) is revision
True

# Another repo => "main" is resolved again for that repo
>>> resolve_revision("openai-community/gpt2-medium", revision=revision).resolved
'6dcaa7a952f72f9298047fd5137cd6e4f05f41da'

📚 Documentation: Manage cache — Pin a revision

  • [Download] A ResolvedRevision only pins the repo it was resolved for by @Wauplin in #4767

🤖 Inference

  • Do not require the conversational tag for chat-completion on hf-inference: any model with pipeline tag text-generation or image-text-to-text can now be served, mirroring the updated provider-side rules by @hanouticelina in #4784

🐛 Bug and typo fixes

  • [Download] Follow redirects between Hub hosts when resolving files, instead of failing with a misleading connection error (redirects to CDNs are still not followed, so LFS metadata is preserved and Authorization is never forwarded off-Hub) by @hanouticelina in #4739

📖 Documentation

  • [Sandbox] Mark the Sandbox API as experimental across docstrings, guides, package reference and CLI help, and clarify that shared sandboxes are intended for workloads within the same trust boundary by @Wauplin in #4783

🏗️ Internal

  • [Release] Refresh OpenCode model cache (opencode models --refresh) before validating RELEASE_NOTES_MODEL, fixing release CI failures on cold runners by @Wauplin in #4761
  • [Release] Consolidate the HF CLI skill sync into the release workflow, removing the duplicate standalone sync workflow that opened two PRs on huggingface/skills per release by @hanouticelina in #4765
  • [CI] The close-unscoped-community-prs workflow stays available for manual runs only: enforcement was enabled and then disabled again within this release by @hanouticelina and @Wauplin in #4766, #4800
  • [CI] Fix torch_1.11 case-arm label to match the renamed matrix entry, restoring torch installation on that lane by @MohammedAlkindi in #4776
  • Post-release: bump version to 1.30.0.dev0 by @huggingface-hub-bot[bot] in #4763