Centralize hf:// URI parsing by Wauplin · Pull Request #4158 · huggingface/huggingface_hub

Wauplin · 2026-04-28T08:44:02Z

Summary

Adds a single source of truth for parsing hf://... URIs, addressing #3971.

New HfUri dataclass and parse_hf_uri helper, plus shared constants in huggingface_hub.constants (HF_PROTOCOL, HfUriType, HF_URI_TYPE_PREFIXES).
Separate HfMount dataclass and parse_hf_mount helper for volume mount specifications (hf://...:<MOUNT_PATH>[:ro|:rw]). Mount logic is cleanly separated from URI parsing — HfUri is a pure location identifier, HfMount wraps a HfUri with a mount path and read-only flag.
A reference page documenting the canonical syntax for both URIs and mounts, and what does (and does not) qualify as a HF URI — strict-by-design (plural-only type prefixes, no canonical repos without namespace, etc.).
Pure string parser, no network calls. Round-trippable via HfUri.to_uri() and HfMount.to_uri().

>>> from huggingface_hub import parse_hf_uri
>>> parse_hf_uri("hf://datasets/namespace/my-dataset@refs/pr/3/train.json")
HfUri(type='dataset', id='namespace/my-dataset', revision='refs/pr/3', path_in_repo='train.json')

>>> from huggingface_hub import parse_hf_mount
>>> parse_hf_mount("hf://buckets/my-org/my-bucket/sub/dir:/mnt:ro")
HfMount(source=HfUri(type='bucket', id='my-org/my-bucket', revision=None, path_in_repo='sub/dir'), mount_path='/mnt', read_only=True)

Follow-ups (not in this PR)

Migrate existing call sites to the new parser:
- HfFileSystem.resolve_path (keeps its 1- vs 2-segment network fallback on top of the parser)
- _parse_hf_copy_handle in hf_api.py (the spot already carries a # TODO referencing Harmonize hf:// parsing logic #3971)
- parse_volumes in cli/_cli_utils.py
- _split_bucket_id_and_prefix / _parse_bucket_path / _is_bucket_path in _buckets.py and _parse_bucket_argument / _is_hf_handle in cli/buckets.py
- Volume.to_hf_handle formatter in _space_api.py (mirror of HfMount.to_uri)
Decide what to do with repo_type_and_id_from_hf_id (used by RepoUrl): it's a mixed parser accepting both hf:// URIs and https://huggingface.co/... URLs. Likely split into two helpers, with the URI half delegating to parse_hf_uri.
Possibly extend the parser later to accept HTTPS Hub URLs, if the RepoUrl migration would benefit.

Note

Medium Risk
Adds new public parsing APIs and stricter URI semantics (e.g., requiring namespace/name), which could affect downstream users who adopt the new helpers or rely on older, looser examples.

Overview
Introduces a new centralized hf://... parser in utils/_hf_uris.py, including frozen HfUri/HfMount dataclasses with canonical round-tripping (to_uri) and strict validation/error reporting via a new HfUriError.

Exports HfUri and parse_hf_uri from the package (huggingface_hub and huggingface_hub.utils), adds shared URI constants/types in constants.py, and adds comprehensive unit tests plus a new docs reference page (package_reference/hf_uris) linked from the docs toctree. CLI volume help text/examples are updated to reflect namespaced model URIs (no more hf://gpt2).

^{Reviewed by Cursor Bugbot for commit 116f6ea. Bugbot is set up for automated code reviews on this repo. Configure here.}

Introduces a single ``parse_hf_uri`` helper and ``HfUri`` dataclass under ``huggingface_hub.utils``, plus a reference page documenting the canonical ``hf://`` syntax and what does (and does not) qualify as a HF URI. Related constants (``HF_PROTOCOL``, ``HfUriType``, ``HF_URI_TYPE_PREFIXES``) live in ``huggingface_hub.constants``. Addresses #3971. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

bot-ci-comment · 2026-04-28T08:50:04Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Adds a dedicated `HfUriError(ValueError)` in `huggingface_hub.errors` and raises it from `parse_hf_uri` instead of plain `ValueError`. Inheriting from `ValueError` keeps backward compatibility for callers that catch the latter. Also drops the unnecessary `from __future__ import annotations` in the new modules — the project requires Python 3.10+ where `str | None` works at runtime, no forward refs are involved, and only 4/136 src files use it. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Switch from `[\`name\`][huggingface_hub.utils.module.name]` to the shorter `[\`name\`]` form throughout the new module's docstrings and reference page, matching the convention used elsewhere. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

Use the `type` and `id` fields directly. The `is_repo`/`is_bucket` booleans remain available to disambiguate when needed. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

More descriptive — pairs naturally with parse_hf_uri. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

t # Lines starting with '#' will be ignored, and an empty message aborts

…nvert refs - to_uri() now URL-encodes '/' in non-special revisions so 'feature/foo' round-trips correctly. - _SPECIAL_REFS_REVISION_REGEX accepts '-' and '.' in convert ref names (e.g. 'refs/convert/parquet-v2').

cursor · 2026-04-28T11:06:51Z

+    "spaces": "space",
+    "kernels": "kernel",
+    "buckets": "bucket",
+}


New constant duplicates existing REPO_TYPES_MAPPING data

Low Severity

HF_URI_TYPE_PREFIXES duplicates the four repo-type entries already defined in REPO_TYPES_MAPPING (same file), only adding "buckets": "bucket". If a new repo type is later added to REPO_TYPES_MAPPING, a developer could easily forget to update HF_URI_TYPE_PREFIXES, causing the URI parser to silently reject valid URIs. Building HF_URI_TYPE_PREFIXES from REPO_TYPES_MAPPING (e.g. {**REPO_TYPES_MAPPING, "buckets": "bucket"}) would keep a single source of truth for repo-type mappings.

Additional Locations (1)

src/huggingface_hub/constants.py#L118-L124

^{Reviewed by Cursor Bugbot for commit 63df690. Configure here.}

not an issue IMO

hanouticelina

Made a first pass, looks good overall! mostly cosmetic comments

hanouticelina · 2026-04-28T16:15:07Z

+# (Pull Request refs) and 'refs/convert/<name>' (e.g. parquet conversions).
+# The conversion name allows the typical git ref characters '[a-zA-Z0-9_.-]'
+# so names like 'parquet-v2' or 'duckdb.v1' round-trip correctly.
+_SPECIAL_REFS_REVISION_REGEX = re.compile(r"^refs/(?:convert/[\w.-]+|pr/\d+)")


it's different from SPECIAL_REFS_REVISION_REGEX defined in hf_api.py right? let's maybe drop the one in hf_api.py, make this one public and import it there?

no it's basically the same but I did not want to touch any existing code. The SPECIAL_REFS_REVISION_REGEX regex you are referring to will disappear once we use parse_hf_uri everywhere

cursor · 2026-04-29T07:47:35Z

+# (Pull Request refs) and 'refs/convert/<name>' (e.g. parquet conversions).
+# The conversion name allows the typical git ref characters '[a-zA-Z0-9_.-]'
+# so names like 'parquet-v2' or 'duckdb.v1' round-trip correctly.
+_SPECIAL_REFS_REVISION_REGEX = re.compile(r"^refs/(?:convert/[\w.-]+|pr/\d+)")


Duplicated special-refs regex with divergent matching behavior

Low Severity

_SPECIAL_REFS_REVISION_REGEX in _hf_uris.py uses [\w.-]+ for convert ref names, while the existing SPECIAL_REFS_REVISION_REGEX in hf_api.py uses \w+. The new pattern matches hyphens and dots (e.g. refs/convert/parquet-v2) that the old one rejects. Having two subtly different regexes for the same semantic concept risks inconsistent behavior across code paths that haven't yet migrated to the new parser.

^{Reviewed by Cursor Bugbot for commit 439df11. Configure here.}

as mentioned above (#4158 (comment)), SPECIAL_REFS_REVISION_REGEX will soon disappear so don't worry

cursor · 2026-04-29T08:26:07Z

+    """Parse the body of a bucket URI: 'namespace/name[/path]'."""
+    if "@" in location:
+        raise HfUriError(uri=raw, msg="Bucket URIs do not support a revision marker ('@').")
+    location = location.strip("/")


strip("/") silently accepts double slashes after type prefix

Low Severity

location.strip("/") in both _parse_bucket_body and _parse_repo_body removes leading slashes that result from double-slash inputs like hf://models//org/model or hf://buckets//org/b. After _split_type returns rest = /org/model (from the double slash), strip silently normalizes it to org/model, causing the URI to parse successfully. This is inconsistent with the explicit rejection of // within paths (e.g. hf://models/org/m//sub), which the PR reviewer specifically requested to be treated as errors.

Additional Locations (1)

src/huggingface_hub/utils/_hf_uris.py#L313-L314

^{Reviewed by Cursor Bugbot for commit 0586cc5. Configure here.}

Co-Authored-By: Claude Opus 4.6 <[email protected]>

Wauplin · 2026-04-29T08:41:42Z

Thanks for the review @hanouticelina ! I have addressed all of your comments.

Also as discussed in DMs, I have split the logic between HfUri and HfMount as it didn't feel right to me to include mount logic into HfUri. We now have parse_hf_mount which depends on parse_hf_uri to parse the mount source. This split should make the API more consistent while keeping the core logic unified.

hanouticelina

Thank you! left two small nits, it looks good and works as expected 🙌

Wauplin · 2026-04-29T13:04:56Z

Thank you! Will merge if CI gets green and move forward on the integration :)

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

There are 4 total unresolved issues (including 3 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 116f6ea. Configure here.}

cursor · 2026-04-29T13:16:37Z

+            try:
+                validate_repo_id(self.id)
+            except HFValidationError as e:
+                raise HfUriError(uri=uri, msg=str(e)) from e


Bucket IDs with empty segments bypass validation

Medium Severity

The __post_init__ validation for HfUri checks self.id.count("/") != 1 but doesn't verify both segments are non-empty. For non-bucket types, validate_repo_id catches IDs like "org/" or "/name" via its regex. But bucket types explicitly skip validate_repo_id, so directly constructing HfUri(type="bucket", id="org/") or HfUri(type="bucket", id="/b") silently produces an invalid object. The parser's _parse_bucket_body has the correct not parts[0] or not parts[1] guard, but __post_init__ doesn't replicate it for the bucket case.

^{Reviewed by Cursor Bugbot for commit 116f6ea. Configure here.}

huggingface-hub-bot · 2026-04-30T11:58:06Z

This PR has been shipped as part of the v1.13.0 release.

julien-c · 2026-04-30T13:54:30Z

this doc page is very cool

Wauplin changed the title ~~[Utils] Centralize hf:// URI parsing~~ Centralize hf:// URI parsing Apr 28, 2026

Wauplin and others added 11 commits April 28, 2026 10:56

useless

53d328b

s

4240e6d

update docs

5e7fec6

remove some useless tests

3b7d187

syntax

b9f2bea

[Utils] Drop repo_id/repo_type/bucket_id properties from HfUri

cc04d29

Use the `type` and `id` fields directly. The `is_repo`/`is_bucket` booleans remain available to disambiguate when needed. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

[Utils] Rename HfUri.to_string() to to_uri()

7af227f

More descriptive — pairs naturally with parse_hf_uri. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

docs formatting

00fdab8

cleaner

50d46a5

Wauplin requested a review from hanouticelina April 28, 2026 10:11

Wauplin marked this pull request as ready for review April 28, 2026 10:12

Wauplin mentioned this pull request Apr 28, 2026

Centralize hf:// URI parsing into utils/_hf_uri.py #3994

Closed

cursor Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread src/huggingface_hub/utils/_hf_uris.py

Wauplin added 3 commits April 28, 2026 12:20

btter ?

3f65ccd

better revision handling

ffef3af

Merge branch 'main' into harmonize-hf-handles

37c0a91

t # Lines starting with '#' will be ignored, and an empty message aborts

cursor Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread src/huggingface_hub/utils/_hf_uris.py Outdated

cursor Bot reviewed Apr 28, 2026

View reviewed changes

Merge branch 'main' into harmonize-hf-handles

038034e

cursor Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread tests/test_utils_hf_uris.py Outdated

Wauplin and others added 4 commits April 28, 2026 14:16

duplicate test

c7ca6d4

Merge branch 'main' into harmonize-hf-handles

42d8679

Merge branch 'main' into harmonize-hf-handles

dcc36f7

forbid canonical urls

1eeb168

cursor Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread src/huggingface_hub/utils/_hf_uris.py

hanouticelina reviewed Apr 28, 2026

View reviewed changes

Wauplin added 2 commits April 29, 2026 09:39

uri passed to HfUriError

2ab4537

why not

439df11

cursor Bot reviewed Apr 29, 2026

View reviewed changes

add some validation at the HfUri level directly

0586cc5

cursor Bot reviewed Apr 29, 2026

View reviewed changes

split HfMount out of HfUri

0521a36

Co-Authored-By: Claude Opus 4.6 <[email protected]>

Wauplin requested a review from hanouticelina April 29, 2026 08:42

hanouticelina approved these changes Apr 29, 2026

View reviewed changes

Comment thread src/huggingface_hub/utils/_hf_uris.py Outdated

Comment thread src/huggingface_hub/utils/_hf_uris.py Outdated

Wauplin and others added 2 commits April 29, 2026 15:04

nits

29e443e

Merge branch 'main' into harmonize-hf-handles

116f6ea

cursor Bot reviewed Apr 29, 2026

View reviewed changes

Wauplin merged commit 5fb553d into main Apr 29, 2026
21 checks passed

Wauplin deleted the harmonize-hf-handles branch April 29, 2026 15:29

julien-c reviewed Apr 30, 2026

View reviewed changes

Comment thread docs/source/en/package_reference/hf_uris.md

Copy link
Copy Markdown

Member

julien-c Apr 30, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this doc page is very cool

Wauplin reacted with heart emoji

Wauplin mentioned this pull request May 5, 2026

[Core] Migrate hf:// URI parsing to centralized parse_hf_uri #4189

Open

Conversation

Wauplin commented Apr 28, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Follow-ups (not in this PR)

Uh oh!

bot-ci-comment Bot commented Apr 28, 2026

Uh oh!

Uh oh!

Uh oh!

cursor Bot Apr 28, 2026

Choose a reason for hiding this comment

New constant duplicates existing REPO_TYPES_MAPPING data

Uh oh!

Wauplin Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

hanouticelina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hanouticelina Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

Wauplin Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Apr 29, 2026

Choose a reason for hiding this comment

Duplicated special-refs regex with divergent matching behavior

Uh oh!

Wauplin Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Apr 29, 2026

Choose a reason for hiding this comment

strip("/") silently accepts double slashes after type prefix

Uh oh!

Wauplin commented Apr 29, 2026

Uh oh!

hanouticelina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Wauplin commented Apr 29, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Apr 29, 2026

Choose a reason for hiding this comment

Bucket IDs with empty segments bypass validation

Uh oh!

Uh oh!

huggingface-hub-bot Bot commented Apr 30, 2026

Uh oh!

julien-c Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Wauplin commented Apr 28, 2026 •

edited by cursor Bot

Loading

New constant duplicates existing `REPO_TYPES_MAPPING` data

Wauplin Apr 28, 2026 •

edited

Loading