Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: huggingface/xet-core
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.5.2
Choose a base ref
...
head repository: huggingface/xet-core
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.6.0
Choose a head ref
  • 15 commits
  • 82 files changed
  • 6 contributors

Commits on Jul 17, 2026

  1. xet_session: box bridged futures to cap type-layout depth (#901)

    ## Problem
    
    Every public `XetSession` async operation (`commit`, `upload_from_path`,
    `download_file_to_path`, `finish`, `finalize_ingestion`, stream writes,
    …) routes its work future through the bridge layer:
    
    ```
    public async API
      └─ TaskRuntime::bridge_async(_finalizing)   (xet_pkg/.../task_runtime.rs)
           └─ TaskRuntime::run_inner_async
                └─ XetRuntime::bridge_async        (xet_runtime/.../native.rs)
                     ├─ External       => Ok(fut.await)          ← inlines the deep future
                     └─ OwnedThreadPool => bridge_to_owned(..)   ← spawn(), type-erased
    ```
    
    On the inline-await paths — the `External` arm on native, and
    `run_inner_async` on wasm (no runtime offload) — the work future (`async
    move { inner.commit().await }`, i.e. the full `xet_client`/`xet_data`
    ingestion/CAS chain) is awaited inline, so its entire nested type
    becomes part of the bridge's state machine and propagates up through
    every layer into the caller.
    
    Both match arms compile unconditionally, so the `External` arm inflates
    the layout even for `OwnedThreadPool` consumers. For deep operations
    this can push a consumer's own future past rustc's type-layout
    query-depth limit (`error: queries overflow the depth limit!`) at the
    call site — no downstream code change involved, just the depth of what
    it awaits.
    
    ## Fix
    
    Box the future on the two inline paths so its deep type is heap-erased
    instead of embedded:
    
    - `XetRuntime::bridge_async` (native) `External` arm →
    `Ok(Box::pin(fut).await)`
    - `TaskRuntime::run_inner_async` (wasm) → box `fut` before the `select!`
    
    The `OwnedThreadPool` arm already type-erases via `spawn`, so it's
    untouched (no extra allocation on that path). This collapses the layout
    depth once for all public async APIs and all consumers, so downstream
    crates no longer need `Box::pin` at call sites or `#![recursion_limit]`
    bumps.
    
    ## Testing
    
    - `cargo build -p hf-xet -p xet-runtime` ✅
    - `cargo check --target wasm32-unknown-unknown` (via
    `wasm/hf_xet_thin_wasm`) ✅
    - `cargo clippy -p hf-xet -p xet-runtime` ✅ (clean)
    - `cargo +nightly fmt` ✅
    
    <!-- CURSOR_SUMMARY -->
    ---
    
    > [!NOTE]
    > **Low Risk**
    > Compile-time typing change with a small extra allocation on
    External/wasm paths; owned-pool bridging and API semantics are
    unchanged.
    > 
    > **Overview**
    > Public `XetSession` async work flows through bridge layers that used
    to **inline** very deep futures on certain paths, which could trigger
    rustc’s `queries overflow the depth limit!` at consumer call sites.
    > 
    > **`XetRuntime::bridge_async`** on native **External** mode now awaits
    `Box::pin(fut)` instead of `fut` directly.
    **`TaskRuntime::run_inner_async`** on **wasm** boxes the work future
    before the cancellation `select!`, since wasm has no runtime offload.
    **OwnedThreadPool** bridging is unchanged (still type-erased via
    `spawn`).
    > 
    > Behavior and cancellation semantics stay the same; the tradeoff is one
    heap allocation on External and wasm inline paths so downstream crates
    don’t need local `Box::pin` or higher recursion limits.
    > 
    > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
    c26ee6c. Bugbot is set up for automated
    code reviews on this repo. Configure
    [here](https://www.cursor.com/dashboard/bugbot).</sup>
    <!-- /CURSOR_SUMMARY -->
    assafvayner authored Jul 17, 2026
    Configuration menu
    Copy the full SHA
    e1d1d7d View commit details
    Browse the repository at this point in the history
  2. Use LazyLock instead of lazy_static (#900)

    <!-- CURSOR_SUMMARY -->
    > [!NOTE]
    > **Low Risk**
    > Mechanical std-library migration with preserved lazy-init semantics;
    the postcard default-features change only affects dependency features,
    not obvious runtime logic.
    > 
    > **Overview**
    > Replaces the **`lazy_static`** crate with **`std::sync::LazyLock`**
    across the workspace and drops **`lazy_static`** from workspace and
    crate manifests.
    > 
    > **`test_configurable_constants!`** in `xet_runtime` now defines
    env-overridable constants as **`LazyLock`** instead of
    **`lazy_static!`**, with the same debug-only **`HF_XET_*`** behavior;
    **`HIGH_PERFORMANCE`** and privilege detection follow the same pattern.
    Call sites (SIGINT/runtime globals in **`hf_xet`**, profiling session,
    shard filename regex, **`MAX_CHUNK_SIZE`**, simulation URL reference
    time, etc.) are updated accordingly, and the duplicate
    **`MAX_CHUNK_SIZE`** lazy static in **`xet_data`** chunking is removed
    in favor of the core-structures definition.
    > 
    > Workspace **`postcard`** is pinned with **`default-features = false`**
    (still **`alloc`**), which prunes **`heapless`** and related transitive
    deps from **`Cargo.lock`**. **`FN_CALL_ID`** in the remote CAS client is
    a plain **`AtomicU64`** static rather than a lazy wrapper.
    > 
    > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
    0e667fd. Bugbot is set up for automated
    code reviews on this repo. Configure
    [here](https://www.cursor.com/dashboard/bugbot).</sup>
    <!-- /CURSOR_SUMMARY -->
    
    ---------
    
    Co-authored-by: Assaf Vayner <[email protected]>
    Expyron and assafvayner authored Jul 17, 2026
    Configuration menu
    Copy the full SHA
    39d109e View commit details
    Browse the repository at this point in the history
  3. Bump sha2 to 0.11 (#899)

    ## Summary
    - bump workspace sha2 dependency to 0.11
    - remove the obsolete sha2 asm feature from xet-data
    - add a canonical byte-order constructor on DataHash/Sha256 and use it
    for sha2 0.11 digest bytes
    
    ## Tests
    - cargo check -p xet-data
    - cargo test -p xet-core-structures test_hash_from_be_bytes
    - cargo test -p xet-data --features simulation sha_tests
    - cargo fmt --check -p xet-data -p xet-core-structures
    - git diff --check
    
    <!-- CURSOR_SUMMARY -->
    ---
    
    > [!NOTE]
    > **Medium Risk**
    > Changes how SHA-256 digests are converted into merkle hashes (byte
    order vs hex round-trip); incorrect mapping would break content
    verification, though tests cover the path.
    > 
    > **Overview**
    > Upgrades the workspace **`sha2`** dependency from **0.10** to **0.11**
    and refreshes related lockfiles (including **`hf_xet`** and WASM
    crates). **`sha2` 0.11** no longer ships the **`asm`** / **`sha2-asm`**
    path, so **`xet-data`** stops enabling **`asm`** on non-Windows and uses
    plain workspace **`sha2`** everywhere.
    > 
    > **`Sha256Generator`** now maps digests with
    **`Sha256::from_be_bytes`** instead of formatting hex and parsing back.
    **`DataHash`** gains **`from_be_bytes`** (canonical hex byte order →
    internal **`[u64; 4]`** layout), with a unit test.
    > 
    > A standalone **`simulation/sha256_bench`** crate plus
    **`benches/results.md`** document throughput comparing **0.10.9+asm** vs
    **0.11** to justify dropping **`asm`**. The root workspace **removes
    `simulation` from `members`** and **excludes** the whole
    **`simulation`** tree (replacing the narrower **`chunk_cache_bench`**
    exclude).
    > 
    > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
    b79ece9. Bugbot is set up for automated
    code reviews on this repo. Configure
    [here](https://www.cursor.com/dashboard/bugbot).</sup>
    <!-- /CURSOR_SUMMARY -->
    assafvayner authored Jul 17, 2026
    Configuration menu
    Copy the full SHA
    d1b5cdf View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2026

  1. Fix all 21 open Dependabot alerts across all lockfiles (#903)

    (big line diff is Cargo.locks)
    
    Resolves all 21 open Dependabot alerts by bumping vulnerable
    dependencies across the four affected lockfiles. All changes verified
    with `cargo check` (workspace, benches, wasm32 target, and hf_xet with
    the `profiling` feature); no behavior changes outside of one profiling
    output-dir naming detail noted below.
    
    ## Changes
    
    ### openssl 0.10.79 → 0.10.81 (GHSA-phqj-4mhp-q6mq, moderate)
    Lockfile-only; transitive via `native-tls`. In `chunk_cache_bench`,
    openssl leaves the dependency graph entirely (see sccache change below).
    Resolves alerts
    [#167](https://github.com/huggingface/xet-core/security/dependabot/167),
    [#168](https://github.com/huggingface/xet-core/security/dependabot/168),
    [#169](https://github.com/huggingface/xet-core/security/dependabot/169).
    
    ### rustls-webpki → 0.103.13 (GHSA-82j2-j2ch-gfr8 high,
    GHSA-pwjx-qhcg-rvj4 moderate, GHSA-xgp8-3hg3-c2mh, GHSA-965h-392x-2mh5
    low)
    Lockfile bump in `wasm/hf_xet_thin_wasm`. In
    `simulation/chunk_cache_bench` there was a second unfixable copy
    (0.102.8 via `rustls 0.22 ← redis 0.25 ← opendal 0.48 ← sccache 0.8`),
    eliminated by bumping `sccache = "0.16"`. Resolves alerts
    [#112](https://github.com/huggingface/xet-core/security/dependabot/112),
    [#123](https://github.com/huggingface/xet-core/security/dependabot/123),
    [#124](https://github.com/huggingface/xet-core/security/dependabot/124),
    [#128](https://github.com/huggingface/xet-core/security/dependabot/128),
    [#132](https://github.com/huggingface/xet-core/security/dependabot/132),
    [#156](https://github.com/huggingface/xet-core/security/dependabot/156),
    [#159](https://github.com/huggingface/xet-core/security/dependabot/159).
    
    ### rand 0.9.2 → 0.9.4, 0.8.5 → 0.8.6 (GHSA-cq8v-f236-94qc, low)
    Lockfile-only targeted bumps in all four lockfiles; all vulnerable
    copies were transitive (direct dep is rand 0.10). Resolves alerts
    [#133](https://github.com/huggingface/xet-core/security/dependabot/133),
    [#134](https://github.com/huggingface/xet-core/security/dependabot/134),
    [#135](https://github.com/huggingface/xet-core/security/dependabot/135),
    [#137](https://github.com/huggingface/xet-core/security/dependabot/137),
    [#144](https://github.com/huggingface/xet-core/security/dependabot/144),
    [#154](https://github.com/huggingface/xet-core/security/dependabot/154).
    
    ### atty removal via criterion 0.4 → 0.7 (GHSA-g98v-hv3f-hcfr, low, no
    patched release exists)
    `atty`'s only parent was `criterion 0.4` (dev/bench-only) in `xet_data`
    and `chunk_cache_bench`; criterion 0.5+ dropped it. Bench code needed no
    changes. Resolves alerts
    [#2](https://github.com/huggingface/xet-core/security/dependabot/2),
    [#80](https://github.com/huggingface/xet-core/security/dependabot/80).
    
    ### protobuf 2.28.0 → 3.7.2 via pprof 0.14 → 0.15 (GHSA-2gh3-rmm4-6rq5,
    moderate)
    `protobuf 2.x` has no patched release; `pprof 0.15` moved to protobuf
    3.x. Side note: the `profiling` feature of `hf_xet` didn't compile on
    main (`chrono` was used but never declared); fixed by switching the
    profile output dir name from a formatted local timestamp to epoch
    seconds. Resolves alert
    [#32](https://github.com/huggingface/xet-core/security/dependabot/32).
    
    ### tar 0.4.45 → 0.4.46 (GHSA-3pv8-6f4r-ffg2, moderate)
    Lockfile-only, transitive via sccache. Resolves alert
    [#172](https://github.com/huggingface/xet-core/security/dependabot/172).
    
    ### jsonwebtoken removal via `sccache` default features off
    (GHSA-h395-gr6q-cpjc, moderate)
    No fixed version is reachable through the dependency chain (`sccache →
    opendal → reqsign → jsonwebtoken ^9`; the fix is the semver-major
    10.3.0, which nothing accepts yet). However, jsonwebtoken only enters
    via sccache's cloud-storage backend features, which the benchmark never
    uses — it only needs `sccache::lru_disk_cache`. Setting
    `default-features = false` on sccache removes jsonwebtoken, opendal,
    reqsign, and openssl from `chunk_cache_bench`'s graph entirely (-913
    lockfile lines). Resolves alert
    [#89](https://github.com/huggingface/xet-core/security/dependabot/89).
    
    Note: the `chunk_cache_bench` lockfile diff is large because of the
    sccache upgrade + feature trim, and the lockfile was also stale vs. its
    manifest (orphaned `csv`/`csv-core` entries removed).
    
    <!-- CURSOR_SUMMARY -->
    ---
    
    > [!NOTE]
    > **Low Risk**
    > Changes are dependency and lockfile updates plus
    dev/bench/profiling-only paths; the only user-visible tweak is profiling
    output directory naming when `--features profiling` is enabled.
    > 
    > **Overview**
    > Closes open Dependabot findings by refreshing **Cargo.lock** files and
    bumping a few direct dev/bench/profiling dependencies. Production crates
    mostly pick up transitive fixes (e.g. **openssl**, **rustls-webpki**,
    **rand**, **crossbeam-epoch**); **criterion** moves **0.4 → 0.7** in
    `xet_data` and `chunk_cache_bench`, which drops the unmaintained
    **atty** chain from benchmarks.
    > 
    > For **`hf_xet`**, optional **`pprof` 0.14 → 0.15** moves the protobuf
    stack to 3.x and fixes the profiling build; profile output folders under
    `profiles/` now use **Unix epoch seconds** instead of a
    `chrono`-formatted local timestamp.
    > 
    > **`chunk_cache_bench`** upgrades **`sccache` to 0.16** with
    **`default-features = false`**, trimming cloud-storage deps (e.g.
    **jsonwebtoken**, **opendal**, **openssl**) from that crate’s graph
    while still using its disk LRU cache in benches. **`.cargo/audit.toml`**
    drops several advisory ignores that are no longer needed and narrows the
    **async-std** ignore rationale.
    > 
    > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
    c611e36. Bugbot is set up for automated
    code reviews on this repo. Configure
    [here](https://www.cursor.com/dashboard/bugbot).</sup>
    <!-- /CURSOR_SUMMARY -->
    
    ## cargo-audit cleanup
    
    `.cargo/audit.toml` shrinks from 8 ignores to 2. Removed: both atty
    entries and rand (fixed above), atomic-polyfill (was already gone from
    the lockfile), crossbeam-epoch (bumped 0.9.18 → 0.9.20,
    RUSTSEC-2026-0204), and scc (serial_test 3.4 → 3.5 dropped scc entirely,
    RUSTSEC-2026-0205). The two remaining ignores are rsa
    (RUSTSEC-2023-0071, Marvin — no fixed release exists; dev-only via
    russh) and async-std (RUSTSEC-2025-0052 — intentional dev-dep of xet_pkg
    for non-tokio executor bridge tests). `cargo audit -D warnings` passes.
    assafvayner authored Jul 20, 2026
    Configuration menu
    Copy the full SHA
    ec93b47 View commit details
    Browse the repository at this point in the history
  2. Remove stale tests and update cargo test command (#905)

    Remove stale tests since the test binary tool "x" was moved.
    Also add "internal-tools" feature to the "cargo test --workspace"
    command because these the compilation of test binary tools are now gated
    behind this feature.
    seanses authored Jul 20, 2026
    Configuration menu
    Copy the full SHA
    3f36c21 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    98e9348 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    075687d View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2026

  1. Stream shard finalization progress via POST /v2/shards (#884)

    ## Summary
    
    Adds client support for `POST /v2/shards`, a streaming NDJSON variant of
    shard upload that reports server-side finalization progress (validation
    against xorb metadata, then durable commit to S3/DynamoDB) instead of
    blocking silently until the whole thing completes. This closes a
    visibility gap for large shards, where server-side finalization can take
    a while after the bytes have already been transferred.
    
    - `RemoteClient::upload_shard` auto-detects V2 vs V1: it tries
    `/v2/shards` first and falls back to `/v1/shards` on `404`/`501`,
    caching the detected version per client (same pattern already used for
    the reconstruction API). A new `client.shard_api_version` config option
    (`HF_XET_CLIENT_SHARD_API_VERSION`) can force a specific version with no
    fallback.
    - New `ShardUploadEvent` wire type (`Validating`, `Committing{stage}`,
    `Result`, `Error`) models the NDJSON frames sent by the server. It has a
    custom, deliberately partial `PartialOrd`/`precede` so the client can
    safely ignore stale, duplicate, or out-of-order frames instead of
    naively overwriting already-observed state.
    - New `shard_upload_v2` module parses the response body frame-by-frame
    (correctly handling frames split across TCP chunks), invokes an optional
    progress callback per frame, and resolves once a terminal
    `Result`/`Error` frame arrives.
    
    ## Progress tracking refactor
    
    This also reworks the internal progress-tracking types to plug in shard
    finalization progress alongside existing file/xorb progress:
    
    - Added `ShardUploadProgress`: tracks shard byte-transfer progress plus
    the new server-reported validation/commit counters
    (`total_shard_validation_entries[_completed]`,
    `total_shards_uploaded_to_store`, `total_shards_synced`,
    `total_shards_completed`), keyed per-shard by `UniqueId` so
    out-of-order/duplicate NDJSON frames can't regress already-recorded
    state.
    - Introduced `UploadGroupProgress`, combining file/xorb progress
    (`GroupProgress`) and the new `ShardUploadProgress` into a single handle
    now owned by `CompletionTracker` (previously `FileUploadSession` held a
    bare `GroupProgress` directly).
    - `CompletionTracker::register_new_file` now creates the per-file
    `ItemProgressUpdater` internally instead of requiring callers to
    construct one and pass it in, simplifying call sites in
    `FileUploadSession`.
    - Removed two redundant aliases on `ItemProgressUpdater`
    (`report_bytes_written`, `report_transfer_progress`) in favor of their
    canonical names (`report_bytes_completed`,
    `report_transfer_bytes_completed`).
    
    ## Other changes
    
    - `upload_xorb` now moves the serialized xorb bytes (`mem::take` +
    `Bytes::from`) instead of cloning them before streaming.
    - Simulation/local test server and all `Client` impls (`LocalClient`,
    `MemoryClient`, `RemoteSimulationClient`, `SimulationControlClient`,
    `LocalTestServer`) updated to thread the new `progress_callback`
    parameter through `upload_shard`, and the simulation router's `/shards`
    route was consolidated under the `/v1` nest.
    - Substantial new unit test coverage: `ShardUploadEvent` serde
    round-trips and a full `PartialOrd`/`precede` ordering matrix,
    `ShardUploadProgress::update` behavior under
    out-of-order/skipped/duplicate frames (including a multi-shard
    aggregation test), and `shard_upload_v2` NDJSON parsing edge cases
    (mid-line frame splits, blank lines, missing terminal frame, callback
    ordering).
    
    
    ## Related PRs
    CAS server side:
    huggingface-internal/xetcas#1140
    huggingface_hub side:
    huggingface/huggingface_hub#4478
    
    Resolves client side of
    huggingface-internal/xetcas#996
    seanses authored Jul 23, 2026
    Configuration menu
    Copy the full SHA
    87ca7ad View commit details
    Browse the repository at this point in the history
  2. fix: stack overflow in xet-session from unboxed futures (#908)

    ## Summary
    
    `AuthGroupBuilder<T>::build()`'s compiler-generated future was large
    enough to overflow a thread's default stack on platforms with a small
    default (e.g. ~524 KiB for non-main threads on macOS).
    
    Root cause: `bridge_async`/`bridge_async_finalizing` and their
    downstream helpers were `async fn`s generic over `F`, so a large future
    crossing one of these boundaries by value got duplicated across nested
    generator frames instead of being shared — the same failure mode #901
    partially addressed but didn't fully close.
    
    **Fix:** box the future once, as early as possible (in
    `xet_session::TaskRuntime`), and pass the boxed pointer through
    downstream layers.
    
    | | Before | After |
    |---|---|---|
    | `AuthGroupBuilder::build()` future size | 23,664 bytes | ~536–800
    bytes (see below) |
    | Min. viable stack (release build) | ~128–256 KiB | ~65.5–66 KiB |
    
    One crate boundary further downstream,
    `xet_runtime::XetRuntime::bridge_async`/`bridge_to_owned` can either
    re-box or stay generic over `F` — we measured both
    (`-Zprint-type-sizes`, nightly, debug build):
    
    | `AuthGroupBuilder<T>` instantiation | Re-box at `xet_runtime` boundary
    | Generic (current) |
    |---|---:|---:|
    | `download_stream_group` | 536 bytes | 808 bytes |
    | `file_download_group` | 528 bytes | 800 bytes |
    | `upload_commit` | 528 bytes | 800 bytes |
    
    Both are trivially safe vs. the 23,664-byte pre-fix size. Re-boxing is
    ~34% smaller and makes that boundary self-defending for any caller, not
    just `xet_session` — we opted for the simpler single-boxing-point
    approach instead, accepting the larger (still tiny) future and leaving
    "box early" as the caller's responsibility.
    
    ---------
    
    Co-authored-by: di <[email protected]>
    kszucs and seanses authored Jul 23, 2026
    Configuration menu
    Copy the full SHA
    fe93e98 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2026

  1. Embed input in the DataHash parse Error (#914)

    ## Summary
    - Include the failing input in DataHash parse errors so download
    failures like [#895](#895)
    surface `Unable to parse string as hex hash value (got '...')` instead
    of an opaque message, helps debugging.
    - Extract hex/bytes parse errors into `merklehash/error.rs` as a
    `thiserror` `DataHashError` enum (`InvalidHex` / `InvalidBytes`), both
    carrying the bad input for display. This has been our code style.
    - Wire `CoreError::HashParsing` to `DataHashError` (replacing the unused
    `Infallible` stub) and map it to `XetError::DataIntegrity`.
    seanses authored Jul 29, 2026
    Configuration menu
    Copy the full SHA
    a3f7e87 View commit details
    Browse the repository at this point in the history
  2. Allow selecting the reqwest TLS backend via hf-xet features (#917)

    requested by users of hf-xet transitively through hf-hub
    vllm-project/vllm#48575 (comment)
    
    ## What
    
    Adds feature-based TLS backend selection to the `hf-xet` crate (and
    `xet-data`), so consumers can switch the reqwest TLS backend from rustls
    to the platform-native TLS stack.
    
    `xet-client` already exposed `rustls-tls` / `native-tls` /
    `native-tls-vendored` features, and `hf-xet` forwarded the native ones —
    but since `hf-xet` depended on `xet-client` with default features on,
    `rustls-tls` was always enabled and could never be opted out of (cargo
    features are additive).
    
    ## Changes
    
    - `xet_pkg` (`hf-xet`) and `xet_data`: depend on `xet-client` (and
    `xet-data`) with `default-features = false`
    - Both crates re-expose the backend choice as their own features, with
    `default = ["rustls-tls"]`
    
    Usage:
    
    ```toml
    hf-xet = { version = "1.5.4", default-features = false, features = ["native-tls"] }
    ```
    
    Note: even without `default-features = false`, enabling `native-tls`
    switches the runtime backend — reqwest 0.13's `TlsBackend::default()`
    prefers native-tls when both backends are compiled in. The
    `default-features = false` path additionally drops rustls from the
    dependency tree entirely.
    
    ## Behavior
    
    - Default builds unchanged: rustls remains the default backend for all
    existing consumers (all internal dependency edges use default features).
    - `Cargo.lock` unchanged — native-tls pins were already present.
    
    ## Verification
    
    - `cargo check -p hf-xet` → `cargo tree -i rustls` shows rustls
    (unchanged default)
    - `cargo check -p hf-xet --no-default-features --features native-tls`
    compiles; `cargo tree -i rustls` finds no rustls, `-i native-tls` shows
    native-tls wired via hyper-tls → reqwest
    - `cargo check --workspace` passes
    
    <!-- CURSOR_SUMMARY -->
    ---
    
    > [!NOTE]
    > **Low Risk**
    > Build-time dependency/feature wiring only; default rustls behavior is
    unchanged for normal consumers.
    > 
    > **Overview**
    > **Enables `hf-xet` and `xet-data` consumers to choose the reqwest TLS
    stack** instead of always pulling in rustls through transitive default
    features.
    > 
    > Both crates now depend on `xet-client` (and `hf-xet` on `xet-data`)
    with **`default-features = false`**, and re-export **`rustls-tls`**,
    **`native-tls`**, and **`native-tls-vendored`** with **`default =
    ["rustls-tls"]`**. `hf-xet`’s TLS features forward to both `xet-client`
    and `xet-data` so the whole stack stays aligned.
    > 
    > Default behavior stays rustls for existing dependents. With
    `default-features = false` and `native-tls` (or vendored), rustls can be
    dropped from the dependency tree entirely.
    > 
    > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
    8a20af9. Bugbot is set up for automated
    code reviews on this repo. Configure
    [here](https://www.cursor.com/dashboard/bugbot).</sup>
    <!-- /CURSOR_SUMMARY -->
    assafvayner authored Jul 29, 2026
    Configuration menu
    Copy the full SHA
    7a26750 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2026

  1. fix: scope pyo3 auto-initialize to dev-dependencies (#891) (#923)

    Fixes #891.
    
    ## Problem
    
    `hf_xet` enables pyo3's `auto-initialize` feature. pyo3's build script
    hard-fails when that feature is combined with a Python built
    `--disable-shared`, which is the standard configuration for manylinux
    build environments:
    
    ```
    error: The `auto-initialize` feature is not supported when linking Python
    statically instead of with a shared library.
    ```
    
    The check has no `extension-module` carve-out, so it fires even though
    we build as an extension module. This blocks downstream builds of
    `huggingface-hub` and anything above it.
    
    ## Why the suggested one-line fix is insufficient
    
    The issue suggests simply dropping the feature. That breaks `cargo test`
    for this crate:
    
    ```
    test result: FAILED. 10 passed; 26 failed
    
    panicked at pyo3-0.29.0/src/interpreter_lifecycle.rs:134:
    assertion `left != right` failed: The Python interpreter is not initialized
    and the `auto-initialize` feature is not enabled.
    ```
    
    26 of the 36 unit tests call `Python::attach` from a plain Rust test
    binary, where no interpreter is running yet. pyo3 has an escape hatch
    for exactly this case, but it is gated on `CARGO_PRIMARY_PACKAGE`, which
    Cargo sets only when pyo3 itself is the package under test — it never
    applies to a downstream crate.
    
    ## Fix
    
    Keep the feature, but scope it to test builds:
    
    ```toml
    [dependencies]
    pyo3 = { version = "0.29", features = ["abi3-py38"] }
    
    [dev-dependencies]
    pyo3 = { version = "0.29", features = ["auto-initialize"] }
    ```
    
    Cargo's v2/v3 feature resolver only unifies dev-dependency features when
    test targets are built, so the cdylib build never sees it.
    
    Verified by reading the resolved feature set on pyo3's `custom-build`
    unit — the unit that actually runs the build-script check — via `cargo
    build --unit-graph`:
    
    | Build | `auto-initialize` |
    | --- | --- |
    | `cargo build --features extension-module` (how maturin builds it) |
    absent |
    | `cargo build` | absent |
    | `cargo test` | present |
    
    ## Impact
    
    **No functional change to the shipped wheel.** Both variants go through
    the same one-shot `Once` in pyo3: with the feature, the first
    `Python::attach` calls `Py_InitializeEx` only `if Py_IsInitialized() ==
    0`; without it, the first call asserts `Py_IsInitialized() != 0`. An
    extension module always has a live interpreter by the time our code
    runs, so that first call is a no-op either way and every later call
    short-circuits on the completed `Once`.
    
    No new dependencies, and `Cargo.lock` is unchanged — feature selection
    is not recorded there.
    
    ## Tests
    
    Adds `hf_xet/tests/pyo3_features.rs`, asserting the manifest invariant
    directly. Release CI builds wheels against interpreters that *do* expose
    a shared libpython, so it would not catch someone re-adding
    `auto-initialize` to `[dependencies]` — the failure would only resurface
    downstream. The guard was mutation-tested: it fails when the feature is
    re-added and passes on the fix.
    
    All 36 existing unit tests plus the 2 new tests pass; `cargo +nightly
    fmt --check` is clean and clippy reports no new warnings.
    
    🤖 Generated with [Claude Code](https://claude.com/claude-code)
    
    https://claude.ai/code/session_01MAo54vCMnUhYQJUA82ueEw
    
    <!-- CURSOR_SUMMARY -->
    ---
    
    > [!NOTE]
    > **Low Risk**
    > Build and test wiring only; shipped wheel behavior is unchanged
    because the extension always runs under an already-initialized
    interpreter.
    > 
    > **Overview**
    > Fixes **#891** by moving pyo3's **`auto-initialize`** feature out of
    **`[dependencies]`** and into **`[dev-dependencies]`**, with manifest
    comments explaining that the cdylib extension never starts an
    interpreter but manylinux **`--disable-shared`** builds fail when the
    feature is on the main dependency.
    > 
    > **`cargo test`** still gets **`auto-initialize`** via the
    dev-dependency so tests that call **`Python::attach`** without a running
    interpreter keep working; wheel/cdylib builds should no longer hit
    pyo3's static-Python build-script error.
    > 
    > Adds **`hf_xet/tests/pyo3_features.rs`** to parse **`Cargo.toml`** and
    assert **`auto-initialize`** is absent from **`[dependencies]`** and
    present under **`[dev-dependencies]`**, since release CI may not catch a
    regression.
    > 
    > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
    b19a787. Bugbot is set up for automated
    code reviews on this repo. Configure
    [here](https://www.cursor.com/dashboard/bugbot).</sup>
    <!-- /CURSOR_SUMMARY -->
    
    Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
    rajatarya and claude authored Aug 3, 2026
    Configuration menu
    Copy the full SHA
    f3d85bd View commit details
    Browse the repository at this point in the history
  2. Local simulation test for v2 shards upload API (#910)

    Adds local simulation coverage for the `POST /v2/shards` NDJSON
    streaming endpoint, following our established pattern of exercising new
    CAS-facing APIs against the in-process `LocalServer`/`LocalTestServer`
    harness rather than only unit-testing client-side parsing logic.
    seanses authored Aug 3, 2026
    Configuration menu
    Copy the full SHA
    1284224 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    0f689ef View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    de71453 View commit details
    Browse the repository at this point in the history
Loading