Repository navigation
Comparing changes
Open a pull request
base repository: huggingface/xet-core
base: v1.5.2
head repository: huggingface/xet-core
compare: v1.6.0
- 15 commits
- 82 files changed
- 6 contributors
Commits on Jul 17, 2026
-
xet_session: box bridged futures to cap type-layout depth (#901)
## Problem Every public `XetSession` async operation (`commit`, `upload_from_path`, `download_file_to_path`, `finish`, `finalize_ingestion`, stream writes, …) routes its work future through the bridge layer: ``` public async API └─ TaskRuntime::bridge_async(_finalizing) (xet_pkg/.../task_runtime.rs) └─ TaskRuntime::run_inner_async └─ XetRuntime::bridge_async (xet_runtime/.../native.rs) ├─ External => Ok(fut.await) ← inlines the deep future └─ OwnedThreadPool => bridge_to_owned(..) ← spawn(), type-erased ``` On the inline-await paths — the `External` arm on native, and `run_inner_async` on wasm (no runtime offload) — the work future (`async move { inner.commit().await }`, i.e. the full `xet_client`/`xet_data` ingestion/CAS chain) is awaited inline, so its entire nested type becomes part of the bridge's state machine and propagates up through every layer into the caller. Both match arms compile unconditionally, so the `External` arm inflates the layout even for `OwnedThreadPool` consumers. For deep operations this can push a consumer's own future past rustc's type-layout query-depth limit (`error: queries overflow the depth limit!`) at the call site — no downstream code change involved, just the depth of what it awaits. ## Fix Box the future on the two inline paths so its deep type is heap-erased instead of embedded: - `XetRuntime::bridge_async` (native) `External` arm → `Ok(Box::pin(fut).await)` - `TaskRuntime::run_inner_async` (wasm) → box `fut` before the `select!` The `OwnedThreadPool` arm already type-erases via `spawn`, so it's untouched (no extra allocation on that path). This collapses the layout depth once for all public async APIs and all consumers, so downstream crates no longer need `Box::pin` at call sites or `#![recursion_limit]` bumps. ## Testing - `cargo build -p hf-xet -p xet-runtime` ✅ - `cargo check --target wasm32-unknown-unknown` (via `wasm/hf_xet_thin_wasm`) ✅ - `cargo clippy -p hf-xet -p xet-runtime` ✅ (clean) - `cargo +nightly fmt` ✅ <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Compile-time typing change with a small extra allocation on External/wasm paths; owned-pool bridging and API semantics are unchanged. > > **Overview** > Public `XetSession` async work flows through bridge layers that used to **inline** very deep futures on certain paths, which could trigger rustc’s `queries overflow the depth limit!` at consumer call sites. > > **`XetRuntime::bridge_async`** on native **External** mode now awaits `Box::pin(fut)` instead of `fut` directly. **`TaskRuntime::run_inner_async`** on **wasm** boxes the work future before the cancellation `select!`, since wasm has no runtime offload. **OwnedThreadPool** bridging is unchanged (still type-erased via `spawn`). > > Behavior and cancellation semantics stay the same; the tradeoff is one heap allocation on External and wasm inline paths so downstream crates don’t need local `Box::pin` or higher recursion limits. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit c26ee6c. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->Configuration menu - View commit details
-
Copy full SHA for e1d1d7d - Browse repository at this point
Copy the full SHA e1d1d7dView commit details -
Use
LazyLockinstead oflazy_static(#900)<!-- CURSOR_SUMMARY --> > [!NOTE] > **Low Risk** > Mechanical std-library migration with preserved lazy-init semantics; the postcard default-features change only affects dependency features, not obvious runtime logic. > > **Overview** > Replaces the **`lazy_static`** crate with **`std::sync::LazyLock`** across the workspace and drops **`lazy_static`** from workspace and crate manifests. > > **`test_configurable_constants!`** in `xet_runtime` now defines env-overridable constants as **`LazyLock`** instead of **`lazy_static!`**, with the same debug-only **`HF_XET_*`** behavior; **`HIGH_PERFORMANCE`** and privilege detection follow the same pattern. Call sites (SIGINT/runtime globals in **`hf_xet`**, profiling session, shard filename regex, **`MAX_CHUNK_SIZE`**, simulation URL reference time, etc.) are updated accordingly, and the duplicate **`MAX_CHUNK_SIZE`** lazy static in **`xet_data`** chunking is removed in favor of the core-structures definition. > > Workspace **`postcard`** is pinned with **`default-features = false`** (still **`alloc`**), which prunes **`heapless`** and related transitive deps from **`Cargo.lock`**. **`FN_CALL_ID`** in the remote CAS client is a plain **`AtomicU64`** static rather than a lazy wrapper. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 0e667fd. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> --------- Co-authored-by: Assaf Vayner <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for 39d109e - Browse repository at this point
Copy the full SHA 39d109eView commit details -
## Summary - bump workspace sha2 dependency to 0.11 - remove the obsolete sha2 asm feature from xet-data - add a canonical byte-order constructor on DataHash/Sha256 and use it for sha2 0.11 digest bytes ## Tests - cargo check -p xet-data - cargo test -p xet-core-structures test_hash_from_be_bytes - cargo test -p xet-data --features simulation sha_tests - cargo fmt --check -p xet-data -p xet-core-structures - git diff --check <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Medium Risk** > Changes how SHA-256 digests are converted into merkle hashes (byte order vs hex round-trip); incorrect mapping would break content verification, though tests cover the path. > > **Overview** > Upgrades the workspace **`sha2`** dependency from **0.10** to **0.11** and refreshes related lockfiles (including **`hf_xet`** and WASM crates). **`sha2` 0.11** no longer ships the **`asm`** / **`sha2-asm`** path, so **`xet-data`** stops enabling **`asm`** on non-Windows and uses plain workspace **`sha2`** everywhere. > > **`Sha256Generator`** now maps digests with **`Sha256::from_be_bytes`** instead of formatting hex and parsing back. **`DataHash`** gains **`from_be_bytes`** (canonical hex byte order → internal **`[u64; 4]`** layout), with a unit test. > > A standalone **`simulation/sha256_bench`** crate plus **`benches/results.md`** document throughput comparing **0.10.9+asm** vs **0.11** to justify dropping **`asm`**. The root workspace **removes `simulation` from `members`** and **excludes** the whole **`simulation`** tree (replacing the narrower **`chunk_cache_bench`** exclude). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit b79ece9. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
Configuration menu - View commit details
-
Copy full SHA for d1b5cdf - Browse repository at this point
Copy the full SHA d1b5cdfView commit details
Commits on Jul 20, 2026
-
Fix all 21 open Dependabot alerts across all lockfiles (#903)
(big line diff is Cargo.locks) Resolves all 21 open Dependabot alerts by bumping vulnerable dependencies across the four affected lockfiles. All changes verified with `cargo check` (workspace, benches, wasm32 target, and hf_xet with the `profiling` feature); no behavior changes outside of one profiling output-dir naming detail noted below. ## Changes ### openssl 0.10.79 → 0.10.81 (GHSA-phqj-4mhp-q6mq, moderate) Lockfile-only; transitive via `native-tls`. In `chunk_cache_bench`, openssl leaves the dependency graph entirely (see sccache change below). Resolves alerts [#167](https://github.com/huggingface/xet-core/security/dependabot/167), [#168](https://github.com/huggingface/xet-core/security/dependabot/168), [#169](https://github.com/huggingface/xet-core/security/dependabot/169). ### rustls-webpki → 0.103.13 (GHSA-82j2-j2ch-gfr8 high, GHSA-pwjx-qhcg-rvj4 moderate, GHSA-xgp8-3hg3-c2mh, GHSA-965h-392x-2mh5 low) Lockfile bump in `wasm/hf_xet_thin_wasm`. In `simulation/chunk_cache_bench` there was a second unfixable copy (0.102.8 via `rustls 0.22 ← redis 0.25 ← opendal 0.48 ← sccache 0.8`), eliminated by bumping `sccache = "0.16"`. Resolves alerts [#112](https://github.com/huggingface/xet-core/security/dependabot/112), [#123](https://github.com/huggingface/xet-core/security/dependabot/123), [#124](https://github.com/huggingface/xet-core/security/dependabot/124), [#128](https://github.com/huggingface/xet-core/security/dependabot/128), [#132](https://github.com/huggingface/xet-core/security/dependabot/132), [#156](https://github.com/huggingface/xet-core/security/dependabot/156), [#159](https://github.com/huggingface/xet-core/security/dependabot/159). ### rand 0.9.2 → 0.9.4, 0.8.5 → 0.8.6 (GHSA-cq8v-f236-94qc, low) Lockfile-only targeted bumps in all four lockfiles; all vulnerable copies were transitive (direct dep is rand 0.10). Resolves alerts [#133](https://github.com/huggingface/xet-core/security/dependabot/133), [#134](https://github.com/huggingface/xet-core/security/dependabot/134), [#135](https://github.com/huggingface/xet-core/security/dependabot/135), [#137](https://github.com/huggingface/xet-core/security/dependabot/137), [#144](https://github.com/huggingface/xet-core/security/dependabot/144), [#154](https://github.com/huggingface/xet-core/security/dependabot/154). ### atty removal via criterion 0.4 → 0.7 (GHSA-g98v-hv3f-hcfr, low, no patched release exists) `atty`'s only parent was `criterion 0.4` (dev/bench-only) in `xet_data` and `chunk_cache_bench`; criterion 0.5+ dropped it. Bench code needed no changes. Resolves alerts [#2](https://github.com/huggingface/xet-core/security/dependabot/2), [#80](https://github.com/huggingface/xet-core/security/dependabot/80). ### protobuf 2.28.0 → 3.7.2 via pprof 0.14 → 0.15 (GHSA-2gh3-rmm4-6rq5, moderate) `protobuf 2.x` has no patched release; `pprof 0.15` moved to protobuf 3.x. Side note: the `profiling` feature of `hf_xet` didn't compile on main (`chrono` was used but never declared); fixed by switching the profile output dir name from a formatted local timestamp to epoch seconds. Resolves alert [#32](https://github.com/huggingface/xet-core/security/dependabot/32). ### tar 0.4.45 → 0.4.46 (GHSA-3pv8-6f4r-ffg2, moderate) Lockfile-only, transitive via sccache. Resolves alert [#172](https://github.com/huggingface/xet-core/security/dependabot/172). ### jsonwebtoken removal via `sccache` default features off (GHSA-h395-gr6q-cpjc, moderate) No fixed version is reachable through the dependency chain (`sccache → opendal → reqsign → jsonwebtoken ^9`; the fix is the semver-major 10.3.0, which nothing accepts yet). However, jsonwebtoken only enters via sccache's cloud-storage backend features, which the benchmark never uses — it only needs `sccache::lru_disk_cache`. Setting `default-features = false` on sccache removes jsonwebtoken, opendal, reqsign, and openssl from `chunk_cache_bench`'s graph entirely (-913 lockfile lines). Resolves alert [#89](https://github.com/huggingface/xet-core/security/dependabot/89). Note: the `chunk_cache_bench` lockfile diff is large because of the sccache upgrade + feature trim, and the lockfile was also stale vs. its manifest (orphaned `csv`/`csv-core` entries removed). <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Changes are dependency and lockfile updates plus dev/bench/profiling-only paths; the only user-visible tweak is profiling output directory naming when `--features profiling` is enabled. > > **Overview** > Closes open Dependabot findings by refreshing **Cargo.lock** files and bumping a few direct dev/bench/profiling dependencies. Production crates mostly pick up transitive fixes (e.g. **openssl**, **rustls-webpki**, **rand**, **crossbeam-epoch**); **criterion** moves **0.4 → 0.7** in `xet_data` and `chunk_cache_bench`, which drops the unmaintained **atty** chain from benchmarks. > > For **`hf_xet`**, optional **`pprof` 0.14 → 0.15** moves the protobuf stack to 3.x and fixes the profiling build; profile output folders under `profiles/` now use **Unix epoch seconds** instead of a `chrono`-formatted local timestamp. > > **`chunk_cache_bench`** upgrades **`sccache` to 0.16** with **`default-features = false`**, trimming cloud-storage deps (e.g. **jsonwebtoken**, **opendal**, **openssl**) from that crate’s graph while still using its disk LRU cache in benches. **`.cargo/audit.toml`** drops several advisory ignores that are no longer needed and narrows the **async-std** ignore rationale. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit c611e36. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> ## cargo-audit cleanup `.cargo/audit.toml` shrinks from 8 ignores to 2. Removed: both atty entries and rand (fixed above), atomic-polyfill (was already gone from the lockfile), crossbeam-epoch (bumped 0.9.18 → 0.9.20, RUSTSEC-2026-0204), and scc (serial_test 3.4 → 3.5 dropped scc entirely, RUSTSEC-2026-0205). The two remaining ignores are rsa (RUSTSEC-2023-0071, Marvin — no fixed release exists; dev-only via russh) and async-std (RUSTSEC-2025-0052 — intentional dev-dep of xet_pkg for non-tokio executor bridge tests). `cargo audit -D warnings` passes.
Configuration menu - View commit details
-
Copy full SHA for ec93b47 - Browse repository at this point
Copy the full SHA ec93b47View commit details -
Remove stale tests and update cargo test command (#905)
Remove stale tests since the test binary tool "x" was moved. Also add "internal-tools" feature to the "cargo test --workspace" command because these the compilation of test binary tools are now gated behind this feature.
Configuration menu - View commit details
-
Copy full SHA for 3f36c21 - Browse repository at this point
Copy the full SHA 3f36c21View commit details -
Configuration menu - View commit details
-
Copy full SHA for 98e9348 - Browse repository at this point
Copy the full SHA 98e9348View commit details -
Configuration menu - View commit details
-
Copy full SHA for 075687d - Browse repository at this point
Copy the full SHA 075687dView commit details
Commits on Jul 23, 2026
-
Stream shard finalization progress via POST /v2/shards (#884)
## Summary Adds client support for `POST /v2/shards`, a streaming NDJSON variant of shard upload that reports server-side finalization progress (validation against xorb metadata, then durable commit to S3/DynamoDB) instead of blocking silently until the whole thing completes. This closes a visibility gap for large shards, where server-side finalization can take a while after the bytes have already been transferred. - `RemoteClient::upload_shard` auto-detects V2 vs V1: it tries `/v2/shards` first and falls back to `/v1/shards` on `404`/`501`, caching the detected version per client (same pattern already used for the reconstruction API). A new `client.shard_api_version` config option (`HF_XET_CLIENT_SHARD_API_VERSION`) can force a specific version with no fallback. - New `ShardUploadEvent` wire type (`Validating`, `Committing{stage}`, `Result`, `Error`) models the NDJSON frames sent by the server. It has a custom, deliberately partial `PartialOrd`/`precede` so the client can safely ignore stale, duplicate, or out-of-order frames instead of naively overwriting already-observed state. - New `shard_upload_v2` module parses the response body frame-by-frame (correctly handling frames split across TCP chunks), invokes an optional progress callback per frame, and resolves once a terminal `Result`/`Error` frame arrives. ## Progress tracking refactor This also reworks the internal progress-tracking types to plug in shard finalization progress alongside existing file/xorb progress: - Added `ShardUploadProgress`: tracks shard byte-transfer progress plus the new server-reported validation/commit counters (`total_shard_validation_entries[_completed]`, `total_shards_uploaded_to_store`, `total_shards_synced`, `total_shards_completed`), keyed per-shard by `UniqueId` so out-of-order/duplicate NDJSON frames can't regress already-recorded state. - Introduced `UploadGroupProgress`, combining file/xorb progress (`GroupProgress`) and the new `ShardUploadProgress` into a single handle now owned by `CompletionTracker` (previously `FileUploadSession` held a bare `GroupProgress` directly). - `CompletionTracker::register_new_file` now creates the per-file `ItemProgressUpdater` internally instead of requiring callers to construct one and pass it in, simplifying call sites in `FileUploadSession`. - Removed two redundant aliases on `ItemProgressUpdater` (`report_bytes_written`, `report_transfer_progress`) in favor of their canonical names (`report_bytes_completed`, `report_transfer_bytes_completed`). ## Other changes - `upload_xorb` now moves the serialized xorb bytes (`mem::take` + `Bytes::from`) instead of cloning them before streaming. - Simulation/local test server and all `Client` impls (`LocalClient`, `MemoryClient`, `RemoteSimulationClient`, `SimulationControlClient`, `LocalTestServer`) updated to thread the new `progress_callback` parameter through `upload_shard`, and the simulation router's `/shards` route was consolidated under the `/v1` nest. - Substantial new unit test coverage: `ShardUploadEvent` serde round-trips and a full `PartialOrd`/`precede` ordering matrix, `ShardUploadProgress::update` behavior under out-of-order/skipped/duplicate frames (including a multi-shard aggregation test), and `shard_upload_v2` NDJSON parsing edge cases (mid-line frame splits, blank lines, missing terminal frame, callback ordering). ## Related PRs CAS server side: huggingface-internal/xetcas#1140 huggingface_hub side: huggingface/huggingface_hub#4478 Resolves client side of huggingface-internal/xetcas#996Configuration menu - View commit details
-
Copy full SHA for 87ca7ad - Browse repository at this point
Copy the full SHA 87ca7adView commit details -
fix: stack overflow in xet-session from unboxed futures (#908)
## Summary `AuthGroupBuilder<T>::build()`'s compiler-generated future was large enough to overflow a thread's default stack on platforms with a small default (e.g. ~524 KiB for non-main threads on macOS). Root cause: `bridge_async`/`bridge_async_finalizing` and their downstream helpers were `async fn`s generic over `F`, so a large future crossing one of these boundaries by value got duplicated across nested generator frames instead of being shared — the same failure mode #901 partially addressed but didn't fully close. **Fix:** box the future once, as early as possible (in `xet_session::TaskRuntime`), and pass the boxed pointer through downstream layers. | | Before | After | |---|---|---| | `AuthGroupBuilder::build()` future size | 23,664 bytes | ~536–800 bytes (see below) | | Min. viable stack (release build) | ~128–256 KiB | ~65.5–66 KiB | One crate boundary further downstream, `xet_runtime::XetRuntime::bridge_async`/`bridge_to_owned` can either re-box or stay generic over `F` — we measured both (`-Zprint-type-sizes`, nightly, debug build): | `AuthGroupBuilder<T>` instantiation | Re-box at `xet_runtime` boundary | Generic (current) | |---|---:|---:| | `download_stream_group` | 536 bytes | 808 bytes | | `file_download_group` | 528 bytes | 800 bytes | | `upload_commit` | 528 bytes | 800 bytes | Both are trivially safe vs. the 23,664-byte pre-fix size. Re-boxing is ~34% smaller and makes that boundary self-defending for any caller, not just `xet_session` — we opted for the simpler single-boxing-point approach instead, accepting the larger (still tiny) future and leaving "box early" as the caller's responsibility. --------- Co-authored-by: di <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for fe93e98 - Browse repository at this point
Copy the full SHA fe93e98View commit details
Commits on Jul 29, 2026
-
Embed input in the DataHash parse Error (#914)
## Summary - Include the failing input in DataHash parse errors so download failures like [#895](#895) surface `Unable to parse string as hex hash value (got '...')` instead of an opaque message, helps debugging. - Extract hex/bytes parse errors into `merklehash/error.rs` as a `thiserror` `DataHashError` enum (`InvalidHex` / `InvalidBytes`), both carrying the bad input for display. This has been our code style. - Wire `CoreError::HashParsing` to `DataHashError` (replacing the unused `Infallible` stub) and map it to `XetError::DataIntegrity`.
Configuration menu - View commit details
-
Copy full SHA for a3f7e87 - Browse repository at this point
Copy the full SHA a3f7e87View commit details -
Allow selecting the reqwest TLS backend via hf-xet features (#917)
requested by users of hf-xet transitively through hf-hub vllm-project/vllm#48575 (comment) ## What Adds feature-based TLS backend selection to the `hf-xet` crate (and `xet-data`), so consumers can switch the reqwest TLS backend from rustls to the platform-native TLS stack. `xet-client` already exposed `rustls-tls` / `native-tls` / `native-tls-vendored` features, and `hf-xet` forwarded the native ones — but since `hf-xet` depended on `xet-client` with default features on, `rustls-tls` was always enabled and could never be opted out of (cargo features are additive). ## Changes - `xet_pkg` (`hf-xet`) and `xet_data`: depend on `xet-client` (and `xet-data`) with `default-features = false` - Both crates re-expose the backend choice as their own features, with `default = ["rustls-tls"]` Usage: ```toml hf-xet = { version = "1.5.4", default-features = false, features = ["native-tls"] } ``` Note: even without `default-features = false`, enabling `native-tls` switches the runtime backend — reqwest 0.13's `TlsBackend::default()` prefers native-tls when both backends are compiled in. The `default-features = false` path additionally drops rustls from the dependency tree entirely. ## Behavior - Default builds unchanged: rustls remains the default backend for all existing consumers (all internal dependency edges use default features). - `Cargo.lock` unchanged — native-tls pins were already present. ## Verification - `cargo check -p hf-xet` → `cargo tree -i rustls` shows rustls (unchanged default) - `cargo check -p hf-xet --no-default-features --features native-tls` compiles; `cargo tree -i rustls` finds no rustls, `-i native-tls` shows native-tls wired via hyper-tls → reqwest - `cargo check --workspace` passes <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Build-time dependency/feature wiring only; default rustls behavior is unchanged for normal consumers. > > **Overview** > **Enables `hf-xet` and `xet-data` consumers to choose the reqwest TLS stack** instead of always pulling in rustls through transitive default features. > > Both crates now depend on `xet-client` (and `hf-xet` on `xet-data`) with **`default-features = false`**, and re-export **`rustls-tls`**, **`native-tls`**, and **`native-tls-vendored`** with **`default = ["rustls-tls"]`**. `hf-xet`’s TLS features forward to both `xet-client` and `xet-data` so the whole stack stays aligned. > > Default behavior stays rustls for existing dependents. With `default-features = false` and `native-tls` (or vendored), rustls can be dropped from the dependency tree entirely. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 8a20af9. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY -->
Configuration menu - View commit details
-
Copy full SHA for 7a26750 - Browse repository at this point
Copy the full SHA 7a26750View commit details
Commits on Aug 3, 2026
-
fix: scope pyo3 auto-initialize to dev-dependencies (#891) (#923)
Fixes #891. ## Problem `hf_xet` enables pyo3's `auto-initialize` feature. pyo3's build script hard-fails when that feature is combined with a Python built `--disable-shared`, which is the standard configuration for manylinux build environments: ``` error: The `auto-initialize` feature is not supported when linking Python statically instead of with a shared library. ``` The check has no `extension-module` carve-out, so it fires even though we build as an extension module. This blocks downstream builds of `huggingface-hub` and anything above it. ## Why the suggested one-line fix is insufficient The issue suggests simply dropping the feature. That breaks `cargo test` for this crate: ``` test result: FAILED. 10 passed; 26 failed panicked at pyo3-0.29.0/src/interpreter_lifecycle.rs:134: assertion `left != right` failed: The Python interpreter is not initialized and the `auto-initialize` feature is not enabled. ``` 26 of the 36 unit tests call `Python::attach` from a plain Rust test binary, where no interpreter is running yet. pyo3 has an escape hatch for exactly this case, but it is gated on `CARGO_PRIMARY_PACKAGE`, which Cargo sets only when pyo3 itself is the package under test — it never applies to a downstream crate. ## Fix Keep the feature, but scope it to test builds: ```toml [dependencies] pyo3 = { version = "0.29", features = ["abi3-py38"] } [dev-dependencies] pyo3 = { version = "0.29", features = ["auto-initialize"] } ``` Cargo's v2/v3 feature resolver only unifies dev-dependency features when test targets are built, so the cdylib build never sees it. Verified by reading the resolved feature set on pyo3's `custom-build` unit — the unit that actually runs the build-script check — via `cargo build --unit-graph`: | Build | `auto-initialize` | | --- | --- | | `cargo build --features extension-module` (how maturin builds it) | absent | | `cargo build` | absent | | `cargo test` | present | ## Impact **No functional change to the shipped wheel.** Both variants go through the same one-shot `Once` in pyo3: with the feature, the first `Python::attach` calls `Py_InitializeEx` only `if Py_IsInitialized() == 0`; without it, the first call asserts `Py_IsInitialized() != 0`. An extension module always has a live interpreter by the time our code runs, so that first call is a no-op either way and every later call short-circuits on the completed `Once`. No new dependencies, and `Cargo.lock` is unchanged — feature selection is not recorded there. ## Tests Adds `hf_xet/tests/pyo3_features.rs`, asserting the manifest invariant directly. Release CI builds wheels against interpreters that *do* expose a shared libpython, so it would not catch someone re-adding `auto-initialize` to `[dependencies]` — the failure would only resurface downstream. The guard was mutation-tested: it fails when the feature is re-added and passes on the fix. All 36 existing unit tests plus the 2 new tests pass; `cargo +nightly fmt --check` is clean and clippy reports no new warnings. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01MAo54vCMnUhYQJUA82ueEw <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Low Risk** > Build and test wiring only; shipped wheel behavior is unchanged because the extension always runs under an already-initialized interpreter. > > **Overview** > Fixes **#891** by moving pyo3's **`auto-initialize`** feature out of **`[dependencies]`** and into **`[dev-dependencies]`**, with manifest comments explaining that the cdylib extension never starts an interpreter but manylinux **`--disable-shared`** builds fail when the feature is on the main dependency. > > **`cargo test`** still gets **`auto-initialize`** via the dev-dependency so tests that call **`Python::attach`** without a running interpreter keep working; wheel/cdylib builds should no longer hit pyo3's static-Python build-script error. > > Adds **`hf_xet/tests/pyo3_features.rs`** to parse **`Cargo.toml`** and assert **`auto-initialize`** is absent from **`[dependencies]`** and present under **`[dev-dependencies]`**, since release CI may not catch a regression. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit b19a787. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: Claude Opus 5 (1M context) <[email protected]>
Configuration menu - View commit details
-
Copy full SHA for f3d85bd - Browse repository at this point
Copy the full SHA f3d85bdView commit details -
Local simulation test for v2 shards upload API (#910)
Adds local simulation coverage for the `POST /v2/shards` NDJSON streaming endpoint, following our established pattern of exercising new CAS-facing APIs against the in-process `LocalServer`/`LocalTestServer` harness rather than only unit-testing client-side parsing logic.
Configuration menu - View commit details
-
Copy full SHA for 1284224 - Browse repository at this point
Copy the full SHA 1284224View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0f689ef - Browse repository at this point
Copy the full SHA 0f689efView commit details -
Configuration menu - View commit details
-
Copy full SHA for de71453 - Browse repository at this point
Copy the full SHA de71453View commit details
This comparison is taking too long to generate.
Unfortunately it looks like we can’t render this comparison for you right now. It might be too big, or there might be something weird with your repository.
You can try running this command locally to see the comparison on your machine:
git diff v1.5.2...v1.6.0