Repository navigation
Releases
[hf-xet v1.8.0]: Direct-to-S3 xorb uploads (experimental)
✨ New Features
- Experimental direct-to-S3 xorb uploads: xorbs are uploaded straight to S3 through short-lived upload grants issued by CAS, which validates each xorb before storing it. The server is enabling this gradually; other uploads use the existing path (by @assafvayner in #1004 #1005 #1008 #1029)
- Firewall note: allow
s3.us-east-1.amazonaws.com(ands3.eu-west-3.amazonaws.comfor repos stored in the EU). If S3 can't be reached, uploads fall back to the existing path. - Set
HF_XET_CLIENT_LEGACY_DIRECT_XORB_UPLOAD=1to always use the existing path.
- Firewall note: allow
🐛 Bug Fixes
- Downloads no longer request reconstruction information past the end of the file if file size is known ahead (by @seanses in #995)
- Fix a partially written file being published after a write error (by @k1nkreet in #1067) - 1st contribution, thanks!
Full Changelog: v1.7.1...v1.8.0
[hf-xet v1.7.1]: Improved dedupe in long upload session
Improvements and bug fixes
- Fix a bug that dedup silently stops once the in-memory chunk index reaches size limit, this bug previously lead a long upload session by MicroAGI to slowdown dramatically.
- Fix a bug that commit and download-group construction errors, such as a token-refresh 403 (bad CAS token refresh URL or providing a HF_TOKEN with nonmatching access), cause all subsequent group constructions to fail.
- Respect
HF_XET_MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES(aliased asHF_XET_DATA_MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES) that controls minimum number of chunks to wait for between generating global dedup queries. - Simulation and comment fixes, dependency bumps.
Full Changelog: v1.7.0...v1.7.1
[hf-xet v1.7.0]: Telemetry, Dynamic Download Buffers, and fixes
✨ New Features and Improvements
- Added client-side telemetry reports for each upload/download, see docs here (by @sirahd, @rajatarya in #932 #933 #934 #935 #936 #937 #987)
- Utilizing dynamic download buffers based on available RAM in machine/container, see docs here (by @rajatarya in #943)
🐛 Bug Fixes and Enhancements
- Release the GIL while waiting for the next stream chunk (by @hanouticelina in #990) - 1st contribution, thanks!
- Fix double-counting when reducing fragmentation (by @jgodlew in #931)
- Cap untrusted shard memory allocations (by @seanses in #941)
- Bump dependencies (by @rajatarya @sirahd in #950 #963 #954)
- Simulation enhancements (by @sirahd in #958 #962)
- agents.md file for agent-based contribution guidance (by @seanses in #985)
- Support rustls-no-provider TLS feature (by @valkum in #957) - 1st contribution, thanks!
New Contributors
- @valkum made their first contribution in #957
- @hanouticelina made their first contribution in #990
Full Changelog: v1.6.0...v1.7.0
[hf-xet v1.6.0] Stream shard finalization progress
Improvements
- Stream shard finalization progress via
POST /v2/shards, so large-shard validation/commit is visible instead of blocking silently after upload - Expose shard upload progress (validation, commit stages, completion) through upload progress tracking, including Python ShardUploadProgressReport
- Minor upload efficiency: avoid cloning xorb bytes before streaming
- Reduce xet_session future size to avoid compilation failure and stack overflows
- Allow selecting the reqwest TLS backend via hf-xet features
- Drop pyo3
auto-initializefeature to avoid build failures when that feature is combined with a Python built --disable-shared
New Contributors
Full Changelog: v1.5.2...v1.6.0
[hf-xet v1.5.2] Fix HTTP Request Retry
Problems addressed
- This release fixes a bug in HTTP request retry configuration that could cause apparent hangs under poor network conditions when a request was retried more than twice. Related PR: #893
New Contributors
Full Changelog: v1.5.1...v1.5.2
[hf-xet v1.5.1] Progress Reporting Improvement
Problems addressed
-
Missing final progress update — Progress was only emitted on a fixed poll interval, so the last snapshot could be skipped when upload/download finished between ticks (e.g. UI stuck below 100%).
-
Panic on shutdown — The background progress thread used Python::attach while the interpreter was tearing down, which can panic when the GIL is no longer available.
Full Changelog: 1.5.0...v1.5.1
[hf-xet v1.5.0] Session based API
Replaces the old upload_files / download_files / hash_files Python functions with a new object-oriented API that exposes XetSession and its child objects directly as PyO3 classes. This gives Python callers full control over session lifecycle, connection pooling, and progress reporting.
The previous module-level functions are kept under hf_xet/src/legacy/ and remain importable as from hf_xet import upload_files etc., but now emit DeprecationWarning.
New Python API
import hf_xet
# Optional: create a custom config (immutable; use .with_config() to derive updates)
config = hf_xet.XetConfig().with_config("data.max_concurrent_file_ingestion", 8)
# Create session; config is optional (defaults to XetConfig() with HF_XET_* env overrides)
session = hf_xet.XetSession(config=config)
# Upload — multiple files, bytes, and streaming within one commit
with session.new_upload_commit(
endpoint="https://cas.xethub.hf.co",
token="jwt", token_expiry_unix_secs=9999999999,
token_refresh_url="https://…/xet-write-token/main",
token_refresh_headers={"Authorization": "Bearer hf_…"},
) as commit:
h1 = commit.start_upload_file("/path/to/model.bin")
h2 = commit.start_upload_file("/path/to/tokenizer.json", sha256="f2358d9a…")
h3 = commit.start_upload_bytes(b"...", name="config.json")
with commit.start_upload_stream(name="big.bin") as stream:
for chunk in produce_chunks():
stream.write(chunk)
# on normal exit: wait_to_finish() is called automatically
# on exception: abort() is called automatically
# SHA-256 sentinels
commit.start_upload_file("/path/to/model.bin", sha256=hf_xet.COMPUTE_SHA256) # default
commit.start_upload_file("/path/to/model.bin", sha256=hf_xet.SKIP_SHA256) # skip
# Progress callback — receives (GroupProgressReport, dict[UniqueID, ItemProgressReport])
def on_progress(group, items):
bar.n = group.total_bytes_completed
bar.refresh()
with session.new_upload_commit(
token_refresh_url="https://…/xet-write-token/main",
token_refresh_headers={"Authorization": "Bearer hf_…"},
progress_callback=on_progress,
progress_interval_ms=100,
) as commit:
commit.start_upload_file("/path/to/model.bin")
# File download — multiple files within one group (downloads run concurrently)
file_info_a = hf_xet.XetFileInfo(hash_a, size_a)
file_info_b = hf_xet.XetFileInfo(hash_b, size_b)
with session.new_file_download_group(
token_refresh_url="https://…/xet-read-token/main",
token_refresh_headers={"Authorization": "Bearer hf_…"},
) as group:
group.start_download_file(file_info_a, dest_path_a)
group.start_download_file(file_info_b, dest_path_b)
# Streaming download — start/end are both optional (open-ended ranges supported)
group = session.new_download_stream_group(
token_refresh_url="https://…/xet-read-token/main",
token_refresh_headers={"Authorization": "Bearer hf_…"},
)
for chunk in group.download_stream(file_info): # whole file
f.write(chunk)
for chunk in group.download_stream(file_info, start=1024): # 1024 .. EOF
f.write(chunk)
for chunk in group.download_stream(file_info, start=0, end=4096): # 0 .. 4096
f.write(chunk)
for offset, chunk in group.download_unordered_stream(file_info):
buf[offset:offset+len(chunk)] = chunk
# Ctrl-C handling
try:
with session.new_upload_commit(...) as commit:
commit.start_upload_file("model.bin")
except KeyboardInterrupt:
session.sigint_abort()
raiseFull Changelog: v1.4.3...1.5.0
[hf-xet v1.4.3] File Truncation Hotfix
Bug Fixes and Enhancements
Full Changelog: v1.4.2...v1.4.3
[hf-xet v1.4.2]: Performance Optimizations and critical fixes
This release includes significant improvements in performance and critical fixes in both uploading and downloading large files. It is strongly recommended to upgrade to this release.
Note: v1.4.2 replaces v1.4.1 which had a packaging issue (version bump was merged after the release was built). The code is identical.
New Features and Improvements
- Upload optimization avoids unnecessary duplicate SHA-256 computation (@XciD in #679 and #705)
- Download buffer memory is now scaled by active downloads (@hoytak in #666)
Bug Fixes and Enhancements
- Resolve potential deadlock / stall on large file downloads (@XciD in #698)
- Shard Upload extended timeout (was silently timing out on large uploads previously) (@rajatarya in #685)
- Improvements to monitoring and simulation/testing (@seanses in #617, @hoytak in #681)
What's Changed
- Scale download buffer memory limit by number of active downloads by @hoytak in #666
- Feature to monitor client process system usage by @seanses in #617
- Streaming data writer by @hoytak in #656
- Fix command injection in release workflow (CVE) by @XciD in #677
- feat: accept pre-computed SHA-256 in upload_files() by @XciD in #678
- XetSession API by @seanses in #657
- Naming clarification: A Xorb is a data object, CAS is the remote server. by @hoytak in #680
- Fix for incorrect error propagation on truncated download stream. by @hoytak in #683
- Simulation interface for LocalTestServer: supports deletion, direct access, data dumps, etc. by @hoytak in #681
- Rework simulation pipeline for adaptive concurrency and connection resiliency. by @hoytak in #648
- feat: add skip_sha256 option to SingleFileCleaner by @XciD in #679
- fix: prevent download stall on large file reconstruction by @XciD in #698
- fix: per-request timeout for shard uploads (XET-885) by @rajatarya in #685
- version bump to 1.4.0 by @rajatarya in #699
- Code reorganization towards release of xet cargo package by @hoytak in #693
- Record API changes in api_changes/updates__.md by @hoytak in #689
- Rework the interface for session task to get result from registered upload by @seanses in #690
- feat: expose skip_sha256 parameter in Python upload API by @XciD in #705
- version bump to 1.4.1 by @rajatarya in #707
- chore: version bump to 1.4.2 by @XciD in #712
Full Changelog: v1.3.2...v1.4.2
[hf-xet v1.4.1]: Performance Optimizations and critical fixes
This release includes significant improvements in performance and critical fixes in both uploading and downloading large files. It is strongly recommended to upgrade to this release.
✨ New Features and Improvements
- Upload optimization avoids unnecessary duplicate SHA-256 computation (@XciD in #679 and #705)
- Download buffer memory is now scaled by active downloads (@hoytak in #666)
🐛 Bug Fixes and Enhancements
- Resolve potential deadlock / stall on large file downloads (@XciD in #698)
- Shard Upload extended timeout (was silently timing out on large uploads previously) (@rajatarya in #685)
- Improvements to monitoring and simulation/testing (@seanses in #617, @hoytak in #681)
What's Changed
- Scale download buffer memory limit by number of active downloads by @hoytak in #666
- Feature to monitor client process system usage by @seanses in #617
- Streaming data writer by @hoytak in #656
- Fix command injection in release workflow (CVE) by @XciD in #677
- feat: accept pre-computed SHA-256 in upload_files() by @XciD in #678
- XetSession API by @seanses in #657
- Naming clarification: A Xorb is a data object, CAS is the remote server. by @hoytak in #680
- Fix for incorrect error propagation on truncated download stream. by @hoytak in #683
- Simulation interface for LocalTestServer: supports deletion, direct access, data dumps, etc. by @hoytak in #681
- Rework simulation pipeline for adaptive concurrency and connection resiliency. by @hoytak in #648
- feat: add skip_sha256 option to SingleFileCleaner by @XciD in #679
- fix: prevent download stall on large file reconstruction by @XciD in #698
- fix: per-request timeout for shard uploads (XET-885) by @rajatarya in #685
- version bump to 1.4.0 by @rajatarya in #699
- Code reorganization towards release of xet cargo package by @hoytak in #693
- Record API changes in api_changes/updates__.md by @hoytak in #689
- Rework the interface for session task to get result from registered upload by @seanses in #690
- feat: expose skip_sha256 parameter in Python upload API by @XciD in #705
Full Changelog: v1.3.2...v1.4.1