Skip to content

Releases

[hf-xet v1.8.0]: Direct-to-S3 xorb uploads (experimental)

Choose a tag to compare

@github-actions github-actions released this 30 Oct 10:00
e551f0a

✨ New Features

  • Experimental direct-to-S3 xorb uploads: xorbs are uploaded straight to S3 through short-lived upload grants issued by CAS, which validates each xorb before storing it. The server is enabling this gradually; other uploads use the existing path (by @assafvayner in #1004 #1005 #1008 #1029)
    • Firewall note: allow s3.us-east-1.amazonaws.com (and s3.eu-west-3.amazonaws.com for repos stored in the EU). If S3 can't be reached, uploads fall back to the existing path.
    • Set HF_XET_CLIENT_LEGACY_DIRECT_XORB_UPLOAD=1 to always use the existing path.

🐛 Bug Fixes

  • Downloads no longer request reconstruction information past the end of the file if file size is known ahead (by @seanses in #995)
  • Fix a partially written file being published after a write error (by @k1nkreet in #1067) - 1st contribution, thanks!

Full Changelog: v1.7.1...v1.8.0

[hf-xet v1.7.1]: Improved dedupe in long upload session

Choose a tag to compare

@github-actions github-actions released this 28 Oct 15:06
f466508

Improvements and bug fixes

  • Fix a bug that dedup silently stops once the in-memory chunk index reaches size limit, this bug previously lead a long upload session by MicroAGI to slowdown dramatically.
  • Fix a bug that commit and download-group construction errors, such as a token-refresh 403 (bad CAS token refresh URL or providing a HF_TOKEN with nonmatching access), cause all subsequent group constructions to fail.
  • Respect HF_XET_MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES (aliased as HF_XET_DATA_MIN_SPACING_BETWEEN_GLOBAL_DEDUP_QUERIES) that controls minimum number of chunks to wait for between generating global dedup queries.
  • Simulation and comment fixes, dependency bumps.

Full Changelog: v1.7.0...v1.7.1

[hf-xet v1.7.0]: Telemetry, Dynamic Download Buffers, and fixes

Choose a tag to compare

@github-actions github-actions released this 06 Oct 20:11
4b48571

✨ New Features and Improvements

🐛 Bug Fixes and Enhancements

New Contributors

Full Changelog: v1.6.0...v1.7.0

[hf-xet v1.6.0] Stream shard finalization progress

Choose a tag to compare

@github-actions github-actions released this 03 Aug 22:31
de71453

Improvements

  • Stream shard finalization progress via POST /v2/shards, so large-shard validation/commit is visible instead of blocking silently after upload
  • Expose shard upload progress (validation, commit stages, completion) through upload progress tracking, including Python ShardUploadProgressReport
  • Minor upload efficiency: avoid cloning xorb bytes before streaming
  • Reduce xet_session future size to avoid compilation failure and stack overflows
  • Allow selecting the reqwest TLS backend via hf-xet features
  • Drop pyo3 auto-initialize feature to avoid build failures when that feature is combined with a Python built --disable-shared

New Contributors

Full Changelog: v1.5.2...v1.6.0

[hf-xet v1.5.2] Fix HTTP Request Retry

Choose a tag to compare

@github-actions github-actions released this 16 Jul 17:26
cb96dfe

Problems addressed

  • This release fixes a bug in HTTP request retry configuration that could cause apparent hangs under poor network conditions when a request was retried more than twice. Related PR: #893

New Contributors

Full Changelog: v1.5.1...v1.5.2

[hf-xet v1.5.1] Progress Reporting Improvement

Choose a tag to compare

@github-actions github-actions released this 08 Jun 22:57
9dc8c18

Problems addressed

  • Missing final progress update — Progress was only emitted on a fixed poll interval, so the last snapshot could be skipped when upload/download finished between ticks (e.g. UI stuck below 100%).

  • Panic on shutdown — The background progress thread used Python::attach while the interpreter was tearing down, which can panic when the GIL is no longer available.

Full Changelog: 1.5.0...v1.5.1

[hf-xet v1.5.0] Session based API

Choose a tag to compare

@github-actions github-actions released this 06 May 06:17
6ed5a00

Replaces the old upload_files / download_files / hash_files Python functions with a new object-oriented API that exposes XetSession and its child objects directly as PyO3 classes. This gives Python callers full control over session lifecycle, connection pooling, and progress reporting.

The previous module-level functions are kept under hf_xet/src/legacy/ and remain importable as from hf_xet import upload_files etc., but now emit DeprecationWarning.

New Python API

import hf_xet

# Optional: create a custom config (immutable; use .with_config() to derive updates)
config = hf_xet.XetConfig().with_config("data.max_concurrent_file_ingestion", 8)

# Create session; config is optional (defaults to XetConfig() with HF_XET_* env overrides)
session = hf_xet.XetSession(config=config)

# Upload — multiple files, bytes, and streaming within one commit
with session.new_upload_commit(
        endpoint="https://cas.xethub.hf.co",
        token="jwt", token_expiry_unix_secs=9999999999,
        token_refresh_url="https://…/xet-write-token/main",
        token_refresh_headers={"Authorization": "Bearer hf_…"},
    ) as commit:
    h1 = commit.start_upload_file("/path/to/model.bin")
    h2 = commit.start_upload_file("/path/to/tokenizer.json", sha256="f2358d9a…")
    h3 = commit.start_upload_bytes(b"...", name="config.json")

    with commit.start_upload_stream(name="big.bin") as stream:
    for chunk in produce_chunks():
        stream.write(chunk)
# on normal exit: wait_to_finish() is called automatically
# on exception:   abort() is called automatically

# SHA-256 sentinels
commit.start_upload_file("/path/to/model.bin", sha256=hf_xet.COMPUTE_SHA256)  # default
commit.start_upload_file("/path/to/model.bin", sha256=hf_xet.SKIP_SHA256)     # skip

# Progress callback — receives (GroupProgressReport, dict[UniqueID, ItemProgressReport])
def on_progress(group, items):
    bar.n = group.total_bytes_completed
    bar.refresh()

with session.new_upload_commit(
        token_refresh_url="https://…/xet-write-token/main",
        token_refresh_headers={"Authorization": "Bearer hf_…"},
        progress_callback=on_progress,
        progress_interval_ms=100,
    ) as commit:
    commit.start_upload_file("/path/to/model.bin")

# File download — multiple files within one group (downloads run concurrently)
file_info_a = hf_xet.XetFileInfo(hash_a, size_a)
file_info_b = hf_xet.XetFileInfo(hash_b, size_b)
with session.new_file_download_group(
        token_refresh_url="https://…/xet-read-token/main",
        token_refresh_headers={"Authorization": "Bearer hf_…"},
    ) as group:
    group.start_download_file(file_info_a, dest_path_a)
    group.start_download_file(file_info_b, dest_path_b)

# Streaming download — start/end are both optional (open-ended ranges supported)
group = session.new_download_stream_group(
    token_refresh_url="https://…/xet-read-token/main",
    token_refresh_headers={"Authorization": "Bearer hf_…"},
)
for chunk in group.download_stream(file_info):                    # whole file
    f.write(chunk)
for chunk in group.download_stream(file_info, start=1024):        # 1024 .. EOF
    f.write(chunk)
for chunk in group.download_stream(file_info, start=0, end=4096): # 0 .. 4096
    f.write(chunk)
for offset, chunk in group.download_unordered_stream(file_info):
    buf[offset:offset+len(chunk)] = chunk

# Ctrl-C handling
try:
    with session.new_upload_commit(...) as commit:
        commit.start_upload_file("model.bin")
except KeyboardInterrupt:
    session.sigint_abort()
    raise

Full Changelog: v1.4.3...1.5.0

[hf-xet v1.4.3] File Truncation Hotfix

Choose a tag to compare

@github-actions github-actions released this 31 Mar 22:39

Bug Fixes and Enhancements

  • fix: truncate local file on full-file download to prevent corruption (@XciD in #764)

Full Changelog: v1.4.2...v1.4.3

[hf-xet v1.4.2]: Performance Optimizations and critical fixes

Choose a tag to compare

@github-actions github-actions released this 13 Mar 06:58
bcce76b

This release includes significant improvements in performance and critical fixes in both uploading and downloading large files. It is strongly recommended to upgrade to this release.

Note: v1.4.2 replaces v1.4.1 which had a packaging issue (version bump was merged after the release was built). The code is identical.

New Features and Improvements

  • Upload optimization avoids unnecessary duplicate SHA-256 computation (@XciD in #679 and #705)
  • Download buffer memory is now scaled by active downloads (@hoytak in #666)

Bug Fixes and Enhancements

  • Resolve potential deadlock / stall on large file downloads (@XciD in #698)
  • Shard Upload extended timeout (was silently timing out on large uploads previously) (@rajatarya in #685)
  • Improvements to monitoring and simulation/testing (@seanses in #617, @hoytak in #681)

What's Changed

  • Scale download buffer memory limit by number of active downloads by @hoytak in #666
  • Feature to monitor client process system usage by @seanses in #617
  • Streaming data writer by @hoytak in #656
  • Fix command injection in release workflow (CVE) by @XciD in #677
  • feat: accept pre-computed SHA-256 in upload_files() by @XciD in #678
  • XetSession API by @seanses in #657
  • Naming clarification: A Xorb is a data object, CAS is the remote server. by @hoytak in #680
  • Fix for incorrect error propagation on truncated download stream. by @hoytak in #683
  • Simulation interface for LocalTestServer: supports deletion, direct access, data dumps, etc. by @hoytak in #681
  • Rework simulation pipeline for adaptive concurrency and connection resiliency. by @hoytak in #648
  • feat: add skip_sha256 option to SingleFileCleaner by @XciD in #679
  • fix: prevent download stall on large file reconstruction by @XciD in #698
  • fix: per-request timeout for shard uploads (XET-885) by @rajatarya in #685
  • version bump to 1.4.0 by @rajatarya in #699
  • Code reorganization towards release of xet cargo package by @hoytak in #693
  • Record API changes in api_changes/updates__.md by @hoytak in #689
  • Rework the interface for session task to get result from registered upload by @seanses in #690
  • feat: expose skip_sha256 parameter in Python upload API by @XciD in #705
  • version bump to 1.4.1 by @rajatarya in #707
  • chore: version bump to 1.4.2 by @XciD in #712

Full Changelog: v1.3.2...v1.4.2

[hf-xet v1.4.1]: Performance Optimizations and critical fixes

Choose a tag to compare

@github-actions github-actions released this 12 Mar 22:29
0fb930c

This release includes significant improvements in performance and critical fixes in both uploading and downloading large files. It is strongly recommended to upgrade to this release.

✨ New Features and Improvements

  • Upload optimization avoids unnecessary duplicate SHA-256 computation (@XciD in #679 and #705)
  • Download buffer memory is now scaled by active downloads (@hoytak in #666)

🐛 Bug Fixes and Enhancements

  • Resolve potential deadlock / stall on large file downloads (@XciD in #698)
  • Shard Upload extended timeout (was silently timing out on large uploads previously) (@rajatarya in #685)
  • Improvements to monitoring and simulation/testing (@seanses in #617, @hoytak in #681)

What's Changed

  • Scale download buffer memory limit by number of active downloads by @hoytak in #666
  • Feature to monitor client process system usage by @seanses in #617
  • Streaming data writer by @hoytak in #656
  • Fix command injection in release workflow (CVE) by @XciD in #677
  • feat: accept pre-computed SHA-256 in upload_files() by @XciD in #678
  • XetSession API by @seanses in #657
  • Naming clarification: A Xorb is a data object, CAS is the remote server. by @hoytak in #680
  • Fix for incorrect error propagation on truncated download stream. by @hoytak in #683
  • Simulation interface for LocalTestServer: supports deletion, direct access, data dumps, etc. by @hoytak in #681
  • Rework simulation pipeline for adaptive concurrency and connection resiliency. by @hoytak in #648
  • feat: add skip_sha256 option to SingleFileCleaner by @XciD in #679
  • fix: prevent download stall on large file reconstruction by @XciD in #698
  • fix: per-request timeout for shard uploads (XET-885) by @rajatarya in #685
  • version bump to 1.4.0 by @rajatarya in #699
  • Code reorganization towards release of xet cargo package by @hoytak in #693
  • Record API changes in api_changes/updates__.md by @hoytak in #689
  • Rework the interface for session task to get result from registered upload by @seanses in #690
  • feat: expose skip_sha256 parameter in Python upload API by @XciD in #705

Full Changelog: v1.3.2...v1.4.1