Skip to content

feat(python): add list_with_delimiter, delete_file, and read_range to LanceFileSession#7426

Merged
westonpace merged 8 commits into
lance-format:mainfrom
jmhsieh:jon/add-non-recursive-delimited-list-to-lancefilesession
Jun 23, 2026
Merged

feat(python): add list_with_delimiter, delete_file, and read_range to LanceFileSession#7426
westonpace merged 8 commits into
lance-format:mainfrom
jmhsieh:jon/add-non-recursive-delimited-list-to-lancefilesession

Conversation

@jmhsieh

@jmhsieh jmhsieh commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds three primitives to LanceFileSession (lance.file), each a binding-exposure of an existing object_store capability (Rust → PyO3 → Python). They let callers route listing, deletion, and ranged reads through the dataset's object_store session instead of PyArrow's AzureFileSystem.

APIs added

1. list_with_delimiter(path=None) -> ListResult

Non-recursive, path-segment-delimited list of a single directory level. Unlike the existing recursive list(), it returns only the immediate children of path: child "directories" as common_prefixes and direct child files as objects (a ListResult dataclass), all session-relative.

result = session.list_with_delimiter("subdir")
result.common_prefixes  # immediate child "directories"
result.objects          # immediate child files
  • lance-io: add ObjectStore::list_with_delimiter(prefix) -> Result<ListResult>.

2. delete_file(path) -> None

Delete an object by session-relative path. Deleting a missing path is a no-op (idempotent) rather than an error, so it is safe on best-effort cleanup paths.

session.delete_file("subdir/checkpoint.lance")  # removes it
session.delete_file("already_gone.lance")        # no-op, no error

3. read_range(path, offset, length) -> bytes

Single ranged read of a byte slice from a file. Reading a missing object raises OSError, consistent with download_file.

data = session.read_range("data/file-0.lance", offset=1024, length=256)  # -> bytes

Why this matters

These remove the remaining PyArrow AzureFileSystem / filesystem_from_uri dependencies from geneva's checkpoint and range-blob paths:

  • The recursive-only list() forced the flat-namespace bf-dir enumeration fallback to scan the whole checkpoint subtree; list_with_delimiter makes that fallback bounded again.
  • LanceFileSession had no delete, so checkpoint purge fell back to PyArrow and silently became a no-op on flat / unreachable-DFS accounts; delete_file makes purge functional and blob-only.
  • The range-blob reader's last PyArrow handle is replaced by read_range.

Testing

  • test_session_list_with_delimiter — non-recursive semantics (base level returns immediate files + child dirs but not their contents; subdir descends one level; trailing-slash equivalence; empty for a missing prefix).
  • test_session_delete_file — top-level delete, nested delete, idempotent no-op on missing.
  • test_session_read_range — mid-file range, start-of-file, up-to-end, zero-length read, and OSError on a missing object.
  • All existing session tests pass.
  • cargo fmt; cargo clippy -- -D warnings clean (default features).

🤖 Generated with Claude Code

Expose a blob-only, non-recursive list on LanceFileSession that returns
the immediate child prefixes (and objects) of a path, mirroring
object_store's list_with_delimiter. Unlike the existing recursive list,
this descends exactly one directory level and, on Azure, is served
entirely from the blob endpoint (no hierarchical-namespace/DFS probe).

- lance-io: add ObjectStore::list_with_delimiter wrapper exposing the
  underlying ListResult primitive.
- python: add LanceFileSession.list_with_delimiter returning a ListResult
  dataclass (common_prefixes, objects) of session-relative paths.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@github-actions github-actions Bot added A-python Python bindings A-encoding Encoding, IO, file reader/writer enhancement New feature or request labels Jun 23, 2026
@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 3 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-io/src/object_store.rs 0.00% 3 Missing ⚠️

📢 Thoughts on this report? Let us know!

@jmhsieh
jmhsieh marked this pull request as ready for review June 23, 2026 18:16
jmhsieh and others added 2 commits June 23, 2026 11:48
Add a delete primitive to LanceFileSession so callers can remove objects
through the object_store session instead of going through PyArrow's
AzureFileSystem. The underlying object_store crate already supports
delete; this is a binding-exposure change.

- python (Rust): add LanceFileSession.delete_file(path), routing to
  object_store delete with a session-relative path. Deleting a missing
  object is a no-op (idempotent) rather than an error.
- python (wrapper): add the delete_file wrapper and type stub.

On Azure this talks to the blob endpoint only and never probes the
hierarchical-namespace (DFS) endpoint.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add a byte-range random read to LanceFileSession so callers can read a
slice of a data file through the object_store session instead of going
through PyArrow's AzureFileSystem. The underlying object_store crate
already supports ranged GETs; this is a binding-exposure change.

- python (Rust): add LanceFileSession.read_range(path, offset, length)
  -> bytes, routing to a single ranged read with a session-relative path.
  Reading a missing object raises OSError, consistent with download_file.
- python (wrapper): add the read_range wrapper and type stub.

On Azure this talks to the blob endpoint only and never probes the
hierarchical-namespace (DFS) endpoint.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@jmhsieh jmhsieh changed the title feat(python): add non-recursive delimited list to LanceFileSession feat(python): add list_with_delimiter, delete_file, and read_range to LanceFileSession Jun 23, 2026

@westonpace westonpace left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great. My only concern is all these comments specifically mention Azure but I think this feature is generic to any object storage. Can we just remove these mentions to Azure entirely? No need to specify exactly how the call performs on the backend.

Also, one small question on the delete.

Comment thread python/python/lance/file.py Outdated
Comment thread python/python/lance/file.py Outdated
Comment thread python/python/lance/file.py Outdated
Comment thread python/src/file.rs Outdated
Comment thread python/src/file.rs Outdated
Comment thread python/src/file.rs Outdated
Comment thread python/src/file.rs Outdated
Comment thread rust/lance-io/src/object_store.rs Outdated
jmhsieh and others added 2 commits June 23, 2026 12:37
Address review feedback: the new session methods are generic to any
object store, so remove the storage-backend-specific notes about the
blob endpoint and hierarchical-namespace probing from the
list_with_delimiter, delete_file, and read_range docs.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Address review feedback: rather than swallowing the not-found error to
make delete idempotent, mirror the object_store API directly and let the
error propagate. Deleting a missing path now raises OSError, consistent
with download_file and read_range.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
@jmhsieh
jmhsieh requested a review from westonpace June 23, 2026 19:46

@westonpace westonpace left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@westonpace
westonpace merged commit f3638cc into lance-format:main Jun 23, 2026
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer A-python Python bindings enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants