feat(python): add list_with_delimiter, delete_file, and read_range to LanceFileSession#7426
Merged
westonpace merged 8 commits intoJun 23, 2026
Conversation
Expose a blob-only, non-recursive list on LanceFileSession that returns the immediate child prefixes (and objects) of a path, mirroring object_store's list_with_delimiter. Unlike the existing recursive list, this descends exactly one directory level and, on Azure, is served entirely from the blob endpoint (no hierarchical-namespace/DFS probe). - lance-io: add ObjectStore::list_with_delimiter wrapper exposing the underlying ListResult primitive. - python: add LanceFileSession.list_with_delimiter returning a ListResult dataclass (common_prefixes, objects) of session-relative paths. Co-Authored-By: Claude Opus 4.8 <[email protected]>
…-delimited-list-to-lancefilesession
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
jmhsieh
marked this pull request as ready for review
June 23, 2026 18:16
Add a delete primitive to LanceFileSession so callers can remove objects through the object_store session instead of going through PyArrow's AzureFileSystem. The underlying object_store crate already supports delete; this is a binding-exposure change. - python (Rust): add LanceFileSession.delete_file(path), routing to object_store delete with a session-relative path. Deleting a missing object is a no-op (idempotent) rather than an error. - python (wrapper): add the delete_file wrapper and type stub. On Azure this talks to the blob endpoint only and never probes the hierarchical-namespace (DFS) endpoint. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Add a byte-range random read to LanceFileSession so callers can read a slice of a data file through the object_store session instead of going through PyArrow's AzureFileSystem. The underlying object_store crate already supports ranged GETs; this is a binding-exposure change. - python (Rust): add LanceFileSession.read_range(path, offset, length) -> bytes, routing to a single ranged read with a session-relative path. Reading a missing object raises OSError, consistent with download_file. - python (wrapper): add the read_range wrapper and type stub. On Azure this talks to the blob endpoint only and never probes the hierarchical-namespace (DFS) endpoint. Co-Authored-By: Claude Opus 4.8 <[email protected]>
westonpace
requested changes
Jun 23, 2026
westonpace
left a comment
Member
There was a problem hiding this comment.
Looks great. My only concern is all these comments specifically mention Azure but I think this feature is generic to any object storage. Can we just remove these mentions to Azure entirely? No need to specify exactly how the call performs on the backend.
Also, one small question on the delete.
Address review feedback: the new session methods are generic to any object store, so remove the storage-backend-specific notes about the blob endpoint and hierarchical-namespace probing from the list_with_delimiter, delete_file, and read_range docs. Co-Authored-By: Claude Opus 4.8 <[email protected]>
Address review feedback: rather than swallowing the not-found error to make delete idempotent, mirror the object_store API directly and let the error propagate. Deleting a missing path now raises OSError, consistent with download_file and read_range. Co-Authored-By: Claude Opus 4.8 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds three primitives to
LanceFileSession(lance.file), each a binding-exposure of an existingobject_storecapability (Rust → PyO3 → Python). They let callers route listing, deletion, and ranged reads through the dataset'sobject_storesession instead of PyArrow'sAzureFileSystem.APIs added
1.
list_with_delimiter(path=None) -> ListResultNon-recursive, path-segment-delimited list of a single directory level. Unlike the existing recursive
list(), it returns only the immediate children ofpath: child "directories" ascommon_prefixesand direct child files asobjects(aListResultdataclass), all session-relative.lance-io: addObjectStore::list_with_delimiter(prefix) -> Result<ListResult>.2.
delete_file(path) -> NoneDelete an object by session-relative path. Deleting a missing path is a no-op (idempotent) rather than an error, so it is safe on best-effort cleanup paths.
3.
read_range(path, offset, length) -> bytesSingle ranged read of a byte slice from a file. Reading a missing object raises
OSError, consistent withdownload_file.Why this matters
These remove the remaining PyArrow
AzureFileSystem/filesystem_from_uridependencies from geneva's checkpoint and range-blob paths:list()forced the flat-namespace bf-dir enumeration fallback to scan the whole checkpoint subtree;list_with_delimitermakes that fallback bounded again.LanceFileSessionhad no delete, so checkpoint purge fell back to PyArrow and silently became a no-op on flat / unreachable-DFS accounts;delete_filemakes purge functional and blob-only.read_range.Testing
test_session_list_with_delimiter— non-recursive semantics (base level returns immediate files + child dirs but not their contents; subdir descends one level; trailing-slash equivalence; empty for a missing prefix).test_session_delete_file— top-level delete, nested delete, idempotent no-op on missing.test_session_read_range— mid-file range, start-of-file, up-to-end, zero-length read, andOSErroron a missing object.cargo fmt;cargo clippy -- -D warningsclean (default features).🤖 Generated with Claude Code