Describe the bug
Dataset.read_blobs(..., preserve_order=True) raises an internal RuntimeError whenever the last requested address(es) resolve to blobs with no payload bytes — a valid size-0 (empty) blob or a null blob:
RuntimeError: Encountered internal error. Please file a bug report at
https://github.com/lance-format/lance/issues. planned blob read stream
completed before selection index 1 was produced,
rust/lance/src/dataset/blob.rs:1724:35
The panic fires because the planned read stream schedules no bytes for those rows, so the stream completes before the preserve order machinery has produced every selection index.
Related inconsistencies observed on the same inputs:
- With
preserve_order=False there is no panic, but the empty and null rows
are silently omitted from the results.
- The same empty blob is returned as
(addr, b"") when a payload read
happens to be scheduled after it in the plan (e.g. requesting
[payload, empty, payload]), so whether an empty row appears in the results
depends on plan ordering rather than the row itself.
This looks like the read_blobs-by-address analogue of #7716 (fixed for scans by #7717): empty values schedule no reads, but result assignment still expects every selection index to be produced.
Reproduction
import tempfile
import lance
import pyarrow as pa
tmp = tempfile.mkdtemp()
table = pa.Table.from_arrays(
[
pa.array([0, 1, 2, 3], type=pa.int32()),
lance.blob_array([b"hello", b"", None, b"world"]),
],
schema=pa.schema([pa.field("id", pa.int32()), lance.blob_field("b")]),
)
ds = lance.write_dataset(table, f"{tmp}/t.lance", data_storage_version="2.2")
addrs = ds.to_table(columns=[], with_row_address=True)["_rowaddr"].to_pylist()
hello, empty, null, world = addrs
def show(label: str, addresses: list[int], preserve_order: bool) -> None:
try:
pairs = ds.read_blobs("b", addresses=addresses, preserve_order=preserve_order)
print(f"{label}: {[(int(a), bytes(p)) for a, p in pairs]}")
except Exception as e:
print(f"{label}: {type(e).__name__}: {e}")
# Panics: the last scheduled read(s) have no payload bytes.
show("[payload, empty] preserve_order=True ", [hello, empty], True)
show("[payload, empty, null] preserve_order=True ", [hello, empty, null], True)
show("[empty] preserve_order=True ", [empty], True)
# No panic, but silently omits the empty and null rows instead:
show("[payload, empty, null] preserve_order=False", [hello, empty, null], False)
# Inconsistent: the SAME empty blob IS returned as (addr, b"") when a payload
# read happens to be scheduled after it in the plan.
show("[payload, empty, payload] preserve_order=True", [hello, empty, world], True)
Observed output (pylance 9.1.0-beta.4):
[payload, empty] preserve_order=True : RuntimeError: ... planned blob read stream completed before selection index 1 was produced, .../rust/lance/src/dataset/blob.rs:1724:35
[payload, empty, null] preserve_order=True : RuntimeError: ... planned blob read stream completed before selection index 1 was produced, .../rust/lance/src/dataset/blob.rs:1724:35
[empty] preserve_order=True : RuntimeError: ... planned blob read stream completed before selection index 0 was produced, .../rust/lance/src/dataset/blob.rs:1724:35
[payload, empty, null] preserve_order=False: [(0, b'hello')]
[payload, empty, payload] preserve_order=True: [(0, b'hello'), (1, b''), (3, b'world')]
Expected behavior
No internal panic, and consistent semantics for empty/null addresses that do not depend on preserve_order or plan ordering. Ideally read_blobs returns one pair per requested address — (addr, b"") for a valid size-0 descriptor and (addr, None) (or a documented omission) for a null descriptor.
Impact
Any by-address blob read over rows that may contain empty or null values can hit the panic in production. We hit it in Geneva's carry-forward path, which re-reads existing blob columns by row address; callers can work around it by consulting descriptors first and only requesting addresses with size > 0, but the panic is easy to trigger for anyone using the API naively.
Environment
- pylance 9.1.0-beta.4
- python 3.12, macOS (darwin 25.5)
- dataset written with
data_storage_version="2.2"
Describe the bug
Dataset.read_blobs(..., preserve_order=True)raises an internalRuntimeErrorwhenever the last requested address(es) resolve to blobs with no payload bytes — a valid size-0 (empty) blob or a null blob:The panic fires because the planned read stream schedules no bytes for those rows, so the stream completes before the preserve order machinery has produced every selection index.
Related inconsistencies observed on the same inputs:
preserve_order=Falsethere is no panic, but the empty and null rowsare silently omitted from the results.
(addr, b"")when a payload readhappens to be scheduled after it in the plan (e.g. requesting
[payload, empty, payload]), so whether an empty row appears in the resultsdepends on plan ordering rather than the row itself.
This looks like the
read_blobs-by-address analogue of #7716 (fixed for scans by #7717): empty values schedule no reads, but result assignment still expects every selection index to be produced.Reproduction
Observed output (pylance 9.1.0-beta.4):
Expected behavior
No internal panic, and consistent semantics for empty/null addresses that do not depend on
preserve_orderor plan ordering. Ideallyread_blobsreturns one pair per requested address —(addr, b"")for a valid size-0 descriptor and(addr, None)(or a documented omission) for a null descriptor.Impact
Any by-address blob read over rows that may contain empty or null values can hit the panic in production. We hit it in Geneva's carry-forward path, which re-reads existing blob columns by row address; callers can work around it by consulting descriptors first and only requesting addresses with
size > 0, but the panic is easy to trigger for anyone using the API naively.Environment
data_storage_version="2.2"