Skip to content

read_blobs panics with an internal error when trailing addresses resolve to empty or null blobs #7899

Description

@justinrmiller

Describe the bug

Dataset.read_blobs(..., preserve_order=True) raises an internal RuntimeError whenever the last requested address(es) resolve to blobs with no payload bytes — a valid size-0 (empty) blob or a null blob:

RuntimeError: Encountered internal error. Please file a bug report at
https://github.com/lance-format/lance/issues. planned blob read stream
completed before selection index 1 was produced,
rust/lance/src/dataset/blob.rs:1724:35

The panic fires because the planned read stream schedules no bytes for those rows, so the stream completes before the preserve order machinery has produced every selection index.

Related inconsistencies observed on the same inputs:

  • With preserve_order=False there is no panic, but the empty and null rows
    are silently omitted from the results.
  • The same empty blob is returned as (addr, b"") when a payload read
    happens to be scheduled after it in the plan (e.g. requesting
    [payload, empty, payload]), so whether an empty row appears in the results
    depends on plan ordering rather than the row itself.

This looks like the read_blobs-by-address analogue of #7716 (fixed for scans by #7717): empty values schedule no reads, but result assignment still expects every selection index to be produced.

Reproduction

import tempfile

import lance
import pyarrow as pa

tmp = tempfile.mkdtemp()
table = pa.Table.from_arrays(
    [
        pa.array([0, 1, 2, 3], type=pa.int32()),
        lance.blob_array([b"hello", b"", None, b"world"]),
    ],
    schema=pa.schema([pa.field("id", pa.int32()), lance.blob_field("b")]),
)
ds = lance.write_dataset(table, f"{tmp}/t.lance", data_storage_version="2.2")
addrs = ds.to_table(columns=[], with_row_address=True)["_rowaddr"].to_pylist()
hello, empty, null, world = addrs


def show(label: str, addresses: list[int], preserve_order: bool) -> None:
    try:
        pairs = ds.read_blobs("b", addresses=addresses, preserve_order=preserve_order)
        print(f"{label}: {[(int(a), bytes(p)) for a, p in pairs]}")
    except Exception as e:
        print(f"{label}: {type(e).__name__}: {e}")


# Panics: the last scheduled read(s) have no payload bytes.
show("[payload, empty]        preserve_order=True ", [hello, empty], True)
show("[payload, empty, null]  preserve_order=True ", [hello, empty, null], True)
show("[empty]                 preserve_order=True ", [empty], True)

# No panic, but silently omits the empty and null rows instead:
show("[payload, empty, null]  preserve_order=False", [hello, empty, null], False)

# Inconsistent: the SAME empty blob IS returned as (addr, b"") when a payload
# read happens to be scheduled after it in the plan.
show("[payload, empty, payload] preserve_order=True", [hello, empty, world], True)

Observed output (pylance 9.1.0-beta.4):

[payload, empty]        preserve_order=True : RuntimeError: ... planned blob read stream completed before selection index 1 was produced, .../rust/lance/src/dataset/blob.rs:1724:35
[payload, empty, null]  preserve_order=True : RuntimeError: ... planned blob read stream completed before selection index 1 was produced, .../rust/lance/src/dataset/blob.rs:1724:35
[empty]                 preserve_order=True : RuntimeError: ... planned blob read stream completed before selection index 0 was produced, .../rust/lance/src/dataset/blob.rs:1724:35
[payload, empty, null]  preserve_order=False: [(0, b'hello')]
[payload, empty, payload] preserve_order=True: [(0, b'hello'), (1, b''), (3, b'world')]

Expected behavior

No internal panic, and consistent semantics for empty/null addresses that do not depend on preserve_order or plan ordering. Ideally read_blobs returns one pair per requested address — (addr, b"") for a valid size-0 descriptor and (addr, None) (or a documented omission) for a null descriptor.

Impact

Any by-address blob read over rows that may contain empty or null values can hit the panic in production. We hit it in Geneva's carry-forward path, which re-reads existing blob columns by row address; callers can work around it by consulting descriptors first and only requesting addresses with size > 0, but the panic is easy to trigger for anyone using the API naively.

Environment

  • pylance 9.1.0-beta.4
  • python 3.12, macOS (darwin 25.5)
  • dataset written with data_storage_version="2.2"

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions