A blob-encoded large_binary column written through the raw LanceFileWriter at version="2.1" or "2.2" cannot round-trip empty (zero-length) values, and its null descriptors leak packed rep/def levels through the dataset-level descriptor read.

Root cause (2.1+ structural blob encoding, rust/lance-encoding/src/encodings/logical/primitive/blob.rs):

• The writer stores a null as {position: (def << 16) | rep, size: 0} (rep/def levels smuggled into position) and a valid empty value as {position: 0, size: 0}.
• BlobPageScheduler::schedule_ranges schedules a read only for size > 0 rows. The load task then assigns read results to every blob with def == 0 — but a valid empty value also has def == 0 and scheduled no read. It consumes the next blob's read result; at the end of the page the iterator runs dry and bytes_iter.next().expect_ok() fails with Expected option to have value. Depending on ordering this is a loud crash or silent misassignment of payload bytes to the wrong rows.
• Any file containing at least one empty blob value is affected. Null-only files read back fine (e.g. [b"B", None] round-trips).
• Separately, reading such a file's blob column at the dataset descriptor level surfaces the null rows' raw packed descriptor — e.g. {position: 65536, size: 0} for a top-level null (def=1) — indistinguishable from an empty payload, so nullness is silently lost to descriptor-level consumers (files written by lance.write_dataset store the {position: 1, size: 0} null marker instead).

Repro

import os, tempfile
import lance.file
import pyarrow as pa

schema = pa.schema(
    [pa.field("b", pa.large_binary(), metadata={b"lance-encoding:blob": b"true"})]
)
batch = pa.record_batch(
    {"b": pa.array([b"B", b"BB", None, b""], pa.large_binary())}, schema=schema
)
for ver in ["2.0", "2.1", "2.2"]:
    p = os.path.join(tempfile.mkdtemp(), "f.lance")
    with lance.file.LanceFileWriter(p, schema, version=ver) as w:
        w.write_batch(batch)
    try:
        t = lance.file.LanceFileReader(p).read_all().to_table()
        vals = [(v.as_py() if v.is_valid else None) for v in t["b"].combine_chunks()]
        print(f"{ver}: read OK {vals}")
    except Exception as e:
        print(f"{ver}: read CRASHED: {str(e)[:110]}")

Output (pylance 9.0.0-beta.19; also reproduces on a build of current main):

2.0: read OK [b'B', b'BB', None, b'']
2.1: read CRASHED: External error: Encountered internal error. Please file a bug report at https://github.com/lance-fo
2.2: read CRASHED: External error: Encountered internal error. Please file a bug report at https://github.com/lance-fo

Bisection over value compositions: every case containing b"" crashes ([b"B", b""], [b""], [None, b""], ...); null-only cases round-trip ([b"B", None], [None]).

Expected

Empty blob values round-trip like they do at 2.0, and a null blob cell is distinguishable from an empty payload at every read level.

Update: dataset-writer files are affected too

The trigger is not limited to the raw LanceFileWriter path — lance.write_dataset routes blob columns through the same structural encoder, so any 2.1+ blob column containing at least one empty value produces a file the structural value reader cannot decode:

ds = lance.write_dataset(
    pa.table({"b": [b"XY", b"", b"Z"]}, schema=schema), uri, data_storage_version="2.1"
)
ds.to_table()          # descriptors fine: [{0,2}, {0,0}, {64,1}]
ds.take_blobs("b", indices=[0, 1, 2])   # fine: [b"XY", b"", b"Z"]
lance.file.LanceFileReader(data_file).read_all()   # CRASH: Expected option to have value

Descriptor-level reads and take_blobs are unaffected; the failure is confined to readers that assemble blob values through BlobPageScheduler (e.g. file-level value reads, blob-as-binary scan paths). One bound worth noting: a page-load shard containing an empty value always over-consumes its read-result iterator (pulls = payloads + empties), so the decode task always fails loudly — the transient in-memory misassignment never surfaces as silently wrong bytes.

Compaction is affected: dataset.optimize.compact_files() on a 2.1 table containing an empty blob value fails with the same internal error (compaction assembles blob values through the structural reader), so affected tables permanently fail maintenance until the read-side fix ships:

ds = lance.write_dataset(pa.table({"b": [b"XY", b"", b"Z"]}, schema=schema), uri,
                         data_storage_version="2.1", max_rows_per_file=1)
ds.optimize.compact_files()   # OSError: ... Expected option to have value