feat: support list blob scans by Xuanwo · Pull Request #7664 · lance-format/lance

Xuanwo · 2026-07-07T10:10:26Z

Summary

This adds scan-based support for List<Blob> and nested blob leaves such as Struct<List<Blob>>.

Descriptor scans preserve the original Arrow nesting and expose blob leaves as descriptors without loading payload bytes. BlobHandling::AllBinary materializes only blob leaves to binary values while keeping the surrounding List / Struct layout intact.

This keeps the existing one-blob-per-row APIs unchanged and avoids adding a public random-access API for list blob values.

Summary by CodeRabbit

New Features
- Added Blob V2 binary view support for list and nested fields, with correct scan/filtered read/take output schema behavior.
- Improved Blob V2 payload resolution for inline, dedicated, packed, and external sources.
Bug Fixes
- Stricter Blob V2 detection and more consistent conversion/unload behavior.
- Improved projection and schema intersection handling for Blob V2, including nested cases and better type-ignore behavior.
- Preserved nulls, list offsets, and descriptor validity; ensured the binary-view marker is handled consistently.
Tests
- Expanded Blob V2 coverage for binary/view materialization, descriptor projection, and nested list/struct scenarios.

codecov · 2026-07-07T15:15:49Z

Codecov Report

❌ Patch coverage is 81.48148% with 190 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/blob.rs	82.17%	94 Missing and 50 partials ⚠️
rust/lance/src/io/exec/scan.rs	29.41%	36 Missing ⚠️
rust/lance-core/src/datatypes/field.rs	92.22%	6 Missing and 1 partial ⚠️
rust/lance/src/io/exec/take.rs	92.30%	0 Missing and 2 partials ⚠️
rust/lance-core/src/datatypes/schema.rs	94.73%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

# Conflicts: # rust/lance-core/src/datatypes/schema.rs

coderabbitai · 2026-07-10T08:34:48Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 9142a0b7-ac7c-4e75-afb3-79c434f30fef

📥 Commits

Reviewing files that changed from the base of the PR and between 1ca127d and e554285.

📒 Files selected for processing (2)

rust/lance-core/src/datatypes/field.rs
rust/lance/src/dataset/blob.rs

📝 Walkthrough

Walkthrough

Blob V2 handling validates descriptors more strictly, supports binary views for nested list fields, preserves descriptor nulls, and materializes binary payloads during filtered, unfiltered, and take execution paths.

Changes

Blob V2 binary flow

Layer / File(s)	Summary
Blob field handling and projection policy `rust/lance-core/src/datatypes/field.rs`, `rust/lance-core/src/datatypes/schema.rs`	Blob V2 identity checks, binary conversion, projection and intersection handling, binary-loading policies, and unload-transition tests are updated.
Descriptor nulls and list preprocessing `rust/lance-encoding/src/encodings/logical/blob.rs`, `rust/lance/src/dataset/blob.rs`	Descriptor validity mirrors source struct nulls, while list and large-list blob fields are recursively normalized and preprocessed.
Blob V2 descriptor traversal and materialization `rust/lance/src/dataset/blob.rs`	Nested descriptor traversal is validated, payload locations are resolved through a read context, and descriptor arrays and lists are materialized into binary payload arrays.
Scan, filtered read, and take integration `rust/lance/src/io/exec/filtered_read.rs`, `rust/lance/src/io/exec/scan.rs`, `rust/lance/src/io/exec/take.rs`	Execution paths use descriptor schemas and row addresses when needed, materialize binary batches, and expose public binary output schemas.

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ScanPlanner
  participant FragmentReader
  participant BlobV2ReadContext
  participant BlobMaterializer
  ScanPlanner->>FragmentReader: select descriptor schema and row addresses
  FragmentReader->>BlobMaterializer: provide decoded descriptor batch
  BlobMaterializer->>BlobV2ReadContext: resolve descriptor payload locations
  BlobV2ReadContext->>BlobMaterializer: return payload bytes
  BlobMaterializer->>ScanPlanner: return public binary batch

Possibly related issues

Blob v2: filtering on a blob field panics in Field::apply_projection instead of returning a count/error #7707 — The changes update blob projection, unloading, and Blob V2 schema handling related to the reported Field::apply_projection panic.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding scan support for list-shaped blob fields.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

Create stacked PR
Commit on current branch

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch xuanwo/list-blob-scan

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance-core/src/datatypes/field.rs`:
- Around line 603-611: In the relevant field conversion logic, replace the bare
unwrap on LogicalType::try_from(&DataType::LargeBinary) with expect containing a
clear reason, and replace the duplicated "packed" and "lance-encoding:packed"
metadata removals with iteration over the existing PACKED_KEYS constant used by
is_packed_struct.

In `@rust/lance/src/dataset/blob.rs`:
- Around line 2338-2361: Update the blob-v2 filtered-read materialization around
materialize_blob_v2_binary_array and FilteredReadExec schema handling so
requested system columns (_rowid, _row_last_updated_at_version, and
_row_created_at_version) are preserved alongside _rowaddr. Ensure
projection.to_bare_schema() does not drop them, and that the produced columns
and output schema match FilteredReadExec::schema() for every requested system
column.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 02fd15aa-ede3-45ce-859c-38adabfb62c3

📥 Commits

Reviewing files that changed from the base of the PR and between 1946a5a and 3c08f7e.

📒 Files selected for processing (6)

rust/lance-core/src/datatypes/field.rs
rust/lance-core/src/datatypes/schema.rs
rust/lance-encoding/src/encodings/logical/blob.rs
rust/lance/src/dataset/blob.rs
rust/lance/src/io/exec/filtered_read.rs
rust/lance/src/io/exec/scan.rs

BubbleCal

Blob payload reads are serialized across list elements.

BubbleCal · 2026-07-10T09:18:31Z

+    for (idx, row_addr) in row_addrs.iter().copied().enumerate() {
+        if descriptions.is_null(idx) || columns.kinds.is_null(idx) {
+            builder.append_null();
            continue;
        }

-        match kind {
+        let kind = BlobKind::try_from(columns.kinds.value(idx))?;
+        if matches!(kind, BlobKind::Inline)
+            && columns.positions.value(idx) == 0
+            && columns.sizes.value(idx) == 0
+        {
+            builder.append_value([]);
+            continue;
+        }
+
+        let entry = read_context
+            .collect_entry(&columns, idx, idx, row_addr)
+            .await?;
+        if let Some(entry) = entry {
+            let data = entry.file.read().await?;
+            builder.append_value(data.as_ref());


Blob payloads are awaited one at a time here, so a list with N remote blobs needs roughly N sequential round trips and BlobSource cannot coalesce the reads. Filtered scans also pay this cost before applying the residual filter.

lance 9.0.0-rc.1 picked up lance-format/lance#7664, which stopped the blob v2 encoder from emitting a redundant rep/def validity layer. PrimitiveStructuralEncoder already derives validity via extract_validity, so v8 pushed two layers where the packed-struct decoder reads one -- it consumed the bogus AllValidItem at index 0 and the null was gone. The practical effect is that lance <= 8 never round-tripped a null blob at all: it wrote the null away, and every reader (v8 included) saw a non-null zero-filled descriptor. Verified by writing a null blob with lance 8 and reading it back with both versions -- identical zero-filled result, so v9 changes nothing for existing data and there is no read-side compatibility break. Only new writes differ, and they are now correct. So these assertions were encoding the bug: they checked the descriptor was zero-filled, which is precisely the signature of the dropped null, while the test names promised the null was preserved. Assert the null descriptor instead, and add the value-level null check the names always implied but could not pass before. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

feat: support list blob scans

27e2981

github-actions Bot added A-encoding Encoding, IO, file reader/writer enhancement New feature or request labels Jul 7, 2026

Xuanwo marked this pull request as ready for review July 7, 2026 10:15

Xuanwo added 2 commits July 7, 2026 18:15

Merge branch 'main' into xuanwo/list-blob-scan

3bf7ff8

fix: preserve legacy blob descriptor semantics

2149a0a

Merge remote-tracking branch 'origin/main' into xuanwo/list-blob-scan

3c08f7e

# Conflicts: # rust/lance-core/src/datatypes/schema.rs

coderabbitai Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread rust/lance-core/src/datatypes/field.rs Outdated

Comment thread rust/lance/src/dataset/blob.rs Outdated

BubbleCal reviewed Jul 10, 2026

View reviewed changes

fix: address list blob scan review feedback

1ca127d

BubbleCal reviewed Jul 10, 2026

View reviewed changes

Comment thread rust/lance-core/src/datatypes/schema.rs

fix: preserve blob descriptor children in projections

e554285

BubbleCal approved these changes Jul 13, 2026

View reviewed changes

Xuanwo merged commit e7bce1e into main Jul 13, 2026
34 checks passed

Xuanwo deleted the xuanwo/list-blob-scan branch July 13, 2026 09:58

Xuanwo mentioned this pull request Jul 22, 2026

Blob v2: filtering on a blob field panics in Field::apply_projection instead of returning a count/error #7707

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: support list blob scans#7664

feat: support list blob scans#7664
Xuanwo merged 6 commits into
mainfrom
xuanwo/list-blob-scan

Xuanwo commented Jul 7, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

codecov Bot commented Jul 7, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

BubbleCal left a comment •

edited

Loading

Uh oh!

BubbleCal Jul 10, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Xuanwo commented Jul 7, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Summary by CodeRabbit

Uh oh!

codecov Bot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

coderabbitai Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Sequence Diagram(s)

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

BubbleCal left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BubbleCal Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Xuanwo commented Jul 7, 2026 •

edited by coderabbitai Bot

Loading

codecov Bot commented Jul 7, 2026 •

edited

Loading

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading

BubbleCal left a comment •

edited

Loading

BubbleCal Jul 10, 2026 •

edited

Loading