docs: specify data overlay files for the table format by wjones127 · Pull Request #7381 · lance-format/lance

wjones127 · 2026-06-19T22:56:04Z

Adds a specification for data overlay files: small files attached to a fragment that supply new values for a subset of (row offset, field) cells without rewriting the base data files. They make cell-level updates cheap when only a small fraction of rows and/or columns change.

This PR is spec + proto only — no read/write implementation yet. It is also explicitly experimental. The released libraries will not produce tables with this feature enabled. Once the implementation is done in the library, we will vote on the final design before releasing. This is similar to how we have done file format updates.

Changes

protos/table.proto
- Rework DataOverlayFile: a oneof coverage { bytes shared_offset_bitmap | FieldCoverage field_coverage } to support both dense (rectangular) and sparse overlays; add the FieldCoverage message.
- Rename read_version → committed_version (uint64), with effective/commit-stamped semantics so overlay-vs-index ordering is correct.
- Drop the in-file offset key column in favor of rank-based addressing off the coverage bitmap.
- Document reader feature flag 64 (and previously-undocumented 16/32).
docs/src/format/table/data_overlay_file.md (new): full specification — coverage/resolution, deletion precedence, NULL-override, layout + rank addressing, dense vs. sparse, versioning, field-aware index exclusion with flat re-evaluation, the correctness invariant, both compaction modes, row lineage, a worked example (write → read → index query → sparse write → read → compaction), and a guidance stub with open questions.
docs/src/format/table/index.md: concise overview + link to the new spec (replacing the earlier inline sketch).

Out of scope / follow-ups

Write transaction shape (new Operation variant in transaction.proto + Rust).
Writer support for unequal-length columns (needed for single-file sparse overlays).
Coverage bitmap external spill for very large coverage.
Per-fragment vs. per-table overlays / LSM analogy (open question in the doc).

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- Added documentation for experimental Data Overlay Files, including storage, versioning, querying, compaction, and transaction behavior.
- Added format and transaction schema support for describing data overlays.
- Added navigation links to the new documentation.
Bug Fixes
- Datasets using unsupported data overlays are now explicitly rejected instead of risking stale results.
Documentation
- Expanded guidance on invalidated index results and overlay-related filtering scenarios.

github-actions · 2026-06-19T22:56:13Z

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

Start a vote following the Lance community voting process.
Format specification modifications need 3 binding +1 votes (excluding the
proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
Once the vote passes, link the completed vote in this PR. It should not be
merged until the vote is linked.

wjones127 · 2026-06-22T21:18:20Z

See discussion: #7401

codecov · 2026-06-22T22:07:13Z

Codecov Report

❌ Patch coverage is 85.71429% with 3 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/transaction.rs	82.35%	2 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Adds the in-memory + commit machinery for data overlay files (per the spec in lance-format#7381), the foundation the scanner/take/index/compaction work builds on. - `DataOverlayFile` / `OverlayCoverage` (dense `shared_offset_bitmap` and sparse per-field) with protobuf round-trip, attached to `Fragment.overlays`. - Reader feature flag 64 (`FLAG_DATA_OVERLAY_FILES`): set whenever any fragment carries overlays, so a reader that does not understand them refuses the dataset instead of returning stale base values. - `Operation::DataOverlay` transaction op: appends overlays to a fragment's list (preserving concurrently-written overlays) and stamps each overlay's `committed_version` to the new dataset version at commit time (re-stamped on retry). Conflict rules mirror DataReplacement — permissive against appends, deletes, column rewrites, index builds, and other overlays; conflicts only with row-rewriting compaction of the same fragment. Scan-side merge, take, and end-to-end write+read tests follow in the same PR branch. Part of the Data Overlay Files feature (OSS-1322). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

hamersaw · 2026-06-25T03:23:16Z

+The query then proceeds as:
+
+1. Run the index search as usual, producing candidate rows.
+2. Remove any candidate in the exclusion set. (Its indexed value may be stale.)
+3. **Re-evaluate** the excluded rows against their current values — the same flat
+   path already used for the unindexed tail of fragments. For a scalar predicate
+   this re-applies the filter; for a vector query it re-scores the row's current
+   vector. Rows that still match are added back to the result.
+
+Step 3 is what makes exclusion correct rather than merely safe: removing a row
+from index candidates without re-evaluating it would silently drop a row that
+should match under its new value.


IIUC this may not be enough to ensure accurate results in case of overlay updates -- if that is indeed the goal here. This covers the case where an query matching row was updated to no longer match, but what about one where the row previously didn't match and the updated value does? I think the only way to ensure 100% correctness is to brute force every update in the overlay.

Yeah I see. We need to brute force a full exclusion set, not just the intersection between the candidate rows and the full exclusion set. Good point, I will update.

## What The v2 file writer advanced every column from a single global row counter, so a single file could only hold columns of equal length. Sparse [data overlay files](#7381) need columns whose item counts differ *within one file* (each field covers a different set of rows). This is technically allowed by the file format but unused today. This PR adds that capability: - **`FileWriter::write_column(column_index, array)`** — writes one column, advancing only that field's row counter. Called across multiple calls a field appends; a field never written ends up as a zero-length column; an empty array is a no-op. `write_batch` is unchanged — it still advances all fields together, so ordinary rectangular files round-trip exactly as before. - **`FileReader::column_num_rows(column_index)`** — exposes a single column's length (sum of its pages' row counts). Always derivable from page metadata; this just surfaces it. - **Reader projection-length validation** — the reader combines a projection's columns into rectangular batches, so they must share a length. A projection whose columns differ in length is now rejected up front with a descriptive error (naming each column's length) instead of panicking inside the decoder. `RangeFull`/`RangeFrom` resolve to the projected common length rather than `self.num_rows` (the file's longest column), so a single short column reads back at its own length. Rectangular files are unaffected (the common length equals `num_rows`). Per-field lengths are computed from each top-level field's root (first) physical column, which equals its top-level row count for primitive, struct, and list fields in both v2.0 and v2.1. ## Tests `lance-file` lib suite (81 tests) + doctest pass; `cargo fmt` and `cargo clippy --tests -D warnings` clean. New/extended, all rstest over V2_0 + V2_1 where relevant: - `test_write_columns_unequal_lengths`: writes a 5-row, a 1-row, and a 0-row column in one file; asserts per-column row counts; reads each back at its own length; random access (sorted indices) within the longer column; empty-array writes are a no-op on both written and unwritten columns. - `test_read_unequal_length_projection`: equal-length projections full-scan; mismatched-length projections error before any batch (error names each column's length); single-column `RangeFull`/`RangeFrom`/`RangeTo` resolve to that column's own length; out-of-range bounds error. - `test_read_nested_columns_under_validation`: struct and list columns (which map to multiple physical columns) still read under the new validation path. - `test_write_column_validation_errors`: missing explicit schema, out-of-bounds field index, and a null written into a non-nullable field are each rejected. - `test_blocking_read_unequal_length`: the blocking read path applies the same validation. - `test_write_batch_keeps_equal_lengths`: rectangular files still produce equal-length columns (backwards compatible). ## Acceptance criteria (OSS-1323) - [x] v2 writer can write a single file with columns of differing item counts (no shared global row counter) - [x] v2 reader reads each column back independently at its own length - [x] Random access by position within a shorter/longer column returns correct values - [x] Per-column row counts derivable from file metadata (`column_num_rows`) - [x] Round-trip tests incl. zero-length and single-row columns alongside longer ones - [x] Existing equal-length files round-trip unchanged Closes #7404 (OSS-1323). Root of the Data Overlay Files stack; OSS-1322 builds on this. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

wjones127 · 2026-06-30T23:52:42Z

I've prototyped the basic write, scan, and take paths.

Write: feat: data overlay files — model, feature flag, and write/commit path #7535 (comment)
- Overlay files have a significant impact on manifest size when > 5% of rows have overlays and they are sparse rather than contiguous.
- Overlays are significantly faster and more IO efficient for targeted updates than update or merge insert.
Scan & Take: feat: resolve data overlay files on the take (and scan) read path #7536 (comment)
- Overlays have negligible impact on take performance
- Overlays can have a large impact on scan performance. Having 16 overlays can make scanning 3.5x slower.

Add a specification for data overlay files: small files attached to a fragment that supply new values for a subset of (row offset, field) cells without rewriting the base data files, for cheap cell-level updates. - protos/table.proto: rework DataOverlayFile with a dense/sparse coverage oneof (shared_offset_bitmap vs new FieldCoverage), rename read_version to committed_version (effective, commit-stamped), and document rank-based addressing with no offset column. Document reader feature flag 64. - docs: add data_overlay_file.md (full spec, worked example, guidance stub) and link it from the table format overview. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Add the `DataOverlay` operation (and `DataOverlayGroup`) to attach overlay files to fragments without rewriting their base data. Mirrors the `DataReplacement` batch shape, appends to each fragment's `overlays` list, and documents permissive conflict semantics: concurrent overlays, appends, deletes, and column rewrites are compatible; row-rewrites, compaction, and overlay->base folds conflict. committed_version is left 0 by the writer and stamped at commit time. Proto only — Rust/Python bindings deferred. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

The table/transaction proto changes generate new fields and an Operation variant. This wires the minimum needed to compile without implementing overlay support: - Emit empty `overlays` when converting fragments to proto. - Reject the `DataOverlay` transaction operation with NotSupported on read. Datasets that use overlays set reader feature flag 64, which already falls in the unknown-flag range rejected by `can_read_dataset`, so the library refuses them at the feature-flag layer. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Add an experimental admonition to the data overlay file spec and the table-format overview, with a TODO to name the first released version that supports the feature once implementation lands. Add TODO comments on the guidance subsections noting they will be filled in as benchmarking and implementation progress. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Adds the in-memory + commit machinery for data overlay files (per the spec in #7381), the foundation the scanner/take/index/compaction work builds on. - `DataOverlayFile` / `OverlayCoverage` (dense `shared_offset_bitmap` and sparse per-field) with protobuf round-trip, attached to `Fragment.overlays`. - Reader feature flag 64 (`FLAG_DATA_OVERLAY_FILES`): set whenever any fragment carries overlays, so a reader that does not understand them refuses the dataset instead of returning stale base values. - `Operation::DataOverlay` transaction op: appends overlays to a fragment's list (preserving concurrently-written overlays) and stamps each overlay's `committed_version` to the new dataset version at commit time (re-stamped on retry). Conflict rules mirror DataReplacement — permissive against appends, deletes, column rewrites, index builds, and other overlays; conflicts only with row-rewriting compaction of the same fragment. Scan-side merge, take, and end-to-end write+read tests follow in the same PR branch. Part of the Data Overlay Files feature (OSS-1322). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

westonpace

Looks like a cool feature. I have some questions, some of which I think are probably blocking (but maybe I'm misunderstanding) but only because they need to be clarified. I don't see anything insurmountable.

westonpace · 2026-07-07T16:55:11Z

+Precedence among overlays is determined by:
+
+1. `committed_version` — higher wins (see [Versioning](#versioning-and-ordering)).
+2. Position in `DataFragment.overlays` as a tiebreaker — a later entry is newer.


Does this mean a single commit can add multiple overlays that target the same field? Is there a reason for this? Or are we just being exhaustive?

It's possible, but not encouraged. This is mostly just trying to be exhaustive in the spec.

westonpace · 2026-07-07T17:03:45Z

+`data_file.fields`. It does **not** store a row-offset key column. The position of
+a covered offset's value within its column is the **rank** of that offset in the
+field's coverage bitmap — the number of set bits below it. For a Roaring bitmap
+this is an O(1) operation, so random access to any cell is a rank computation


I'm not sure I believe that rank is O(1).

westonpace · 2026-07-07T19:15:29Z

+<details>
+<summary>DataOverlayFile protobuf message</summary>
+
+```protobuf
+%%% proto.message.DataOverlayFile %%%
+```
+
+</details>


This is already in data_overlay_file.md. Do we really want it here too?

westonpace · 2026-07-07T19:17:41Z

+  // Attach overlay files to fragments, supplying new values for a subset of
+  // (row offset, field) cells without rewriting the fragments' base data files.
+  // See the DataOverlayFile message in table.proto and the Data Overlay Files
+  // specification for resolution, coverage, and versioning rules.


What operation is used to merge overlays (overlay->overlay compaction)? This operation or a different one?

It can't be this one, as this one appends overlays and doesn't remove any. We'll create another operation for that.

westonpace · 2026-07-07T19:19:53Z

+  //  * DataReplacement or column-rewrite (Update with REWRITE_COLUMNS) of the
+  //    same field: COMPATIBLE. Both preserve physical row addresses, so overlay
+  //    offsets stay valid; the overlay is newer and wins its covered cells, and
+  //    the version gate excludes those cells from any rebuilt index.


What if I have a column with an overlay file. This has been the stable state for some time. Later I want to replace the column with DataReplacement. Doesn't this conflict rule mean that the overlay will still sit on top? In other words, there is no way to overwrite an overlay with a DataReplacment until I've folded it in with a rewrite?

I think this is talking what if I try to add a data overlay but a concurrent operation already made a data replacement or update. In those cases it is compatible.

But I think you are right the other direction needs care: if I"m trying to do a data replacement but another operation just did an overlay, I either need to drop/rewrite that overlay or raise a conflict error.

I'm not actually worried about "just did an overlay". The overlay could be days or months old.

This isn't a "conflict error and retry the operation" failure but a more lasting "there is no way to paste over an overlay" problem.

If we had one overlay file per field (which I'm not recommending) we could say "as part of a DataReplacement commit you should remove any overlay files for that field" but I don't know how this works when there is an overlay file that covers multiple fields (how to say that overlay file is only valid for a subset of fields)

I see. This is actually something I addressed in #7535

how to say that overlay file is only valid for a subset of fields

It's actually the same as when you need to invalidate a field in a datafile: you change the field id to -2 to mask it out. If all fields at -2 in the file, then you can drop the data file or overlay.

Got it, and I see that in 7535. I think that's probably worth mentioning in this specification doc.

westonpace · 2026-07-07T19:21:15Z

+  // See the DataOverlayFile message in table.proto and the Data Overlay Files
+  // specification for resolution, coverage, and versioning rules.
+  //
+  // Conflict semantics (intentionally permissive, like DataReplacement). Against


Conflict semantics should probably be in a spec doc somewhere (transaction.md) and not here

Co-authored-by: Weston Pace <[email protected]> Co-authored-by: Will Jones <[email protected]>

Move the DataOverlay conflict-resolution rules out of the transaction.proto comment into transaction.md as a DataOverlay operation + Compatibility section, matching the pattern used for every other operation. Consolidate the reader-side overlay handling in the index doc and make re-evaluation explicit. Mark the feature experimental, drop the open-questions section, and cross-link the three docs. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Remove the standalone Correctness invariant section (redundant with the operational description and the compaction warning) and keep its one meaningful claim: a write's overlay committed_version always exceeds a pre-existing index's dataset_version, so exclusion is always sufficient. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Adds the in-memory + commit machinery for data overlay files (per the spec in #7381), the foundation the scanner/take/index/compaction work builds on. - `DataOverlayFile` / `OverlayCoverage` (dense `shared_offset_bitmap` and sparse per-field) with protobuf round-trip, attached to `Fragment.overlays`. - Reader feature flag 64 (`FLAG_DATA_OVERLAY_FILES`): set whenever any fragment carries overlays, so a reader that does not understand them refuses the dataset instead of returning stale base values. - `Operation::DataOverlay` transaction op: appends overlays to a fragment's list (preserving concurrently-written overlays) and stamps each overlay's `committed_version` to the new dataset version at commit time (re-stamped on retry). Conflict rules mirror DataReplacement — permissive against appends, deletes, column rewrites, index builds, and other overlays; conflicts only with row-rewriting compaction of the same fragment. Scan-side merge, take, and end-to-end write+read tests follow in the same PR branch. Part of the Data Overlay Files feature (OSS-1322). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

The Physical layout note claimed rank on a Roaring bitmap is O(1) with no binary search. roaring-rs rank is O(number of containers) plus a within-container lookup that does binary-search (array) or popcount (bitmap), so neither claim holds. Drop the complexity claim and keep the mechanism: rank plus one value fetch, no stored offset column. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

westonpace

I don't have any further blocking questions

wkalt · 2026-07-10T04:28:53Z

+  The merged overlay takes the **maximum** `committed_version` of its inputs, so
+  the exclusion semantics are preserved. Indexes can still be re-used, but they
+  may now need to exclude more rows. This is cheap to write and does not touch
+  the base.


I think we can state that merged overlays must be contiguous in the version record?

wkalt · 2026-07-10T04:37:17Z

+```
+
+This touches one field (`age`) for one row, so the writer emits a dense overlay
+and commits it as version 2. Fragment `0` gains:


nitpick: I think the single-cell case makes the choice of sparse vs dense a little arbitrary. We might be able to strengthen this example by updating more than one row/col.

wkalt

@wjones127 thanks for pushing this through! Very excited for this feature. Just two minor comments.

Overlay->overlay compaction stamps the merged overlay with the maximum committed_version of its inputs, which only preserves precedence when the merged overlays are contiguous in committed_version. State that constraint with an example. Also widen the worked example's first UPDATE to touch two rows so the dense-vs-sparse choice is motivated rather than arbitrary, and let it demonstrate that the full exclusion set is re-evaluated. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

coderabbitai · 2026-07-10T19:25:26Z

📝 Walkthrough

Walkthrough

The change defines experimental data overlay schemas and specifications, documents transaction compatibility, index invalidation, resolution, and compaction rules, and rejects overlay transactions in the current Rust implementation while initializing overlay fields explicitly.

Changes

Data Overlay Support

Layer / File(s)	Summary
Overlay schema and format semantics `protos/table.proto`, `docs/src/format/table/data_overlay_file.md`, `docs/src/format/table/index.md`, `docs/src/format/table/.pages`	Adds overlay metadata, dense and sparse coverage definitions, precedence rules, feature-flag documentation, and navigation for the new specification.
Overlay transaction contracts `protos/transaction.proto`, `docs/src/format/table/transaction.md`	Adds overlay transaction messages and documents fragment attachment, concurrent overlay preservation, version stamping, and compatibility rules.
Query and compaction behavior `docs/src/format/index/index.md`, `docs/src/format/table/data_overlay_file.md`	Documents field-aware index exclusion, row re-evaluation, overlay resolution, lineage, compaction modes, and index coverage updates.
Unsupported runtime handling `rust/lance/src/dataset/transaction.rs`, `rust/lance-table/src/format/fragment.rs`, `rust/lance-table/benches/manifest_intern.rs`	Rejects `DataOverlay` transactions as unsupported, tests the rejection, and initializes serialized overlay lists as empty in existing paths.

Estimated code review effort: 3 (Moderate) | ~30 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding data overlay file support/specification for the table format.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

docs/src/format/table/data_overlay_file.md (1)
214-311: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Keep this format reference text-only and use an executable schema definition.

The worked example adds SQL/protobuf code blocks and a markdown schema table. Replace it with concise prose and express the shown schema as a pyarrow schema definition.

As per coding guidelines, format docs must be concise and text-only, must not include code examples, and must express file schemas as pyarrow schema definitions rather than markdown tables.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docs/src/format/table/data_overlay_file.md` around lines 214 - 311, Replace
the worked example in the overlay lifecycle section with concise prose only:
remove the SQL statements, protobuf-like DataOverlayFile blocks, and markdown
schema table. Define the shown table and overlay file schemas using executable
pyarrow schema definitions, while retaining the essential lifecycle, coverage,
versioning, read, and index-query behavior in text.
Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance/src/dataset/transaction.rs`:
- Around line 6200-6216: Update test_data_overlay_operation_rejected to capture
the returned error and assert it is Error::NotSupported with a message
containing either “data overlay” or the reader feature flag 64 identifier,
ensuring the test validates the specific rejection reason.

---

Nitpick comments:
In `@docs/src/format/table/data_overlay_file.md`:
- Around line 214-311: Replace the worked example in the overlay lifecycle
section with concise prose only: remove the SQL statements, protobuf-like
DataOverlayFile blocks, and markdown schema table. Define the shown table and
overlay file schemas using executable pyarrow schema definitions, while
retaining the essential lifecycle, coverage, versioning, read, and index-query
behavior in text.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b5a0d66f-9dae-48e7-ab92-26bb5d071685

📥 Commits

Reviewing files that changed from the base of the PR and between bf4f32b and f079f39.

📒 Files selected for processing (10)

docs/src/format/index/index.md
docs/src/format/table/.pages
docs/src/format/table/data_overlay_file.md
docs/src/format/table/index.md
docs/src/format/table/transaction.md
protos/table.proto
protos/transaction.proto
rust/lance-table/benches/manifest_intern.rs
rust/lance-table/src/format/fragment.rs
rust/lance/src/dataset/transaction.rs

coderabbitai · 2026-07-10T19:29:01Z

+    #[test]
+    fn test_data_overlay_operation_rejected() {
+        // Overlay files are not supported by this version of the library. A
+        // transaction carrying the DataOverlay operation must be rejected rather
+        // than silently ignored, mirroring the feature-flag-64 rejection.
+        let message = pb::Transaction {
+            read_version: 1,
+            uuid: Uuid::new_v4().to_string(),
+            operation: Some(pb::transaction::Operation::DataOverlay(
+                pb::transaction::DataOverlay { groups: vec![] },
+            )),
+            ..Default::default()
+        };
+
+        let result = Transaction::try_from(message);
+        assert!(matches!(result, Err(Error::NotSupported { .. })));
+    }


📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Assert the expected error message as well as the variant.

The test currently verifies only Error::NotSupported, so an unrelated NotSupported error could pass. Assert that the message identifies unsupported data overlays or reader feature flag 64.

As per coding guidelines: “Assert on both the error variant and the message content in tests; do not check only is_err().”

Proposed test assertion

- let result = Transaction::try_from(message); - assert!(matches!(result, Err(Error::NotSupported { .. }))); + let err = match Transaction::try_from(message) { + Err(err) => err, + Ok(_) => panic!("expected data overlay transaction to be rejected"), + }; + assert!(matches!(&err, Error::NotSupported { .. })); + assert!(err + .to_string() + .contains("data overlay files are not supported"));

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

#[test]

fn test_data_overlay_operation_rejected() {

// Overlay files are not supported by this version of the library. A

// transaction carrying the DataOverlay operation must be rejected rather

// than silently ignored, mirroring the feature-flag-64 rejection.

let message = pb::Transaction {

read_version: 1,

uuid: Uuid::new_v4().to_string(),

operation: Some(pb::transaction::Operation::DataOverlay(

pb::transaction::DataOverlay { groups: vec![] },

)),

..Default::default()

};

let result = Transaction::try_from(message);

assert!(matches!(result, Err(Error::NotSupported { .. })));

}

#[test]

fn test_data_overlay_operation_rejected() {

// Overlay files are not supported by this version of the library. A

// transaction carrying the DataOverlay operation must be rejected rather

// than silently ignored, mirroring the feature-flag-64 rejection.

let message = pb::Transaction {

read_version: 1,

uuid: Uuid::new_v4().to_string(),

operation: Some(pb::transaction::Operation::DataOverlay(

pb::transaction::DataOverlay { groups: vec![] },

)),

..Default::default()

};

let err = match Transaction::try_from(message) {

Err(err) => err,

Ok(_) => panic!("expected data overlay transaction to be rejected"),

};

assert!(matches!(&err, Error::NotSupported { .. }));

assert!(err

.to_string()

.contains("data overlay files are not supported"));

}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@rust/lance/src/dataset/transaction.rs` around lines 6200 - 6216, Update test_data_overlay_operation_rejected to capture the returned error and assert it is Error::NotSupported with a message containing either “data overlay” or the reader feature flag 64 identifier, ensuring the test validates the specific rejection reason.

Source: Coding guidelines

Adds the in-memory + commit machinery for data overlay files (per the spec in lance-format#7381), the foundation the scanner/take/index/compaction work builds on. - `DataOverlayFile` / `OverlayCoverage` (dense `shared_offset_bitmap` and sparse per-field) with protobuf round-trip, attached to `Fragment.overlays`. - Reader feature flag 64 (`FLAG_DATA_OVERLAY_FILES`): set whenever any fragment carries overlays, so a reader that does not understand them refuses the dataset instead of returning stale base values. - `Operation::DataOverlay` transaction op: appends overlays to a fragment's list (preserving concurrently-written overlays) and stamps each overlay's `committed_version` to the new dataset version at commit time (re-stamped on retry). Conflict rules mirror DataReplacement — permissive against appends, deletes, column rewrites, index builds, and other overlays; conflicts only with row-rewriting compaction of the same fragment. Scan-side merge, take, and end-to-end write+read tests follow in the same PR branch. Part of the Data Overlay Files feature (OSS-1322). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

…#7535) > **Supersedes #7407.** Relocated into `lance-format/lance` (head + base in-repo) so the OSS-1324 PR can stack on it directly and show a clean, specific diff. Data model, feature flag, and write/commit foundation for [Data Overlay Files](#7381) (OSS-1322). **Stacked on #7381** (spec/proto). To keep this diff clean, the base is an in-repo mirror of #7381's branch (`will/data-overlay-spec-base`); retarget to `main` once #7381 lands. (#7406 / OSS-1323, needed for the sparse overlay sink, has merged.) ## What's here - **Data model** (`lance-table/src/format/overlay.rs`) — `DataOverlayFile` + `OverlayCoverage` on `Fragment.overlays`, dense (`shared_offset_bitmap`) and sparse (per-field). Lives in its own module: a small public surface (the two types, `dense`/`sparse`, `coverage_for_field`) with the coverage / rank / parse-once / newest-last invariants documented at the module level and the serde/proto/Roaring plumbing kept private. Coverage bitmaps are parsed once on load into `Arc<RoaringBitmap>` (not re-deserialized per access), and a fragment's overlays are stable-sorted by `committed_version` (newest last) on load so resolution can rely on the ordering. Protobuf/serde round-trip + `coverage_for_field` tested. - **Feature flag 64** (`FLAG_DATA_OVERLAY_FILES`) — set when any fragment has overlays. Release-gated: treated as an unknown flag in release builds (so release readers/writers refuse overlay datasets) unless `LANCE_ENABLE_DATA_OVERLAY_FILES` is set; debug builds understand it. Tested (release-gating policy is asserted profile-independently). - **`Operation::DataOverlay` transaction** — appends overlays to a fragment (preserving concurrently-written ones) and stamps `committed_version` at commit, re-stamped on retry. Protobuf round-trip + multi-fragment `build_manifest` (distinct targets + untargeted pass-through) tested. - **Conflict resolution** (v2 `CommitConflictResolver`) — permissive like `DataReplacement`: compatible with append / column-rewrite / index-build / data-replacement / other overlays, and with deletes/updates that leave the overlaid fragment in place. Retryable when a concurrent op row-rewrites or consumes the overlays on an overlaid fragment (`Rewrite`/`Merge`) or *removes* one (`Update`/`Delete` removal); incompatible with whole-dataset `Overwrite`/`Restore` and with `UpdateMemWalState` (matching the spec — both commit directions now agree). Tested (both-direction matrix, including remove-fragment and `Rewrite`×overlay). The fragment read path refuses overlays until the scan merge lands (OSS-1324). ## Follow-ups on this stack - [ ] `DataOverlayFileWriter` — streaming dense+sparse sink, used internally behind `update`/`merge_insert`. - [ ] `validate()` overlay checks — value-column length vs. coverage cardinality, dtype vs. schema (I/O-bound; cheap structural invariants are enforced at parse/commit time). - [ ] Scan/take merge (OSS-1324), resolved at read time fetching only the touched coverage ranks. Blocks OSS-1324 (take/scan), OSS-1325 (index masking), OSS-1326 (compaction). 🤖 Generated with [Claude Code](https://claude.com/claude-code)  ## Summary by CodeRabbit * **New Features** * Added experimental data overlay file support with opt-in release gating via an environment variable. * Introduced first-class `DataOverlay` transaction support to append per-fragment overlay updates during commits, including newest-last overlay ordering validation. * **Bug Fixes** * Improved overlay conflict handling across rewrites, data replacements, and row-moving updates, including deferred row-overlap validation. * Prevented reads from proceeding on fragments containing overlay data when overlay merge is in progress. * **Documentation** * Updated transaction compatibility docs with clearer overlay stacking rules, rewrite/replacement overlay interactions, and added conflict scenarios.  --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

github-actions Bot added the A-format On-disk format: protos and format spec docs label Jun 19, 2026

github-actions Bot added the documentation Improvements or additions to documentation label Jun 19, 2026

wjones127 marked this pull request as ready for review June 22, 2026 22:12

linear-code Bot mentioned this pull request Jun 23, 2026

V2 File writers and readers can support columns of varying lengths #7404

Closed

6 tasks

This was referenced Jun 23, 2026

feat(file): v2 writer/reader support columns of unequal length #7406

Merged

feat: data overlay files — model, feature flag, and write/commit path #7407

Closed

wjones127 requested review from westonpace and wkalt June 24, 2026 17:56

hamersaw reviewed Jun 25, 2026

View reviewed changes

wjones127 mentioned this pull request Jun 30, 2026

feat: data overlay files — model, feature flag, and write/commit path #7535

Merged

3 tasks

wjones127 and others added 4 commits July 6, 2026 13:11

wjones127 force-pushed the feat-patch-files branch from 633b1d0 to 16a9bea Compare July 6, 2026 20:44

westonpace requested changes Jul 7, 2026

View reviewed changes

westonpace reviewed Jul 7, 2026

View reviewed changes

Comment thread protos/table.proto

wjones127 commented Jul 7, 2026

View reviewed changes

Comment thread docs/src/format/table/index.md Outdated

wjones127 and others added 3 commits July 7, 2026 14:27

Apply suggestions from code review

e8341c3

Co-authored-by: Weston Pace <[email protected]> Co-authored-by: Will Jones <[email protected]>

westonpace approved these changes Jul 9, 2026

View reviewed changes

wkalt reviewed Jul 10, 2026

View reviewed changes

wkalt approved these changes Jul 10, 2026

View reviewed changes

coderabbitai Bot reviewed Jul 10, 2026

View reviewed changes

wjones127 merged commit fbd22b4 into lance-format:main Jul 10, 2026
36 checks passed

Uh oh!

Conversation

wjones127 commented Jun 19, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Out of scope / follow-ups

Summary by CodeRabbit

Uh oh!

github-actions Bot commented Jun 19, 2026

Uh oh!

wjones127 commented Jun 22, 2026

Uh oh!

codecov Bot commented Jun 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjones127 commented Jun 30, 2026

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjones127 Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wkalt left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot commented Jul 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

coderabbitai Bot Jul 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

wjones127 commented Jun 19, 2026 •

edited by coderabbitai Bot

Loading

codecov Bot commented Jun 22, 2026 •

edited

Loading

wjones127 Jul 7, 2026 •

edited

Loading

coderabbitai Bot commented Jul 10, 2026 •

edited

Loading