Skip to content

fix: relax DataReplacement conflict with concurrent Delete/Update#7433

Merged
wkalt merged 2 commits into
lance-format:mainfrom
brendanclement:datareplacement-conflict-only-on-removal
Jun 24, 2026
Merged

fix: relax DataReplacement conflict with concurrent Delete/Update#7433
wkalt merged 2 commits into
lance-format:mainfrom
brendanclement:datareplacement-conflict-only-on-removal

Conversation

@brendanclement

Copy link
Copy Markdown
Contributor

#7373 made a DataReplacement conflict with any concurrent Update/Delete that touched a target fragment. That is too conservative: a delete that only adds a deletion vector leaves the positional column file aligned, and build_manifest rebases the replacement onto the current fragment, preserving the deletion vector.

A DataReplacement vs a concurrent Delete/Update now conflicts only when:

  • the target fragment is removed outright -- returns a non-retryable IncompatibleTransaction so the caller can drop it and commit the rest;
  • a row-rewriting (RewriteRows) update moves the matched rows out to new fragments the positional file does not cover; or
  • the update rewrote one of the replaced fields in place (fields_modified).

Deletion-vector-only deletes/updates and non-overlapping column rewrites are now allowed.

lance-format#7373 made a DataReplacement conflict with any concurrent Update/Delete that
touched a target fragment. That is too conservative: a delete that only adds a
deletion vector leaves the positional column file aligned, and build_manifest
rebases the replacement onto the current fragment, preserving the deletion
vector.

A DataReplacement vs a concurrent Delete/Update now conflicts only when:
- the target fragment is removed outright -- returns a non-retryable
  IncompatibleTransaction so the caller can drop it and commit the rest;
- a row-rewriting (RewriteRows) update moves the matched rows out to new
  fragments the positional file does not cover; or
- the update rewrote one of the replaced fields in place (fields_modified).

Deletion-vector-only deletes/updates and non-overlapping column rewrites are
now allowed.
@github-actions github-actions Bot added the bug Something isn't working label Jun 24, 2026
Update the DataReplacement compatibility table to match the refined conflict
resolver: full fragment removal by Delete/Update is now a non-retryable
conflict, Update is a retryable conflict only when it moves rows out of a
target fragment or rewrites one of the replaced fields, and a deletion-vector
-only Delete/Update is compatible.
@github-actions

Copy link
Copy Markdown
Contributor

Important

This PR touches the Lance format specification.

Substantive changes to the format specification — the .proto definitions
and the spec docs under docs/src/format/ — require a PMC vote before merge.
Minor edits such as typo fixes, wording, or formatting are excluded; use your
judgment.

If this is a meaningful format change:

  • Start a vote following the Lance community voting process.
    Format specification modifications need 3 binding +1 votes (excluding the
    proposer), held on GitHub Discussions, with a minimum voting period of 1 week.
  • Once the vote passes, link the completed vote in this PR. It should not be
    merged until the vote is linked.

@github-actions github-actions Bot added the A-format On-disk format: protos and format spec docs label Jun 24, 2026
@wkalt
wkalt merged commit fa2b755 into lance-format:main Jun 24, 2026
30 checks passed
@codecov

codecov Bot commented Jun 24, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.20000% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance/src/io/commit/conflict_resolver.rs 99.20% 0 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-format On-disk format: protos and format spec docs bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants