Skip to content

feat(mem_wal): non-blocking ShardWriter::delete_no_wait#7483

Merged
jackye1995 merged 3 commits into
lance-format:mainfrom
hamersaw:feat/mem-wal-delete-no-wait
Jun 26, 2026
Merged

feat(mem_wal): non-blocking ShardWriter::delete_no_wait#7483
jackye1995 merged 3 commits into
lance-format:mainfrom
hamersaw:feat/mem-wal-delete-no-wait

Conversation

@hamersaw

Copy link
Copy Markdown
Contributor

What

Adds ShardWriter::delete_no_wait, the delete analog of put_no_wait: it inserts the tombstone into the in-memory tier (visible to reads on this writer the instant it returns) and hands back the durability watcher without awaiting it. delete becomes a thin wrapper that awaits the watcher.

Why

Lets a caller hold an external lock across only the in-memory tombstone insert and await durability after releasing it — matching how put_no_wait is already used so flushes still coalesce. A downstream WAL serializes deletes against partial-update merges under a per-bucket lock; with the blocking delete it would hold that lock across durability, and delete_no_wait lets it release first.

Tests

  • test_shard_writer_delete_no_wait_visible_before_durability

🤖 Generated with Claude Code

Add `ShardWriter::delete_no_wait`, the delete analog of `put_no_wait`: it
inserts the tombstone into the in-memory tier (visible to reads on this writer
the instant it returns) and hands back the durability watcher *without*
awaiting it, so a caller can hold an external lock across only the in-memory
insert and await durability after releasing it. `delete` becomes a thin
wrapper that awaits the watcher.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@github-actions github-actions Bot added the enhancement New feature or request label Jun 25, 2026
hamersaw and others added 2 commits June 25, 2026 20:40
…ntee

The CI test `test_shard_writer_delete_no_wait_visible_before_durability`
read through the index-driven LSM scan *before* awaiting the durability
watcher and asserted the tombstone was already folded out. But the
no-wait path appends the tombstone to the batch store synchronously and
defers both the PK-index update and the `max_visible_batch_position`
advance to the async WAL flush — which is exactly what awaiting the
watcher waits for. The LSM read is gated on that watermark, so the
tombstone cannot be folded until the flush lands. The test encoded a
guarantee the system does not make and lost the race deterministically
in CI.

Replace it with two tests for the actual contract:
- durable: tombstone is in the in-memory tier immediately (batch-store
  `row_count`), watcher returned without blocking, and the LSM read
  folds the delete only after `watcher.wait()`.
- non-durable: no watcher is returned, but the tombstone still lands in
  the in-memory tier.

Also correct the `delete_no_wait`/`put_no_wait` doc comments, which
overclaimed immediate "visible to reads on this writer" — true only at
the batch-store level, not the index-driven LSM read path.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@jackye1995 jackye1995 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@jackye1995
jackye1995 merged commit a1cae2d into lance-format:main Jun 26, 2026
30 checks passed
@hamersaw
hamersaw deleted the feat/mem-wal-delete-no-wait branch June 26, 2026 15:36
BubbleCal pushed a commit that referenced this pull request Jun 30, 2026
## What

Adds `ShardWriter::delete_no_wait`, the delete analog of `put_no_wait`:
it inserts the tombstone into the in-memory tier (visible to reads on
this writer the instant it returns) and hands back the durability
watcher *without* awaiting it. `delete` becomes a thin wrapper that
awaits the watcher.

## Why

Lets a caller hold an external lock across only the in-memory tombstone
insert and await durability after releasing it — matching how
`put_no_wait` is already used so flushes still coalesce. A downstream
WAL serializes deletes against partial-update merges under a per-bucket
lock; with the blocking `delete` it would hold that lock across
durability, and `delete_no_wait` lets it release first.

## Tests

- `test_shard_writer_delete_no_wait_visible_before_durability`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
BubbleCal pushed a commit that referenced this pull request Jun 30, 2026
Backport of the following PRs:
- #7362
- #7417
- #7482
- #7483
- #7489

This PR backports the changes from the original PRs to the release/v8.0
branch.

---------

Co-authored-by: Dan Rammer <[email protected]>
Co-authored-by: Lance Release Bot <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants