## Summary

Adds row-level **DELETE** to the in-memory WAL (`mem_wal`), expressed as
**tombstone (sentinel) rows**: a delete appends the primary key plus
`_tombstone = true` (null elsewhere). A tombstone is the newest value
for its PK — it wins newest-per-PK resolution (suppressing the older
real row) and is then silently dropped from query results.

This is the lance (OSS) half of the WAL delete feature. The sophon half
(compaction hard-delete + serving) consumes this API separately.

## What's here

- **`_tombstone` column** — lance-owned, non-nullable `Boolean` appended
to the memtable/generation schema only (never the base table).
`ShardWriter::open` extends the caller's base schema internally; callers
keep passing the base schema and never name the column.
- **Write path** — `ShardWriter::put` injects `_tombstone = false`; new
**`ShardWriter::delete(keys)`** builds the tombstone batch from a
key-only input. Legacy WAL entries (written before this change) are
back-filled on replay.
- **HNSW null-vector fix** — mem_wal's HNSW rejected null vectors
outright, so it couldn't handle a nullable vector column at all (a
pre-existing bug, independent of deletes). `append_batch` now skips null
rows like the base-table index, compacting survivors while preserving
their original offsets in `row_ids` so search still resolves to the
right row.
- **Read paths**
- *Filtered read*: fold `NOT _tombstone` into the predicate per WAL arm
(only when the source carries the column), so it runs before the
per-source pushdown limit and post-dedup in the active arm.
- *Point lookup*: carry `_tombstone` through and filter **after**
`CoalesceFirstExec` (per-source filtering would let the coalesce fall
through and resurrect the row); the plan-free BTree fast path
short-circuits a tombstone hit to not-found.
- *Vector / FTS*: no filter needed — a null-payload tombstone is never
an index candidate; the deleted real row is suppressed by the existing
block-list (cross-gen) / `NewestPkFilter` (within active).

## Notable design points

- Fold/carry `_tombstone` **only when the source schema has the column**
— back-compat for legacy generations and existing tests for free.
- `_tombstone` is non-nullable so the point-lookup base arm can
synthesize a matching `Literal(false)` for `CoalesceFirstExec`'s
exact-schema check.
- Delete requires non-PK columns to be nullable (a tombstone nulls
them); surfaced as a clear error otherwise.
- Caveat documented in code: a delete-heavy burst between compactions
can under-deliver a `LIMIT`/top-k query (uncompensated block-list drops)
— recall degradation, not wrong content; self-limiting as compaction
drains tombstones.

## Testing

19 new tests; full `mem_wal` suite green (417 passed, 0 failed). `cargo
fmt` + `cargo clippy -p lance --tests` clean. Coverage: filtered-read
masking + filter + limit, point-lookup resurrection guards (fast + plan
paths), delete-then-reinsert, delete-of-absent, HNSW null-vector
handling (mid-batch + all-null), write-path injection/round-trip and
validation.

Draft — opening for early review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

## What

Adds `LsmPointLookupPlanner::lookup_keep_tombstone` and
`lookup_many_keep_tombstone` — point-lookup variants that carry the
`_tombstone` marker through instead of filtering deleted keys out.

## Why

The existing `lookup` / `lookup_many` collapse two distinct states —
**deleted in the fresh tier** and **never written** — into a single
`None`. That's correct for a normal reader, but a caller doing a
read-on-write merge needs to tell them apart: a fresh-deleted PK must be
treated as *absent* (so a later partial update resurrects it from
carried columns + NULLs), while a never-written PK falls back to the
base row.

These variants return the deleted row with `_tombstone = true`; absent
keys still return `None`. Implemented by refactoring `plan_lookup` to
share a `plan_lookup_coalesced` helper and skipping the post-coalesce
tombstone filter, so the hot read path is unchanged.

## Tests

- `test_lookup_keep_tombstone_returns_deleted_row_with_marker`
- `test_lookup_many_keep_tombstone_includes_tombstoned_keys`

## Context

Prerequisite for a downstream WAL partial-column-update resurrection
fix: a partial update arriving after a delete must resurrect carried
columns + NULL untouched, never stale pre-delete base data.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

## What

Adds `ShardWriter::delete_no_wait`, the delete analog of `put_no_wait`:
it inserts the tombstone into the in-memory tier (visible to reads on
this writer the instant it returns) and hands back the durability
watcher *without* awaiting it. `delete` becomes a thin wrapper that
awaits the watcher.

## Why

Lets a caller hold an external lock across only the in-memory tombstone
insert and await durability after releasing it — matching how
`put_no_wait` is already used so flushes still coalesce. A downstream
WAL serializes deletes against partial-update merges under a per-bucket
lock; with the blocking `delete` it would hold that lock across
durability, and `delete_no_wait` lets it release first.

## Tests

- `test_shard_writer_delete_no_wait_visible_before_durability`

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

#7489)

## What

The WAL fresh-tier vector and FTS search arms apply a per-generation
recency filter (`NewestPkFilterExec`) to every in-memory memtable — but
they only apply the cross-generation block-list (`PkBlockFilterExec`) to
flushed/base sources, never to in-memory ones. This PR applies the
block-list to in-memory arms too.

## Why

`LsmDataSourceCollector::in_memory_sources` emits **both** the live
active memtable **and** every frozen-awaiting-flush memtable as
`ActiveMemTable` sources, each with its own `index_store`. The recency
filter queries only *that generation's* PK index, so it is purely
within-generation.

When a PK is inserted in one generation and later deleted or updated in
a **newer** in-memory generation, the older (frozen) generation's arm
keeps the stale matching row: the newer generation's tombstone/update
lives in a different `index_store` the frozen arm never consults. This
happens whenever an insert and its later delete/update straddle a freeze
boundary — common under a short flush interval, which freezes memtables
continuously.

The cross-generation block-list (`NEWER(G)`) is the only mechanism that
resolves this, and `compute_source_block_lists` already computes it for
in-memory sources (snapshot-bounded by each generation's
`max_visible_row`, so a not-yet-visible write can't over-block). The
arms just discarded it.

## Change

- `vector_search.rs`: after `NewestPkFilterExec`, also wrap in
`PkBlockFilterExec` when the source has a non-empty block set.
- `fts_search.rs`: unify the dedup loop so every source applies the
block-list when blocked; the within-generation recency filter is already
applied inside `build_source_plan` for in-memory sources.

The newest memtable per shard has no newer generation, so its block set
is empty (`None`) and it is unaffected — only older frozen generations
are now filtered. Each in-memory generation gets both within-generation
recency and cross-generation supersession.

## Tests

- `cross_gen_stale_version_blocked_by_newer_memtable` (vector): a PK at
a near vector in a frozen generation and at a far vector in the active
generation. Before the fix the stale near copy leaks alongside the
newest far version; after, only the newest survives.
- `cross_gen_stale_update_blocked_by_newer_memtable` (FTS): a PK
matching "alpha" in a frozen generation, updated to "beta" in the active
generation; an "alpha" search must not return the stale frozen row.

Both fail before this change and pass after. The existing
single-generation recency tests
(`keeps_only_the_newest_visible_position_per_pk`,
`active_stale_update_predicate_crossing_leaks`) still pass.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>