## Summary

Adds row-level **DELETE** to the in-memory WAL (`mem_wal`), expressed as
**tombstone (sentinel) rows**: a delete appends the primary key plus
`_tombstone = true` (null elsewhere). A tombstone is the newest value
for its PK — it wins newest-per-PK resolution (suppressing the older
real row) and is then silently dropped from query results.

This is the lance (OSS) half of the WAL delete feature. The sophon half
(compaction hard-delete + serving) consumes this API separately.

## What's here

- **`_tombstone` column** — lance-owned, non-nullable `Boolean` appended
to the memtable/generation schema only (never the base table).
`ShardWriter::open` extends the caller's base schema internally; callers
keep passing the base schema and never name the column.
- **Write path** — `ShardWriter::put` injects `_tombstone = false`; new
**`ShardWriter::delete(keys)`** builds the tombstone batch from a
key-only input. Legacy WAL entries (written before this change) are
back-filled on replay.
- **HNSW null-vector fix** — mem_wal's HNSW rejected null vectors
outright, so it couldn't handle a nullable vector column at all (a
pre-existing bug, independent of deletes). `append_batch` now skips null
rows like the base-table index, compacting survivors while preserving
their original offsets in `row_ids` so search still resolves to the
right row.
- **Read paths**
- *Filtered read*: fold `NOT _tombstone` into the predicate per WAL arm
(only when the source carries the column), so it runs before the
per-source pushdown limit and post-dedup in the active arm.
- *Point lookup*: carry `_tombstone` through and filter **after**
`CoalesceFirstExec` (per-source filtering would let the coalesce fall
through and resurrect the row); the plan-free BTree fast path
short-circuits a tombstone hit to not-found.
- *Vector / FTS*: no filter needed — a null-payload tombstone is never
an index candidate; the deleted real row is suppressed by the existing
block-list (cross-gen) / `NewestPkFilter` (within active).

## Notable design points

- Fold/carry `_tombstone` **only when the source schema has the column**
— back-compat for legacy generations and existing tests for free.
- `_tombstone` is non-nullable so the point-lookup base arm can
synthesize a matching `Literal(false)` for `CoalesceFirstExec`'s
exact-schema check.
- Delete requires non-PK columns to be nullable (a tombstone nulls
them); surfaced as a clear error otherwise.
- Caveat documented in code: a delete-heavy burst between compactions
can under-deliver a `LIMIT`/top-k query (uncompensated block-list drops)
— recall degradation, not wrong content; self-limiting as compaction
drains tombstones.

## Testing

19 new tests; full `mem_wal` suite green (417 passed, 0 failed). `cargo
fmt` + `cargo clippy -p lance --tests` clean. Coverage: filtered-read
masking + filter + limit, point-lookup resurrection guards (fast + plan
paths), delete-then-reinsert, delete-of-absent, HNSW null-vector
handling (mid-batch + all-null), write-path injection/round-trip and
validation.

Draft — opening for early review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

Backport of the following PRs:
- #7362
- #7417
- #7482
- #7483
- #7489

This PR backports the changes from the original PRs to the release/v8.0
branch.

---------

Co-authored-by: Dan Rammer <[email protected]>
Co-authored-by: Lance Release Bot <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>