Lance: v7.0.0-beta.6 (commit b66eee73)
Build: Linux x86_64, release, c7i.12xlarge
Symptom
When repeated merge_insert upserts via the MemWAL ShardWriter path push the active MemTable past max_memtable_rows, the first triggered flush errors out:
[ERROR lance::dataset::mem_wal::write] Task 'memtable_flusher' error handling message:
Dataset already exists:
s3://<bucket>/.../bench_sync_no-idx.lance?/_mem_wal/<shard_uuid>/bda7e4a9_gen_1,
/home/ubuntu/.cargo/git/checkouts/lance-.../rust/lance/src/dataset/write/insert.rs:286:28
thread 'lancedb-tokio-worker' panicked at
/home/ubuntu/.cargo/git/checkouts/lance-.../rust/lance/src/dataset/mem_wal/util.rs:79:14:
watch channel closed: RecvError(())
The subsequent panic at mem_wal/util.rs:79 propagates as RustPanic: rust future panicked: unknown error to the caller, killing the merge_insert call mid-stream.
Reproducer
Configured a ShardWriter via the LanceDB MemWAL merge_insert path with:
durable_write = true (default)max_memtable_rows = 1_000_000max_memtable_batches = 10_000_000- Single shard (unsharded MemWAL spec)
Upsert workload writes 100-row batches sequentially; once the cumulative row count crosses 1M, the flusher attempts to write a fragment generation file at <shard>/<frag_id>_gen_1 and finds the path already populated. From the path it looks like the fragment ID is reused or the generation counter is not advanced before the flush fires.
Branch / repro script with config: https://github.com/touch-of-grey/lancedb/blob/MergeInsertShardWriter/ (LanceDB is consuming the released v7.0.0-beta.6, so this is a Lance-side bug surfacing through that consumer).
Likely culprits
- The
_gen_1 suffix repeats across flushes — generation counter not bumped before the second flush attempts its write?
- Or fragment ID is stable across flushes when it should be unique-per-flush.
mem_wal/util.rs:79 has an unwrap() on a watch receive that should propagate the flusher's error instead of panicking the worker.
Happy to gather more diagnostics or try a candidate fix if you can point at the right spot.
Lance:
v7.0.0-beta.6(commitb66eee73)Build: Linux x86_64, release, c7i.12xlarge
Symptom
When repeated
merge_insertupserts via the MemWALShardWriterpath push the active MemTable pastmax_memtable_rows, the first triggered flush errors out:The subsequent panic at
mem_wal/util.rs:79propagates asRustPanic: rust future panicked: unknown errorto the caller, killing themerge_insertcall mid-stream.Reproducer
Configured a
ShardWritervia the LanceDB MemWALmerge_insertpath with:durable_write = true(default)max_memtable_rows = 1_000_000max_memtable_batches = 10_000_000Upsert workload writes 100-row batches sequentially; once the cumulative row count crosses 1M, the flusher attempts to write a fragment generation file at
<shard>/<frag_id>_gen_1and finds the path already populated. From the path it looks like the fragment ID is reused or the generation counter is not advanced before the flush fires.Branch / repro script with config: https://github.com/touch-of-grey/lancedb/blob/MergeInsertShardWriter/ (LanceDB is consuming the released
v7.0.0-beta.6, so this is a Lance-side bug surfacing through that consumer).Likely culprits
_gen_1suffix repeats across flushes — generation counter not bumped before the second flush attempts its write?mem_wal/util.rs:79has anunwrap()on a watch receive that should propagate the flusher's error instead of panicking the worker.Happy to gather more diagnostics or try a candidate fix if you can point at the right spot.