feat: add Lance-native MemWAL HNSW for shard writer by touch-of-grey · Pull Request #6795 · lance-format/lance

touch-of-grey · 2026-05-15T08:01:24Z

Summary

This PR adds a Lance-native in-memory HNSW implementation for MemWAL and wires it into the async ShardWriter path.

The benchmark shape follows @jackye1995's suggestion to use a FineWeb-like baseline: a text payload around 1.5 KiB plus a 1024-dim f32 vector column, or about 5760 bytes per row. I used hnswlib's construction hot path as a reference while adapting the implementation to Lance MemWAL's requirements: multi-reader/single-writer access, reader-visible publication during writes, and reuse of the vector data already held by the MemTable instead of copying vectors into a separate HNSW-owned buffer.

Main changes:

add rust/lance/src/dataset/mem_wal/hnsw/ with a MemWAL-oriented HNSW graph and Arrow-backed vector storage
update mem_wal/index/hnsw.rs to use the new graph for active MemTable vector search
add WAL queue stats so benchmarks can distinguish WAL flush lag from final close/drain work
add native Rust benchmarks under rust/lance/benches/, including side-by-side HNSW/hnswlib comparison scaffolding and the shard-writer WAL backpressure benchmark
add --schema-shape fineweb|vector_only to the native shard-writer benchmark so we can run both the FineWeb-shaped case and the older small id + 1024-dim vector case requested in review

Benchmark Summary So Far

Baseline before this work was roughly:

safe durable async+index throughput: ~3.66 MB/s (batch=50, 512 KiB WAL, no backpressure)
previous peak with manageable backpressure: ~6.17 MB/s (batch=100, 2 MiB WAL, ~24s drain)
bottleneck identified as active in-memory HNSW insert throughput, not S3 bandwidth

Best current WAL-queue results on c7i.16xlarge, S3 bucket jack-devland-build, FineWeb-shaped rows, --skip-close, and explicit WAL queue stats:

batch	WAL	target rows/s	actual rows/s	MB/s	final WAL queue	max WAL queue	thread setting
10	16 MiB	8000	8000	46.08	0.310%	1.526%	rayon=48, tokio=16
100	8 MiB	6000	6000	34.56	0.130%	1.070%	rayon=64, tokio=16
1000	8 MiB	8000	8000	46.08	0.200%	1.500%	rayon=64, tokio=8

Interpretation: for these paced single-shard runs, WAL flush can keep up at 34-46 MB/s without accumulating material WAL backlog. This is about 7.5x over the previous 6.17 MB/s peak and about 12.6x over the earlier 3.66 MB/s no-backpressure point on the same FineWeb-shaped workload.

Thread ablation suggests high Tokio worker counts are not required for this path; the important CPU knob is the Rayon pool used by async index construction. Moderate Tokio settings (4-16) were enough in the tested single-writer workload.

Small-schema follow-up for @jackye1995's comment, using --schema-shape vector_only (id + 1024-dim f32 vector, default row estimate 4160 bytes) on the 12xlarge EC2 runner, S3 bucket jack-devland-build, --skip-close, and the same WAL-queue stats:

batch	WAL	target rows/s	actual rows/s	MB/s	p99 ms	final WAL queue	max WAL queue	thread setting
10	16 MiB	8000	7999.855	33.279	0.020	0.481%	1.998%	rayon=64, tokio=64
10	16 MiB	11000	10999.782	45.759	0.019	0.481%	2.495%	rayon=64, tokio=64
100	8 MiB	6000	5999.954	24.960	0.042	0.340%	1.400%	rayon=64, tokio=32
100	8 MiB	9000	8999.850	37.439	0.045	0.340%	2.100%	rayon=64, tokio=32
1000	8 MiB	8000	7999.913	33.280	0.084	0.100%	1.800%	rayon=64, tokio=8
1000	8 MiB	11000	10999.909	45.760	0.154	4.600%	4.600%	rayon=64, tokio=8

This is not a full max-throughput sweep for the small schema, but it confirms the benchmark can run the requested older shape and that it reaches the same ~45.8 MB/s region at 11k rows/s, with the batch=1000 high-rate case starting to show more WAL queue backlog than the smaller batches.

Analysis Artifacts

Saved local analysis:

/Users/jackye/ai/analysis/lance/jack-mem-wal-hnsw/wal-queue-all-runs-20260515/RESULTS.md
/Users/jackye/ai/analysis/lance/jack-mem-wal-hnsw/thread-ablation-20260515/RESULTS.md
/Users/jackye/ai/analysis/lance/jack-mem-wal-hnsw/vector-only-pr-20260515/

S3 result artifacts:

s3://jack-devland-build/memwal-walqueue-panel-20260514T171240Z/_bench_results/
s3://jack-devland-build/memwal-walqueue-supplement-20260514T182826Z/_bench_results/
s3://jack-devland-build/memwal-walqueue-higher-20260515T001817Z/_bench_results/
s3://jack-devland-build/memwal-thread-ablation-20260515T060229Z/_bench_results/
s3://jack-devland-build/memwal-vector-only-pr-20260515T090611Z/_bench_results/

Validation

cargo fmt --all --check
cargo fmt --manifest-path python/Cargo.toml --all --check
cargo check -p lance --bench mem_wal_shard_writer_backpressure
cargo check --manifest-path python/Cargo.toml --lib
cargo clippy -p lance --tests -- -D warnings
cargo test -p lance dataset::mem_wal::index::hnsw::tests::test_index_concurrent_insert_and_search -- --exact

cc @hamersaw

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

github-actions · 2026-05-15T08:11:53Z

ACTION NEEDED
Lance follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

For details on the error please inspect the "PR Title Check" action.

jackye1995 · 2026-05-15T08:18:12Z

use a FineWeb-like baseline: a text payload around 1.5 KiB plus a 1024-dim f32 vector column, or about 5760 bytes per row.

Thanks for doing this, super informative! I think we probably want to test both this and also keep the old small schema of just a id + 1024 dim vector case, just to make sure we have knowledge of the range of throughput depending on the per-row size. (technically the same comment for the FTS PR you have, we should test both this schema and a small schema with just id + 1.5kb text).

touch-of-grey · 2026-05-15T08:25:06Z

Addressed this in ff768f1: the native shard-writer benchmark now has --schema-shape fineweb|vector_only. Default remains the FineWeb-shaped payload, and vector_only uses the small id + vec schema with a row-byte estimate derived from vector dim unless --row-bytes is passed explicitly. I also updated the PR summary to call this out; next benchmark panels can run both shapes side by side.

codecov · 2026-05-15T09:24:58Z

Codecov Report

❌ Patch coverage is 81.28693% with 285 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/dataset/mem_wal/hnsw/graph.rs	83.27%	123 Missing and 25 partials ⚠️
rust/lance/src/dataset/mem_wal/index/hnsw.rs	63.94%	62 Missing and 13 partials ⚠️
rust/lance/src/dataset/mem_wal/hnsw/storage.rs	80.21%	52 Missing and 3 partials ⚠️
...ce/src/dataset/mem_wal/memtable/scanner/builder.rs	66.66%	1 Missing and 2 partials ⚠️
rust/lance/src/dataset/mem_wal/memtable/flush.rs	93.10%	2 Missing ⚠️
.../lance/src/dataset/mem_wal/memtable/batch_store.rs	96.87%	0 Missing and 1 partial ⚠️
rust/lance/src/dataset/mem_wal/write.rs	98.78%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

touch-of-grey · 2026-05-15T09:27:42Z

Follow-up for the small-schema part of @jackye1995's comment: I ran the native Rust shard-writer benchmark with --schema-shape vector_only (id + 1024-dim f32 vector, default 4160 bytes/row) on the 12xlarge EC2 runner against jack-devland-build.

batch	WAL	target rows/s	actual rows/s	MB/s	p99 ms	final WAL queue	max WAL queue
10	16 MiB	8000	7999.855	33.279	0.020	0.481%	1.998%
10	16 MiB	11000	10999.782	45.759	0.019	0.481%	2.495%
100	8 MiB	6000	5999.954	24.960	0.042	0.340%	1.400%
100	8 MiB	9000	8999.850	37.439	0.045	0.340%	2.100%
1000	8 MiB	8000	7999.913	33.280	0.084	0.100%	1.800%
1000	8 MiB	11000	10999.909	45.760	0.154	4.600%	4.600%

Artifacts:

local: /Users/jackye/ai/analysis/lance/jack-mem-wal-hnsw/vector-only-pr-20260515/
S3: s3://jack-devland-build/memwal-vector-only-pr-20260515T090611Z/_bench_results/

This is not a full max-throughput sweep, but it verifies the benchmark covers both requested schema shapes and gives an initial small-schema range to compare against the FineWeb-shaped results.

touch-of-grey · 2026-05-15T20:41:32Z

I ran the native Rust MemWAL shard-writer benchmark for both schema shapes based on suggestion from @jackye1995.

FineWeb-shaped baseline:
~1.5 KiB text + 1024-dim f32 vector, about 5760 bytes/row.

batch	WAL	target rows/s	actual rows/s	MB/s	final WAL queue	max WAL queue
10	16 MiB	8000	8000	46.08	0.310%	1.526%
100	8 MiB	6000	6000	34.56	0.130%	1.070%
1000	8 MiB	8000	8000	46.08	0.200%	1.500%

Small vector-only schema:
id + 1024-dim f32 vector, about 4160 bytes/row.

batch	WAL	target rows/s	actual rows/s	MB/s	p99 ms	final WAL queue	max WAL queue
10	16 MiB	8000	7999.855	33.279	0.020	0.481%	1.998%
10	16 MiB	11000	10999.782	45.759	0.019	0.481%	2.495%
100	8 MiB	6000	5999.954	24.960	0.042	0.340%	1.400%
100	8 MiB	9000	8999.850	37.439	0.045	0.340%	2.100%
1000	8 MiB	8000	7999.913	33.280	0.084	0.100%	1.800%
1000	8 MiB	11000	10999.909	45.760	0.154	4.600%	4.600%

Takeaways:

The native Rust benchmark now covers both requested schema shapes.
On the FineWeb-shaped workload, paced single-shard async indexed writes sustain roughly 34-46 MB/s without material WAL backlog.
The small vector-only schema reaches the same ~45.8 MB/s region at 11k rows/s; the batch=1000 high-rate case shows more WAL queue buildup than smaller batches.
This confirms the previous Python/LanceDB numbers were not primarily Python-binding limited for this path; the Rust-native path is able to push similar or higher throughput while exposing explicit WAL queue lag.

Previously flush_with_indexes wrote N+1 manifest files: one for the data fragment via Dataset::write, then one per index via apply_commit. Now all index metadata is collected without committing and the v1 manifest is overwritten in-place with write_manifest_file_to_path to produce a single-version dataset containing both data and all indexes.

jackye1995

thanks for the work! I added one missing part about flushing memtable as a single commit also here.

@jackye1995

## Summary This PR adds a Lance-native in-memory HNSW implementation for MemWAL and wires it into the async `ShardWriter` path. The benchmark shape follows @jackye1995's suggestion to use a FineWeb-like baseline: a text payload around 1.5 KiB plus a 1024-dim `f32` vector column, or about 5760 bytes per row. I used hnswlib's construction hot path as a reference while adapting the implementation to Lance MemWAL's requirements: multi-reader/single-writer access, reader-visible publication during writes, and reuse of the vector data already held by the MemTable instead of copying vectors into a separate HNSW-owned buffer. Main changes: - add `rust/lance/src/dataset/mem_wal/hnsw/` with a MemWAL-oriented HNSW graph and Arrow-backed vector storage - update `mem_wal/index/hnsw.rs` to use the new graph for active MemTable vector search - add WAL queue stats so benchmarks can distinguish WAL flush lag from final close/drain work - add native Rust benchmarks under `rust/lance/benches/`, including side-by-side HNSW/hnswlib comparison scaffolding and the shard-writer WAL backpressure benchmark - add `--schema-shape fineweb|vector_only` to the native shard-writer benchmark so we can run both the FineWeb-shaped case and the older small `id + 1024-dim vector` case requested in review ## Benchmark Summary So Far Baseline before this work was roughly: - safe durable async+index throughput: ~3.66 MB/s (`batch=50`, 512 KiB WAL, no backpressure) - previous peak with manageable backpressure: ~6.17 MB/s (`batch=100`, 2 MiB WAL, ~24s drain) - bottleneck identified as active in-memory HNSW insert throughput, not S3 bandwidth Best current WAL-queue results on `c7i.16xlarge`, S3 bucket `jack-devland-build`, FineWeb-shaped rows, `--skip-close`, and explicit WAL queue stats: | batch | WAL | target rows/s | actual rows/s | MB/s | final WAL queue | max WAL queue | thread setting | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | | 10 | 16 MiB | 8000 | 8000 | 46.08 | 0.310% | 1.526% | rayon=48, tokio=16 | | 100 | 8 MiB | 6000 | 6000 | 34.56 | 0.130% | 1.070% | rayon=64, tokio=16 | | 1000 | 8 MiB | 8000 | 8000 | 46.08 | 0.200% | 1.500% | rayon=64, tokio=8 | Interpretation: for these paced single-shard runs, WAL flush can keep up at 34-46 MB/s without accumulating material WAL backlog. This is about 7.5x over the previous 6.17 MB/s peak and about 12.6x over the earlier 3.66 MB/s no-backpressure point on the same FineWeb-shaped workload. Thread ablation suggests high Tokio worker counts are not required for this path; the important CPU knob is the Rayon pool used by async index construction. Moderate Tokio settings (`4-16`) were enough in the tested single-writer workload. Small-schema follow-up for @jackye1995's comment, using `--schema-shape vector_only` (`id + 1024-dim f32 vector`, default row estimate 4160 bytes) on the 12xlarge EC2 runner, S3 bucket `jack-devland-build`, `--skip-close`, and the same WAL-queue stats: | batch | WAL | target rows/s | actual rows/s | MB/s | p99 ms | final WAL queue | max WAL queue | thread setting | | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | | 10 | 16 MiB | 8000 | 7999.855 | 33.279 | 0.020 | 0.481% | 1.998% | rayon=64, tokio=64 | | 10 | 16 MiB | 11000 | 10999.782 | 45.759 | 0.019 | 0.481% | 2.495% | rayon=64, tokio=64 | | 100 | 8 MiB | 6000 | 5999.954 | 24.960 | 0.042 | 0.340% | 1.400% | rayon=64, tokio=32 | | 100 | 8 MiB | 9000 | 8999.850 | 37.439 | 0.045 | 0.340% | 2.100% | rayon=64, tokio=32 | | 1000 | 8 MiB | 8000 | 7999.913 | 33.280 | 0.084 | 0.100% | 1.800% | rayon=64, tokio=8 | | 1000 | 8 MiB | 11000 | 10999.909 | 45.760 | 0.154 | 4.600% | 4.600% | rayon=64, tokio=8 | This is not a full max-throughput sweep for the small schema, but it confirms the benchmark can run the requested older shape and that it reaches the same ~45.8 MB/s region at 11k rows/s, with the `batch=1000` high-rate case starting to show more WAL queue backlog than the smaller batches. ## Analysis Artifacts Saved local analysis: - `/Users/jackye/ai/analysis/lance/jack-mem-wal-hnsw/wal-queue-all-runs-20260515/RESULTS.md` - `/Users/jackye/ai/analysis/lance/jack-mem-wal-hnsw/thread-ablation-20260515/RESULTS.md` - `/Users/jackye/ai/analysis/lance/jack-mem-wal-hnsw/vector-only-pr-20260515/` S3 result artifacts: - `s3://jack-devland-build/memwal-walqueue-panel-20260514T171240Z/_bench_results/` - `s3://jack-devland-build/memwal-walqueue-supplement-20260514T182826Z/_bench_results/` - `s3://jack-devland-build/memwal-walqueue-higher-20260515T001817Z/_bench_results/` - `s3://jack-devland-build/memwal-thread-ablation-20260515T060229Z/_bench_results/` - `s3://jack-devland-build/memwal-vector-only-pr-20260515T090611Z/_bench_results/` ## Validation - `cargo fmt --all --check` - `cargo fmt --manifest-path python/Cargo.toml --all --check` - `cargo check -p lance --bench mem_wal_shard_writer_backpressure` - `cargo check --manifest-path python/Cargo.toml --lib` - `cargo clippy -p lance --tests -- -D warnings` - `cargo test -p lance dataset::mem_wal::index::hnsw::tests::test_index_concurrent_insert_and_search -- --exact` cc @hamersaw --------- Co-authored-by: Jack Ye <[email protected]>

claude Bot reviewed May 15, 2026

View reviewed changes

touch-of-grey changed the title ~~Add Lance-native MemWAL HNSW for shard writer~~ feat: native MemWAL HNSW for shard writer May 15, 2026

github-actions Bot added the enhancement New feature or request label May 15, 2026

touch-of-grey changed the title ~~feat: native MemWAL HNSW for shard writer~~ feat: add Lance-native MemWAL HNSW for shard writer May 15, 2026

github-actions Bot added the A-python Python bindings label May 15, 2026

touch-of-grey added 10 commits May 15, 2026 22:47

Add Lance HNSW prototype and benchmarks

a27153b

Integrate lance-hnsw with MemWAL shard writer

abb73fa

Move MemWAL HNSW into lance crate

c406ec0

Add native MemWAL shard writer backpressure bench

17644b5

Fix MemWAL HNSW benchmark capacity sizing

252a2ac

Add WAL queue stats to MemWAL benchmark

6eaa12c

fix: sync MemWAL Python stats fields

511899b

style: format Python MemWAL stats

048c90d

fix: remove redundant HNSW test clone

ff8bd76

fix: cap MemWAL HNSW search visibility

f1881f8

touch-of-grey force-pushed the LanceHNSW branch from 357e9e2 to f1881f8 Compare May 16, 2026 05:55

jackye1995 approved these changes May 16, 2026

View reviewed changes

jackye1995 merged commit 708cfe7 into lance-format:main May 16, 2026
28 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: add Lance-native MemWAL HNSW for shard writer#6795

feat: add Lance-native MemWAL HNSW for shard writer#6795
jackye1995 merged 11 commits into
lance-format:mainfrom
touch-of-grey:LanceHNSW

touch-of-grey commented May 15, 2026 •

edited

Loading

Uh oh!

claude Bot left a comment

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

jackye1995 commented May 15, 2026

Uh oh!

touch-of-grey commented May 15, 2026

Uh oh!

codecov Bot commented May 15, 2026 •

edited

Loading

Uh oh!

touch-of-grey commented May 15, 2026

Uh oh!

touch-of-grey commented May 15, 2026

Uh oh!

jackye1995 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

touch-of-grey commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Benchmark Summary So Far

Analysis Artifacts

Validation

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

github-actions Bot commented May 15, 2026

Uh oh!

jackye1995 commented May 15, 2026

Uh oh!

touch-of-grey commented May 15, 2026

Uh oh!

codecov Bot commented May 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

touch-of-grey commented May 15, 2026

Uh oh!

touch-of-grey commented May 15, 2026

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

touch-of-grey commented May 15, 2026 •

edited

Loading

codecov Bot commented May 15, 2026 •

edited

Loading