feat: builder-style MemWAL initialization API#6815
Merged
Conversation
Add a `writer_defaults` map to `MemWalIndexDetails` so the default ShardWriter configuration is persisted and resumes across writers, recorded at `initialize_mem_wal` time via `MemWalConfig.writer_defaults`.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
github-actions
Bot
added
enhancement
jackye1995
reviewed
May 17, 2026
| }; | ||
|
|
||
| let config = lance::dataset::mem_wal::MemWalConfig { | ||
| writer_defaults: Default::default(), |
Contributor
There was a problem hiding this comment.
I think at this point we should probably use a builder style. Some logic in your lancedb PR can also be moved here like the logic to do sharding with hash bucket, not shard, etc.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Replace the MemWalConfig / MemWalShardConfig structs and the initialize_mem_wal / initialize_mem_wal_with_shards pair with a fluent InitializeMemWalBuilder returned by Dataset::initialize_mem_wal. The builder exposes high-level sharding strategies — bucket_sharding(column, num_buckets) and unsharded() — owning the ShardingSpec construction and validation that previously lived in the LanceDB layer. num_shards is now derived from the sharding choice instead of being a separate input.
touch-of-grey
changed the title
identity_sharding(column) shards by the raw value of a scalar column (the "identity" transform), for data already partitioned by that column. It validates that the column exists and is a scalar type usable as a shard key, and records an open-ended shard count.
jackye1995
reviewed
May 17, 2026
| /// Requires the dataset schema to have at least one field with | ||
| /// the `lance-schema:unenforced-primary-key` metadata. | ||
| #[pyo3(signature=(maintained_indexes=None, region_spec=None))] | ||
| #[pyo3(signature=(maintained_indexes=None))] |
Contributor
There was a problem hiding this comment.
Should make sure python can take all input segments in this function and then invoke the builder in the binding
jackye1995
reviewed
May 17, 2026
| )) | ||
| /// These are defaults only; an individual writer may still override any | ||
| /// value at runtime in its own (non-persisted) `ShardWriterConfig`. | ||
| pub fn writer_defaults<I, K, V>(mut self, defaults: I) -> Self |
Contributor
There was a problem hiding this comment.
We should just move all the ones you had in lancedb over here to allow setting specific config defaults. We can leave a add_writer_default to add arbitrary config defaults.
The Python initialize_mem_wal binding now accepts every builder input — sharding strategy (bucket / identity / unsharded), maintained indexes, and ShardWriter config defaults — and invokes InitializeMemWalBuilder. Adds a mem_wal_index_details binding for reading the recorded MemWAL details back. The builder's writer-defaults API is replaced by writer_config_defaults, which takes a ShardWriterConfig and persists its tunable fields, plus add_writer_default for arbitrary keys.
jackye1995
reviewed
May 17, 2026
| // configuration. These are defaults only: an individual writer may | ||
| // still override any value at runtime in its own ShardWriterConfig | ||
| // (which is not persisted). | ||
| map<string, string> writer_defaults = 11; |
Contributor
There was a problem hiding this comment.
we should also rename it as writer_config_defaults
jackye1995
reviewed
May 17, 2026
| /// | ||
| /// Use this for keys not covered by | ||
| /// [`writer_config_defaults`](Self::writer_config_defaults). | ||
| pub fn add_writer_default(mut self, key: impl Into<String>, value: impl Into<String>) -> Self { |
Contributor
There was a problem hiding this comment.
similarly, add_writer_config_default
Renames the persisted MemWalIndexDetails field (protobuf field 11), the InitializeMemWalBuilder API, and the Python binding accordingly; add_writer_default becomes add_writer_config_default.
jackye1995
approved these changes
May 17, 2026
jackye1995
left a comment
jackye1995
left a comment
Contributor
There was a problem hiding this comment.
this looks great! thanks for the fixes
touch-of-grey
added a commit
to touch-of-grey/lancedb
that referenced
this pull request
May 18, 2026
Now that lance-format/lance#6815 has merged, point the Lance dependency at the main branch and rewrite set_lsm_write_spec to call the new InitializeMemWalBuilder (bucket_sharding / unsharded), removing the local shard-spec construction boilerplate.
touch-of-grey
added a commit
to touch-of-grey/lancedb
that referenced
this pull request
May 18, 2026
The MemWAL builder changes (lance-format/lance#6815) are now released in v7.0.0-beta.12; replace the temporary main-branch dependency with the tag.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the struct-based MemWAL initialization API with a fluent builder on
Dataset, and persists defaultShardWriterconfiguration in the MemWAL index.Dataset::initialize_mem_wal()returns anInitializeMemWalBuilder:dataset .initialize_mem_wal() .bucket_sharding("id", 16) .maintained_indexes(["id_btree"]) .writer_config_defaults(ShardWriterConfig::default().with_durable_write(false)) .execute() .await?;bucket_sharding(column, num_buckets),unsharded(), andidentity_sharding(column)are high-level sharding strategies. They own theShardingSpecconstruction and validation that callers previously did byhand;
num_shardsis derived from the sharding choice.writer_config_defaults(ShardWriterConfig)records the tunable writerconfiguration as the persisted defaults — a new
map<string, string> writer_config_defaultsfield on theMemWalIndexDetailsprotobuf message —so every writer, across processes and restarts, starts from the same
defaults.
add_writer_config_defaultrecords arbitrary extra keys.initialize_mem_walbinding accepts the full builder surface(sharding, maintained indexes, writer-config defaults) and invokes the
builder; a new
mem_wal_index_detailsbinding reads the recorded details back.MemWalConfig,MemWalShardConfig,initialize_mem_wal_with_shards.Context
While reviewing lancedb/lancedb#3396, @jackye1995 noted that per-writer tuning
knobs are runtime configuration and should not be persisted as part of the
sharding spec. That holds for a writer's live
ShardWriterConfig, which staysnon-persisted. This PR records only the defaults: without a persisted
default, writers would be configured independently and could silently drift
apart.
@jackye1995 also asked for a builder style so bucket sharding is easy to set,
moving the hash-bucket spec logic out of the LanceDB layer into Lance. The
builder owns all three sharding strategies (bucket, unsharded, identity), so
LanceDB's
set_lsm_write_speccan call the builder directly instead ofhand-building shard specs.