Summary

In v7.0.0-rc.1, InitializeMemWalBuilder::bucket_sharding no longer enforces any relationship between the bucket column and the dataset's unenforced primary key. This appears to be an unintended over-relaxation introduced by #6848, whose description states the opposite.

What #6848's description says

• allow MemWAL initialization on append-only tables without unenforced primary key metadata
• keep bucket sharding constrained to the primary key when a primary key exists, but allow no-PK tables to bucket by a non-nested column

What the merged code actually does

bucket_sharding_spec in rust/lance/src/dataset/mem_wal/api.rs validates only:

1. num_buckets is in [1, 1024]
2. column exists on the dataset
3. column has a supported scalar type

There is no check that a primary key exists, and no check that column equals the PK column — even when a PK is defined. Diff vs v7.0.0-beta.13:

-    let pk_fields = dataset.schema().unenforced_primary_key();
-    let pk = match pk_fields.as_slice() {
-        [single] => *single,
-        _ => return Err(Error::invalid_input(
-            "bucket_sharding requires a single-column unenforced primary key; ...")),
-    };
-    if pk.name.as_str() != column {
-        return Err(Error::invalid_input(format!(
-            "bucket_sharding: column '{}' does not match the unenforced primary key column '{}'", ...)));
+    let source_field = dataset.schema().field(column).ok_or_else(|| { ... })?;
+    let data_type = source_field.data_type();
+    if !is_bucket_sharding_supported_type(&data_type) { ... }

The "allow no-PK tables to bucket by a non-nested column" half landed; the "keep bucket sharding constrained to the primary key when a primary key exists" half did not.

Why this matters

For a PK table, bucket_sharding is meant to hash-bucket by the primary key so that all rows sharing a primary key deterministically land in the same shard. With the constraint gone, a caller can now bucket_sharding on an arbitrary scalar column unrelated to the PK; rows with the same PK can then be routed to different shards. #6848's own updated doc comments still describe identity sharding as relying on "every primary key maps consistently to a single value of column", so this invariant is clearly still expected to matter.

Impact

Discovered while bumping LanceDB to v7.0.0-rc.1 (lancedb/lancedb#3428). LanceDB's LsmWriteSpec::Bucket is documented as "column must equal the table's currently-set single-column unenforced primary key", and its tests assert both rejections (no-PK and column≠PK). Both assertions now fail against rc.1.

Question

Is the full removal of the PK relationship intentional (in which case the #6848 description and the surrounding doc comments should be corrected), or should bucket_sharding keep rejecting a bucket column that doesn't match the primary key when one exists?

cc @touch-of-grey