feat: implement vector index details by wjones127 · Pull Request #6099 · lance-format/lance

wjones127 · 2026-03-04T20:27:33Z

Cache vector index configuration within the index metadata, such as the distance type and build parameters.

Previously, to determine things like the distance type or index type of a vector index, the index file itself had to be opened. This PR stores that information in VectorIndexDetails within the manifest's index_details field, which is fetched and cached eagerly when loading the manifest.

Old indexes have this field left blank. When blank, the details are extracted from the index files and cached. This migration happens on the first write with a new library version.

What's stored in VectorIndexDetails

Core build parameters (typed fields — required for any runtime to build the index):

metric_type
target_partition_size (IVF)
hnsw_index_config — max_connections, construction_ef, max_level (HNSW)
compression — PQ/SQ/RQ/flat, including num_bits, num_sub_vectors, rotation_type

Runtime hints (map<string, string> runtime_hints):
Optional build preferences that don't affect index structure. Stored so a background rebuild process can reproduce the original configuration. Runtimes that don't recognize a key must silently ignore it. Only non-default values are written.

Keys use reverse-DNS namespacing: lance.* for core Lance hints, other prefixes for runtime-specific hints (e.g., lancedb.accelerator for GPU acceleration in LanceDB Enterprise).

Current lance.* hints: lance.ivf.max_iters, lance.ivf.sample_rate, lance.ivf.shuffle_partition_batches, lance.ivf.shuffle_partition_concurrency, lance.pq.max_iters, lance.pq.sample_rate, lance.pq.kmeans_redos, lance.sq.sample_rate, lance.hnsw.prefetch_distance, lance.skip_transpose.

Also adds apply_runtime_hints() to read hints back into build params for future rebuild logic.

Closes #5963

github-actions · 2026-03-04T20:29:10Z

PR Review: feat: implement vector index details

P0: `Bits` enum doesn't cover RaBitQ's actual bit-width

RaBitQ in this codebase uses 1 bit per dimension (see rust/lance-index/src/vector/bq/builder.rs:40 where num_bits: 1). The Bits enum only has BIT_8 = 0 and BIT_4 = 1 — there's no BIT_1 variant. This means RabitQuantization.num_bits cannot represent the actual quantization used by RaBitQ today.

Either:

Add a BIT_1 = 2 variant, or
Remove num_bits from RabitQuantization since RaBitQ is always 1-bit in practice (the field in the Rust struct exists but is always set to 1).

P1: No corresponding Rust code to populate the new fields

All existing call sites create VectorIndexDetails::default() (e.g., rust/lance/src/index.rs:361, rust/lance/src/dataset/index.rs:93). This PR only changes the proto — is there a follow-up PR planned to actually populate these fields during index creation? Without that, describe_indices() still won't have the metadata. Consider noting this in the PR description or linking a follow-up issue.

P1: `target_partition_size` may be better as `uint32` or have docs clarifying semantics

target_partition_size is uint64 but IVF partition counts are typically small numbers (sqrt of dataset size). More importantly, is this the target number of partitions (i.e. num_partitions) or the target size per partition? The issue mentions num_partitions — if this represents partition count, the field name is misleading. Consider renaming to num_partitions to match the terminology used elsewhere in the codebase and in the issue description.

Minor

The Bits enum name is generic — if other proto messages in this file or future messages need a Bits enum with different semantics, there could be a naming collision. Since it's nested inside VectorIndexDetails this is scoped, but worth noting.
The comment on VectorIndexDetails still says "Empty details messages for older indexes" — should be updated to reflect the message is no longer empty.

codecov · 2026-03-04T21:21:11Z

Codecov Report

❌ Patch coverage is 92.93219% with 74 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
rust/lance/src/index/vector/details.rs	93.36%	32 Missing and 27 partials ⚠️
rust/lance/src/dataset/scanner.rs	25.00%	9 Missing ⚠️
rust/lance/src/index.rs	93.18%	4 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

Previously, vector indices returned index_type "Unknown" and empty details in describe_indices(). This populates VectorIndexDetails at creation time from build params, derives a human-readable index type string (e.g. "IVF_PQ"), serializes details as JSON, and infers details from index files on disk as a fallback for legacy indices. Also changes proto num_bits from Bits enum to uint32 to support RQ's default of 1 bit, and adds rotation_type to RabitQuantization. Co-Authored-By: Claude Opus 4.6 <[email protected]>

Replace imperative serde_json::Map construction with #[derive(Serialize)] structs for clearer, more maintainable JSON serialization. This also adds the missing rotation_type field to RQ compression output. Add snapshot-style unit tests that assert exact JSON strings to guard backwards compatibility of the describe_indices() output format. Co-Authored-By: Claude Opus 4.6 <[email protected]>

Previously, vector index details for legacy indices were only inferred lazily in describe_indices(). This moves inference to load_indices() and migrate_indices(), so details are populated before caching and persisted into new manifest versions. Inference runs once per index name, concurrently. Co-Authored-By: Claude Opus 4.6 <[email protected]>

Also handles the case where index_details is None (very old indices) by checking if the indexed field is a vector type. Moves inference outside the cache-miss branch in load_indices so it also runs on indices that were opportunistically cached during Dataset::open. Co-Authored-By: Claude Opus 4.6 <[email protected]>

- Fix compression oneof field numbers (5,6,7 -> 4,5,6) to avoid gap - Add comment that target_partition_size = 0 means unset - Extract infer_missing_vector_details helper to deduplicate logic between load_indices and migrate_indices Co-Authored-By: Claude Opus 4.6 <[email protected]>

…etails # Conflicts: # rust/lance/src/index/append.rs # rust/lance/src/index/create.rs # rust/lance/src/index/vector.rs

wjones127 · 2026-03-05T19:46:16Z

+// Details for vector indexes.
+message VectorIndexDetails {
+  enum VectorMetricType {
+    L2 = 0;
+    COSINE = 1;
+    DOT = 2;
+    HAMMING = 3;
+  }
+
+  VectorMetricType metric_type = 1;
+
+  // 0 means unset (unknown or not applicable).
+  uint64 target_partition_size = 2;
+
+  optional HnswIndexDetails hnsw_index_config = 3;
+
+  message ProductQuantization {
+    uint32 num_bits = 1;
+    uint32 num_sub_vectors = 2;
+  }
+  message ScalarQuantization {
+    uint32 num_bits = 1;
+  }
+  message RabitQuantization {
+    enum RotationType {
+      FAST = 0;
+      MATRIX = 1;
+    }
+    uint32 num_bits = 1;
+    RotationType rotation_type = 2;
+  }
+
+  // An unset compression oneof means flat / no quantization.
+  oneof compression {
+    ProductQuantization pq = 4;
+    ScalarQuantization sq = 5;
+    RabitQuantization rq = 6;
+  }
+}
+
+// Hierarchical Navigable Small World (HNSW) index details, used as an optional configuration for IVF indexes.
+message HnswIndexDetails {
+  // The maximum number of outgoing edges per node in the HNSW graph. Higher values
+  // means more connections, better recall, but more memory and slower builds.
+  // Referred to as "M" in the HNSW literature.
+  uint32 max_connections = 1;
+  // "construction exploration factor": The size of the dynamic list used during
+  // index construction.
+  uint32 construction_ef = 2;
+}


This representation works for the existing set of vector indices, but wondering if this is good for future plans. The current internal design has the concept of "stages", so you could have something like ivf-ivf-pq or hnsw-pq (no IVF). Some sequence of stages just doesn't make sense, like pq-pq. So I'm not sure the stages representation makes sense.

I was thinking it could be a tree-like system, where IVF and HNSW could have children, but PQ and SQ can't.

What do you think? @BubbleCal @eddyxu

I've always seen it as three layers. The first is partitioning (IVF is really the only choice here). The second is searching within partition (flat vs hnsw) and the third is quantization (pq, rq, sq, etc.) So I don't think the concept of stages makes sense. For example, I don't see ivf-ivf-pq. I see ivf (layers=2) - pq.

…x.proto These messages belong with other index-related protos. After the move, VectorIndexDetails reuses the existing top-level VectorMetricType enum instead of defining its own nested copy. Rust imports updated from lance_table::format::pb to lance_index::pb throughout. Co-Authored-By: Claude Opus 4.6 <[email protected]>

…rd, type_url - Fix inverted comment on hnsw_index_config field in index.proto - Use tracing::warn! instead of log::warn! in details.rs - Prefer non-empty index_details when carrying forward in append - Revert describe_indices to original chunk_by pattern - Update Python test type_url to match new proto package Co-Authored-By: Claude Opus 4.6 <[email protected]>

westonpace

I'm not 100% sure we can get away with changing the protobuf type URL. Can you create a compatibility test to ensure that old versions can read new indexes created with these new details?

westonpace · 2026-03-05T22:39:24Z

 }

+// Details for vector indexes, stored in the manifest's index_details field.
+message VectorIndexDetails {


Where are the IVF details? was it hierarchical? How many partitions in each stage?.

I'm using VectorIndexDetails here for the IVF stage. I suppose I could make it a substruct to make it clearer. The only parameter right now is target_partition_size.

How many partitions in each stage?

What do you mean by this?

westonpace · 2026-03-05T22:41:39Z

+}
+
+// Hierarchical Navigable Small World (HNSW) index details, used as an optional configuration for IVF indexes.
+message HnswIndexDetails {


Minor nit: I think of "index details" as the top-level message describing a type of index. This is a nested message that cannot stand on its own so maybe just HnswParameters?

Yeah that's fair.

westonpace · 2026-03-05T22:42:18Z

-    assert info.type_url == "/lance.table.VectorIndexDetails"
-    # This is currently Unknown because vector indices are not yet handled by plugins
-    assert info.index_type == "Unknown"
+    assert info.type_url == "/lance.index.pb.VectorIndexDetails"


Regrettably, I do not think we can change the type URL for backwards compatibility reasons.

I can move that back if it's necessary.

westonpace · 2026-03-05T22:44:03Z

+            // Carry forward existing index details, preferring the first segment
+            // that has populated (non-empty) details.


I think this is a safe assumption but we are assuming all segments have the same details right?

Should be. Ideally, we would create some top-level index configuration that is deduplicated across segments, but that's a complex format change for another day.

westonpace · 2026-03-05T22:44:46Z

+use vector::details::{
+    derive_vector_index_type, infer_missing_vector_details, vector_details_as_json,
+};
+pub(crate) use vector::details::{vector_index_details, vector_index_details_default};


Should these be at the top of the file?

Yeah I should move those.

wjones127 · 2026-03-05T22:51:42Z

I'm not 100% sure we can get away with changing the protobuf type URL. Can you create a compatibility test to ensure that old versions can read new indexes created with these new details?

We have one here that goes back to 0.29.1.beta2:

lance/python/python/tests/compat/test_vector_indices.py

Lines 25 to 32 in db07e77

    
           @compat_test(min_version="0.29.1.beta2") 
        
           class PqVectorIndex(UpgradeDowngradeTest): 
        
               """Test PQ (Product Quantization) vector index compatibility.""" 
        
               def __init__(self, path: Path): 
        
                   self.path = path 
        
               def create(self):

I wonder if we need to check earlier than that to see incompatabilities.

wjones127 · 2026-03-09T21:03:32Z

Should be able to implement this TODO:

lance/rust/lance/src/dataset/scanner.rs

Lines 3422 to 3433 in de393a2

    
           // TODO: Once we do https://github.com/lance-format/lance/issues/5231, we 
        
           // should be able to get the metric type directly from the index metadata, 
        
           // at least for newer indexes. 
        
           let idx = self 
        
               .dataset 
        
               .open_vector_index( 
        
                   q.column.as_str(), 
        
                   &index.uuid.to_string(), 
        
                   &NoOpMetricsCollector, 
        
               ) 
        
               .await?; 
        
           let index_metric = idx.metric_type();

wjones127 · 2026-03-13T22:36:28Z

@westonpace I had claude go through and see if this breaks backwards compat as is. Other than the type_url, it claims that the indexes don't have different compatibility from what's on main:

  Results                                                                                                                                                                                                                                                                                                               
                                                                                                                                                                                                                                                                                                                      
  The protobuf move (VectorIndexDetails from table.proto to index.proto) does not break backward compatibility. All failures seen are pre-existing and identical when reading indices written by main vs this branch.                                                                                                   
                                                                                                                                                                                                                                                                                                                        
  ┌───────────────┬────────────────────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐                                                                                                                                                  
  │ Version range │              Behavior              │                                                    Cause                                                    │                                                                                                                                                  
  ├───────────────┼────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤                                                                                                                                                  
  │ 0.8.0         │ Can't open dataset                 │ Old manifest format                                                                                         │                                                                                                                                                  
  ├───────────────┼────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤                                                                                                                                                  
  │ 0.9.0-0.10.0  │ Opens, panics on search            │ Very old index format                                                                                       │
  ├───────────────┼────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 0.11.0-0.14.0 │ Import error                       │ NumPy 1.x/2.x incompatibility                                                                               │
  ├───────────────┼────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 0.15.0-0.16.0 │ Works (brute force)                │ Doesn't recognize index, falls back to scan                                                                 │
  ├───────────────┼────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 0.17.0-0.29.0 │ Opens, sees index, fails to search │ Pre-existing index format incompatibility (missing field num_bits, 2-D tensor shape) — same failure on main │
  ├───────────────┼────────────────────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
  │ 0.30.0+       │ Works                              │ Full compatibility                                                                                          │
  └───────────────┴────────────────────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

  The earliest version that can successfully use an IVF_PQ index written by this branch is 0.30.0 — but this is the same boundary as main. The registry lookup at rust/lance-index/src/registry.rs:98 uses details.type_url.split('.').next_back() which extracts just "VectorIndexDetails" regardless of package, so
  the package name change is transparent.

westonpace

Sweet. Lets go for it.

…etails

Previously, `scanner.rs:3425` called `open_vector_index` just to get the metric type for metric compatibility checks. This required expensive deserialization of the index file. Now we read the metric type directly from `IndexMetadata.index_details` (a `VectorIndexDetails` proto) added on the feat/vector-index-details branch. This provides a fast path for newer indices without I/O. For legacy indices without populated details (empty proto value bytes), we fall back to the original expensive path. Adds `metric_type_from_index_metadata` helper in `details.rs` that: - Returns `None` for missing or empty details (legacy indices) - Converts `VectorIndexDetails.metric_type` to `DistanceType` for populated details - Uses the existing `From<VectorMetricType> for DistanceType` impl Changes `matching_index` tuple from `(index, idx, index_metric)` to `(index, index_metric)` since `idx` is only used in the fallback path. Fixes lance-format#5231 Co-Authored-By: Claude Haiku 4.5 <[email protected]>

BubbleCal · 2026-03-16T18:25:46Z

+  }
+
+  // An unset compression oneof means flat / no quantization.
+  oneof compression {


maybe worth adding index_version for each one.
Found it's complicated for vector index to maintain compatibility when introducing breaking changes because it's hard to get the index version.

The other scalar index has index_version already today.

- Rename HnswIndexDetails -> HnswParameters (westonpace) - Move imports to top of index.rs (westonpace) - Add index_version field to VectorIndexDetails proto (BubbleCal) - Add explicit FlatCompression message instead of using unset oneof (westonpace) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…e into feat/vector-index-details

…etails # Conflicts: # rust/lance/src/index/append.rs # rust/lance/src/index/vector/ivf.rs

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

VectorIndexDetails moved from table.proto to index.proto, so the Python binding needs to reference lance_index::pb instead of lance_table::format::pb. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Adds a `map<string, string> runtime_hints` field to `VectorIndexDetails` for storing optional build preferences that don't affect index structure (e.g., KMeans iterations, shuffle concurrency, GPU accelerator). These are needed so a background index rebuild process can reproduce the original build configuration. Keys use reverse-DNS namespacing: `lance.*` for core Lance hints, `lancedb.*` for LanceDB-specific hints (e.g., `lancedb.accelerator`). Runtimes that don't recognize a key must silently ignore it. Only non-default values are written to keep the map minimal. Also adds `apply_runtime_hints()` to read hints back into build params. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

…uild Adds `vector_params_from_details()` to reconstruct a full `VectorIndexParams` from stored `VectorIndexDetails` (core spec fields + runtime hints). This enables future index rebuild logic to reproduce the original build config from the manifest without re-opening index files. Also wires the `lance.skip_transpose` hint into `optimize_vector_indices_v2` so incremental rebuilds honour the original skip_transpose preference rather than silently reverting to false on each append. Adds Python tests validating that non-default build params appear as `runtime_hints` in `describe_indices()` output, and that default values are omitted. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

- `python/src/dataset.rs`: `max_iters` kwarg was not forwarded to `IvfBuildParams`/`PQBuildParams` in `prepare_vector_index_params`, so it was silently ignored and never stored as a runtime hint - `python/src/indices.rs`: wrong proto path for `VectorIndexDetails` (`lance_table::format::pb` → `lance_index::pb`) - `java/lance-jni/src/utils.rs`: missing `runtime_hints` field in `VectorIndexParams` struct literal Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

wjones127 · 2026-04-10T21:57:13Z

@westonpace @BubbleCal do you want to take another look? I changed up the format a little bit to handle other index parameters.

BubbleCal · 2026-04-13T06:42:37Z

+
+#[derive(Serialize)]
+#[serde(tag = "type", rename_all = "lowercase")]
+enum CompressionDetailsJson {


missed Flat

I think that's covered by None in compression: Option<CompressionDetailsJson>,

…etails

…etails # Conflicts: # rust/lance/src/index/vector/ivf.rs

- Add parametrised round-trip test across (subindex, quantizer, metric) for VectorIndexDetails. - Fix convert_v3_metadata_to_details: read HNSW under HNSW_METADATA_KEY (not IVF_PARTITION_KEY); read quantizer metadata from the auxiliary file under STORAGE_METADATA_KEY; derive index shape from INDEX_METADATA_SCHEMA_KEY.index_type. Tighten the legacy-inference test to assert the inferred type and compression match the original. - Drop skip_transpose from runtime_hints. It's a build-time flag for distributed intermediate shards, not something a finished index should persist. Removes the with_transpose plumbing in optimize_vector_indices. - Mark vector_params_from_details / apply_runtime_hints as #[allow(dead_code)] with a TODO to wire them into a future Dataset::rebuild_index method. - Move in-function infer_missing_vector_details import to top of commit.rs. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

- build_segment_plan: fall back to vector_index_details_default() when a source segment has no index_details, instead of erroring out. The empty placeholder triggers re-inference on the next manifest load. - IndexDescriptionImpl::try_new: report the system index type for frag-reuse / mem-wal indices via infer_system_index_type, instead of falling through to the plugin lookup that returns "Unknown". Fixes test_index_segment_builder_vector_segments_without_index_details and test_describe_indices_matches_list_indices_for_frag_reuse. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

The default-comparison logic for omitting hints was fragile (could drift if upstream defaults changed) and offered little benefit. Always record the actual value so the stored details capture exactly what the index was built with.

Hints are now always written, so this assertion no longer holds.

This needs lance-format/lance#6099 to work. Closes #3140 --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* feat(rust): support DataFusion Expr for table row deletions (#3415) Modified the parameter of delete to a Predicate that could be constructed from either datafusion Expr, from str (to support SQL predicate), or from String to support python and javascript bindings. When a datafusion Expr is used, it avoids the overhead of serializing to SQL and re-parsing when callers already have an Expr (e.g. from query planning). The native implementation uses lance's `DeleteBuilder::from_expr`. The remote implementation converts the Expr to SQL via `expr_to_sql_string` before sending to the server, consistent with the existing query and count_rows paths. Closes #3204 Signed-off-by: Yuval Lifshitz <[email protected]> Co-authored-by: Claude Code <[email protected]> * feat(remote): send read freshness headers for remote table consistency (#3439) Closes client side work of #3370 ### Summary - Plumbs `read_consistency_interval` from `ConnectBuilder` through `RestfulLanceDbClient` so remote reads attach an `x-lancedb-min-timestamp` freshness header. None = no header (default), zero = "now", positive = `now - interval`. - Adds per-table `FreshnessState` on `RemoteTable`: write responses (`update`, `delete`, `merge_insert`, `add_columns`, `alter_columns`, `drop_columns`) track the committed version, and the next read sends `x-lancedb-min-version` so the server's cache honors read-your-write. - `checkout(v)` / `checkout_tag(t)` / `checkout_latest()` / `restore()` reset the freshness state appropriately; the validating `/describe/` and tag-resolve requests are sent without freshness headers so they don't carry stale state. - Updates Rust, Python, and Node docstrings and calls out that stronger consistency raises per-read latency and cost. ### Testing - Unit tests cover default behavior, interval=0, positive interval, checkout_latest baseline, min_version-after-write, checkout clears state, and the two no-stale-header invariants on `checkout(v)` and `checkout_tag(t)`. - Ran smoke tests against local remote table to verify functionality * chore(deps): bump the rust-minor-patch group across 1 directory with 2 updates (#3440) Bumps the rust-minor-patch group with 2 updates in the / directory: [serde_json](https://github.com/serde-rs/json) and [aws-smithy-runtime](https://github.com/smithy-lang/smithy-rs). Updates `serde_json` from 1.0.149 to 1.0.150 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/serde-rs/json/releases">serde_json's releases</a>.</em></p> <blockquote> <h2>v1.0.150</h2> <ul> <li>Reject non-string enum object keys (<a href="https://redirect.github.com/serde-rs/json/issues/1324">#1324</a>, thanks <a href="https://github.com/puneetdixit200"><code>@puneetdixit200</code></a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/serde-rs/json/commit/a1ae73ac6a6940a4a57c673aebaa13ed4dfe3e8c"><code>a1ae73a</code></a> Release 1.0.150</li> <li><a href="https://github.com/serde-rs/json/commit/1a360b0a6c003912afc3503c834b0edd798bca28"><code>1a360b0</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/json/issues/1324">#1324</a> from puneetdixit200/reject-non-string-enum-keys</li> <li><a href="https://github.com/serde-rs/json/commit/2037b634f9dccbddc11cff189ebeb5854fa0e01c"><code>2037b63</code></a> Reject non-string enum object keys</li> <li><a href="https://github.com/serde-rs/json/commit/5d30df60e916e9b8fc46c74794007ff271fdfbbf"><code>5d30df6</code></a> Resolve manual_assert_eq pedantic clippy lint</li> <li><a href="https://github.com/serde-rs/json/commit/dc8003a88e7142529cf4a7429c4778af31dadf50"><code>dc8003a</code></a> Raise required compiler for preserve_order feature to 1.85</li> <li><a href="https://github.com/serde-rs/json/commit/a42fa980f8556cda36d896fa3713544b2e5eaa2c"><code>a42fa98</code></a> Unpin CI miri toolchain</li> <li><a href="https://github.com/serde-rs/json/commit/684a60eba18abfc0e0f7ddb0c2cd39f8f60249cf"><code>684a60e</code></a> Pin CI miri to nightly-2026-02-11</li> <li><a href="https://github.com/serde-rs/json/commit/7c7da3302b6b1cdab7f11ea49ca1a74422ab4551"><code>7c7da33</code></a> Raise required compiler to Rust 1.71</li> <li><a href="https://github.com/serde-rs/json/commit/acf4850e2969f1caccab2c4727a90ed006ba35bb"><code>acf4850</code></a> Simplify Number::is_f64</li> <li><a href="https://github.com/serde-rs/json/commit/6b8ceab565dcfe4f83dfaacd287d11c8bd8f306c"><code>6b8ceab</code></a> Resolve unnecessary_map_or clippy lint</li> <li>Additional commits viewable in <a href="https://github.com/serde-rs/json/compare/v1.0.149...v1.0.150">compare view</a></li> </ul> </details> <br /> Updates `aws-smithy-runtime` from 1.11.1 to 1.11.3 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(rerankers): inverted scores and incorrect missing-FTS penalty in LinearCombinationReranker (#3437) ## Problem `LinearCombinationReranker.merge_results` has two related bugs that make it return **inverted relevance rankings** — the least relevant document ranks first (closes #3154). ### Bug 1 — `_combine_score` subtracts from 1, inverting the final ranking ```python def _combine_score(self, vector_score, fts_score): return 1 - (self.weight * vector_score + (1 - self.weight) * fts_score) ``` Both `vector_score` (already converted via `_invert_score`) and `fts_score` (BM25 relevance) are in **higher-is-better** space. Wrapping the weighted average in `1 - (...)` flips the direction: a perfectly matching document (`vector_score=1, fts_score=1`) gets `_relevance_score = 0.0`, while a non-matching document gets a high score. ### Bug 2 — Documents missing an FTS score are rewarded, not penalised ```python fts_score = result.get("_score", fill) # fill=1.0 by default ``` When a document has no FTS match, `fts_score = fill = 1.0`. In `_combine_score` (with the bug-1 formula), this large value becomes a **negative penalty** via `1 - (... + 0.3 * 1.0)`, counterintuitively *boosting* the document's score. By contrast, missing vector results correctly receive `_invert_score(fill) = 0.0` (penalised). ## Fix **Bug 1** — remove the `1 -` inversion from `_combine_score`: ```python def _combine_score(self, vector_score, fts_score): return self.weight * vector_score + (1 - self.weight) * fts_score ``` **Bug 2** — use `1 - fill` for missing FTS scores so both penalties are symmetric (mirror of what `_invert_score(fill)` already does for missing vector scores): ```python fts_score = result.get("_score", 1 - fill) # was: fill ``` With `fill=1.0` (default): `1 - 1.0 = 0.0` — missing-FTS entries contribute `0` to the FTS term, identical to how missing-vector entries contribute `0` to the vector term. ## Verification Concrete example from the issue. With `weight=0.7`, `fill=1.0`: | Document | `_distance` | `_score` | Old `_relevance_score` | New `_relevance_score` | |----------|-------------|----------|------------------------|------------------------| | `apple orange` | 0.0 (best) | 2.41 (only FTS) | 0.30 (**wrong: ranked 2nd**) | 1.42 (**correct: ranked 1st**) | | `banana grape` | 0.9999 (worst) | — | 0.70 (**wrong: ranked 1st**) | 0.00 (**correct: ranked last**) | ## Tests Two regression tests added to `python/python/tests/test_rerankers.py`: - `test_linear_combination_best_match_ranks_first` — the document with the smallest distance **and** an FTS match must have the highest `_relevance_score`. - `test_linear_combination_missing_fts_is_penalised` — a document with any FTS score must beat an otherwise-equal document with no FTS match. --------- Co-authored-by: Will Jones <[email protected]> * fix: remove primary key constraint from MemWAL bucket sharding (#3435) ## Summary - Bump lance dependency from `v7.0.0-beta.13` to `v7.0.0-rc.1` - Remove PK constraint from `LsmWriteSpec::Bucket` docs and `Table::set_lsm_write_spec` docs - Remove test assertions that expected rejection when no PK is set or when bucket column != PK Closes https://github.com/lance-format/lance/issues/6917 * fix: allow appending arrow.json data into lance.json tables (#3429) When a table is created with `pa.json_()` (PyArrow's JSON extension type), it is stored internally as `lance.json` (LargeBinary with `lance.json` extension metadata). Calling `table.add()` with `pa.json_()` data failed with: ``` RuntimeError: lance error: Append with different schema: `data` should have type json but type was large_binary ``` `build_field_exprs` in `rust/lancedb/src/table/datafusion/cast.rs` saw that the input field (`Utf8` with `arrow.json` metadata) differed from the table field (`LargeBinary` with `lance.json` metadata). Since `can_cast_types(Utf8, LargeBinary)` is true, it inserted a DataFusion `Utf8 → LargeBinary` cast. That cast preserved the input field's `arrow.json` extension metadata instead of adopting the table's `lance.json` metadata, so lance-core detected a schema mismatch and rejected the append. This adds a special case in `build_field_exprs`: when the input is `arrow.json` and the table field is `lance.json`, the expression is passed through unchanged. Lance-core's write path already handles the `arrow.json → lance.json` conversion (including JSONB encoding), so no DataFusion cast is needed. Fixes #3144 Continues #3291 from a fork (the original author's branch could not be pushed to). The original commits are preserved; an additional commit fixes the CI failures on that PR — formatting, a missing trait import, and read-back assertions that assumed binary storage when a lance.json column is read back as `Utf8`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: yunju.lly <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * chore: update lance dependency to v7.1.0-beta.4 (#3450) ## Summary - Updates Lance Rust workspace dependencies to `v7.1.0-beta.4` using `ci/set_lance_version.py`. - Updates the Java `lance-core` dependency property to `7.1.0-beta.4`. - Triggering Lance tag: https://github.com/lance-format/lance/releases/tag/v7.1.0-beta.4 ## Verification - `cargo clippy --workspace --tests --all-features -- -D warnings` - `cargo fmt --all` Co-authored-by: Daniel Rammer <[email protected]> * ci: fix pypi publish on mac/windows/arm (#3449) The python-v0.32.0 publish run failed on every build matrix entry. Three independent issues: 1. **Mac and Windows**: `pypa/gh-action-pypi-publish` only runs on Linux, but was being called inline from each build job. 2. **Linux (all arches)**: `pypa/gh-action-pypi-publish` derives its docker image name from `github.action_repository`, which is empty when the action is invoked from inside a composite action (actions/runner#2473 — pypa's own `action.yml` references this bug). It falls back to `github.repository`, generating `docker://ghcr.io/lancedb/lancedb:<tag>`, which doesn't exist → `denied`. Only the ARM matrix entry surfaced this because it failed first and cancel-cascaded the rest. 3. **Windows**: `upload-artifact` in `build_windows_wheel` pointed at `python\target\wheels`, but maturin writes to the workspace-root `target/wheels`. The artifact was always empty. Also, `pypi-publish.yml` passed a `vcpkg_token` input that the composite doesn't declare. ## Changes - Build jobs (linux/mac/windows) now upload their wheels as `actions/upload-artifact` artifacts. - New Linux `publish` job downloads all wheel artifacts and runs the Fury or PyPA publish step directly (not via a composite), so `github.action_repository` resolves correctly. - Delete the unused `upload_wheel` composite action. - Drop the broken upload-artifact step inside `build_windows_wheel`. - Remove the bogus `vcpkg_token` input. - Fury upload now loops over all wheels instead of just the first. - Bump `actions/checkout`, `actions/upload-artifact`, `actions/download-artifact` to current major versions (Node 24) to clear deprecation warnings. - Bump Windows job timeout 60 → 90 minutes; previous run was cancel-timing-out on a 60m cap. - Use `rust-lld` as the Windows MSVC linker via `CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER`. `link.exe` is single-threaded and the long pole on Windows builds. Fixes #3445 ## Test plan - [x] Open this PR — `paths` filter triggers a dry-run build on all three platforms. - [x] Verify all three builds produce wheels. - [x] Confirm the `pypa/gh-action-pypi-publish` container actually starts (the actions/runner#2473 bug) via the `publish-dry-run` job pointed at TestPyPI. - [x] **REMOVE BEFORE MERGE**: drop the `publish-dry-run` job and the now-redundant `actions/upload-artifact` runs on PRs (currently always-on so the dry-run has wheels to publish). - [ ] After merge, cherry-pick onto `python-v0.32.0` and force-push the tag to re-trigger the publish. * ci: drop manylinux2_17 wheel builds (#3455) manylinux2_17 reached EOL in 2024 and pyarrow stopped publishing 2_17 wheels long ago. We already build manylinux2_28 wheels, so drop the 2_17 matrix entries. Fixes #3452 Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * perf: migrate list_indices to use Lance's describe_indices (#3108) This needs https://github.com/lance-format/lance/pull/6099 to work. Closes #3140 --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * feat(python): unify sync create_index API to match async API (#2882) ## Summary - Transitions `LanceTable` and `RemoteTable` to use the unified `create_index()` API matching `AsyncTable` - Deprecates `create_scalar_index()` and `create_fts_index()` with deprecation warnings - Adds detection logic to distinguish legacy vs new API calls - Adds `@overload` decorators for type checker compatibility - Adds `accelerator` parameter to IVF config classes for GPU support **New API:** ```python table.create_index("vec", config=IvfPq(distance_type="l2")) table.create_index("col", config=BTree()) table.create_index("text_col", config=FTS(with_position=True)) ``` **Legacy API (deprecated):** ```python table.create_index("l2", vector_column_name="vec") # emits DeprecationWarning table.create_scalar_index("col", index_type="BTREE") # deprecated table.create_fts_index("text_col") # deprecated ``` Fixes #2879 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <[email protected]> * test(python): fix remote create_index schema fixture (#3462) The latest main Python workflow fails across multiple matrix jobs because `test_remote_create_index_new_api` opens a remote table whose mocked schema only exposes `id`, while the new `create_index(..., config=...)` path validates the requested indexed columns. This updates the remote-table fixture to include the indexed columns used by the smoke test and checks the emitted column payloads, keeping the test aligned with the schema-aware API path. * chore: upgrade Rust toolchain to 1.95.0 (#3390) Bumps the pinned toolchain in `rust-toolchain.toml` from 1.94.0 to 1.95.0. Fixes new lints surfaced by clippy on 1.95.0: - `manual_checked_ops` — fragment size mean in `table.rs` uses `checked_div` - `explicit_counter_loop` — shuffle test loop in `shuffle.rs` No rustc warnings were introduced. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]> * feat(table): route merge_insert through the MemWAL LSM write path (#3354) ## Summary When an `LsmWriteSpec` is installed on a table (#3396), `merge_insert` upsert calls are dispatched through Lance's MemWAL `ShardWriter` (LSM-style append) instead of the standard merge path. - **`use_lsm_write`** — a `merge_insert` builder option, default `true`; set it `false` to use the standard path for a call even when a spec is set. - **`assume_pre_sharded`** — a `merge_insert` builder option, default `false`; skips the per-row shard check and routes by the first row only. - **`close_lsm_writers`** — drains and closes the table's cached MemWAL shard writers. - The `merge_insert` **`on`** columns default to, and are validated against, the table's unenforced primary key. - Shard writers are cached alongside the dataset (in `DatasetConsistencyWrapper`) and reused for the session. - `MergeResult` gains **`num_rows`** — on the LSM path the insert/update breakdown is unknown until compaction, so only the total is reported. Routing covers all three sharding strategies — bucket (murmur3, Iceberg-compatible), identity, and unsharded. Each `merge_insert` call targets a single shard; the whole input is collected and validated before a single atomic `ShardWriter::put`, so a validation failure leaves the MemWAL untouched. Bindings: Python (`merge_insert(...).use_lsm_write(...)` / `.assume_pre_sharded(...)`, `Table.close_lsm_writers`) and TypeScript (`mergeInsert(...).useLsmWrite(...)` / `.assumePreSharded(...)`, `Table.closeLsmWriters`). ## Context Reconstructed from the original #3354 branch onto current `main`: the branch predated the #3394 (unenforced primary key) / #3396 (`LsmWriteSpec`) split and has been rebuilt on that merged foundation. Depends on Lance `v7.0.0-beta.13`. The MemWAL read path (reading un-flushed shard data back into queries) and remote (LanceDB Cloud) LSM support are follow-ups. --------- Co-authored-by: Jack Ye <[email protected]> * chore: update Lance to v7.2.0-beta.1 (#3461) Update the Rust workspace Lance git dependencies and Java lance-core dependency to v7.2.0-beta.1. This keeps LanceDB aligned with the latest Lance beta release and refreshes the Cargo lockfile for the new Lance dependency graph. * chore(deps): bump the rust-minor-patch group with 5 updates (#3465) Bumps the rust-minor-patch group with 5 updates: | Package | From | To | | --- | --- | --- | | [log](https://github.com/rust-lang/log) | `0.4.29` | `0.4.30` | | [serde_json](https://github.com/serde-rs/json) | `1.0.149` | `1.0.150` | | [http](https://github.com/hyperium/http) | `1.4.0` | `1.4.1` | | [uuid](https://github.com/uuid-rs/uuid) | `1.23.1` | `1.23.2` | | [aws-smithy-runtime](https://github.com/smithy-lang/smithy-rs) | `1.11.1` | `1.11.3` | Updates `log` from 0.4.29 to 0.4.30 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/log/releases">log's releases</a>.</em></p> <blockquote> <h2>0.4.30</h2> <h3>What's Changed</h3> <ul> <li>Support capturing of <code>std::net</code> types by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/724">rust-lang/log#724</a></li> </ul> <h3>New Contributors</h3> <ul> <li><a href="https://github.com/V0ldek"><code>@V0ldek</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/log/pull/720">rust-lang/log#720</a></li> <li><a href="https://github.com/woodruffw"><code>@woodruffw</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/log/pull/723">rust-lang/log#723</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rust-lang/log/compare/0.4.29...0.4.30">https://github.com/rust-lang/log/compare/0.4.29...0.4.30</a></p> <h3>Notable Changes</h3> <ul> <li>MSRV is bumped to 1.71.0 in <a href="https://redirect.github.com/rust-lang/log/pull/723">rust-lang/log#723</a></li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/log/blob/master/CHANGELOG.md">log's changelog</a>.</em></p> <blockquote> <h2>[0.4.30] - 2026-05-21</h2> <h3>What's Changed</h3> <ul> <li>Support capturing of <code>std::net</code> types by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/724">rust-lang/log#724</a></li> </ul> <h3>New Contributors</h3> <ul> <li><a href="https://github.com/V0ldek"><code>@V0ldek</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/log/pull/720">rust-lang/log#720</a></li> <li><a href="https://github.com/woodruffw"><code>@woodruffw</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/log/pull/723">rust-lang/log#723</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rust-lang/log/compare/0.4.29...0.4.30">https://github.com/rust-lang/log/compare/0.4.29...0.4.30</a></p> <h3>Notable Changes</h3> <ul> <li>MSRV is bumped to 1.71.0 in <a href="https://redirect.github.com/rust-lang/log/pull/723">rust-lang/log#723</a></li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/rust-lang/log/commit/9c55760b499b18e81de7df5f3c13a67d5661131d"><code>9c55760</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/log/issues/725">#725</a> from rust-lang/cargo/0.4.30</li> <li><a href="https://github.com/rust-lang/log/commit/d1acb0585c0f6af5dc466eb255187cd6d3b7359e"><code>d1acb05</code></a> update docs on current MSRV and note latest bump in changelog</li> <li><a href="https://github.com/rust-lang/log/commit/50682937b0d9ec9a18c4c9b0510d889762e20e34"><code>5068293</code></a> prepare for 0.4.30 release</li> <li><a href="https://github.com/rust-lang/log/commit/7ccd873cb50de97690d46f69d8744a61f0b87c46"><code>7ccd873</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/log/issues/724">#724</a> from rust-lang/feat/net-to-value</li> <li><a href="https://github.com/rust-lang/log/commit/923dfaaf00dca352efe45930ae009d9a22526597"><code>923dfaa</code></a> fix up test cfgs</li> <li><a href="https://github.com/rust-lang/log/commit/ecb7de8daf7feec9dcf0d31cecc8523b31a8d104"><code>ecb7de8</code></a> gate net value impls on std</li> <li><a href="https://github.com/rust-lang/log/commit/67bb4f6d2e377b0008b740631124f292e80d4e5d"><code>67bb4f6</code></a> run fmt</li> <li><a href="https://github.com/rust-lang/log/commit/25f49fe3d31e7a0797652ad4bacaff633f7237cd"><code>25f49fe</code></a> rework net type capturing</li> <li><a href="https://github.com/rust-lang/log/commit/7087dcb95cb925364b4ba1da0d7c0eead9356dfc"><code>7087dcb</code></a> feat: impl ToValue for core::net types</li> <li><a href="https://github.com/rust-lang/log/commit/67bc7e32c68a4a8908d1016693418f12b43bab90"><code>67bc7e3</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/log/issues/723">#723</a> from woodruffw-forks/ww/ci</li> <li>Additional commits viewable in <a href="https://github.com/rust-lang/log/compare/0.4.29...0.4.30">compare view</a></li> </ul> </details> <br /> Updates `serde_json` from 1.0.149 to 1.0.150 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/serde-rs/json/releases">serde_json's releases</a>.</em></p> <blockquote> <h2>v1.0.150</h2> <ul> <li>Reject non-string enum object keys (<a href="https://redirect.github.com/serde-rs/json/issues/1324">#1324</a>, thanks <a href="https://github.com/puneetdixit200"><code>@puneetdixit200</code></a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/serde-rs/json/commit/a1ae73ac6a6940a4a57c673aebaa13ed4dfe3e8c"><code>a1ae73a</code></a> Release 1.0.150</li> <li><a href="https://github.com/serde-rs/json/commit/1a360b0a6c003912afc3503c834b0edd798bca28"><code>1a360b0</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/json/issues/1324">#1324</a> from puneetdixit200/reject-non-string-enum-keys</li> <li><a href="https://github.com/serde-rs/json/commit/2037b634f9dccbddc11cff189ebeb5854fa0e01c"><code>2037b63</code></a> Reject non-string enum object keys</li> <li><a href="https://github.com/serde-rs/json/commit/5d30df60e916e9b8fc46c74794007ff271fdfbbf"><code>5d30df6</code></a> Resolve manual_assert_eq pedantic clippy lint</li> <li><a href="https://github.com/serde-rs/json/commit/dc8003a88e7142529cf4a7429c4778af31dadf50"><code>dc8003a</code></a> Raise required compiler for preserve_order feature to 1.85</li> <li><a href="https://github.com/serde-rs/json/commit/a42fa980f8556cda36d896fa3713544b2e5eaa2c"><code>a42fa98</code></a> Unpin CI miri toolchain</li> <li><a href="https://github.com/serde-rs/json/commit/684a60eba18abfc0e0f7ddb0c2cd39f8f60249cf"><code>684a60e</code></a> Pin CI miri to nightly-2026-02-11</li> <li><a href="https://github.com/serde-rs/json/commit/7c7da3302b6b1cdab7f11ea49ca1a74422ab4551"><code>7c7da33</code></a> Raise required compiler to Rust 1.71</li> <li><a href="https://github.com/serde-rs/json/commit/acf4850e2969f1caccab2c4727a90ed006ba35bb"><code>acf4850</code></a> Simplify Number::is_f64</li> <li><a href="https://github.com/serde-rs/json/commit/6b8ceab565dcfe4f83dfaacd287d11c8bd8f306c"><code>6b8ceab</code></a> Resolve unnecessary_map_or clippy lint</li> <li>Additional commits viewable in <a href="https://github.com/serde-rs/json/compare/v1.0.149...v1.0.150">compare view</a></li> </ul> </details> <br /> Updates `http` from 1.4.0 to 1.4.1 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/hyperium/http/releases">http's releases</a>.</em></p> <blockquote> <h2>v1.4.1</h2> <h2>tl;dr</h2> <ul> <li>Fix <code>PathAndQuery::from_static()</code> and <code>from_shared()</code> to reject inputs that do not start with <code>/</code>.</li> <li>Fix <code>Extend</code> for <code>HeaderMap</code> to clamp max size hint and not overflow.</li> <li>Fix <code>header::IntoIter</code> that could use-after-free if the generic value type could panic on drop.</li> <li>Fix <code>header::{IterMut, ValuesIterMut}</code> to not violate stacked borrows.</li> </ul> <h2>What's Changed</h2> <ul> <li>chore(header): fix clippy::assign_op_pattern by <a href="https://github.com/rxc-amzn"><code>@rxc-amzn</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/806">hyperium/http#806</a></li> <li>ci: pin itoa in msrv job by <a href="https://github.com/seanmonstar"><code>@seanmonstar</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/813">hyperium/http#813</a></li> <li>Remove unnecessary explicit lifetimes by <a href="https://github.com/jplatte"><code>@jplatte</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/815">hyperium/http#815</a></li> <li>chore(ci): update to actions/checkout@v6 by <a href="https://github.com/tottoto"><code>@tottoto</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/819">hyperium/http#819</a></li> <li>tests: update to rand 0.10 by <a href="https://github.com/tottoto"><code>@tottoto</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/818">hyperium/http#818</a></li> <li>refactor: Remove usage of float instruction by <a href="https://github.com/AurelienFT"><code>@AurelienFT</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/823">hyperium/http#823</a></li> <li>refactor(uri): consolidate PathAndQuery::from_shared and from_static by <a href="https://github.com/seanmonstar"><code>@seanmonstar</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/825">hyperium/http#825</a></li> <li>fix(uri): reject Path::from_shared/from_static if doesn't start with slash by <a href="https://github.com/seanmonstar"><code>@seanmonstar</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/826">hyperium/http#826</a></li> <li>Rephrase comment by <a href="https://github.com/daalfox"><code>@daalfox</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/827">hyperium/http#827</a></li> <li>Fix typo in request builder docs by <a href="https://github.com/vleksis"><code>@vleksis</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/831">hyperium/http#831</a></li> <li>fix: clamp Extend size hint so HeaderMap reserve cannot overflow by <a href="https://github.com/SAY-5"><code>@SAY-5</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/833">hyperium/http#833</a></li> <li>fix(headers): fix stacked borrows for IterMut/ValuesIterMut by <a href="https://github.com/seanmonstar"><code>@seanmonstar</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/837">hyperium/http#837</a></li> <li>fix(header): use a set_len guard in IntoIter drop by <a href="https://github.com/seanmonstar"><code>@seanmonstar</code></a> in <a href="https://redirect.github.com/hyperium/http/pull/838">hyperium/http#838</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/rxc-amzn"><code>@rxc-amzn</code></a> made their first contribution in <a href="https://redirect.github.com/hyperium/http/pull/806">hyperium/http#806</a></li> <li><a href="https://github.com/AurelienFT"><code>@AurelienFT</code></a> made their first contribution in <a href="https://redirect.github.com/hyperium/http/pull/823">hyperium/http#823</a></li> <li><a href="https://github.com/daalfox"><code>@daalfox</code></a> made their first contribution in <a href="https://redirect.github.com/hyperium/http/pull/827">hyperium/http#827</a></li> <li><a href="https://github.com/vleksis"><code>@vleksis</code></a> made their first contribution in <a href="https://redirect.github.com/hyperium/http/pull/831">hyperium/http#831</a></li> <li><a href="https://github.com/SAY-5"><code>@SAY-5</code></a> made their first contribution in <a href="https://redirect.github.com/hyperium/http/pull/833">hyperium/http#833</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/hyperium/http/compare/v1.4.0...v1.4.1">https://github.com/hyperium/http/compare/v1.4.0...v1.4.1</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/hyperium/http/blob/master/CHANGELOG.md">http's changelog</a>.</em></p> <blockquote> <h1>1.4.1 (May 25, 2026)</h1> <ul> <li>Fix <code>PathAndQuery::from_static()</code> and <code>from_shared()</code> to reject inputs that do not start with <code>/</code>.</li> <li>Fix <code>Extend</code> for <code>HeaderMap</code> to clamp max size hint and not overflow.</li> <li>Fix <code>header::IntoIter</code> that could use-after-free if the generic value type could panic on drop.</li> <li>Fix <code>header::{IterMut, ValuesIterMut}</code> to not violate stacked borrows.</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/hyperium/http/commit/a24c968ba3b53c4c9953164235664cab9e8fa315"><code>a24c968</code></a> v1.4.1</li> <li><a href="https://github.com/hyperium/http/commit/bc3b0441be3065fc2653e9b3b1392c0fed873482"><code>bc3b044</code></a> fix(header): use a set_len guard in IntoIter drop (<a href="https://redirect.github.com/hyperium/http/issues/838">#838</a>)</li> <li><a href="https://github.com/hyperium/http/commit/1b968dc519c49b1922bc546c95f33900e684f4ab"><code>1b968dc</code></a> fix(header): fix stacked borrows for IterMut/ValuesIterMut (<a href="https://redirect.github.com/hyperium/http/issues/837">#837</a>)</li> <li><a href="https://github.com/hyperium/http/commit/6e2dd42a15d4c1711baa2191bd1d15022e1e2e9c"><code>6e2dd42</code></a> fix: clamp Extend size hint so HeaderMap reserve cannot overflow (<a href="https://redirect.github.com/hyperium/http/issues/833">#833</a>)</li> <li><a href="https://github.com/hyperium/http/commit/68e0abb052a243a5530ad4c404cb0b169a7ecb4a"><code>68e0abb</code></a> docs: fix typo in request builder docs (<a href="https://redirect.github.com/hyperium/http/issues/831">#831</a>)</li> <li><a href="https://github.com/hyperium/http/commit/29dd307b3e382a4343fc917fa3c41125ac50dfb8"><code>29dd307</code></a> docs(extensions): rephrase internal comment (<a href="https://redirect.github.com/hyperium/http/issues/827">#827</a>)</li> <li><a href="https://github.com/hyperium/http/commit/ae48fb55b090b4859d38a3a49a8332b83492d7c1"><code>ae48fb5</code></a> fix(uri): reject Path::from_shared/from_static if doesn't start with slash (#...</li> <li><a href="https://github.com/hyperium/http/commit/1ad200ec4ce5ec714005d500f8b0cea39c6c16f5"><code>1ad200e</code></a> refactor(uri): consolidate PathAndQuery::from_shared and from_static (<a href="https://redirect.github.com/hyperium/http/issues/825">#825</a>)</li> <li><a href="https://github.com/hyperium/http/commit/d59d939f928c6d836f5c87940f01399cb45cddb9"><code>d59d939</code></a> refactor: Remove usage of float instruction (<a href="https://redirect.github.com/hyperium/http/issues/823">#823</a>)</li> <li><a href="https://github.com/hyperium/http/commit/ed680c4d90a514b7f427efc99b61e60632811d2f"><code>ed680c4</code></a> tests: update to rand 0.10 (<a href="https://redirect.github.com/hyperium/http/issues/818">#818</a>)</li> <li>Additional commits viewable in <a href="https://github.com/hyperium/http/compare/v1.4.0...v1.4.1">compare view</a></li> </ul> </details> <br /> Updates `uuid` from 1.23.1 to 1.23.2 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/uuid-rs/uuid/releases">uuid's releases</a>.</em></p> <blockquote> <h2>v1.23.2</h2> <h2>What's Changed</h2> <ul> <li>Improve error messages for ambiguous formats by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/882">uuid-rs/uuid#882</a></li> <li>Prepare for 1.23.2 release by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/uuid-rs/uuid/pull/883">uuid-rs/uuid#883</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/uuid-rs/uuid/compare/v1.23.1...v1.23.2">https://github.com/uuid-rs/uuid/compare/v1.23.1...v1.23.2</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/uuid-rs/uuid/commit/d11965705f88ae2546e0d277dac8f52f47e5694f"><code>d119657</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/883">#883</a> from uuid-rs/cargo/v1.23.2</li> <li><a href="https://github.com/uuid-rs/uuid/commit/0651cfcb895d5d0b7e21edba621422bf446d585f"><code>0651cfc</code></a> prepare for 1.23.2 release</li> <li><a href="https://github.com/uuid-rs/uuid/commit/e8dea0c1fdc69e066cff93957e441022acfcb90f"><code>e8dea0c</code></a> Merge pull request <a href="https://redirect.github.com/uuid-rs/uuid/issues/882">#882</a> from uuid-rs/fix/error-msgs</li> <li><a href="https://github.com/uuid-rs/uuid/commit/bdc429a8c731a067b0d49c8890c6209dbb9f02db"><code>bdc429a</code></a> fix up serde messages</li> <li><a href="https://github.com/uuid-rs/uuid/commit/d4342e400df7adb17028b499a53a96228951baec"><code>d4342e4</code></a> make indexes 0 based and fix up more error messages</li> <li><a href="https://github.com/uuid-rs/uuid/commit/4ad479fc20fd09f34467e00adf176d4fdbdf9161"><code>4ad479f</code></a> work on more accurate parser errors</li> <li>See full diff in <a href="https://github.com/uuid-rs/uuid/compare/v1.23.1...v1.23.2">compare view</a></li> </ul> </details> <br /> Updates `aws-smithy-runtime` from 1.11.1 to 1.11.3 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * chore: bump lance to 7.2.0-beta.3 (#3471) This updates the workspace Lance dependencies from `v7.1.0-beta.4` to `v7.2.0-beta.3` and refreshes `Cargo.lock`. The lockfile now points at Lance commit `7c070f760fa8e24c8015cb2afbd22c5e6b7898e8` and includes the transitive dependency updates required by the new beta. * Bump version: 0.33.0-beta.1 → 0.33.1-beta.0 * Bump version: 0.30.0-beta.1 → 0.30.1-beta.0 * feat(python): support remote tables in PyTorch dataloaders (#3432) This PR makes remote LanceDB tables usable from PyTorch multiprocessing workers. Remote tables now carry enough safe JSON connection state to reopen themselves after pickle/spawn or fork, and permutations lazily rebuild their reader from restored tables instead of trying to reuse process-local handles. This addresses the remote-table gap in the PyTorch dataset path while preserving the explicit connection factory escape hatch for custom worker-side credential loading or non-serializable header providers. Validated with targeted remote table, permutation, and PyTorch DataLoader tests. * ci: move Lance dependency bump flow into skill (#3475) Moves the Lance dependency bump process into an in-repository skill so local agents and GitHub Actions share the same workflow definition. The update workflow is now an explicit, optional-tag entrypoint; latest-release resolution, duplicate PR handling, Java/Rust dependency updates, and Sophon follow-up are documented in the skill and backed by a small deterministic helper. * feat: add update_field_metadata to edit per-field metadata (#3482) ### Summary Adds update_field_metadata to the client SDK (Rust core, Python, and TypeScript) so clients can edit per-field (column) Arrow metadata (schema.fields[].metadata) ### Testing - added unit tests - ran E2E against a local server on both local and remote tables (set → merge → delete), across Python sync/async and TypeScript ### Next steps - deprecate replace_field_metadata in the python lancedb favor of this (typescript didn't have replace_field_metadata method). This matches Lance's API direction (Lance already deprecated replace_field_metadata for update_field_metadata) * feat: deprecate replace_field_metadata for update_field_metadata (#3484) ### Summary Deprecates the Python replace_field_metadata (on Table and AsyncTable) in favor of update_field_metadata. Mirrors Lance, which already deprecated Dataset.replace_field_metadata for update_field_metadata. Stacked on top of #3482 as this was a follow-up task after adding update_field_metadata * feat(python): support blob modes in query to_pandas (#3487) ## Feature - What is the new feature? - Adds `blob_mode` support to sync and async Python query `to_pandas()` APIs. - Enables plain scan queries to return blob columns as lazy `BlobFile` objects, raw bytes, or blob descriptions. - Lets namespace-backed local tables use Lance native blob-aware pandas conversion for lazy blobs. - Why do we need this feature? - Table and Lance dataset/scanner APIs already support blob-aware pandas conversion, but LanceDB query builders did not expose that capability. - Geneva and other callers should be able to use query-level `to_pandas(blob_mode=...)` without manually constructing Lance scanners. - How does it work? - Plain scan queries route through Lance scanner native `to_pandas(blob_mode=...)`, preserving filter, projection, limit, offset, row id, and alias/expression projection behavior. - Non-native query shapes keep existing Arrow fallback semantics and raise a clear error when they return blob columns with `blob_mode="lazy"` or `blob_mode="bytes"`. - Focused tests cover table/query blob modes, filter/select/limit/offset/alias query cases, async query behavior, vector-query error boundaries, and namespace-backed lazy blobs. ## Validation - `cd python && .venv/bin/maturin develop --uv --extras tests,dev --profile dev` - `cd python && uv run --frozen --no-sync pytest python/tests/test_table.py::test_table_to_pandas_blob_modes python/tests/test_table.py::test_async_table_to_pandas_blob_bytes python/tests/test_query.py::test_plain_scan_query_to_pandas_blob_modes python/tests/test_query.py::test_plain_scan_query_to_pandas_blob_projection python/tests/test_query.py::test_async_plain_scan_query_to_pandas_blob_projection python/tests/test_query.py::test_vector_query_to_pandas_blob_mode_requires_native_path python/tests/test_namespace.py::TestNamespaceConnection::test_table_to_pandas_blob_lazy_through_namespace -q` - `cd python && uv run --frozen --no-sync ruff format --check .` - `cd python && uv run --frozen --no-sync ruff check .` - `git diff --check` * Bump version: 0.33.1-beta.0 → 0.33.1-beta.1 * Bump version: 0.30.1-beta.0 → 0.30.1-beta.1 * fix(rerankers/mrr): raise ValueError on empty vector_results list (#3469) ## What's broken `MRRReranker.rerank_multivector([])` raises `IndexError: list index out of range`. The crash happens on line 128 (the `all()` type-homogeneity check passes vacuously on an empty iterable) and on line 134 which accesses `vector_results[0]` unconditionally, with no prior guard for an empty list. ## Why it happens `all()` over an empty iterable returns `True`, so the type check silently passes and execution falls through to `vector_results[0]` which crashes. ## Fix Added a two-line guard at the top of `rerank_multivector` that raises a clear `ValueError("vector_results must not be empty")` before any indexing occurs. ## Test Added `test_mrr_reranker_empty_input` in `test_rerankers.py` which calls `rerank_multivector([])` and asserts that a `ValueError` with the message "must not be empty" is raised. Fixes #3468 Co-authored-by: Aegis Dev <[email protected]> * fix(rerankers): guard against empty vector_results in RRFReranker.rerank_multivector (#3467) ## What's broken Calling `RRFReranker().rerank_multivector([])` crashes with `IndexError: list index out of range` because the method accesses `vector_results[0]` for the type-homogeneity check before verifying the list is non-empty. The `all()` call passes vacuously on an empty iterable so the crash hits the next lines. ```python from lancedb.rerankers import RRFReranker RRFReranker().rerank_multivector([]) # IndexError: list index out of range ``` ## Why it happens The type check uses `vector_results[0]` as the reference type but never guards against an empty list. `all(...)` short-circuits to `True` when the iterable is empty, so the bad index access on the lines that follow is never reached by the existing guard logic. ## Fix Add an explicit empty-list check before any indexing. * docs: add cross-SDK parity guidance for code review (#3464) Adds a REVIEW.md at the repo root with cross-SDK parity guidance for automated code review. The Claude Code review feature automatically loads `REVIEW.md` as review-only context. This is intentionally a semantic nudge, not a deterministic check, it relies on the reviewer reading the sibling SDK, so it will catch most gaps. * test(python): add regression test for nullable struct with None (#2654) (#3483) ## Summary Regression test for [issue #2654](https://github.com/lancedb/lancedb/issues/2654) — a nullable struct column whose first batch contains only `None` values crashed in `_align_field_types` with `AttributeError: 'pyarrow.lib.DataType' object has no attribute 'fields'`. The actual fix landed in #3394, but no test was added. This PR adds the reproducer from the issue as a test. ## Test plan - `test_add_nullable_struct_with_none`: creates a table with a nullable struct column, adds a row with a non-null struct value, then a row with `None` for the struct field. Verifies both rows land correctly. - Uses Lance file format v2.1 (`new_table_data_storage_version="2.1"`) because nullable structs aren't supported on v2.0. ## Related - #3028 (the original fix attempt, now superseded) * ci: update python lockfile weekly (#3498) Make sure we are getting security fixes in there regularly, and other useful bumps. * feat(rust): support datafusion expressions for merge insert predicates (#3444) ### Description This PR exposes native DataFusion expression support in the Rust SDK's `MergeInsertBuilder` via two new builder methods: `when_matched_update_all_expr` and `when_not_matched_by_source_delete_expr`. For remote LanceDB tables (where operations are serialized over HTTP/JSON to the SaaS backend), native DataFusion expression trees cannot be executed directly. The SDK handles this gracefully by returning a `NotSupported` error. ### Key Changes - **`MergeFilter` Enum**: Introduced a helper enum to store either a SQL string or a native `datafusion_expr::Expr`. - **`MergeInsertBuilder`**: Updated `when_matched_update_all_filt` and `when_not_matched_by_source_delete_filt` fields to store the new enum, and added `when_matched_update_all_expr` and `when_not_matched_by_source_delete_expr` builder methods. - **Execution & Remote Dispatch**: Dispatched the filter variants during local execution, and rejected expression filters with a clean `NotSupported` error in remote table request conversion. - **Testing**: Added a `test_merge_insert_expr` unit test covering conditional updates and deletes with programmatically built DataFusion expressions. ### Verification - Added integration test `test_merge_insert_expr` which successfully compiles and passes. - Formatted and linted the code. Closes #3416 * fix(python): route blob query pandas through scanner (#3491) ## Bug Fix ### What is the bug? `QueryBuilder.to_pandas(blob_mode="descriptions")` could still fall back to `self.to_arrow()` for query outputs with blob columns. Custom query subclasses or wrappers can have `to_arrow()` behavior that is not compatible with pandas blob-description conversion, which can surface as low-level Arrow/list-batch conversion failures. ### What issues or incorrect behavior does the bug cause? Callers need to carry local `to_pandas` or plain-scan adapter special casing for blob descriptions, and scanner-only kwargs such as row addresses and fragment selection are not represented in LanceDB query state. ### How does this PR fix the problem? This PR routes blob-output query `to_pandas()` through the Lance scanner path for `lazy`, `bytes`, and `descriptions` modes when the query is a scanner-backed plain scan. For `blob_mode="descriptions"` with `flatten`, it collects scanner Arrow/table output, applies LanceDB `flatten_columns`, and converts to pandas from there. Non-plain blob query shapes now fail with a clear unsupported error instead of falling into subclass `to_arrow()` behavior. It also adds Python query state and builder methods for scanner-only plain-scan parameters: - `with_row_address()` for `_rowaddr` - `with_fragments(...)` for Lance fragment objects - `fragment_ids([...])` as a convenience wrapper that resolves IDs to Lance fragments ## Validation - `cd python && uv run --no-sync ruff format --check python/lancedb/query.py python/tests/test_query.py` - `cd python && uv run --no-sync ruff check python/lancedb/query.py python/tests/test_query.py` Targeted pytest was intentionally not run locally per maintainer request. * Bump version: 0.33.1-beta.1 → 0.33.1-beta.2 * Bump version: 0.30.1-beta.1 → 0.30.1-beta.2 * chore(deps): bump the rust-minor-patch group with 3 updates (#3499) Bumps the rust-minor-patch group with 3 updates: [log](https://github.com/rust-lang/log), [test-log](https://github.com/d-e-s-o/test-log) and [serial_test](https://github.com/palfrey/serial_test). Updates `log` from 0.4.30 to 0.4.31 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/log/releases">log's releases</a>.</em></p> <blockquote> <h2>0.4.31</h2> <h2>What's Changed</h2> <ul> <li>fix typos in kv compile errors and log documentation by <a href="https://github.com/Isvane"><code>@Isvane</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/726">rust-lang/log#726</a></li> <li>Leverage static str key when possible by <a href="https://github.com/tisonkun"><code>@tisonkun</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/727">rust-lang/log#727</a></li> <li>Prepare for 0.4.31 release by <a href="https://github.com/KodrAus"><code>@KodrAus</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/728">rust-lang/log#728</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/Isvane"><code>@Isvane</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/log/pull/726">rust-lang/log#726</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rust-lang/log/compare/0.4.30...0.4.31">https://github.com/rust-lang/log/compare/0.4.30...0.4.31</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/rust-lang/log/blob/master/CHANGELOG.md">log's changelog</a>.</em></p> <blockquote> <h2>[0.4.31] - 2026-06-02</h2> <h2>What's Changed</h2> <ul> <li>Leverage static str key when possible by <a href="https://github.com/tisonkun"><code>@tisonkun</code></a> in <a href="https://redirect.github.com/rust-lang/log/pull/727">rust-lang/log#727</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/Isvane"><code>@Isvane</code></a> made their first contribution in <a href="https://redirect.github.com/rust-lang/log/pull/726">rust-lang/log#726</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/rust-lang/log/compare/0.4.30...0.4.31">https://github.com/rust-lang/log/compare/0.4.30...0.4.31</a></p> <h2>[Unreleased]</h2> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/rust-lang/log/commit/580839288e5f2babc17e6c36f7d56e60082a47ef"><code>5808392</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/log/issues/728">#728</a> from rust-lang/cargo/0.4.31</li> <li><a href="https://github.com/rust-lang/log/commit/86d739f51a9c59a3cb66a79e695639e6fb41465b"><code>86d739f</code></a> prepare for 0.4.31 release</li> <li><a href="https://github.com/rust-lang/log/commit/c906cfb02e351b59cfe35c0f0be22093086aabb1"><code>c906cfb</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/log/issues/727">#727</a> from tisonkun/leverage-static-str-key-when-possible</li> <li><a href="https://github.com/rust-lang/log/commit/756c279649f79ce0ef8dccf952c5df4017791d1c"><code>756c279</code></a> leverage str literal as well</li> <li><a href="https://github.com/rust-lang/log/commit/3dd250d1537fd7e5974e0802b1025cc3e4561503"><code>3dd250d</code></a> rename Key::from_static_str to from_str_static</li> <li><a href="https://github.com/rust-lang/log/commit/db145979e229549215300f2696fa89b215cb1cab"><code>db14597</code></a> Leverage static str key when possible</li> <li><a href="https://github.com/rust-lang/log/commit/761461a5d0c8ea3d483d79b1de0205c2897318d2"><code>761461a</code></a> Merge pull request <a href="https://redirect.github.com/rust-lang/log/issues/726">#726</a> from Isvane/fix/typos</li> <li><a href="https://github.com/rust-lang/log/commit/48ce372edd343179cb9f4837381bf34c7679db3e"><code>48ce372</code></a> fix typos in kv compile errors and log documentation</li> <li>See full diff in <a href="https://github.com/rust-lang/log/compare/0.4.30...0.4.31">compare view</a></li> </ul> </details> <br /> Updates `test-log` from 0.2.20 to 0.2.21 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/d-e-s-o/test-log/releases">test-log's releases</a>.</em></p> <blockquote> <h2>v0.2.21</h2> <ul> <li>Fixed spans in generated code, improving <code>rust-analyzer</code> interaction</li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/jorendorff"><code>@jorendorff</code></a> made their first contribution in <a href="https://redirect.github.com/d-e-s-o/test-log/pull/68">d-e-s-o/test-log#68</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/d-e-s-o/test-log/compare/v0.2.20...v0.2.21">https://github.com/d-e-s-o/test-log/compare/v0.2.20...v0.2.21</a></p> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/d-e-s-o/test-log/blob/main/CHANGELOG.md">test-log's changelog</a>.</em></p> <blockquote> <h2>0.2.21</h2> <ul> <li>Fixed spans in generated code, improving <code>rust-analyzer</code> interaction</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/d-e-s-o/test-log/commit/b7b9da034578877997e0cbfb59ea11507e0a3da8"><code>b7b9da0</code></a> Bump version to 0.2.21</li> <li><a href="https://github.com/d-e-s-o/test-log/commit/db522dc408e1ac6f04b2d0c89dda7e4b48be0584"><code>db522dc</code></a> Add CHANGELOG entry for <a href="https://redirect.github.com/d-e-s-o/test-log/issues/68">#68</a></li> <li><a href="https://github.com/d-e-s-o/test-log/commit/5e996d9ac66882e6258d1d86df91417336436d14"><code>5e996d9</code></a> Wrap the injected init code, not the original test body</li> <li><a href="https://github.com/d-e-s-o/test-log/commit/c78563c1ca76720571dc5ffe731217adc7e781ed"><code>c78563c</code></a> Retain existing spans for test code</li> <li>See full diff in <a href="https://github.com/d-e-s-o/test-log/compare/v0.2.20...v0.2.21">compare view</a></li> </ul> </details> <br /> Updates `serial_test` from 3.4.0 to 3.5.0 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/palfrey/serial_test/releases">serial_test's releases</a>.</em></p> <blockquote> <h2>v3.5.0</h2> <h2>What's Changed</h2> <ul> <li>Replace scc/sdd with std::sync::Mutex for Miri strict provenance compatibility by <a href="https://github.com/justanotheranonymoususer"><code>@justanotheranonymoususer</code></a> in <a href="https://redirect.github.com/palfrey/serial_test/pull/157">palfrey/serial_test#157</a></li> </ul> <h2>New Contributors</h2> <ul> <li><a href="https://github.com/justanotheranonymoususer"><code>@justanotheranonymoususer</code></a> made their first contribution in <a href="https://redirect.github.com/palfrey/serial_test/pull/157">palfrey/serial_test#157</a></li> </ul> <p><strong>Full Changelog</strong>: <a href="https://github.com/palfrey/serial_test/compare/v3.4.0...v3.5.0">https://github.com/palfrey/serial_test/compare/v3.4.0...v3.5.0</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/palfrey/serial_test/commit/6181f64de942180231fdb9098dd0894bbd9e7472"><code>6181f64</code></a> 3.5.0</li> <li><a href="https://github.com/palfrey/serial_test/commit/480bead2f697707cd2e61214287d5f0b8518d44d"><code>480bead</code></a> Merge pull request <a href="https://redirect.github.com/palfrey/serial_test/issues/157">#157</a> from justanotheranonymoususer/remove-scc-dep</li> <li><a href="https://github.com/palfrey/serial_test/commit/e03019e3cdc79b65daeaa913c99376aed69d4101"><code>e03019e</code></a> Update ci.yml</li> <li><a href="https://github.com/palfrey/serial_test/commit/820c0f3de967d55c52c59cac9ae55345511bf468"><code>820c0f3</code></a> Update ci.yml</li> <li><a href="https://github.com/palfrey/serial_test/commit/62a89b055fb923159c428269c5be999509344cb1"><code>62a89b0</code></a> Only skip file_lock with filesystem access</li> <li><a href="https://github.com/palfrey/serial_test/commit/5ff550164ed6f149fc80230faa8d5b5ded234190"><code>5ff5501</code></a> Update ci.yml</li> <li><a href="https://github.com/palfrey/serial_test/commit/0bd996de9eb044293e149095465701175309942e"><code>0bd996d</code></a> Let's try --all-features</li> <li><a href="https://github.com/palfrey/serial_test/commit/338e4ed891a095e2bfda01572c150449e5f26e73"><code>338e4ed</code></a> Fix formatting</li> <li><a href="https://github.com/palfrey/serial_test/commit/a55cde5d1d1572db2a8e5930d361d95df9796ea0"><code>a55cde5</code></a> Cleanup code_lock.rs</li> <li><a href="https://github.com/palfrey/serial_test/commit/9ad7a8f18c9a598109df5197214669e8680ccb96"><code>9ad7a8f</code></a> Remove unnecessary test leftover changes</li> <li>Additional commits viewable in <a href="https://github.com/palfrey/serial_test/compare/v3.4.0...v3.5.0">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix(python): make LanceDBClientError pickleable (#3470) ## Summary - Add `__reduce__` methods to `LanceDBClientError` and `RetryError` so that instances can be pickled and unpickled correctly - `HttpError` inherits the fix from `LanceDBClientError` since it has no additional `__init__` parameters - Add tests verifying pickle roundtrip for all three exception classes Fixes #3447 ## Test plan - [x] Verified pickle roundtrip for `LanceDBClientError` with and without `status_code` - [x] Verified pickle roundtrip for `HttpError` (subclass, no extra init params) - [x] Verified pickle roundtrip for `RetryError` (subclass with many extra params) - [ ] CI tests pass 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Will Jones <[email protected]> * chore: update lance dependency to v8.0.0-beta.2 (#3500) Updates Lance dependencies to v8.0.0-beta.2 across the Rust workspace and Java lance-core metadata. The update was generated with ci/update_lance_dependency.py and required no compatibility code changes. Lance tag: https://github.com/lance-format/lance/releases/tag/v8.0.0-beta.2 ## ⛔ Merge blocker: legal review required This bump pulls in a new transitive **dev/profiling** dependency chain `inferno v0.11.21` → `pprof v0.15.0` → `lance-testing`, and `inferno` is licensed **CDDL-1.0** (copyleft). To get `cargo-deny` green, `CDDL-1.0` was added to the `deny.toml` allow list. **Do not merge until legal has reviewed and signed off on allowing CDDL-1.0.** The dependency is dev/test-only and not distributed, but the allow-list addition still requires legal approval per our policy. --------- Co-authored-by: Daniel Rammer <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * feat(remote): implement set/unset_lsm_write_spec REST variant (#3501) ## Summary Wires `RemoteTable::set_lsm_write_spec` / `unset_lsm_write_spec` to the sophon REST endpoints added in [lancedb/sophon#6181](https://github.com/lancedb/sophon/pull/6181), replacing the previous `NotSupported` stubs. - `set_lsm_write_spec` maps the `LsmWriteSpec` onto sophon's request DTO — mode-tagged `sharding` (`unsharded` / `bucket` / `identity`), `maintained_indexes`, and `writer_config_defaults` — and POSTs to `/v1/table/{name}/set_lsm_write_spec/`. - `unset_lsm_write_spec` POSTs to `/v1/table/{name}/unset_lsm_write_spec/`. - Both call `check_mutable` first, matching the other remote mutations. - `maintained_indexes` is sent verbatim (an empty list means "no maintained indexes", matching native semantics). ## Testing - Added mocked-endpoint unit tests for unsharded / bucket / identity set and for unset. - `cargo check --features remote --tests` passes. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * fix(python): run AsyncTable.search embeddings on a dedicated executor (#3459) ## Summary `AsyncTable.search()` computes the query embedding with `loop.run_in_executor(None, ...)`, which uses asyncio's **default** `ThreadPoolExecutor`. That pool is shared with all other `run_in_executor(None, ...)` work, so a slow embedding call — a heavy local model or an HTTP request to an embeddings API — ties up those threads and starves unrelated async I/O under concurrent load. This moves the (potentially blocking) embedding call onto a **dedicated executor**, isolating it from the default pool. Closes #3310. ## Problem `python/lancedb/table.py`, `AsyncTable.search()`: ```python return ( await loop.run_in_executor( None, # asyncio's default executor, shared with other blocking I/O embedding.function.compute_query_embeddings_with_retry, query, ) )[0] ``` Under load, concurrent searches whose embeddings block (or any other code using the default executor) contend for the same small thread pool. ## Change - Add a dedicated `ThreadPoolExecutor(thread_name_prefix="lancedb-embedding")` in `background_loop.py`, exposed via `embedding_executor()`. - Use it in `AsyncTable.search()`'s `make_embedding` instead of the default executor. - Reset the executor in the existing `_reset_after_fork` hook — its worker threads don't survive `fork()`, same as the background even…

draft vector index details

af877d5

github-actions Bot added the enhancement New feature or request label Mar 4, 2026

github-actions Bot added the A-python Python bindings label Mar 4, 2026

wjones127 and others added 8 commits March 4, 2026 15:30

remove flat variant

36fd159

Merge remote-tracking branch 'upstream/main' into feat/vector-index-d…

ca6251f

…etails # Conflicts: # rust/lance/src/index/append.rs # rust/lance/src/index/create.rs # rust/lance/src/index/vector.rs

wip: cleanup

0a8e394

fix python test

dd70fee

wjones127 commented Mar 5, 2026

View reviewed changes

wjones127 mentioned this pull request Mar 5, 2026

perf: migrate list_indices to use Lance's describe_indices lancedb/lancedb#3108

Merged

wjones127 commented Mar 5, 2026

View reviewed changes

Comment thread protos/table.proto Outdated

wjones127 marked this pull request as ready for review March 5, 2026 22:21

wjones127 requested a review from BubbleCal March 5, 2026 22:21

westonpace reviewed Mar 5, 2026

View reviewed changes

wjones127 mentioned this pull request Mar 7, 2026

fix: vector index type shown as unknown in describe_indices #6122

Merged

westonpace approved these changes Mar 13, 2026

View reviewed changes

wjones127 added 3 commits March 16, 2026 10:29

enhance tests

e88ed2f

Merge branch 'main' into feat/vector-index-details

7c986a9

Merge remote-tracking branch 'upstream/main' into feat/vector-index-d…

5e2deaf

…etails

BubbleCal reviewed Mar 16, 2026

View reviewed changes

wjones127 and others added 10 commits March 17, 2026 10:45

Merge branch 'feat/vector-index-details' of github.com:wjones127/lanc…

eccdfa8

…e into feat/vector-index-details

Merge remote-tracking branch 'upstream/main' into feat/vector-index-d…

30f814a

…etails # Conflicts: # rust/lance/src/index/append.rs # rust/lance/src/index/vector/ivf.rs

feat: add max_level to HnswParameters proto

4104b25

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

fix: update Python binding to use VectorIndexDetails from lance_index

b8a99fb

VectorIndexDetails moved from table.proto to index.proto, so the Python binding needs to reference lance_index::pb instead of lance_table::format::pb. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

chore: merge upstream/main into feat/vector-index-details

95c616a

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

clippy

11bc126

github-actions Bot added the A-java Java bindings + JNI label Apr 8, 2026

wjones127 added 2 commits April 8, 2026 10:56

clippy

5702f43

fix compat tests

c3860e6

BubbleCal reviewed Apr 13, 2026

View reviewed changes

wjones127 and others added 4 commits May 6, 2026 15:25

Merge remote-tracking branch 'upstream/main' into feat/vector-index-d…

c4b7ea2

…etails

Merge remote-tracking branch 'upstream/main' into feat/vector-index-d…

3b0f828

…etails # Conflicts: # rust/lance/src/index/vector/ivf.rs

wjones127 commented May 13, 2026

View reviewed changes

Comment thread rust/lance/src/index/vector/details.rs Outdated

wjones127 added 3 commits May 13, 2026 11:50

Always write runtime hints, drop default checks

56d06e8

The default-comparison logic for omitting hints was fragile (could drift if upstream defaults changed) and offered little benefit. Always record the actual value so the stored details capture exactly what the index was built with.

Remove obsolete defaults_omitted python test

5b89d4b

Hints are now always written, so this assertion no longer holds.

Merge branch 'main' into feat/vector-index-details

bcd2183

wjones127 merged commit 6ddd7e2 into lance-format:main May 20, 2026
28 checks passed

		// Carry forward existing index details, preferring the first segment
		// that has populated (non-empty) details.

Uh oh!

Conversation

wjones127 commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What's stored in VectorIndexDetails

Uh oh!

github-actions Bot commented Mar 4, 2026

PR Review: feat: implement vector index details

P0: Bits enum doesn't cover RaBitQ's actual bit-width

P1: No corresponding Rust code to populate the new fields

P1: target_partition_size may be better as uint32 or have docs clarifying semantics

Minor

Uh oh!

codecov Bot commented Mar 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wjones127 commented Mar 5, 2026

Uh oh!

wjones127 commented Mar 9, 2026

Uh oh!

wjones127 commented Mar 13, 2026

Uh oh!

westonpace left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

wjones127 commented Apr 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

wjones127 commented Mar 4, 2026 •

edited

Loading

P0: `Bits` enum doesn't cover RaBitQ's actual bit-width

P1: `target_partition_size` may be better as `uint32` or have docs clarifying semantics

codecov Bot commented Mar 4, 2026 •

edited

Loading