chore: merge upstream lancedb sync (0.32.0-beta.2) by sinianluoye · Pull Request #62 · TheDeltaLab/lancedb

sinianluoye · 2026-07-22T08:02:38Z

Summary

Sync upstream lancedb/lancedb main into this fork (upstream 0.32.0-beta.2, lance v9.1.0-beta.8), resolving all conflicts in favor of fork customizations while adopting upstream changes to rust/ and nodejs/.

Version bump: 0.1.6 → 0.1.7 (rust/lancedb, nodejs, nodejs/npm/*, .bumpversion.toml, Cargo.lock)
lance deps: point at current TheDeltaLab/lance main (c416d29d, v9.1.0-beta.8); version constraint >=7.1.0-beta → >=9.1.0-beta (semver pre-release matching)
[workspace.metadata.upstream].version → 0.32.0-beta.2

Conflict resolution notes

Dropped upstream remote/python/java changes per fork policy (python/, java/, rust/lancedb/src/remote/, related CI and tests deleted)
Kept both upstream Predicate-based delete() API and fork's tracing::instrument spans (dropped the predicate span field since Predicate has no Debug/Display)
Kept both upstream branch APIs (create_branch/checkout_branch/…) and fork's get_version_info/get_version_by_time
Kept upstream list_indices refactor (skips per-index index_statistics) and write_summary shuffle change
nodejs: default features = fork's profiling-otlp + upstream's goosefs/metrics-otel (required by nodejs/src/otel.rs); OAuthConfig re-export dropped (remote-only)
CI: kept fork layout (no python/java/windows jobs); adopted upstream idempotent swap setup, setup-node@v6, and docs-freshness check
Regenerated Cargo.lock, nodejs/pnpm-lock.yaml, and docs/src/js/** (pnpm run docs)

Verification

cargo check --tests --examples / cargo clippy --tests --examples: clean
cargo test -p lancedb --tests: 515 passed, 0 failed
pnpm build, pnpm lint, pnpm tsc: clean
pnpm test: 630 passed, 0 failed

…3415) Modified the parameter of delete to a Predicate that could be constructed from either datafusion Expr, from str (to support SQL predicate), or from String to support python and javascript bindings. When a datafusion Expr is used, it avoids the overhead of serializing to SQL and re-parsing when callers already have an Expr (e.g. from query planning). The native implementation uses lance's `DeleteBuilder::from_expr`. The remote implementation converts the Expr to SQL via `expr_to_sql_string` before sending to the server, consistent with the existing query and count_rows paths. Closes lancedb#3204 Signed-off-by: Yuval Lifshitz <[email protected]> Co-authored-by: Claude Code <[email protected]>

lancedb#3439) Closes client side work of lancedb#3370 ### Summary - Plumbs `read_consistency_interval` from `ConnectBuilder` through `RestfulLanceDbClient` so remote reads attach an `x-lancedb-min-timestamp` freshness header. None = no header (default), zero = "now", positive = `now - interval`. - Adds per-table `FreshnessState` on `RemoteTable`: write responses (`update`, `delete`, `merge_insert`, `add_columns`, `alter_columns`, `drop_columns`) track the committed version, and the next read sends `x-lancedb-min-version` so the server's cache honors read-your-write. - `checkout(v)` / `checkout_tag(t)` / `checkout_latest()` / `restore()` reset the freshness state appropriately; the validating `/describe/` and tag-resolve requests are sent without freshness headers so they don't carry stale state. - Updates Rust, Python, and Node docstrings and calls out that stronger consistency raises per-read latency and cost. ### Testing - Unit tests cover default behavior, interval=0, positive interval, checkout_latest baseline, min_version-after-write, checkout clears state, and the two no-stale-header invariants on `checkout(v)` and `checkout_tag(t)`. - Ran smoke tests against local remote table to verify functionality

…2 updates (lancedb#3440) Bumps the rust-minor-patch group with 2 updates in the / directory: [serde_json](https://github.com/serde-rs/json) and [aws-smithy-runtime](https://github.com/smithy-lang/smithy-rs). Updates `serde_json` from 1.0.149 to 1.0.150 <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/serde-rs/json/releases">serde_json's releases</a>.</em></p> <blockquote> <h2>v1.0.150</h2> <ul> <li>Reject non-string enum object keys (<a href="https://redirect.github.com/serde-rs/json/issues/1324">#1324</a>, thanks <a href="https://github.com/puneetdixit200"><code>@puneetdixit200</code></a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/serde-rs/json/commit/a1ae73ac6a6940a4a57c673aebaa13ed4dfe3e8c"><code>a1ae73a</code></a> Release 1.0.150</li> <li><a href="https://github.com/serde-rs/json/commit/1a360b0a6c003912afc3503c834b0edd798bca28"><code>1a360b0</code></a> Merge pull request <a href="https://redirect.github.com/serde-rs/json/issues/1324">#1324</a> from puneetdixit200/reject-non-string-enum-keys</li> <li><a href="https://github.com/serde-rs/json/commit/2037b634f9dccbddc11cff189ebeb5854fa0e01c"><code>2037b63</code></a> Reject non-string enum object keys</li> <li><a href="https://github.com/serde-rs/json/commit/5d30df60e916e9b8fc46c74794007ff271fdfbbf"><code>5d30df6</code></a> Resolve manual_assert_eq pedantic clippy lint</li> <li><a href="https://github.com/serde-rs/json/commit/dc8003a88e7142529cf4a7429c4778af31dadf50"><code>dc8003a</code></a> Raise required compiler for preserve_order feature to 1.85</li> <li><a href="https://github.com/serde-rs/json/commit/a42fa980f8556cda36d896fa3713544b2e5eaa2c"><code>a42fa98</code></a> Unpin CI miri toolchain</li> <li><a href="https://github.com/serde-rs/json/commit/684a60eba18abfc0e0f7ddb0c2cd39f8f60249cf"><code>684a60e</code></a> Pin CI miri to nightly-2026-02-11</li> <li><a href="https://github.com/serde-rs/json/commit/7c7da3302b6b1cdab7f11ea49ca1a74422ab4551"><code>7c7da33</code></a> Raise required compiler to Rust 1.71</li> <li><a href="https://github.com/serde-rs/json/commit/acf4850e2969f1caccab2c4727a90ed006ba35bb"><code>acf4850</code></a> Simplify Number::is_f64</li> <li><a href="https://github.com/serde-rs/json/commit/6b8ceab565dcfe4f83dfaacd287d11c8bd8f306c"><code>6b8ceab</code></a> Resolve unnecessary_map_or clippy lint</li> <li>Additional commits viewable in <a href="https://github.com/serde-rs/json/compare/v1.0.149...v1.0.150">compare view</a></li> </ul> </details> <br /> Updates `aws-smithy-runtime` from 1.11.1 to 1.11.3 <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/smithy-lang/smithy-rs/commits">compare view</a></li> </ul> </details> <br /> Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore <dependency name> major version` will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself) - `@dependabot ignore <dependency name> minor version` will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself) - `@dependabot ignore <dependency name>` will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself) - `@dependabot unignore <dependency name>` will remove all of the ignore conditions of the specified dependency - `@dependabot unignore <dependency name> <ignore condition>` will remove the ignore condition of the specified dependency and ignore conditions </details> Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…LinearCombinationReranker (lancedb#3437) ## Problem `LinearCombinationReranker.merge_results` has two related bugs that make it return **inverted relevance rankings** — the least relevant document ranks first (closes lancedb#3154). ### Bug 1 — `_combine_score` subtracts from 1, inverting the final ranking ```python def _combine_score(self, vector_score, fts_score): return 1 - (self.weight * vector_score + (1 - self.weight) * fts_score) ``` Both `vector_score` (already converted via `_invert_score`) and `fts_score` (BM25 relevance) are in **higher-is-better** space. Wrapping the weighted average in `1 - (...)` flips the direction: a perfectly matching document (`vector_score=1, fts_score=1`) gets `_relevance_score = 0.0`, while a non-matching document gets a high score. ### Bug 2 — Documents missing an FTS score are rewarded, not penalised ```python fts_score = result.get("_score", fill) # fill=1.0 by default ``` When a document has no FTS match, `fts_score = fill = 1.0`. In `_combine_score` (with the bug-1 formula), this large value becomes a **negative penalty** via `1 - (... + 0.3 * 1.0)`, counterintuitively *boosting* the document's score. By contrast, missing vector results correctly receive `_invert_score(fill) = 0.0` (penalised). ## Fix **Bug 1** — remove the `1 -` inversion from `_combine_score`: ```python def _combine_score(self, vector_score, fts_score): return self.weight * vector_score + (1 - self.weight) * fts_score ``` **Bug 2** — use `1 - fill` for missing FTS scores so both penalties are symmetric (mirror of what `_invert_score(fill)` already does for missing vector scores): ```python fts_score = result.get("_score", 1 - fill) # was: fill ``` With `fill=1.0` (default): `1 - 1.0 = 0.0` — missing-FTS entries contribute `0` to the FTS term, identical to how missing-vector entries contribute `0` to the vector term. ## Verification Concrete example from the issue. With `weight=0.7`, `fill=1.0`: | Document | `_distance` | `_score` | Old `_relevance_score` | New `_relevance_score` | |----------|-------------|----------|------------------------|------------------------| | `apple orange` | 0.0 (best) | 2.41 (only FTS) | 0.30 (**wrong: ranked 2nd**) | 1.42 (**correct: ranked 1st**) | | `banana grape` | 0.9999 (worst) | — | 0.70 (**wrong: ranked 1st**) | 0.00 (**correct: ranked last**) | ## Tests Two regression tests added to `python/python/tests/test_rerankers.py`: - `test_linear_combination_best_match_ranks_first` — the document with the smallest distance **and** an FTS match must have the highest `_relevance_score`. - `test_linear_combination_missing_fts_is_penalised` — a document with any FTS score must beat an otherwise-equal document with no FTS match. --------- Co-authored-by: Will Jones <[email protected]>

…db#3435) ## Summary - Bump lance dependency from `v7.0.0-beta.13` to `v7.0.0-rc.1` - Remove PK constraint from `LsmWriteSpec::Bucket` docs and `Table::set_lsm_write_spec` docs - Remove test assertions that expected rejection when no PK is set or when bucket column != PK Closes lance-format/lance#6917

…3429) When a table is created with `pa.json_()` (PyArrow's JSON extension type), it is stored internally as `lance.json` (LargeBinary with `lance.json` extension metadata). Calling `table.add()` with `pa.json_()` data failed with: ``` RuntimeError: lance error: Append with different schema: `data` should have type json but type was large_binary ``` `build_field_exprs` in `rust/lancedb/src/table/datafusion/cast.rs` saw that the input field (`Utf8` with `arrow.json` metadata) differed from the table field (`LargeBinary` with `lance.json` metadata). Since `can_cast_types(Utf8, LargeBinary)` is true, it inserted a DataFusion `Utf8 → LargeBinary` cast. That cast preserved the input field's `arrow.json` extension metadata instead of adopting the table's `lance.json` metadata, so lance-core detected a schema mismatch and rejected the append. This adds a special case in `build_field_exprs`: when the input is `arrow.json` and the table field is `lance.json`, the expression is passed through unchanged. Lance-core's write path already handles the `arrow.json → lance.json` conversion (including JSONB encoding), so no DataFusion cast is needed. Fixes lancedb#3144 Continues lancedb#3291 from a fork (the original author's branch could not be pushed to). The original commits are preserved; an additional commit fixes the CI failures on that PR — formatting, a missing trait import, and read-back assertions that assumed binary storage when a lance.json column is read back as `Utf8`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: yunju.lly <[email protected]> Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

## Summary - Updates Lance Rust workspace dependencies to `v7.1.0-beta.4` using `ci/set_lance_version.py`. - Updates the Java `lance-core` dependency property to `7.1.0-beta.4`. - Triggering Lance tag: https://github.com/lance-format/lance/releases/tag/v7.1.0-beta.4 ## Verification - `cargo clippy --workspace --tests --all-features -- -D warnings` - `cargo fmt --all` Co-authored-by: Daniel Rammer <[email protected]>

The python-v0.32.0 publish run failed on every build matrix entry. Three independent issues: 1. **Mac and Windows**: `pypa/gh-action-pypi-publish` only runs on Linux, but was being called inline from each build job. 2. **Linux (all arches)**: `pypa/gh-action-pypi-publish` derives its docker image name from `github.action_repository`, which is empty when the action is invoked from inside a composite action (actions/runner#2473 — pypa's own `action.yml` references this bug). It falls back to `github.repository`, generating `docker://ghcr.io/lancedb/lancedb:<tag>`, which doesn't exist → `denied`. Only the ARM matrix entry surfaced this because it failed first and cancel-cascaded the rest. 3. **Windows**: `upload-artifact` in `build_windows_wheel` pointed at `python\target\wheels`, but maturin writes to the workspace-root `target/wheels`. The artifact was always empty. Also, `pypi-publish.yml` passed a `vcpkg_token` input that the composite doesn't declare. ## Changes - Build jobs (linux/mac/windows) now upload their wheels as `actions/upload-artifact` artifacts. - New Linux `publish` job downloads all wheel artifacts and runs the Fury or PyPA publish step directly (not via a composite), so `github.action_repository` resolves correctly. - Delete the unused `upload_wheel` composite action. - Drop the broken upload-artifact step inside `build_windows_wheel`. - Remove the bogus `vcpkg_token` input. - Fury upload now loops over all wheels instead of just the first. - Bump `actions/checkout`, `actions/upload-artifact`, `actions/download-artifact` to current major versions (Node 24) to clear deprecation warnings. - Bump Windows job timeout 60 → 90 minutes; previous run was cancel-timing-out on a 60m cap. - Use `rust-lld` as the Windows MSVC linker via `CARGO_TARGET_X86_64_PC_WINDOWS_MSVC_LINKER`. `link.exe` is single-threaded and the long pole on Windows builds. Fixes lancedb#3445 ## Test plan - [x] Open this PR — `paths` filter triggers a dry-run build on all three platforms. - [x] Verify all three builds produce wheels. - [x] Confirm the `pypa/gh-action-pypi-publish` container actually starts (the actions/runner#2473 bug) via the `publish-dry-run` job pointed at TestPyPI. - [x] **REMOVE BEFORE MERGE**: drop the `publish-dry-run` job and the now-redundant `actions/upload-artifact` runs on PRs (currently always-on so the dry-run has wheels to publish). - [ ] After merge, cherry-pick onto `python-v0.32.0` and force-push the tag to re-trigger the publish.

manylinux2_17 reached EOL in 2024 and pyarrow stopped publishing 2_17 wheels long ago. We already build manylinux2_28 wheels, so drop the 2_17 matrix entries. Fixes lancedb#3452 Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

) This needs lance-format/lance#6099 to work. Closes lancedb#3140 --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…#2882) ## Summary - Transitions `LanceTable` and `RemoteTable` to use the unified `create_index()` API matching `AsyncTable` - Deprecates `create_scalar_index()` and `create_fts_index()` with deprecation warnings - Adds detection logic to distinguish legacy vs new API calls - Adds `@overload` decorators for type checker compatibility - Adds `accelerator` parameter to IVF config classes for GPU support **New API:** ```python table.create_index("vec", config=IvfPq(distance_type="l2")) table.create_index("col", config=BTree()) table.create_index("text_col", config=FTS(with_position=True)) ``` **Legacy API (deprecated):** ```python table.create_index("l2", vector_column_name="vec") # emits DeprecationWarning table.create_scalar_index("col", index_type="BTREE") # deprecated table.create_fts_index("text_col") # deprecated ``` Fixes lancedb#2879 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.5 <[email protected]>

The latest main Python workflow fails across multiple matrix jobs because `test_remote_create_index_new_api` opens a remote table whose mocked schema only exposes `id`, while the new `create_index(..., config=...)` path validates the requested indexed columns. This updates the remote-table fixture to include the indexed columns used by the smoke test and checks the emitted column payloads, keeping the test aligned with the schema-aware API path.

Bumps the pinned toolchain in `rust-toolchain.toml` from 1.94.0 to 1.95.0. Fixes new lints surfaced by clippy on 1.95.0: - `manual_checked_ops` — fragment size mean in `table.rs` uses `checked_div` - `explicit_counter_loop` — shuffle test loop in `shuffle.rs` No rustc warnings were introduced. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-authored-by: Claude Opus 4.7 (1M context) <[email protected]>

…ncedb#3354) ## Summary When an `LsmWriteSpec` is installed on a table (lancedb#3396), `merge_insert` upsert calls are dispatched through Lance's MemWAL `ShardWriter` (LSM-style append) instead of the standard merge path. - **`use_lsm_write`** — a `merge_insert` builder option, default `true`; set it `false` to use the standard path for a call even when a spec is set. - **`assume_pre_sharded`** — a `merge_insert` builder option, default `false`; skips the per-row shard check and routes by the first row only. - **`close_lsm_writers`** — drains and closes the table's cached MemWAL shard writers. - The `merge_insert` **`on`** columns default to, and are validated against, the table's unenforced primary key. - Shard writers are cached alongside the dataset (in `DatasetConsistencyWrapper`) and reused for the session. - `MergeResult` gains **`num_rows`** — on the LSM path the insert/update breakdown is unknown until compaction, so only the total is reported. Routing covers all three sharding strategies — bucket (murmur3, Iceberg-compatible), identity, and unsharded. Each `merge_insert` call targets a single shard; the whole input is collected and validated before a single atomic `ShardWriter::put`, so a validation failure leaves the MemWAL untouched. Bindings: Python (`merge_insert(...).use_lsm_write(...)` / `.assume_pre_sharded(...)`, `Table.close_lsm_writers`) and TypeScript (`mergeInsert(...).useLsmWrite(...)` / `.assumePreSharded(...)`, `Table.closeLsmWriters`). ## Context Reconstructed from the original lancedb#3354 branch onto current `main`: the branch predated the lancedb#3394 (unenforced primary key) / lancedb#3396 (`LsmWriteSpec`) split and has been rebuilt on that merged foundation. Depends on Lance `v7.0.0-beta.13`. The MemWAL read path (reading un-flushed shard data back into queries) and remote (LanceDB Cloud) LSM support are follow-ups. --------- Co-authored-by: Jack Ye <[email protected]>

Update the Rust workspace Lance git dependencies and Java lance-core dependency to v7.2.0-beta.1. This keeps LanceDB aligned with the latest Lance beta release and refreshes the Cargo lockfile for the new Lance dependency graph.

This updates the workspace Lance dependencies from `v7.1.0-beta.4` to `v7.2.0-beta.3` and refreshes `Cargo.lock`. The lockfile now points at Lance commit `7c070f760fa8e24c8015cb2afbd22c5e6b7898e8` and includes the transitive dependency updates required by the new beta.

) This PR makes remote LanceDB tables usable from PyTorch multiprocessing workers. Remote tables now carry enough safe JSON connection state to reopen themselves after pickle/spawn or fork, and permutations lazily rebuild their reader from restored tables instead of trying to reuse process-local handles. This addresses the remote-table gap in the PyTorch dataset path while preserving the explicit connection factory escape hatch for custom worker-side credential loading or non-serializable header providers. Validated with targeted remote table, permutation, and PyTorch DataLoader tests.

Moves the Lance dependency bump process into an in-repository skill so local agents and GitHub Actions share the same workflow definition. The update workflow is now an explicit, optional-tag entrypoint; latest-release resolution, duplicate PR handling, Java/Rust dependency updates, and Sophon follow-up are documented in the skill and backed by a small deterministic helper.

) ### Summary Adds update_field_metadata to the client SDK (Rust core, Python, and TypeScript) so clients can edit per-field (column) Arrow metadata (schema.fields[].metadata) ### Testing - added unit tests - ran E2E against a local server on both local and remote tables (set → merge → delete), across Python sync/async and TypeScript ### Next steps - deprecate replace_field_metadata in the python lancedb favor of this (typescript didn't have replace_field_metadata method). This matches Lance's API direction (Lance already deprecated replace_field_metadata for update_field_metadata)

…cedb#3484) ### Summary Deprecates the Python replace_field_metadata (on Table and AsyncTable) in favor of update_field_metadata. Mirrors Lance, which already deprecated Dataset.replace_field_metadata for update_field_metadata. Stacked on top of lancedb#3482 as this was a follow-up task after adding update_field_metadata

## Feature - What is the new feature? - Adds `blob_mode` support to sync and async Python query `to_pandas()` APIs. - Enables plain scan queries to return blob columns as lazy `BlobFile` objects, raw bytes, or blob descriptions. - Lets namespace-backed local tables use Lance native blob-aware pandas conversion for lazy blobs. - Why do we need this feature? - Table and Lance dataset/scanner APIs already support blob-aware pandas conversion, but LanceDB query builders did not expose that capability. - Geneva and other callers should be able to use query-level `to_pandas(blob_mode=...)` without manually constructing Lance scanners. - How does it work? - Plain scan queries route through Lance scanner native `to_pandas(blob_mode=...)`, preserving filter, projection, limit, offset, row id, and alias/expression projection behavior. - Non-native query shapes keep existing Arrow fallback semantics and raise a clear error when they return blob columns with `blob_mode="lazy"` or `blob_mode="bytes"`. - Focused tests cover table/query blob modes, filter/select/limit/offset/alias query cases, async query behavior, vector-query error boundaries, and namespace-backed lazy blobs. ## Validation - `cd python && .venv/bin/maturin develop --uv --extras tests,dev --profile dev` - `cd python && uv run --frozen --no-sync pytest python/tests/test_table.py::test_table_to_pandas_blob_modes python/tests/test_table.py::test_async_table_to_pandas_blob_bytes python/tests/test_query.py::test_plain_scan_query_to_pandas_blob_modes python/tests/test_query.py::test_plain_scan_query_to_pandas_blob_projection python/tests/test_query.py::test_async_plain_scan_query_to_pandas_blob_projection python/tests/test_query.py::test_vector_query_to_pandas_blob_mode_requires_native_path python/tests/test_namespace.py::TestNamespaceConnection::test_table_to_pandas_blob_lazy_through_namespace -q` - `cd python && uv run --frozen --no-sync ruff format --check .` - `cd python && uv run --frozen --no-sync ruff check .` - `git diff --check`

…ncedb#3469) ## What's broken `MRRReranker.rerank_multivector([])` raises `IndexError: list index out of range`. The crash happens on line 128 (the `all()` type-homogeneity check passes vacuously on an empty iterable) and on line 134 which accesses `vector_results[0]` unconditionally, with no prior guard for an empty list. ## Why it happens `all()` over an empty iterable returns `True`, so the type check silently passes and execution falls through to `vector_results[0]` which crashes. ## Fix Added a two-line guard at the top of `rerank_multivector` that raises a clear `ValueError("vector_results must not be empty")` before any indexing occurs. ## Test Added `test_mrr_reranker_empty_input` in `test_rerankers.py` which calls `rerank_multivector([])` and asserts that a `ValueError` with the message "must not be empty" is raised. Fixes lancedb#3468 Co-authored-by: Aegis Dev <[email protected]>

…ank_multivector (lancedb#3467) ## What's broken Calling `RRFReranker().rerank_multivector([])` crashes with `IndexError: list index out of range` because the method accesses `vector_results[0]` for the type-homogeneity check before verifying the list is non-empty. The `all()` call passes vacuously on an empty iterable so the crash hits the next lines. ```python from lancedb.rerankers import RRFReranker RRFReranker().rerank_multivector([]) # IndexError: list index out of range ``` ## Why it happens The type check uses `vector_results[0]` as the reference type but never guards against an empty list. `all(...)` short-circuits to `True` when the iterable is empty, so the bad index access on the lines that follow is never reached by the existing guard logic. ## Fix Add an explicit empty-list check before any indexing.

Adds a REVIEW.md at the repo root with cross-SDK parity guidance for automated code review. The Claude Code review feature automatically loads `REVIEW.md` as review-only context. This is intentionally a semantic nudge, not a deterministic check, it relies on the reviewer reading the sibling SDK, so it will catch most gaps.

…edb#2654) (lancedb#3483) ## Summary Regression test for [issue lancedb#2654](lancedb#2654) — a nullable struct column whose first batch contains only `None` values crashed in `_align_field_types` with `AttributeError: 'pyarrow.lib.DataType' object has no attribute 'fields'`. The actual fix landed in lancedb#3394, but no test was added. This PR adds the reproducer from the issue as a test. ## Test plan - `test_add_nullable_struct_with_none`: creates a table with a nullable struct column, adds a row with a non-null struct value, then a row with `None` for the struct field. Verifies both rows land correctly. - Uses Lance file format v2.1 (`new_table_data_storage_version="2.1"`) because nullable structs aren't supported on v2.0. ## Related - lancedb#3028 (the original fix attempt, now superseded)

Updates the Rust workspace Lance dependencies and Java lance-core dependency to v9.0.0-beta.24. Lance tag: https://github.com/lance-format/lance/releases/tag/v9.0.0-beta.24

Consolidating skills so we have only one `lancedb` skill, making it easier to install and work with, vs. installing and using different skills for "lancedb-column-metadata", "lancedb-branch-ops", etc. Also deleted lancedb-connect, because the new monoskill uses the python/TS APIs so it doesn't need extra handholding to connect to the REST API. ## does it work? Test 1: do some column metadata operations with 1. no skills, 2. our previous baseline lancedb skill, 3. the baseline lancedb skill with the lancedb-column-metadata skill folded in: ``` ┌───────────────────────────┬──────────────────────┬────────────────────┬───────────────────────┐ │ eval │ no-skill │ lancedb (original) │ lancedb2-incl-columns │ ├───────────────────────────┼──────────────────────┼────────────────────┼───────────────────────┤ │ 2-add-all-metadata-types │ 2.5/5 · 116s · $0.44 │ 0/5 · 154s · $0.60 │ 5/5 · 58s · $0.26 │ ├───────────────────────────┼──────────────────────┼────────────────────┼───────────────────────┤ │ 3-delete-one-metadata-key │ 4/4 · 67s · $0.23 │ 4/4 · 100s · $0.45 │ 4/4 · 45s · $0.20 │ ├───────────────────────────┼──────────────────────┼────────────────────┼───────────────────────┤ │ TOTAL (per rep avg) │ 6.5/9 · 183s · $0.66 │ 4/9 · 254s · $1.05 │ 9/9 · 103s · $0.46 │ └───────────────────────────┴──────────────────────┴────────────────────┴───────────────────────┘ ``` without column-metadata-specific content, it failed because it wrote keys like `description` instead of `lancedb:description`. That's pretty undiscoverable without the skill. Test 2: do some simple branch operations with 1. no skills, 2. our previous baseline lancedb skill, 3. the combined skill (in this PR): ``` ┌───────────────────────────┬────────────────────┬────────────────────┬────────────────────────────────┐ │ eval │ no-skill │ lancedb (original) │ lancedb3-incl-columns-branches │ ├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤ │ 5-create-branch │ 2/2 · 64s · $0.30 │ 2/2 · 57s · $0.34 │ 2/2 · 38s · $0.22 │ ├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤ │ 6-delete-branch │ 2/2 · 38s · $0.21 │ 2/2 · 44s · $0.27 │ 2/2 · 40s · $0.22 │ ├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤ │ 7-switch-branch-and-write │ 1/2 · 96s · $0.49 │ 1/2 · 106s · $0.58 │ 2/2 · 66s · $0.40 │ ├───────────────────────────┼────────────────────┼────────────────────┼────────────────────────────────┤ │ TOTAL (per rep avg) │ 5/6 · 199s · $1.00 │ 5/6 · 207s · $1.20 │ 6/6 · 143s · $0.83 │ └───────────────────────────┴────────────────────┴────────────────────┴────────────────────────────────┘ ``` Test 3: run everything, with the lancedb (original) skill, lancedb(original) + all the separate skills, and lancedb3-incl-columns-branches ``` ┌────────────────────────────────┬──────────────────────┬────────────────────────────────┬─────────────────────────────────────────────────────────────────┐ │ eval │ lancedb (original) │ lancedb3-incl-columns-branches │ all-separate (lancedb + connect + column-metadata + branch-ops) │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 1-pick-column-for-image-search │ 2/2 · 141s · $0.67 │ 2/2 · 128s · $0.43 │ 1/2 · 257s · $0.81 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 2-add-all-metadata-types │ 5/5 · 93s · $0.52 │ 5/5 · 54s · $0.30 │ 5/5 · 40s · $0.24 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 3-delete-one-metadata-key │ 4/4 · 84s · $0.44 │ 4/4 · 36s · $0.24 │ 4/4 · 28s · $0.20 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 4-build-index │ 3/3 · 275s · $0.52 │ 3/3 · 84s · $0.39 │ 3/3 · 68s · $0.50 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 5-create-branch │ 2/2 · 39s · $0.31 │ 2/2 · 41s · $0.20 │ 2/2 · 15s · $0.17 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 6-delete-branch │ 2/2 · 59s · $0.25 │ 2/2 · 69s · $0.29 │ 2/2 · 26s · $0.17 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ 7-switch-branch-and-write │ 1/2 · 84s · $0.46 │ 2/2 · 62s · $0.40 │ 2/2 · 27s · $0.23 │ ├────────────────────────────────┼──────────────────────┼────────────────────────────────┼─────────────────────────────────────────────────────────────────┤ │ TOTAL │ 19/20 · 774s · $3.17 │ 20/20 · 474s · $2.24 │ 19/20 · 462s · $2.32 │ └────────────────────────────────┴──────────────────────┴────────────────────────────────┴─────────────────────────────────────────────────────────────────┘ ``` ("lancedb3-incl-columns-branches" is the combined skill in this PR, all-separate is using the four separate skills.) For overall performance, it helps to have the specialized skills for metadata and branching; doesn't really matter whether they're separate skills or all together. Also doesn't matter much whether it's REST or Python. So let's merge these skills to make it easier for users.

Adds client-side support for analyze_plan distributed metrics modes across Rust, Python, and TypeScript clients. Defaults to aggregate for backward compatibility and sends the remote distributed_metrics parameter only when a non-default mode is requested.

BREAKING CHANGE: splits generated by the permutation data loader will not be the same, due to a change in hash function. Updates the Lance dependencies and Java lance-core to [v9.0.0-rc.1](https://github.com/lance-format/lance/releases/tag/v9.0.0-rc.1). Includes the required DataFusion 54 and Lance file-reader compatibility updates. --------- Co-authored-by: Will Jones <[email protected]> Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…edb#3606) Routes local sync child-namespace operations through the Rust-backed connection instead of the Python namespace-client fallback. Also keeps lazy namespace-client construction for table-to-Lance conversion and preserves public namespace error mappings. Validated locally with ruff format/check and targeted namespace pytest.

…ncedb#3327) Tracks lancedb#3324. On x86_64 CPUs without AVX2 (Sandy Bridge / Ivy Bridge / Westmere on Intel; Bulldozer / Piledriver / Steamroller on AMD), `import lancedb` SIGILLs because the wheel bakes AVX2 + FMA into every compiled function. Per [westonpace's review](lancedb#3324 (comment)), the default `lancedb` wheel stays fast; pre-Haswell users get a separately-published `lancedb-compat` wheel. ## Summary - Adds a `lancedb-compat` matrix entry to `pypi-publish.yml` that builds with `RUSTFLAGS="-C target-cpu=x86-64-v2"` (Nehalem-class baseline). Same Python API (`import lancedb` works) — files install to the same namespace, so the two wheels conflict at install time and users pick one. Same pattern as `psycopg2` / `psycopg2-binary` and `tensorflow` / `tensorflow-cpu`. - Generalizes `build_linux_wheel` and `upload_wheel` composites with optional `package-name` and `rustflags` inputs (defaults preserve the existing 4 `lancedb` matrix entries verbatim). - Documents the choice in `python/README.md`: `pip install lancedb-compat` for pre-Haswell hosts. The default `.cargo/config.toml` baseline is unchanged. ## Sequencing 1. ~~lance-format/lance#6630 merges → runtime SIMD dispatch lands in lance.~~ **Done — merged.** 2. lancedb's lance dep is bumped to a release that includes it (separate PR / normal cadence). 3. This PR's `lancedb-compat` wheel build path starts producing a wheel that runs on pre-Haswell hardware. **Maintainer setup**: register `lancedb-compat` on PyPI and configure trusted publishing. ## Verified end-to-end on Sandy Bridge Xeon E5-2609 Verification was done locally against a fork-pinned lance dep that includes the runtime dispatch implementation, using the same `RUSTFLAGS="-C target-cpu=x86-64-v2"` flags this PR uses in CI: ``` $ RUSTFLAGS="-C target-cpu=x86-64-v2" maturin build --release $ pip install ./target/wheels/lancedb-*.whl $ python verify.py PASS: import + simd dispatch + table create + vector search all work. ``` Pre-fix on the same CPU (default `pip install lancedb`): `Illegal instruction (core dumped)`. Full reproducer (deps + clone + build + verification): https://gist.github.com/tobocop2/2e341358b55c143527416edfdb1e37df. Fork-internal verification PR with the dep bump and full logs: [`tobocop2#2`](tobocop2#2). ## Benchmarks — no regressions on modern CPUs from the lance-side change These are the numbers I ran for the lance PR, confirming the runtime dispatch doesn't slow down the default (`target-cpu=haswell`) wheel that existing users install. Criterion, one machine, one session, base → PR, no `RUSTFLAGS` override. Full methodology, null experiments, and logs: [lance-format/lance#6630 benchmark comment](lance-format/lance#6630 (comment)) and the [logs gist](https://gist.github.com/tobocop2/3c6d0f449cbd736aa2501f89a7fe56a2). | benchmark | EPYC 7B13 (`avx2`, `fma`, no `avx512f`) | Xeon Cascade Lake (`avx512f`) | |---|---|---| | `Cosine(f32, scalar)` *(control)* | +0.04% | +0.09% | | `Cosine(f64, scalar)` | −0.34% | −1.94% | | `Cosine(u8, SIMD)` | +2.30% | +3.63% | | `Dot(f16, SIMD)` | −0.58% | +0.61% | | `Dot(f32, SIMD)` | +0.34% | **−6.08%** | | `Dot(f32, arrow_arity)` | +0.02% | −0.00% | | `L2(f32, scalar)` | −0.10% | −0.02% | | `L2(f32, simd)` (dim 1024) | +2.63% | −0.53% | | **`L2(simd,f32x8)` (dim 8)** | **−45.9%** | **−25.1%** | | `L2(u8, SIMD)` | +0.42% | −3.11% | | `NormL2(f32, SIMD)` | −1.02% | −4.17% | | `NormL2(f64, SIMD)` | +3.51% | −0.58% | Nothing regresses beyond the noise floor. Dim 8 — the PQ sub-vector width — improves 25–46%. --- To be transparent: this isn't my domain of expertise and the lance-side implementation is AI-generated. I verified it works end-to-end on the failing hardware. Happy to roll in feedback.

…ob (lancedb#3677) Hi, and thank you for LanceDB. Small CI supply-chain hardening. In `make-release-commit.yml`, the release job checks out with `LANCEDB_RELEASE_TOKEN` (a push-capable PAT) and its final step pushes the version tag using a third-party action pinned to a **mutable branch**: ```yaml - name: Push new version tag uses: ad-m/github-push-action@master with: github_token: ${{ secrets.LANCEDB_RELEASE_TOKEN }} ``` `@master` can move after review; whatever it points at then runs with that release token in scope. This PR pins it to the commit behind the current release (`v1.3.0` → `881a6320…`), keeping the version visible as a comment. Behavior today is unchanged. For transparency: I used AI assistance to spot and draft this; I verified the workflow and resolved the SHA myself.

Moves the skill from .agents/skills/lancedb to plugins/lancedb/skills/lancedb, as recommended by codex and claude. Install path now is: ### Codex/ChatGPT app Codex: Plugins -> "Create" arrow -> Add plugin marketplace search for lancedb plugin and install ### Codex CLI ``` codex plugin marketplace add lancedb/lancedb codex plugin install lancedb@lancedb # name@marketplace ``` ### Claude app Settings -> Plugins -> Add -> Add marketplace search for lancedb plugin and install ### Claude CLI ``` claude plugin marketplace add lancedb/lancedb claude plugin install lancedb@lancedb ``` Here's how it looks on ChatGPT/Codex app: (the main icon has light and dark modes; the smaller one on the skill doesn't so I made it gray 🤷 ) <img width="764" height="560" alt="Screenshot 2026-07-16 at 2 49 24 PM" src="https://github.com/user-attachments/assets/b82cda16-3392-4740-ac47-b2f187cb2655" />

…b#3625) ## Problem `table.add(dataset)` with a `pyarrow.dataset.Dataset` OOMs the client during bulk ingestion of wide rows (e.g. embedding columns), even against a remote table where the upload itself is streaming. The cause is in `to_scannable`: a `Dataset` is scanned with pyarrow's default scanner settings (`batch_size=131072` rows, `batch_readahead=16`, `fragment_readahead=4`). pyarrow's internal threads prefetch that read-ahead window independently of LanceDB's backpressure, so for wide rows a large fraction of the dataset is held in memory. On the remote path this is then multiplied across the multipart write partitions (one in-flight batch per partition, up to CPU-core count). Reproduced on a 10 GB / 1.55M-row dataset with two 768-dim float32 embeddings: peak client RSS ~11.7 GB for the scan alone (6.8 GB after consuming a *single* batch), ~15.4 GB for the full remote `add()`. ## Fix `to_scannable` now sizes the scanner from an estimate of bytes-per-row derived from the schema: - **Narrow datasets keep pyarrow's defaults** (empty scanner kwargs) — no throughput regression. The bound only engages above ~410 bytes/row. - **Wide rows** get a smaller `batch_size` (~16 MiB/batch) and reduced read-ahead (`batch_readahead=2`, `fragment_readahead=1`) so peak in-flight memory stays near a ~1 GiB budget. Read-ahead (not just batch size) has to drop, because pyarrow pins whole row-group buffers. On the 10 GB dataset this drops peak client RSS to ~1.4 GB, and it stays flat as the dataset grows. The `Dataset`/`LanceDataset` scannables remain rescannable (retry-safe). ## Also: expose `write_parallelism` on `add()` `AddDataBuilder::write_parallelism` already existed in Rust but was not exposed in Python. This PR forwards it through the async, sync, and remote `add()` methods, so users can cap the number of parallel write partitions (each buffers data in flight) to trade throughput for memory on large uploads. ## Tests - `test_scannable.py`: bytes-per-row estimation; narrow → defaults; wide → bounded; `Dataset` reader streams bounded batches and stays rescannable. - `test_table.py`: `write_parallelism` on sync and async `add()`, and that `write_parallelism=0` is rejected. Fixes ENT-1883 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…ncedb#3683) Some additions to our lancedb skill to enable agents to use the jobs methods that we recently added. Eval tests (below, with and without these additions to the skill) suggest that they're helping, mostly to find the right method calls. These are a little unusual because they require REST server connection, they're not yet implemented in the SDKs. ``` ┌─────────────────────┬───────────┬────────────┬─────────────┬──────────┬───────────┬──────────┬───────────┐ │ eval │ grade w/o │ grade with │ improvement │ time w/o │ time with │ cost w/o │ cost with │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ 8-list-running-jobs │ 2.5/3 │ 3/3 │ +0.5 │ 123s │ 29s │ $0.58 │ $0.18 │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ 9-describe-job │ 1/5 │ 5/5 │ +4.0 │ 159s │ 52s │ $0.62 │ $0.25 │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ 10-cancel-job │ 3/3 │ 3/3 │ +0.0 │ 99s │ 35s │ $0.55 │ $0.21 │ ├─────────────────────┼───────────┼────────────┼─────────────┼──────────┼───────────┼──────────┼───────────┤ │ TOTAL │ 6.5/11 │ 11/11 │ +4.5 │ 381s │ 116s │ $1.75 │ $0.65 │ └─────────────────────┴───────────┴────────────┴─────────────┴──────────┴───────────┴──────────┴───────────┘ ``` Failure reasons are because the agent didn't know the right method to call, spent all its turns guessing REST calls, tried to inspect lancedb code, but didn't find the answer in here.

lancedb#3567) ## Summary Fixes lancedb#1653. `infer_vector_column_name` in `util.py` could silently return `None` when `query is None` and `query_type` is not `"fts"` or `"hybrid"`. This `None` then propagated into downstream code, causing a cryptic `TypeError: expected bytes, NoneType found` rather than a clear error message. ## Changes - **Removes the no-op `try/except Exception as e: raise e`** around `inf_vector_column_query` (it was catching and immediately re-raising without adding any value) - - **Adds a `None` guard** after the inference block: if `vector_column_name` is still `None` at this point, raise a clear `ValueError` pointing the user to pass `vector_column_name` explicitly ## Before / After **Before:** cryptic `TypeError: expected bytes, NoneType found` deep in schema lookup code **After:** ``` ValueError: No vector column found in the schema. Please specify the vector column name explicitly via the `vector_column_name` parameter. ``` --------- Co-authored-by: Will Jones <[email protected]>

…ts (lancedb#3630) ## Problem On the remote (LanceDB Cloud) write path, each write partition is uploaded as a **single** `/insert?upload_id=...` request that stays open until the whole partition has been streamed and the server has written it to object storage. For large bulk ingests a partition can be many GB, so a single request can run longer than the client read timeout (default 300s), surfacing as: ``` lancedb.remote.errors.HttpError: operation timed out ``` The server already supports staging **multiple** parts under one `upload_id` (each `/insert` writes a separate transaction that `complete` merges atomically), but the client never used that — it sent one part per partition. ## Change Split each partition into multiple parts of at most `max_bytes_per_request` (Arrow IPC, LZ4-compressed) bytes, each uploaded as its own `/insert?upload_id=...&upload_part_id=...` request. This bounds how long any single request stays open, independent of total data size or write parallelism. Key properties: - **Still streamed, not buffered.** Each part's body is driven through a bounded channel while the request is in flight (`futures::join!` of a producer + the send), so peak memory stays at a couple of batches per partition regardless of the part size. Backpressure from a slow/throttled server still propagates upstream. - **Correct part accounting.** An empty partition still sends exactly one (schema-only) part so `complete` has a transaction to commit; a size cut landing exactly on the end of input does not emit a trailing empty part. - **Multipart only.** The single-request (non-multipart) path is unchanged. ## Config New `ClientConfig::max_bytes_per_request: Option<usize>`, also settable via the `LANCE_CLIENT_MAX_BYTES_PER_REQUEST` environment variable. **Default 1 GiB** (`Some(0)` disables splitting → one request per partition). Python users pick up the default/env automatically through the remote client. ## Tests - `test_multipart_chunked_splits_into_parts`: a 1-byte budget puts each batch in its own part → N requests, each carrying the shared `upload_id` and a distinct `upload_part_id`. - `test_multipart_single_part_when_under_budget`: a large budget keeps the partition in a single request. - Verified end-to-end against a live remote table: a forced-chunked multipart add (many parts) assembles to the correct row count. Related to ENT-1883. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

This PR adds some support for `diff` / `merge` in the remote client as for local tables we stay `NotSupported` until lance-format/lance#7263. This wires the two review-and-land calls against the remote REST API: - `POST /v1/table/{id}/branches/diff` - `POST /v1/table/{id}/branches/merge` Rust gets typed results (`BranchDiff`, `MergeBranchResult`). Python returns the wire JSON, same shape as the REST response. Merge here means promoting a branch's added columns onto `main`. ### Behavior - Remote only. Local raises `NotSupported`. - A rejected merge is not an exception. HTTP 409 still returns `Ok` / a dict with `status="rejected"` and blockers in `diff.mergeBlockers`. - Unknown blocker / status codes parse as `Unknown` so a newer server does not break older clients. - `MergePreview` tolerates missing fields for the same reason. - Merge requests are not retried. 409 is final and carries the body you need. ### Example ```python table = db.open_table("images") table.branches.create("exp") exp = table.branches.checkout("exp") exp.add_columns({"tag": "cast('draft' as string)"}) diff = table.branches.diff("exp") preview = table.branches.merge("exp", dry_run=True) result = table.branches.merge("exp", dry_run=False) if result["status"] == "merged": print("landed at", result["mainVersionAfter"]) elif result["status"] == "rejected": print(result["diff"]["mergeBlockers"]) ``` ### Testing cargo test -p lancedb --features remote diff_branch cargo test -p lancedb --features remote merge_branch --------- Co-authored-by: Cursor <[email protected]>

Updates the Rust workspace Lance dependencies and Java lance-core from v9.1.0-beta.4 to v9.1.0-beta.5. No compatibility fixes were required. Triggering tag: https://github.com/lance-format/lance/releases/tag/v9.1.0-beta.5 --------- Co-authored-by: Lu Qiu <[email protected]> Co-authored-by: Claude Fable 5 <[email protected]>

Updates the Rust workspace Lance dependencies and Java lance-core from v9.1.0-beta.5 to v9.1.0-beta.7, including the generated Cargo lockfile. No LanceDB compatibility changes were required for this release. See the [Lance v9.1.0-beta.7 release](https://github.com/lance-format/lance/releases/tag/v9.1.0-beta.7).

…ancedb#3660) ## What - Replace legacy model names in `WatsonxEmbeddings` with the current supported set: - `ibm/granite-embedding-278m-multilingual` (new default, 768-dim) - `ibm/slate-125m-english-rtrvr-v2` (768-dim) - `ibm/slate-30m-english-rtrvr-v2` (384-dim) - `intfloat/multilingual-e5-large` (1024-dim) - `sentence-transformers/all-minilm-l6-v2` (384-dim) - Add `space_id` field — mutually exclusive with `project_id`, mirrors the existing pattern in `WatsonxReranker` - `project_id` / `space_id` resolution now falls back to `WATSONX_PROJECT_ID` / `WATSONX_SPACE_ID` env vars; exactly one must be supplied ## Why The previously hardcoded models (`ibm/slate-125m-english-rtrvr`, `sentence-transformers/all-minilm-l12-v2`) are legacy and no longer listed as supported by the watsonx.ai platform. `space_id` scoping was already supported by `WatsonxReranker` but was missing from the embeddings counterpart. --------- Co-authored-by: Will Jones <[email protected]>

## Summary - reconstruct foreign Arrow Map schemas from their single sanitized entries field - reject malformed Map types with anything other than one child - preserve the complete Map schema and `keysSorted` value through empty-table creation and IPC round trips across Arrow 15–18 ## Testing - `./node_modules/.bin/jest --runInBand __test__/arrow.test.ts __test__/sanitize.test.ts` - `pnpm lint` - `pnpm build` - `pnpm run docs` Fixes lancedb#2337

Updates Rust workspace Lance dependencies and Java lance-core to v9.1.0-beta.8. Removes MemWAL writer settings that are no longer exposed by Lance. Lance tag: https://github.com/lance-format/lance/releases/tag/v9.1.0-beta.8

- Bump fork version 0.1.6 -> 0.1.7 - Point lance deps at TheDeltaLab/lance main (c416d29d, v9.1.0-beta.8) - Keep fork customizations: no python/java/remote code, @delta-ai/lancedb naming, profiling-otlp default features, fork CI layout - Update [workspace.metadata.upstream].version to 0.32.0-beta.2 Co-Authored-By: Claude <[email protected]>

The merged deny.toml inherited upstream's pyo3 advisory ignores (RUSTSEC-2026-0176/0177), but this fork ships no Python bindings, so no crate matches them and cargo-deny fails with advisory-not-detected. Also drop the stale python/Cargo.toml path filter from the Rust workflow. Co-Authored-By: Claude <[email protected]>

anyhow 1.0.102 -> 1.0.104 (RUSTSEC-2026-0190, unsound downcast_mut) crossbeam-epoch 0.9.18 -> 0.9.20 (RUSTSEC-2026-0204, invalid pointer deref) Co-Authored-By: Claude <[email protected]>

spin 0.9.8 -> 0.9.9 and 0.10.0 -> 0.10.1, matching upstream lancedb#3663. Co-Authored-By: Claude <[email protected]>

yuvalif and others added 30 commits May 26, 2026 11:49

perf: migrate list_indices to use Lance's describe_indices (lancedb#3108

ab982d7

) This needs lance-format/lance#6099 to work. Closes lancedb#3140 --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

chore: update Lance to v7.2.0-beta.1 (lancedb#3461)

5638907

Update the Rust workspace Lance git dependencies and Java lance-core dependency to v7.2.0-beta.1. This keeps LanceDB aligned with the latest Lance beta release and refreshes the Cargo lockfile for the new Lance dependency graph.

Bump version: 0.33.0-beta.1 → 0.33.1-beta.0

60f9615

Bump version: 0.30.0-beta.1 → 0.30.1-beta.0

f20ec99

Bump version: 0.33.1-beta.0 → 0.33.1-beta.1

ac3411e

Bump version: 0.30.1-beta.0 → 0.30.1-beta.1

9483b53

Lance Release and others added 23 commits July 14, 2026 23:27

Bump version: 0.35.0-beta.1 → 0.35.0-beta.2

3fd322a

Bump version: 0.32.0-beta.1 → 0.32.0-beta.2

8a4eaaa

chore: update lance dependency to v9.1.0-beta.4 (lancedb#3690)

8450683

chore: replace lazy_static with LazyLock (lancedb#3679)

82231bf

sinianluoye requested a review from a team July 22, 2026 08:02

github-actions Bot added the chore label Jul 22, 2026

chuang and others added 3 commits July 22, 2026 16:10

fix(ci): bump anyhow and crossbeam-epoch for RUSTSEC advisories

70d739f

anyhow 1.0.102 -> 1.0.104 (RUSTSEC-2026-0190, unsound downcast_mut) crossbeam-epoch 0.9.18 -> 0.9.20 (RUSTSEC-2026-0204, invalid pointer deref) Co-Authored-By: Claude <[email protected]>

fix(ci): bump spin to avoid yanked versions in lock file

2945fe7

spin 0.9.8 -> 0.9.9 and 0.10.0 -> 0.10.1, matching upstream lancedb#3663. Co-Authored-By: Claude <[email protected]>

sinianluoye merged commit d2aa260 into main Jul 22, 2026
26 checks passed

sinianluoye deleted the feat/merge_sync_20260722 branch July 22, 2026 14:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

chore: merge upstream lancedb sync (0.32.0-beta.2)#62

chore: merge upstream lancedb sync (0.32.0-beta.2)#62
sinianluoye merged 186 commits into
mainfrom
feat/merge_sync_20260722

sinianluoye commented Jul 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Uh oh!

Conversation

sinianluoye commented Jul 22, 2026

Summary

Conflict resolution notes

Verification

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants