fix(python): handle scalar-indexed merge inserts#3612
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
github-actions
Bot
added
bug
| builder.when_not_matched_by_source(WhenNotMatchedBySource::Keep); | ||
| } | ||
| builder.use_index(params.use_index); | ||
| builder.use_index(use_index); |
There was a problem hiding this comment.
issue(blocking): this will cause performance regressions in some cases. We should probably fix whatever bug is upstream. I think I'd rather do that.
There was a problem hiding this comment.
In theory, lance-format/lance#7251, should have fixed this. Is your test failing without that change? if so, we should raise a bug issue in the lance-format/lance repo.
There was a problem hiding this comment.
@wjones127 I checked this locally.
With the LanceDB-side fallback removed, the regression still fails on the currently pinned Lance v9.0.0-beta.10: the merge result reports num_updated_rows=0 instead of updating the expected rows.
I also confirmed that v9.0.0-beta.10 already contains lance-format/lance#7251, and tested the same regression against Lance main at dca65b904ce7b6398ff9b3e7aa5c9fbb4951af40 (9.0.0-beta.11). It still fails there with the same num_updated_rows=0 behavior.
So I agree this should be raised upstream in lance-format/lance. I’ll open a focused Lance issue with the LanceDB regression and link it back here before changing this PR further. Does that make sense or any suggestion?
2 participants
Summary
Fix
merge_insert(...).when_matched_update_all()for tables that have a scalar index on the merge key when the key is not one of the leading target fields.The indexed merge path can silently report zero updated rows in that shape. This keeps the indexed path for schemas where the merge keys are leading fields, and falls back to a full scan otherwise.
Adds a Python regression covering a BTREE scalar index on
pathwith partial update data, matching the reported failure mode.Test Plan
python -m pytest python/tests/test_table.py::test_merge_insert_update_all_with_scalar_indexpython -m pytest python/tests/test_table.py::test_merge_insert python/tests/test_table.py::test_merge_insert_subschema python/tests/test_table.py::test_merge_insert_update_all_with_scalar_indexruff check python/tests/test_table.pyruff format --check python/tests/test_table.pycargo fmt --all -- --checkcargo test -p lancedb table::merge::tests::test_merge_insert_use_indexBreaking Changes
None.
Fixes #3515