fix: preserve case in generated field path expressions by skyshineb · Pull Request #7698 · lance-format/lance

skyshineb · 2026-07-09T11:24:54Z

Summary

Fix Lance-generated DataFusion field-path expressions so schema-derived column names preserve exact casing.

Previously, field_path_to_expr used DataFusion col(...), which lowercases unquoted identifiers. This caused generated nullable-vector filters like VECTOR IS NOT NULL to resolve as vector, breaking case-sensitive schemas during
vector index creation / append optimization.

Changes

Build root field-path expressions with Expr::Column(Column::new_unqualified(...)) instead of col(...).
Preserve existing nested field handling through field_newstyle(...).
Add regression coverage for:
- uppercase nullable vector column indexing / append optimization
- exact-case root column expression generation
- mixed-case root plus escaped nested field path containing dots

Testing

cargo test -p lance-datafusion logical_expr::tests::test_field_path_to_expr
cargo test -p lance test_optimize_append_preserves_case_sensitive_nullable_vector_column
cargo fmt --all

<!-- This is an auto-generated comment: release notes by coderabbit.ai -->
## Summary by CodeRabbit

* **Bug Fixes**
* Improved query parsing to correctly preserve case sensitivity for root column names and handle escaped nested field paths (including dots inside escaped segments).
* Fixed an issue where appending data and optimizing indexes could leave unindexed fragments, affecting vector scans.
* Vector search results now remain consistent after append + index optimization when using case-sensitive nullable vector columns.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

coderabbitai · 2026-07-09T11:25:28Z

📝 Walkthrough

Walkthrough

field_path_to_expr now preserves case-sensitive root column names by constructing Expr::Column directly. New tests cover case-sensitive and escaped paths, while an append regression test validates nullable uppercase vector index optimization and search.

Changes

Case-sensitive column expression fix

Layer / File(s)	Summary
Fix column construction and add unit tests `rust/lance-datafusion/src/logical_expr.rs`	Root column expressions use `Column::new_unqualified` instead of `col()`; tests cover case-sensitive roots and backtick-escaped nested paths.
Regression test for append and optimize with uppercase vector column `rust/lance/src/index/append.rs`	Reformats Arrow imports and adds coverage for appending, optimizing, reloading, and querying a nullable `"VECTOR"` column.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	It concisely summarizes the main fix: preserving case in generated field-path expressions.
Linked Issues check	✅ Passed	The fix preserves exact-case root columns in generated expressions and adds a regression test for uppercase nullable vector optimization/search.
Out of Scope Changes check	✅ Passed	The changes stay within the case-preservation bug fix and related regression-test scope.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance-datafusion/src/logical_expr.rs`:
- Around line 289-291: The initial column construction in Expr::Column is
cloning the first segment unnecessarily. Update the path handling in the
relevant logical expression conversion to consume parts with into_iter() (or
otherwise move the first element) so the first segment can be used without
parts[0].clone(), while keeping the later loop over the remaining segments
unchanged.

In `@rust/lance/src/index/append.rs`:
- Around line 1110-1123: The final assertion in the nearest-neighbor test is too
weak and only checks that the query succeeded; update the test around the
scanner/try_collect path to assert the expected number of returned rows instead
of just `result_rows > 0`. Use the same pattern as other append tests like
`test_append_index`, and tighten the check to confirm the `k=10` result count
from `scanner.nearest(...)` is actually returned.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 5614d26f-8559-48bb-aa40-815835fa38f9

📥 Commits

Reviewing files that changed from the base of the PR and between a9f84dd and 9cdf6b7.

📒 Files selected for processing (2)

rust/lance-datafusion/src/logical_expr.rs
rust/lance/src/index/append.rs

codecov · 2026-07-09T12:18:38Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

coderabbitai

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@rust/lance/src/index/append.rs`:
- Around line 1114-1119: Strengthen the assertions in the optimized-index test
after `scanner.try_into_batch()` by verifying the returned `id` column contains
`ROWS as u32 + 5`, in addition to asserting 10 rows. Follow the appended-vector
membership check used by `test_append_index` so the test confirms the appended
row is indexed and returned.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: ba322ffa-3ead-4edd-b5ef-d472b8df6243

📥 Commits

Reviewing files that changed from the base of the PR and between 9cdf6b7 and 76f347a.

📒 Files selected for processing (1)

rust/lance/src/index/append.rs

wjones127

Nice work. I appreciate the regression test.

fix: preserve case in generated field path expressions

9cdf6b7

github-actions Bot added the bug Something isn't working label Jul 9, 2026

coderabbitai Bot reviewed Jul 9, 2026

View reviewed changes

Comment thread rust/lance-datafusion/src/logical_expr.rs

Comment thread rust/lance/src/index/append.rs

test: assert exact vector search result count

76f347a

coderabbitai Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread rust/lance/src/index/append.rs

wjones127 self-requested a review July 10, 2026 17:44

wjones127 approved these changes Jul 13, 2026

View reviewed changes

wjones127 merged commit 5bda9bd into lance-format:main Jul 13, 2026
35 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: preserve case in generated field path expressions#7698

fix: preserve case in generated field path expressions#7698
wjones127 merged 2 commits into
lance-format:mainfrom
skyshineb:fix/preserve-field-path-case

skyshineb commented Jul 9, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Jul 9, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jul 9, 2026

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

wjones127 left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

skyshineb commented Jul 9, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Testing

Uh oh!

coderabbitai Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jul 9, 2026

Codecov Report

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

wjones127 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

skyshineb commented Jul 9, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jul 9, 2026 •

edited

Loading