fix: support nested field paths in native index creation#3408
Conversation
|
ACTION NEEDED Lance follows the Conventional Commits specification for release automation. The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification. For details on the error please inspect the "PR Title Check" action. |
Xuanwo
changed the title
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
|
Docs PR opened: lancedb/docs#245 Added sections to the scalar, vector, and FTS index pages explaining how to create indexes on nested struct fields using dotted paths. |
|
does it count as breaking changes? |
I think it is a grey area. If a user was only using lancedb they could not have created the index in the first place (so list_indices can't be a breaking change). However, some users may have created the index in pylance, in which case they were getting back the old name. Let's label it a breaking change but not worry about it (e.g. don't try and introduce some backwards compatible path). |
| .unwrap(); | ||
| table | ||
| .create_index( | ||
| &["literal.`a.b`"], |
There was a problem hiding this comment.
issue(non-blocking): I think this technically works, but I will note we do have some weird rules around . in column names. We don't allow them at the top-level, but do at nested levels.
There was a problem hiding this comment.
Yep, this is intentional. Top-level dotted fields are rejected, but nested field names may contain ., so this case verifies that escaped nested segments round-trip as literal.a.b`` instead of being split into literal.a.b.
4 participants
Native index creation was resolving requested columns through top-level Arrow schema lookup before handing the request to Lance, which rejected nested paths and could collapse a nested field to its leaf name. This PR resolves index targets with Lance field-path semantics, passes the canonical path through to Lance, and reports indexed columns from field ids as canonical full paths.
This also removes the Python native FTS guard that rejected dotted paths so scalar, vector, and FTS index creation share the same nested-field contract. Related to #3402.