Summary

Lance core can resolve nested field paths and store index metadata by field id, but LanceDB does not yet have an end-to-end nested field contract across indexing, search, metadata, and bindings.

This issue tracks the work needed to make nested fields first-class in LanceDB. It is broader than #929, which only tracks scalar indices on nested columns.

Ordered sub-issues

1. Support nested field paths in native index creation #3402
2. Return canonical nested field paths from index metadata APIs #3403
3. Support nested fields in Python FTS index APIs #3404
4. Define and test nested vector column discovery behavior #3405
5. Lock remote nested field index and search contract #3407
6. Add cross-binding nested field regression matrix #3406

Expected contract

• Public APIs accept Lance field paths, including escaped paths such as `a.b` and `meta-data`.`user-id`.
• API boundaries normalize user input to a canonical field path and field id.
• Index metadata and internal state use field ids as the stable identity.
• list_indices, index stats, table details, and remote responses expose canonical full paths, not leaf names.
• Local and remote tables, Python/Rust/Node bindings, and sync/async APIs behave consistently.

Current gaps

• Native create_index does a top-level Arrow lookup before calling Lance, so paths like metadata.user_id, payload.text, or image.embedding are not reliably accepted.
• Native index creation can pass only the leaf field name to Lance, which loses the parent path.
• list_indices / IndexConfig.columns can return only the leaf name from a field id, which is ambiguous for nested fields and same-name leaves.
• Python create_fts_index currently rejects field names containing ..
• Default vector column discovery only scans top-level fields, so nested vector leaves are not discovered.
• Remote Index::Auto uses a top-level lookup, while explicit remote requests are mostly string passthrough and lack a tested field-path contract.
• Query/search paths may work when a nested field is passed explicitly, but indexed nested search is not covered end-to-end because create/list/index metadata are not reliable yet.
• Append/optimize/fast-search behavior for nested scalar, vector, and FTS indices is not covered by regression tests.

Relevant code areas include:

• rust/lancedb/src/table.rs for native create/list index behavior.
• rust/lancedb/src/index.rs for IndexConfig.columns.
• rust/lancedb/src/utils/mod.rs for default vector column discovery.
• rust/lancedb/src/remote/table.rs for remote create/list/search contracts.
• python/python/lancedb/table.py for Python scalar/vector/FTS wrappers.
• python/src/table.rs and nodejs/src/table.rs for binding surfaces.

Acceptance criteria

• Nested scalar index: create, filter, list, stats, append, and optimize on fields like metadata.user_id.
• Nested vector index: create, explicit search, list, stats, append, and optimize on fields like image.embedding.
• Nested FTS index: create, match/phrase/hybrid search, list, append, and optimize on fields like payload.text.
• Default vector discovery either recursively discovers nested vector leaves or returns a clear error requiring an explicit path.
• list_indices() and index_stats() return canonical full field paths, not leaf names.
• Local and remote tables have the same field-path behavior.
• Python sync/async, Rust, and Node bindings have nested field coverage.
• Tests cover case-insensitive paths, special characters, top-level literal-dot fields, and ambiguous same-name leaves.

Suggested minimum field-name matrix:

• rowId
• `row-id`
• `a.b`
• MetaData.userId
• `meta-data`.`user-id`