Summary

Nullable vector index paths can generate a DataFusion filter expression that lowercases case-sensitive vector column names. This can break index creation/optimization for nullable vector columns named with uppercase or mixed-case
identifiers, such as VECTOR or Vector.

Impact

This affects Lance-generated filters, not user-written SQL. The likely failure mode is optimize_indices() or nullable vector training/optimization failing to resolve the vector column, or targeting the wrong column when a lowercase
case-variant also exists.

Severity: P1 correctness bug in vector index maintenance.

Affected Paths

The nullable vector paths call field_path_to_expr(...).is_not_null():

• rust/lance/src/index/vector/utils.rs:125
• rust/lance/src/index/vector/utils.rs:446
• rust/lance/src/index/append.rs:611

field_path_to_expr builds the root column using DataFusion col(&parts[0]):

• rust/lance-datafusion/src/logical_expr.rs:289

DataFusion col("A") is normalized like col("a"); only quoted identifiers preserve case. Lance also passes ExprFilter::Datafusion through unchanged:

• rust/lance/src/dataset/scanner.rs:486

So this bypasses Lance SQL planner logic that resolves unquoted SQL identifiers against the schema with exact-case preference.

Minimal Repro Shape

1. Create a dataset with a nullable vector column named VECTOR:
   • type: FixedSizeList<Float32, 16>
   • nullable: true
2. Create an IVF_PQ index on VECTOR.
3. Append additional rows.
4. Run optimize_indices().
5. Search on VECTOR.

Expected: index optimization succeeds and search works.

Actual: generated VECTOR IS NOT NULL filter is represented through col("VECTOR"), which DataFusion normalizes to vector, so planning/execution can fail or bind to the wrong field.

Also test the ambiguous case where both vector and VECTOR exist.

Suggested Fix

Make field_path_to_expr preserve the exact parsed field path when building generated expressions. For the root column, avoid DataFusion col(&str) normalization, for example by constructing
Expr::Column(Column::new_unqualified(parts[0].clone())), then applying existing nested-field handling.

Alternatively, quote the root identifier before calling col, but direct Column::new_unqualified is less stringly.

Test Coverage Needed

Add regression coverage for:

• nullable VECTOR vector column with IVF_PQ, append, optimize_indices(), search
• mixed-case Vector
• lowercase vector control case
• non-null uppercase vector column control case
• uppercase nullable vector with actual null rows
• optional ambiguity case: both vector and VECTOR exist