Part of #7750

Depends on #7752.
Wire contract discussion:
#7631. The
validity-polarity text and vote remain a merge gate.

This PR freezes the Lance 2.3 SparseLayout wire contract and lands its
safe, selective reader together with an explicit writer so the contract
is exercised end to end. The reader rejects SparseLayout in pre-2.3
files, validates metadata, domains, semantic sets, descriptors, buffer
ranges, compression headers, chunk sums, and complete chunk consumption,
caches normalized plans, projects range/take selections through nested
layers, reads only intersecting value chunks, and rebuilds no-value
selections without value I/O. SparseLayout adds no format-specific size
or descriptor-complexity quotas for otherwise representable buffers;
callers retain responsibility for resource policy.

The writer can emit sparse pages only when field metadata explicitly
sets `lance-encoding:structural-encoding=sparse` for a Lance 2.3 file.
It does not add automatic selection or change the default encoding
policy. Lance 2.3 is unstable, so tests generate pages at runtime and
round-trip them; this PR contains no checked-in compatibility fixture or
generated test artifact.

Coverage includes nullable primitive and struct,
list/large-list/map/fixed-size-list, null versus empty lists, both
validity polarities, all semantic position/count representations, scan,
range, take, deeply nested structures, no-value selections, selective
value I/O, malformed pages, large representable buffers/descriptors, and
pre-2.3 rejection.

Validation:

- `cargo fmt --all -- --check`
- `cargo check -p lance-encoding -p lance-file --features protoc`
- `protoc --descriptor_set_out=/dev/null --proto_path=protos
protos/encodings_v2_1.proto protos/file2.proto`
- `cargo test -p lance-encoding -p lance-file` (470 + 93 passed; 0
failed)
- focused sparse reader/writer and malformed-page tests
- `uv run mkdocs build`
- `cargo clippy --all --tests --benches -- -D warnings`

---------

Co-authored-by: Weston Pace <[email protected]>

Part of #7750

Depends on #7752.
Wire contract discussion:
#7631. The
validity-polarity text and vote remain a merge gate.

This PR freezes the Lance 2.3 SparseLayout wire contract and lands its
safe, selective reader together with an explicit writer so the contract
is exercised end to end. The reader rejects SparseLayout in pre-2.3
files, validates metadata, domains, semantic sets, descriptors, buffer
ranges, compression headers, chunk sums, and complete chunk consumption,
caches normalized plans, projects range/take selections through nested
layers, reads only intersecting value chunks, and rebuilds no-value
selections without value I/O. SparseLayout adds no format-specific size
or descriptor-complexity quotas for otherwise representable buffers;
callers retain responsibility for resource policy.

The writer can emit sparse pages only when field metadata explicitly
sets `lance-encoding:structural-encoding=sparse` for a Lance 2.3 file.
It does not add automatic selection or change the default encoding
policy. Lance 2.3 is unstable, so tests generate pages at runtime and
round-trip them; this PR contains no checked-in compatibility fixture or
generated test artifact.

Coverage includes nullable primitive and struct,
list/large-list/map/fixed-size-list, null versus empty lists, both
validity polarities, all semantic position/count representations, scan,
range, take, deeply nested structures, no-value selections, selective
value I/O, malformed pages, large representable buffers/descriptors, and
pre-2.3 rejection.

Validation:

- `cargo fmt --all -- --check`
- `cargo check -p lance-encoding -p lance-file --features protoc`
- `protoc --descriptor_set_out=/dev/null --proto_path=protos
protos/encodings_v2_1.proto protos/file2.proto`
- `cargo test -p lance-encoding -p lance-file` (470 + 93 passed; 0
failed)
- focused sparse reader/writer and malformed-page tests
- `uv run mkdocs build`
- `cargo clippy --all --tests --benches -- -D warnings`

---------

Co-authored-by: Weston Pace <[email protected]>