perf(spark): push nested schema pruning down to the native scan (FileFormat connector) - #40

The V2 scan left Spark's nested schema pruning off and the reader
projected whole top-level columns, so reading `s.a` from a struct `s`
read all of `s`.

The scan builder now opts in, and the reader turns the pruned read
schema into a native projection: `select` when only leaf fields are
pruned, which keeps the struct's nulls, and `mask(pack(...),
is_not_null(...))` when a child struct is pruned too. Structs inside
arrays and maps keep their full type, as the native scan cannot project
list elements yet. This adds the `Expression.mask` JNI binding.

The V1 `FileFormat` path is unchanged: Spark's V1 schema pruning rule
only applies to Parquet and ORC.

Signed-off-by: gitRasheed <[email protected]>

The reader trusted that a read struct had the file's fields, in the
file's order and case. With a supplied schema that did not, it read the
wrong field, failed in the native scan, or indexed past the struct's
children, where Parquet returns the right value or null.

Nested fields now resolve by name against each file's own schema under
Spark's case sensitivity, at every level, and are read under the file's
names. A nullable field the file does not carry reads as null; a
non-nullable one fails naming the field. A struct the file carries none
of the read fields of keeps its own nulls, as Parquet does from Spark
4.1. Pushed filters only bind paths the file carries under the same
names, and Spark evaluates the rest.

Signed-off-by: gitRasheed <[email protected]>

The reader converted every column of a file to a Spark type before
projecting, though only names, nullability and struct shapes are matched
against. A column Spark cannot represent, such as DECIMAL(50, 2), failed
a query that never read it. An unsigned integer inside a struct turned
the whole struct into an opaque column, so reading a sibling field read
the whole struct and failed on the unsigned field.

The file schema now keeps struct shapes and leaves every other type
unconverted, so a column Spark cannot read only fails a query that
reads it.

Signed-off-by: gitRasheed <[email protected]>

Signed-off-by: gitRasheed <[email protected]>