feat(spark): prune struct fields inside arrays (FileFormat connector) - #42

…ept lists skip

Signed-off-by: Matt Katz <[email protected]>

A struct reader over a nullable struct expands `$.g` to
`mask($.g, is_not_null($))`. In `list_transform($.g, x -> ...)` that
mask sat between the transform and the field, so the partitioner gave
field `g` only `$.g` and the list reader read every element column.
A list inside nullable struct elements, which is how Spark writes
arrays of structs, never had its elements pruned.

An infallible transform yields the same values whether its list is
masked before or after it, so the reader now masks the whole
transform, as it already did for `get_item` and `select` chains, and
the transform reaches the list reader.

`mask` is now declared infallible: it cannot fail on any input its
dtype check accepts. That lets a transform whose body masks a packed
element struct take this path too. It also lets other infallibility
checks, such as dictionary push-down, evaluate `mask` over values no
row uses.

The new list reader cases cover the expression shapes the Spark
reader builds, and nullable elements at both levels of a nested list.

Signed-off-by: gitRasheed <[email protected]>

Adds `Expression.variable` and `Expression.listTransform`, so a Java
caller can apply an expression to every element of a list. The body
may read only its own parameter; the native lambda rejects anything
else, and the error reaches Java as a RuntimeException.

Signed-off-by: gitRasheed <[email protected]>

A stable write stores each list chunk as one segment, so reading one
field of the structs in an array reads all of them. The list layout
stores offsets, validity and elements apart and splits struct elements
into one column per field, but only the unstable edition permits it.

`vortex.listLayout` (default false) writes it, through
`VortexWriter.Builder.listLayout`. Enabled editions are shared by every
clone of a session, so enabling the unstable edition on the JVM-wide
session would change every later write. The JNI writer instead writes
through a session of its own with that edition enabled. The guide says
these files may hold other unstable encodings and that future versions
may not read them.

Signed-off-by: gitRasheed <[email protected]>

Reading `arr.a` from an `array<struct<...>>` read the whole array: the
scan builder restored each array's full table type, and the reader
projected only through structs.

Pruning now recurses through arrays. Each array whose elements are
pruned is read through `list_transform(arr, e -> ...)`, whose body
keeps the element fields Spark reads in the same way a struct is
pruned: `select` when only leaves are dropped, `pack` when a child is
pruned too, masked with the element's own nulls when the elements are
nullable. Fields resolve against each file's schema as before, which
now keeps array shapes as well as struct shapes, and a nullable element
field the file lacks reads as null.

A file in the list layout then reads only the offsets, validity and
element columns the query uses. A stable file still reads the whole
array.

Signed-off-by: gitRasheed <[email protected]>

A list-layout write ran in a new default session, so it lost every
encoding, layout or function registered on the session the caller
supplied.

It now runs in a fork of that session. `VortexSession::fork` copies the
session's variables into a store of its own, so a variable registered on
the fork is not seen by the original. The enabled editions are then
replaced with a new set holding the session's editions plus the unstable
one; the copied set would still share its contents with the original.

Signed-off-by: gitRasheed <[email protected]>

Spark prunes fields inside arrays only on the V2 path, and keeps whole
arrays after `inline` or `transform` and the inner arrays' fields after
`explode`.

Signed-off-by: gitRasheed <[email protected]>

`NullFilledArrayVector.getArray` read the length of the wrapped
column's array without checking the row, so a null array threw a
NullPointerException where Spark's `ColumnVector` contract returns null.

Signed-off-by: gitRasheed <[email protected]>