perf(scan): merge narrow splits at boundaries only some fields of a struct have - #67

Fixes #61.

Problem

• A scan splits a file wherever any column's chunk ends, and a split decodes the whole chunk of every column it overlaps.
• When a struct's fields chunk at unrelated rows, as list elements and dictionary-encoded columns of a wide nested file do, the union of their boundaries holds spans of a few dozen rows.
• Successive splits then each decode the same chunks of the other fields. On a 180 MB list-layout file, a full scan held 3.6 GiB of native memory, and one 176-row output array referenced 42 MB.

Change

• The struct reader marks soft boundaries. These are boundaries that only some of its fields have. Each field registers into its own RowSplits, and the struct takes their union, marking any boundary not every field has as soft. A single-field struct registers exactly as before.
• SplitBy::LayoutSubSplitting merges spans ending at a soft boundary until they reach DEFAULT_MIN_SPLIT_ROWS (8,192, the writer's row block), or max_rows if that is smaller. It runs before the existing sub-division of wide spans.
• Hard boundaries are never merged. These are boundaries every field has, and those registered by layouts above the struct, such as top-level chunk ends. Files whose columns chunk alike split as before.

Effect

Full scan of one file, with Arrow conversion as JNI does it. Live native bytes are measured with a counting allocator:

Tests

• Struct reader (real layouts written with RepartitionStrategy):
  • fields chunked every 1,000 and 3,000 rows mark only the unshared boundaries soft;
  • the same struct under a top-level chunked layout keeps the chunk end hard;
  • LayoutSubSplitting merges soft boundaries and preserves shared and validity boundaries, for nullable and non-nullable structs.
• merge_small_spans: unit cases plus 1,000 generated cases checking endpoints, hard boundaries, exact coverage, minimum spans at retained soft boundaries, and subsequent subdivision.
• The nullable integration regression failed before the validity-boundary fix.
• Checks: 836 vortex-layout library tests; all-target, all-feature Clippy for vortex-layout and vortex-file with -D warnings; pinned Rust formatting; Java Spotless; full ./gradlew build --no-parallel --max-workers=1 including JNI and Spark checks; patch whitespace. All passed. The full Rust workspace suite was not rerun locally; Linux musl CI passed the workspace tests.

Review follow-up

• Compute the shared-boundary intersection once instead of searching every field for every boundary. The existing 256-column split-collection debug benchmark's warm median fell from about 338 ms to 3 ms.
• Preserve inherited soft marks while keeping this struct's validity boundaries hard, including where they overlap a soft data-field boundary.
• Move soft marks when appending split sets instead of cloning their buffer.

Not done here

• The minimum is a row count. Rows with many list elements are larger than rows with few, so a byte-based minimum would be more precise.
• An earlier variant measured less memory at 16,384 rows; the minimum isn't tuned beyond 8,192.

🤖 Generated with Claude Code