[List layout] Harden metadata, child shapes and offset arithmetic before freeze #55

Harden structural list-layout metadata, child shapes and arithmetic before freezing the accepted wire domain. This tracks F2/F3 of the freeze assessment, coordinated with #54.

Reviewed QC develop at 9f21db7954c7 on 2026-11-04. #57 and #68 are merged. Most of the original implementation gaps are addressed; keep this issue open for the specific remaining items below. The sliced-list normalization fix is already present. #56 is closed by #68.

Completed in merged code

• Reject unknown/non-integer metadata offset PTypes. Metadata validation.
• Deserialization requires exactly parent_rows.checked_add(1) offset rows, rejects overflow and zero-offset structural nodes, and validates child count, List parent dtype, elements dtype and nullable validity length. Child dtypes are checked/supplied through the child adapter. Deserialization.
• Replace ListLayout::new with fallible try_new: enforce element dtype, non-nullable integer offsets with at least one row, and validity presence/dtype/length. Construction.
• Check cumulative element-count addition in the writer and row_range.end + 1 in offset reads. Writer, offset reads.
• Validate consumed decoded offsets before range arithmetic/reconstruction: expected count, actual signed non-negativity (even with forged persisted statistics), monotonicity and terminal element bound. Full, bounded, sparse and length-only paths share this boundary. This is the [List layout] Validate decoded offsets before range calculation and unchecked reconstruction #56 work delivered by fix(layout): check list offsets before the list reader uses them #68; validity-only reads still avoid offsets/elements. Validation.
• Retain metadata, construction, malformed-offset, sliced/nested/null-list and projection regressions. fix(jni): enable only the list layout for vortex.listLayout writes #57 also adds direct zero-row-node and file-level empty-column coverage. These tests were inspected, not rerun for this update.

Remaining work

• Set and enforce the frozen offset domain after [List layout] Specify the freeze contract and qualify empty representations #54's decision. Construction and deserialization currently accept every integer PType; decoded signed values must be non-negative, and the writer emits non-nullable u64. Decide whether to retain that wider reader domain or narrow it, document any experimental-file import policy, then keep both entry points and tests consistent. Acceptance of unreferenced element prefixes/suffixes and direct list-node segments also needs [List layout] Specify the freeze contract and qualify empty representations #54's contract decision before adding rejection rules here.

• Return errors for unrepresentable split coordinates. register_splits still uses checked_add(...).vortex_expect(...) for interior root-coordinate splits and calls root_row_range() for the terminal split. The shared SplitRange::try_new / root_row_range checks local range ordering but not row_offset + end; root-coordinate addition remains unchecked. For example, SplitRange::try_new(u64::MAX, 0..1) is accepted despite an unrepresentable root end. Reject this at the shared construction/bounds boundary or return a controlled error in the consumer. This is a shared API boundary gap, not evidence of a normal scan generating that range.

• Establish an enforced reader request boundary for ranges and masks. projection_evaluation dispatches without validating start <= end <= row_count and mask length equal to range length. project_validity subtracts unchecked; for non-nullable lists it has no child read to reject an invalid request. project_all_bounded adds mask-relative positions to row_range.start, relying on those same invariants. Add a validated request boundary or trace and document an existing enforced caller boundary, then cover reversed/out-of-bounds ranges and short/long masks. Do not describe ordinary validated scans as broken merely because the private helpers rely on established invariants.

• Complete the cumulative row-count audit through child strategies. The element accumulator is checked, but list writing does not independently accumulate outer rows / the required extra offset row. Configurable child writers supply the final counts; the shared ChunkedLayoutStrategy still totals child row counts with unchecked .sum(). Qualify/reject unrepresentable cumulative offsets/validity/outer-row counts, including outer_rows + 1, at the appropriate shared boundary. This is a source-level boundary finding; no enormous stream or production corruption has been reproduced.

• Add focused regressions for the remaining boundaries and uncovered validation branches. Use synthetic counts/helpers rather than huge allocations. Cover split root-offset overflow, invalid range/mask requests, cumulative row-count overflow, offset rebasing overflow, an ordinary missing-offset count, non-List construction, validity presence/dtype/nullability, and matching deserialization rejection cases. Existing constructor cases cover wrong elements dtype, float/nullable/missing offsets and validity row-count mismatch, but do not complete that matrix. Identify which tests fail before a fix; preserve existing sliced/nested/null and pruning coverage.

Arithmetic already justified by the current validation chain

global_offsets uses fallible integer Operator::Add; primitive arithmetic reports overflow as an error. Its materialization preceding the explicit element-count guard is not, by itself, a demonstrated wrapping/panic defect. On reads, checked monotonic/bounded offsets justify rebasing and adjacent subtraction; kept_elements sums disjoint selected intervals bounded by the validated element span. The split-grid helper uses widened u128 arithmetic. Avoid replacing every unchecked expression without tracing its bounds.

This update is source review only. No builds, tests, linters or fuzz campaigns were run, and the remaining boundary examples are not runtime reproductions. The wire specification and empty/degenerate support matrix remain in #54; this issue tracks enforcing that contract and controlled arithmetic failures.