Part of #7750

Summary

• In Lance 2.3+, automatically choose sparse structural pages only when the structural encoding is implicit, the dense mini-block rep/def plan requires top-level page splitting or has a single row over budget, and the sparse writer supports the value path.
• Keep within-budget pages on the dense path without constructing a sparse value block or sparse plan.
• Preserve every explicit structural mode, Lance 2.2 behavior, and dense fallback/error behavior for dictionary, variable packed-struct, and other sparse-ineligible value paths.
• Keep the policy decision separate from sparse serialization; this adds no wire-format fields.

This is the policy/performance layer that turns the already-readable and explicitly-writable sparse format on automatically only for over-budget eligible pages.

Integration

• PR3 #7754 absorbed the readable and explicitly writable sparse format from the closed PR4 #7755. Its squash patch is replayed directly on the current main base before the policy commits in this PR.
• PR1 #7751 is merged; the rebase resolves the stack onto its MiniBlockRepDefBudget naming.

100M-row S3 benchmark

Environment: r7i.8xlarge in us-east-2b, Lance V2.3, 100 files of 1,000,000 rows, 65,536-row write batches, and 1,024-row takes. The final committed harness is at 66c936391aa46bfcd14f7ac1c820878da68fcc0b on xuanwo/sparse-stack-5-auto-sparse-bench-final, based on pre-rebase PR5 head ed0ce68d7f4cceafe6a249c912f7f1da8c4fecd9. The rebase consolidates #7754 and resolves #7751 naming without changing the auto-selection or dense-fallback policy; these measurements remain attributed to that exact benchmarked head and are not relabeled as current-head results.

Cases:

• hnsw: List<UInt32>; the first 1/7 of rows (14,285,714 of 100M) contain 32 consecutive values, followed by an empty tail.
• uniform: List<UInt32>; one singleton non-empty row every 10,000 rows.
• deep: Struct<events: List<Struct<id: Int32, tags: List<Int32>, pair: FixedSizeList<Int32; 2>>>>; non-empty every 4,096 rows with nullable top struct, event list, event struct, id, tags, pair, and pair items.

cold is the first operation on a newly opened Dataset; warm is the immediate repeat on the same Dataset. Times are milliseconds. Bytes are total dataset object bytes. Every dataset has 102 objects, including 100 Lance data files.

deep/full-zip wrote all 100M rows and produced the reported size/page/object counts, then its first full scan panicked at primitive.rs:2588 by unwrapping a missing fixed full-zip decode task. Sparse and mini-block completed all reads. A separate 200K-row check at the exact PR4 base SHA da06b79a7f5eda4f33e2d80990c46766095ba383 independently confirmed that this nested-null deep/full-zip read was already broken before this PR, returning Structural validity has 6 entries for an array with 7 values. The precise failure site varies with this decoder path, but explicit full-zip never enters PR5's automatic sparse candidate closure, so this is not an auto-sparse regression. The policy deliberately does not disguise that behavior as sparse support.