…er.h

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary: BE configure fails the build-hygiene gate on this PR
(COMPILE and BE UT): util/threadpool.h now reaches 1088 TUs, one over
its limit of 1087 (baseline 988 +10%). Master reaches 1083. Five of the
new Lance TUs reach the header through exprs/vexpr_context.h ->
storage/segment/column_reader.h -> io/cache/cached_remote_file_reader.h
-> io/cache/block_file_cache.h, an existing chain this PR does not
touch.

One of them did not need to. lance_record_batch_converter.h included
the whole format_v2/table_reader.h for two structs, ColumnDefinition and
GlobalRowIdContext, which are defined in format_v2/column_data.h, and
that header reaches nothing on the chain. The converter header now
includes column_data.h instead, so lance_record_batch_converter.cpp no
longer reaches util/threadpool.h and the reach is 1087, within the
limit. No gate table changes. The other four TUs (lance_reader.cpp,
lance_runtime_filter_helper.cpp and their unit tests) evaluate conjuncts
and runtime filters, so they include vexpr_context.h on purpose;
lance_reader.h still gets table_reader.h directly.

### Release note

None

### Check List (For Author)

- Test: Manual test
    - build-support/check-build-hygiene.sh passes (threadpool.h reach
      1087 of 1087).
    - With the BE UT compile flags, clang++ -fsyntax-only passes on
      lance_record_batch_converter.cpp, on a TU that includes only its
      header, and on the unity batch that holds it.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68649

Problem Summary:

Reproduction: read an Arrow Date32 array holding a day outside
0000-01-01..9999-12-31 into a Doris DATE column, for example 2932897
(10000-01-01) written by PyArrow into a Lance dataset, or the same value
returned by an ADBC driver, an Arrow stream load or the paimon-rust reader.
The row comes back as a non-null `0000-00-00`, also inside ARRAY, STRUCT and
MAP values.

Root cause: the DATE32 branch of the shared
`DataTypeDateV2SerDe::read_column_from_arrow` ignores the bool returned by
`get_date_from_daynr()` (dropped since apache#48944) and appends the
default-constructed value. It also adds the epoch offset in 32 bits, so
INT32_MAX overflows, and it decodes the payload of NULL slots. The DATE64
branch next to it already rejects out-of-range days.

Fix: port the DATE32 range part of the branch-4.1/4.2 fix apache#68649. NULL slots
append the default value without being decoded, the day number is computed
in 64 bits, and a day number <= 0, above DATE_MAX_DAYNR or rejected by
`get_date_from_daynr()` returns `InvalidArgument("Arrow Date32 value is
outside the Doris DATE range: row=..., days=...")` with the 4.x wording.
ARRAY/STRUCT/MAP leaves get the same check through their element SerDes.
`write_column_to_arrow` and the year-zero calendar mapping (the other part of
apache#68649, and master draft apache#67449) are unchanged; the new tests avoid the days
whose meaning that mapping changes.

Before: out-of-range days read as `0000-00-00`. After: the read fails with
the error above; in-range values and NULLs read as before.

The Lance reader ported in apache#69581 (from the branch-4.2 snapshot, which
already contains apache#68649) relies on this SerDe check.

### Release note

Reading Arrow DATE32 values outside 0000-01-01..9999-12-31 (ADBC, Arrow
stream load, paimon-rust, Lance) now fails instead of returning 0000-00-00.

### Check List (For Author)

- Test: Unit Test
    - New: `DataTypeDateTimeV2SerDeTest.RejectsInvalidDate32Values`,
      `LanceTableReaderTypeTest.RejectsDate32OutsideDorisDateRange`,
      `LanceTableReaderTypeTest.ReadsLastDorisDateAndIgnoresNullDate32Payload`.
    - All pass on macOS arm64 (ASAN), together with `DataTypeDateTimeV2SerDeTest.*`,
      `DataTypeSerDeArrowTest.*`, `DataTypeSerDeArrowValidationTest.*`,
      `ArrowColumnToDorisColumnTest.*`, `ParquetExprTest.*`,
      `IcebergV2ReaderTest.*` and the Lance suites. With master's DATE32 branch
      restored, the first two fail.
    - `DataTypeDateTimeV2SerDeTest` aborts in its setup under macOS ASAN on
      master: `load_data_from_csv()` in `be/test/testutil/test_util.cpp`
      dereferences `std::set::end()`. It ran here with that assertion removed
      locally; that fix goes in a separate PR.
- Behavior changed: Yes. An Arrow DATE32 day outside 0000-01-01..9999-12-31
  used to be returned as a non-null 0000-00-00; the scan now fails with an
  InvalidArgument error, as Arrow DATE64 values already did.
- Does this need documentation: No

Co-authored-by: Gabriel <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…decoding

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

Reproduction: write a Lance dataset with a `pa.decimal256(18, 2)` column and
scan it with the Lance reader added in apache#69581. Lance keeps the 256-bit width
on disk, so the scanner returns an Arrow `Decimal256Array`, and the BE crashes
on the first non-empty batch. The same happens for any `decimal256(p, s)` with
p <= 38, also as a list, large_list, fixed_size_list, struct or map leaf. The
existing fixture only has decimal256 columns with precision 39 and 76, which
is why the tests did not catch it.

Root cause: `arrow_field_to_doris_type()` and the FE pick the Doris type from
the precision alone (DECIMAL32 up to 9, DECIMAL64 up to 18, DECIMAL128I up to
38), while `LanceArrowArrayNormalizer` passes the `Decimal256Array` through
unchanged. `DataTypeDecimalSerDe<TYPE_DECIMAL32/64/128I>::read_column_from_arrow`
casts the array with `dynamic_cast<const arrow::DecimalArray*>`, gets null for
a `Decimal256Array`, and dereferences it in `Value()`.

Fix: at schema binding, `LanceArrowArrayNormalizer::create()` records
`decimal128(p, s)` for a Decimal256 field with p <= 38 and marks it as needing
special handling, which parents inherit through the existing child loop.
`normalize_for_doris()` then narrows the array with
`arrow::compute::Cast(..., CastOptions::Safe())` on the query's Arrow memory
pool, the cast master's ADBC normalizer already uses
(`format/arrow/arrow_array_normalizer.cpp`). Nested parents rebuild their type
around the narrowed child through `set_lance_nested_type()`, as for BFloat16
leaves, before compaction. A value outside the declared precision, which is
invalid Arrow data (for example unscaled 10^18 in decimal(18, 2)), fails with
`narrow Lance Decimal256 field '<name>' to decimal128(18, 2) failed: Decimal
value does not fit in precision 18` instead of being truncated. Precision
39..76 keeps Decimal256 and DECIMAL256 as before.

Before: the BE dereferences a null pointer and crashes. After: the values are
read unchanged, and an out-of-precision value fails with an error that names
the field.

branch-4.1 and branch-4.2 have the same normalizer, so the same change applies
there.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `LanceTableReaderTypeTest.NarrowsDecimal256WithDecimal128PrecisionDuringNormalization`,
      `LanceTableReaderTypeTest.KeepsWideDecimal256UnchangedDuringNormalization`,
      `LanceTableReaderTypeTest.ConvertsDecimal256WithDecimal128PrecisionToDorisDecimals`,
      `LanceTableReaderTypeTest.NarrowsDecimal256LeavesInSlicedNestedArrays`,
      `LanceTableReaderTypeTest.RejectsDecimal256ValuesOutsideDeclaredPrecision`,
      `LanceTableReaderTypeTest.ReadsDecimal256WithDecimal128PrecisionFromLanceDataset`
      (writes a decimal256(18, 2) dataset, checks that Lance stores Decimal256 and
      that the reader returns DECIMAL64 values).
    - All pass on macOS arm64 (ASAN) with the Lance suites. With the narrowing
      disabled, the normalization cases fail and the sliced nested case then
      crashes in `DataTypeDecimalSerDe::read_column_from_arrow`, as reported.
- Behavior changed: No (fixes a crash in the reader this PR adds)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

Reproduction: scan a Lance FLOAT or DOUBLE column that holds a negative NaN
(float bits 0xffc00000, double bits 0xfff8000000000000) under a runtime filter
`x >= c`, for example a MIN-only filter from a nested-loop join on `x >= y`.
Doris treats every NaN as the greatest value regardless of its sign
(`GreaterThanEqualsFloat` in `common/compare.h`), so the row satisfies the
filter, but it is missing from the result.

Root cause: `build_range_filter_sql()` translates the filter into the Lance
SQL `(x >= c)`. Lance evaluates it with Arrow/DataFusion comparison kernels
that order floating values by IEEE 754 totalOrder, where a negative NaN sorts
below every finite value, so Lance drops the row before the scanner evaluates
the Doris residual. The residual can only remove rows, not bring them back.

Fix: `build_range_filter_sql()` returns `std::nullopt` for a `>=` filter whose
slot is FLOAT or DOUBLE, so the filter stays residual-only and is reported
under `LanceRuntimeFilterSkippedIds`, the policy `PaimonRustPredicateConverter`
already applies to floating predicates. `<=` filters are still pushed: Lance
then only returns extra negative-NaN rows, which the residual removes. IN
filters are still pushed: a NaN row never matches a finite member, and sets
with NaN members were already rejected. A MIN_MAX filter now pushes only its
`<=` half.

Before: rows holding a negative NaN were dropped under a floating `>=` runtime
filter. After: Lance returns them and the Doris residual decides, so the
result matches a scan without the pushed filter.

branch-4.1 and branch-4.2 carry the identical helper, so the same change
applies there.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `LanceRuntimeFilterHelperTest.KeepsFloatingMinFiltersResidualOnly`
      (`>=` skipped, `<=` and IN still pushed, SQL of a MIN_MAX pair),
      `LanceTableReaderFilterTest.PreservesNegativeNaNUnderFloatingMinRuntimeFilter`
      (FLOAT and DOUBLE rows 0.5, 1, 2, +NaN, -NaN under `>= 1`; every row Doris
      keeps must come back from Lance).
    - Both pass on macOS arm64 (ASAN) with the Lance suites; with
      `build_range_filter_sql()` as before this change, both fail.
- Behavior changed: No (the Lance reader is added by this PR; floating `>=`
  runtime filters are no longer pushed into Lance, so such scans may read more
  rows)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…file as in branch-4.x

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68813

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68813 changed two things in the 4.x BE, both ported here.

1. Search filter timing. apache#68813 added `TExternalSearchRequest.prefilter`
   (field 5): unset or false means the TVF search filter refines the top-k
   candidates (postfilter), true means it constrains them before top-k
   (prefilter). The 4.x FE sends it for `vector_search()` and
   `full_text_search()`. Without the field, a master BE always prefiltered
   vector search and never prefiltered full-text search, so a 4.x FE paired
   with it got different rows: on the all_types.lance fixture, top_k 1 with
   search filter `row_id >= 3` returns row 4 with prefilter and no row with
   postfilter.
   - `gensrc/thrift/PlanNodes.thrift`: `5: optional bool prefilter` with the
     4.x comment; the stale `TSearchFilter` comment is replaced.
   - `_configure_vector_search()` and `_configure_full_text_search()` pass
     `request.__isset.prefilter && request.prefilter` to
     `lance_scanner_set_prefilter()`, replacing the comment that
     fragment-scoped nearest queries need prefiltering (master's lance-c pin
     4fafcd64525 no longer does).
   - Compatibility (gensrc/AGENTS.md): field 5 is `optional bool prefilter` on
     the branch-4.2 (a7ae6f4) and branch-4.1 (3f911d0) heads;
     branch-4.0 and branch-3.1 have no `TExternalSearchRequest`. No id is
     reused. On master the field replaces the comment that reserves it
     (apache#70075).

2. Profile layout. The snapshot hangs about 45 counters directly under
   `LanceReader`, looks Lance's scan metrics up in two maps, and creates two
   scalar-index timers from Lance's statistics callback the first time they
   are reported, that is from Lance's execution threads while the scan runs.
   As in apache#68813:
   - The counters are grouped under `LanceTiming`, `LanceIO`, `LanceIndex`,
     `LanceVectorIndex`, `LanceScalarIndex`, `LanceExecutionPlan`,
     `LanceScan`, `LanceSearchPlan` and `LanceDataCache` (label counters,
     `TUnit::NONE`, like `RuntimeFilterInfo`).
   - One immutable table, `LANCE_SCAN_METRICS` in `lance_reader_helper.h`,
     drives both registration and the callback lookup. Every metric is
     registered in `_init_scanner_profile()` before the callback is installed,
     so the callback only updates existing counters and every split has the
     same counter tree.
   - The callback is `noexcept` and swallows exceptions, so nothing unwinds
     through the lance-c ABI.
   - `LanceIndexObjectLoads` (`indices_loaded`), `LanceIndexCacheHits` and
     `LanceIndexCacheMisses` are new; the info strings `LanceTopKPlusOffset`
     and `LanceSubstraitFilterBytes` are dropped.
   - Renames: `LanceScannerReadTime` -> `LanceScannerNextTime`,
     `LanceRuntimeFilterSqlTime` -> `LanceRuntimeFilterSqlTranslationTime`,
     `LanceExecutionIOOps` / `IORequests` / `IOBytesRead` -> `LanceIOReadOps` /
     `LanceIOReadRequests` / `LanceIOReadBytes`,
     `LanceIndexPartitionCacheMissLoads` -> `LanceIndexComponentsLoaded`,
     `LancePrefilter*` -> `RowIdPrefilter*` (`LancePrefilterRowIds` ->
     `RowIdPrefilterIds`), `LanceIVFPartitionsRanked` ->
     `LanceIVFPartitionsConsidered`, `LanceIVFPartitionsSearched` ->
     `LanceIndexPartitionsSearched`, `LanceVectorIndexSegmentsSearched` ->
     `LanceVectorIndexDeltasSearched`, `LanceTaskWaitTime` ->
     `LanceScanTaskWaitTime`, `LanceIVFPartitionRankingTime` ->
     `LanceIVFPartitionSelectionTime`, `LanceScalarIndexSegmentPrepareTime` ->
     `LanceScalarIndexCandidateEvaluationTime`.
   - Master differs from 4.x in one line: the callback asserts with
     `DCHECK_LT` that a found metric has a registered counter, where 4.x
     checks it in the `if`; both come from the same array.
   - Not ported: apache#68813's FE change that makes `AggCounter` print nothing for
     `TUnit.NONE`. It belongs to the FE port; until then master's merged
     profile prints the group labels like the existing `RuntimeFilterInfo`.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `LanceTableReaderVectorSearchTest.SearchFilterDefaultsToPostfilter`
      (from apache#68813) and `LanceTableReaderFullTextSearchTest.SearchFilterHonorsPrefilterFlag`
      (top_k 1 with filter `row_id <= 2`: no row with postfilter, row 2 with
      prefilter; 4.x has no BE test for the full-text path).
    - Changed as in apache#68813: `AppliesSearchFilterBeforeTopK` becomes
      `AppliesSearchFilterBeforeTopKWhenPrefilterEnabled`;
      `MultiVectorScoresFiltersOffsetsAndIndexedCoverage` and
      `MultiVectorSegmentTopKMatchesIndependentGlobalOracle` set `prefilter`
      for their filtered cases, and the former checks the `RowIdPrefilter*`
      counters and that a flat split exposes `LanceIndexPartitionLoadTime`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With vector search always prefiltering and
      full-text search never prefiltering, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581. A search filter
  without `prefilter` now postfilters vector search, and full-text search
  honors `prefilter = true`, as on branch-4.1/4.2. Lance profile counters are
  grouped and renamed. Master has no FE that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zhangstar333 <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

… planned

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68707, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45, where
`_configure_vector_search()` passes the request's `use_index` to
`lance_scanner_set_use_index()`. FE decides per split whether to use an index:
a split it plans with index segments carries their UUIDs, and a split without
them is meant to be searched flat. With `use_index = true` and no segments,
Lance instead searches the split with the first index on the column in
manifest order. That index need not be the one FE chose: when FE finds no
index whose metric matches the query, it plans every split flat, and the
snapshot BE still searched them with the mismatched index. On branch-4.2,
apache#68815 also plans distance-range searches over quantized indexes (IVF_SQ) as
flat fallback splits, because quantized distances can drop rows inside the
range; the snapshot BE searches those with the index and returns too few rows.

Fix: port the split-level rule of apache#68707.
- `lance_scanner_set_use_index(scanner, segment_count > 0)`: a split with
  segments searches exactly those, a split without segments searches flat,
  and `LancePlannedFlatSearchFragmentCount` now matches what Lance does.
- A split that carries index segments while the request sets
  `use_index = false` is rejected with InvalidArgument. FE plans segments only
  when `use_index` allows them, on branch-4.1 (4.1.4 onward) and branch-4.2
  alike, so a 4.x FE never sends this combination.
- Not ported: the rest of apache#68707 (searching historical versions, tags and
  branches; rejecting fragment ids missing from the snapshot; naming the
  snapshot in errors and the profile). That comes with the next 4.x catch-up.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New, from apache#68707 (only master's `RuntimeState(TQueryOptions(), globals)`
      constructor differs): `LanceTableReaderVectorSearchTest.SplitWithoutIndexSegmentsSearchesFlat`,
      `LanceTableReaderVectorSearchTest.RejectsIndexSegmentsWhenUseIndexIsFalse`.
    - Changed as in apache#68707: `MultiVectorScoresFiltersOffsetsAndIndexedCoverage`,
      `MultiVectorTopOnePreservesPrecisionAndBatchIndependence` and
      `MultiVectorCosineMasksUndefinedRows` pass the index segments for their
      indexed cases (new helper `make_indexed_lance_range`) and check
      `LanceIndexPartitionsSearched`; `MultiVectorDefaultMetricIsConsistentAcrossFragments`
      checks that every split is searched flat.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c,
      with the Lance suites. With `use_index` taken from the request and the
      rejection removed, the two new cases fail.
- Behavior changed: Yes, on the Lance path added by apache#69581: a vector split
  without index segments is searched flat even when `use_index` is true, and a
  split with segments under `use_index = false` is rejected. Master has no FE
  that plans Lance scans yet.
- Does this need documentation: No

Co-authored-by: zy-kkk <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581, apache#68815

Problem Summary:

apache#69581 ports the Lance reader from the branch-4.2 snapshot cabcf45. After
that snapshot, apache#68815 added `TVectorSearchParams.distance_lower_bound` (6) and
`distance_upper_bound` (7) to branch-4.1 and branch-4.2: FLOAT bounds,
inclusive lower and exclusive upper, which the 4.x FE fills for a
single-vector `vector_search()` with distance bounds. apache#68815 kept the
external-search `schema_version` at 1, so a BE cannot tell a bounded request
from an unbounded one by version. A master BE without the fields ignored them
and returned the unbounded top-k rows, for example rows with `_distance`
below the lower bound.

Fix: port the BE side of apache#68815.
- `gensrc/thrift/PlanNodes.thrift`: add `6: optional double
  distance_lower_bound` and `7: optional double distance_upper_bound` to
  `TVectorSearchParams` with the 4.x comment.
- `LanceTableReader::_validate_external_search_request()` rejects bounds on a
  multi-vector query (NotSupported), a bound that is not a finite FLOAT value
  (NaN, infinity, or beyond FLT_MAX), and a lower bound that is not below the
  upper bound at FLOAT precision (InvalidArgument), before the dataset is
  opened.
- `_configure_vector_search()` narrows the bounds to FLOAT and installs them
  with `lance_scanner_set_distance_range()` after `lance_scanner_nearest()`,
  which clears any earlier range. A missing bound is passed as NULL
  (unbounded on that side). Requests without bounds are unchanged.

Before: distance bounds sent by a 4.x FE were silently ignored. After: the
reader returns only rows with lower <= _distance < upper, still capped by
top_k and offset, and rejects invalid bounds.

Compatibility (gensrc/AGENTS.md): fields 6 and 7 of `TVectorSearchParams` are
`optional double distance_lower_bound` and `optional double
distance_upper_bound` on the branch-4.2 (a7ae6f4) and branch-4.1
(3f911d0) heads, the same ids and types as here. branch-4.0 and
branch-3.1 have no `TVectorSearchParams`, and on master the struct is new in
apache#69581 with fields 1-5. No id is reused, and both fields are optional.

`lance_scanner_set_distance_range()` is in master's lance-c pin
(4fafcd64525), so only the BE changes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New (copied unchanged from apache#68815 except for master's
      `RuntimeState(TQueryOptions(), globals)` constructor):
      `LanceTableReaderVectorSearchTest.ValidatesDistanceRangeBeforeDatasetAccess`,
      `LanceTableReaderVectorSearchTest.AppliesDistanceBoundsAndOffset`,
      `LanceTableReaderVectorSearchTest.DistanceRangeFallbackPreservesExactScores`,
      `LanceTableReaderVectorSearchTest.DistanceRangeCoversIndexedAndAppendedFragments`.
    - Fixtures copied from d306323 (identical on the branch-4.2 head):
      `be/test/format_v2/table/lance/data/distance_range.lance` (11 files),
      `distance_range_dot.lance` (8 files), `distance_range_sq.lance` (8
      files) and their description
      `be/test/format_v2/table/lance/distance_range_fixture.md`.
    - All pass on macOS arm64 (ASAN) against master's pinned lance-c, with the
      Lance suites. `DistanceRangeFallbackPreservesExactScores` needs the
      previous commit: without it the IVF_SQ fallback split is searched with
      the quantized index and loses its only row. With the range not installed,
      the three cases that read rows fail.
- Behavior changed: Yes, on the Lance external-search path added by apache#69581:
  distance bounds are applied instead of ignored, and invalid bounds are
  rejected. Master has no FE that sends Lance search requests yet.
- Does this need documentation: No

Co-authored-by: Gabriel <[email protected]>
Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

…ffset

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

Scenario (traced in the pinned Lance source, not run end to end): a legacy
(v1) Lance dataset holds six rows of `s STRUCT<b BOOLEAN>` in one writer
batch and is scanned with `batch_size = 3`. The v1 plain decoder returns the
Boolean range [start, end) as a bitmap with bit offset `start % 8`, and its
struct and list readers wrap that child unchanged, so the batch of rows 3..5
is a zero-offset struct over a Boolean child at offset 3. `LIST<BOOLEAN>` gets
values at an offset whenever a writer batch is larger than `batch_size`.

Root cause: `compact_lance_array_if_needed()`
passed a STRUCT through unchanged when its own offset was 0 and every child
had its length, and a LIST, LARGE_LIST, MAP or FIXED_SIZE_LIST when its own
offsets were tight, without looking at the children. With
`enable_arrow_input_validation` (true by default) the Doris SerDes reject an
offset at any depth ("non-zero array offset is not supported"), so a valid
scan failed.

Fix: an array is passed through only when it and every descendant start at
offset 0 and reference exactly their child ranges
(`lance_array_layout_is_compact()`); otherwise the existing
`compact_lance_array()` rebuilds it, which copies every level to offset 0.
Checking only the direct struct children would still miss
`STRUCT<s STRUCT<b BOOLEAN>>`, `STRUCT<l LIST<BOOLEAN>>` and a top-level
`LIST<BOOLEAN>`.

Before: such a batch failed the scan. After: it is compacted and read. Arrays
that are offset-free at every level keep the zero-copy path.

branch-4.1 and branch-4.2 carry the identical normalizer, so the same change
applies there.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `LanceTableReaderTypeTest.CompactsOffsetDescendantsOfTightNestedArrays`
      (STRUCT, LIST, MAP, STRUCT<STRUCT> and STRUCT<LIST> arrays that are
      tight at their own level over a Boolean child at offset 3, read through
      the record batch converter with Arrow input validation on) and
      `KeepsCompactNestedArraysUnchangedDuringNormalization` (the same shapes
      without the offset keep their pointers).
    - Both pass with the Lance suites on macOS arm64 (ASAN). With the previous
      checks, or with only the direct struct children checked, the first test
      fails.
- Behavior changed: No (the Lance reader is added by this PR)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

A Lance dataset can hold a zero-field `STRUCT<>` column; the pinned Lance
decoder supports that layout. `arrow_field_to_doris_type()` mapped it, at any
nesting level, to a nullable `DataTypeStruct` without elements, so the schema
fetch reported it as an ordinary STRUCT. Doris cannot materialize that type:
`ColumnStruct::size()` reads its first child (`columns.at(0)`), and the
nullable wrapper checks the nested size when the column is created, so
reading such a column throws `std::out_of_range`.

Fix: the STRUCT branch returns `NotSupported` for a struct without fields.
`convert_arrow_schema_to_doris()` already maps `NotSupported` to the
`DataTypeNothing` sentinel for unsupported columns, so a zero-field struct at
the top level, or inside a struct, list or map, makes that whole column
unsupported, like the other unsupported Lance types, and the remaining
columns of the dataset are unaffected.

Before: the schema fetch reported `STRUCT<>` as a supported STRUCT column.
After: the column is reported unsupported, and its siblings are scanned as
before.

branch-4.1 and branch-4.2 carry the identical mapping, so the same change
applies there.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `LanceTableReaderSchemaTest.MapsZeroFieldStructsAsUnsupported`
      (a top-level `STRUCT<>` and one nested in a struct, a list and a map
      map to the unsupported sentinel; a non-empty struct still maps to
      STRUCT) and `FetchesZeroFieldStructAsUnsupportedAndScansSiblings` (a
      written dataset with an `id` and a `STRUCT<>` column: the schema fetch
      reports the struct unsupported, and a scan of `id` returns both rows).
    - All 102 Lance tests pass on macOS arm64 (ASAN). Without the fix, both
      new tests fail.
- Behavior changed: No (the Lance reader is added by this PR)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>

… rows

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

Arrow leaves the child values under a NULL list, fixed-size-list, map or
struct row unspecified. A FIXED_SIZE_LIST<DATE32, 1> with validity [1, 0, 1]
may hold the days [0, 2932897, 1]; its rows are [1970-01-01], NULL,
[1970-01-02]. Lance's v2.1 miniblock encoding keeps such child data, and
other Arrow producers (ADBC drivers, Arrow stream load, paimon-rust) may as
well.

Root cause: the array, map and struct SerDes pass the whole child range to
the element SerDe, which skips only the child's own NULLs. The DATE32 range
check added by this PR therefore rejected 2932897 (10000-01-01) and failed
the batch, although no visible value is out of range.

Fix: before reading a child, the nested SerDes hand it a validity bitmap
that also clears the slots hidden by a NULL parent row
(`mask_arrow_children_hidden_by_null_parents()`), so the element SerDe
neither validates nor decodes them; the parent's null map hides them as
before. Only the validity buffer is replaced, and nothing is allocated when
no parent row in the range is NULL. This covers DATE32, DATE64 and any other
check an element SerDe makes.

Before: such a batch failed with "Arrow Date32 value is outside the Doris
DATE range". After: it reads as [1970-01-01], NULL, [1970-01-02].

branch-4.1 and branch-4.2 have the same DATE32 check and nested SerDes.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `ArrowParentNullsTest` (without a NULL parent the child itself is
      returned; only the slots of NULL parents are cleared and the child
      keeps its own NULLs; the field of a sliced struct is addressed with its
      offset) and `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`
      (FIXED_SIZE_LIST, LIST with a non-empty NULL segment, STRUCT, MAP and
      LIST<STRUCT> over a hidden 10000-01-01).
    - `ArrowParentNullsTest.*`, `DataTypeSerDeArrowTest.*`,
      `DataTypeSerDeArrowValidationTest.*`, `ArrowColumnToDorisColumnTest.*`,
      the Lance and ADBC suites and `IcebergV2ReaderTest.*` on macOS arm64
      (ASAN): 297 pass, 7 ADBC SQLite cases skip (no SQLite driver here).
      With the three SerDe changes reverted, the Lance test fails with
      "Arrow Date32 value is outside the Doris DATE range: row=1,
      days=2932897".
- Behavior changed: No (visible values are unchanged; the check that hidden
  values tripped is added by this PR)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>

…sks to the query

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

1. The Lance normalizer narrows a Decimal256 leaf of precision <= 38 to
   decimal128 with Arrow's safe cast, and for a nested leaf it cast the whole
   raw child array. The values under a NULL list, fixed-size-list, map or
   struct row, which Arrow leaves unspecified, and for a sliced parent the
   values of the rows outside the slice were precision-checked too. A
   FIXED_SIZE_LIST<decimal256(18,2),1> with validity [1, 0, 1] may hold the
   unscaled children [1, 10^18, 2]; its rows are [0.01], NULL, [0.02], yet
   the hidden 10^18 failed the batch with "Decimal value does not fit in
   precision 18". The SerDe-level masking added for the hidden DATE32 values
   runs only after normalization.

   Fix: before a child goes to a child normalizer that narrows a decimal,
   every child slot that no visible non-NULL parent row references is marked
   NULL (`mask_arrow_child_slots_not_visible()`). Each level masks its own
   children, so the descendants of a hidden slot are hidden too. A visible
   value outside the declared precision is still rejected. A map key cannot
   be NULL, so a Decimal256 map key under a NULL map row is still checked.
   Children that narrow nothing skip the mask.

2. The validity bitmaps of these masks came from Arrow's default pool, which
   the query's memory tracker does not see. They now come from ExecEnv's
   `doris::ArrowMemoryPool`, the pool the Lance converter already uses; unit
   tests that never initialize ExecEnv fall back to the default pool, as the
   converter does.

Before: the batch above failed. After: it reads as [0.01], NULL, [0.02].

branch-4.1 and branch-4.2 do not narrow Lance Decimal256 values and have no
such masks.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `LanceTableReaderTypeTest.NarrowsDecimal256PayloadsHiddenByNullParents`
      (FIXED_SIZE_LIST, LIST, STRUCT, MAP and LIST<STRUCT> over a hidden
      10^18, and a sliced LIST); `RejectsDecimal256ValuesOutsideDeclaredPrecision`
      gains a nested visible out-of-range value; `ArrowParentNullsTest`
      gains 8 cases (each parent type, a sliced parent, no allocation when
      every slot is visible, offsets outside the child, the mask allocated
      from the Doris Arrow pool, allocation failure).
    - `ArrowParentNullsTest.*`, `DataTypeSerDeArrowTest.*`,
      `DataTypeSerDeArrowValidationTest.*`, `ArrowColumnToDorisColumnTest.*`,
      the Lance and ADBC suites and `IcebergV2ReaderTest.*` on macOS arm64
      (ASAN): 306 pass, 7 ADBC SQLite cases skip (no SQLite driver here).
      Mutations: without the normalizer mask the Lance test fails; with the
      mask allocated from the default pool the two pool tests fail; without
      masking the slots outside a sliced parent the slice test and the Lance
      test fail.
    - clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces`
      on the three `DORIS_CHECK` lines of the new function, the macro's
      expansion that existing `DORIS_CHECK`s report too.
- Behavior changed: No (visible values are unchanged; the narrowing that
  hidden values tripped is added by this PR)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

A Lance nested array whose descendants start at an offset is copied into an
offset-zero array before decoding, and the copy runs `ValidateFull()` first,
for the offset and length checks the builder needs. `ValidateFull()` also
checks every non-NULL value of every descendant over its whole length:
UTF-8 for strings, the precision of decimals, whole days for Date64 and the
day range for Time32 / Time64, also in slots that no visible row reaches.
The slice [1, 4) of a FIXED_SIZE_LIST<STRING, 1> with rows ["z"], ["a"],
NULL, ["c"] may hold the child strings ["z", "a", "\xff", "c"]; its rows are
["a"], NULL, ["c"], yet the hidden "\xff" failed the batch with "Invalid
UTF8 sequence". The previous fixes mask such slots only for the element
SerDes and for the Decimal256 leaves the normalizer narrows, so a hidden
value in any other subtree of a compacted array still failed: a string, a
decimal256 of precision above 38, a Date64 or a time.

Fix: `compact_lance_array()` masks the array before validating it:
`mask_arrow_descendants_not_visible()` marks NULL, at every nesting level,
every slot that no visible row reaches, through the per-level
`mask_arrow_child_slots_not_visible()`. `ValidateFull()` and the copy then
run on the masked array, so the offset and length checks stay and an
invalid visible value still fails. The builders copy no child slot of a NULL
list, map or fixed-size-list row, and the masked fields of a NULL struct
row are read as NULL anyway, so every visible value is unchanged. Map keys
cannot be NULL and stay unmasked. A slot that is NULL already needs no
hiding, so the subtrees the normalizer masked before narrowing get no
second bitmap, and an array with nothing hidden allocates nothing.

Before: the slice above failed to read. After: it reads as ["a"], NULL,
["c"].

branch-4.1 and branch-4.2 do not compact Lance arrays this way.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `LanceTableReaderTypeTest.CompactsArraysWithInvalidPayloadsHiddenByNullParents`
      (FIXED_SIZE_LIST<STRING> with invalid UTF-8, LIST<DECIMAL256(40,2)>
      out of precision and LIST<STRUCT<s, d>> under NULL rows and outside
      the slice), `RejectsVisibleInvalidPayloadsInCompactedArrays` (the same
      values visible), `ArrowParentNullsTest.MasksHiddenSlotsAtEveryNestingLevel`
      and `LeavesVisibleAndAlreadyMaskedArraysUnchanged`.
    - `ArrowParentNullsTest.*`, `DataTypeSerDeArrowTest.*`,
      `DataTypeSerDeArrowValidationTest.*`, `ArrowColumnToDorisColumnTest.*`,
      the Lance and ADBC suites and `IcebergV2ReaderTest.*` on macOS arm64
      (ASAN): 310 pass, 7 ADBC SQLite cases skip. Mutations: without the
      mask before validation the compaction test fails; without recursion
      the nesting test and the compaction test fail; masking already-NULL
      slots again fails the no-second-bitmap test.
    - clang-tidy (llvm 22) reports only `readability-inconsistent-ifelse-braces`
      on the three `DORIS_CHECK` lines of the new function.
- Behavior changed: No (visible values are unchanged)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary: cloud_p0 (UBSAN) reported "load of misaligned address
... requires 4 byte alignment" in arrow/array/array_nested.h, from the
entry_slots lambda of DataTypeMapSerDe::read_column_from_arrow through
mask_arrow_children_hidden_by_null_parents (arrow_parent_nulls.cpp): a
Python UDF returned a MAP with a NULL row over an offsets buffer that is
not 4-byte aligned. The serde reads its own offsets with unaligned_load
for that reason, but the mask added in this PR called
MapArray::value_offset(), an aligned int32_t load.
mask_arrow_child_slots_not_visible did the same for LIST, LARGE_LIST and
MAP parents. All of them now read the offsets with unaligned_load, as the
list and map serdes do.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - BE UT ArrowParentNullsTest.MasksNullRowsBehindMisalignedOffsets
      masks NULL LIST / LARGE_LIST / MAP rows whose offsets buffer starts
      at an odd address; DataTypeMapSerDeTest.ArrowMemNotAligned already
      reads such a MAP with a NULL row. Both flag an aligned load only in
      a UBSAN build; locally (ASAN, macOS arm64) the 152 tests of
      ArrowParentNullsTest, DataTypeMapSerDeTest, DataTypeSerDeArrowTest
      and the Lance suites pass.
- Behavior changed: No
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>

…n SQL

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

Reproduction: scan a Lance INT64 column under a runtime filter that mentions
INT64_MIN, for example an ordinary BIGINT join whose build side contains
-9223372036854775808. The IN filter `x IN (-9223372036854775808)` or the
range filter `x <= -9223372036854775808` reaches the Lance scan, and the
query fails reading its first batch instead of returning the matching rows.
A scale-zero DECIMAL128 / DECIMAL256 column fails the same way.

Root cause: `integer_literal_fits_physical_type()` admits every i64 for an
Int64 column, and the scale-zero decimal check admits every value that parses
as an i64, so `to_lance_sql_literal()` emits INT64_MIN. The pinned Lance SQL
parser (lance-datafusion `planner.rs` at 68c12dfd) reads `-N` as the negation
of N and parses N as i64 first. The magnitude 9223372036854775808 does not fit,
so the literal becomes a Float64, and the scalar coercion (`expr.rs`) cannot
turn a Float64 into Int64 or Decimal: the first `lance_scanner_next()` fails
planning. The only existing reader case with INT64_MIN is null-aware, so the
filter was skipped before pushdown and the gap went unnoticed.

Fix: `parse_lance_int64_literal()` parses an integer literal as Lance does and
rejects INT64_MIN. The integer check and the scale-zero decimal check use it,
so a filter that mentions INT64_MIN stays residual-only and is reported under
`LanceRuntimeFilterSkippedIds`; an IN set with that member is rejected as a
unit, as for any member Lance cannot take. Every other i64, INT64_MIN + 1
included, is still pushed.

Before: such a scan failed in Lance planning. After: the filter stays with the
Doris residual and the scan returns the matching rows.

branch-4.1 and branch-4.2 carry the identical helper, so the same change
applies there.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `LanceRuntimeFilterHelperTest.KeepsInt64MinFiltersResidualOnly`
      (IN with an INT64_MIN member, `>=`, `<=` and a scale-zero decimal IN on
      INT64_MIN are skipped; the same filters on INT64_MIN + 1 are pushed with
      the expected SQL),
      `LanceTableReaderFilterTest.KeepsInt64MinRuntimeFiltersOutOfLanceScanner`
      (a Lance dataset holding INT64_MIN read under an INT64 IN, an INT64 `<=`
      and a DECIMAL(38, 0) IN on INT64_MIN; the scan succeeds and reports all
      three as skipped).
    - `run-be-ut.sh --run --filter='Lance*'` on macOS arm64 (ASAN): 106
      tests pass. Restoring `parse_int64()` in either the integer check or
      the scale-zero decimal check makes exactly the two new tests fail,
      the reader case with Lance's "Received literal Float64(...) and
      could not convert" planning error.
- Behavior changed: No (the Lance reader is added by this PR; runtime filters
  that mention INT64_MIN are no longer pushed into Lance, so such scans may
  read more rows)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

Reproduction: set `lance_index_cache_size_bytes = -1` (or
`lance_metadata_cache_size_bytes`) in be.conf and query a Lance table. The
BE starts, and the first Lance scan creates the shared session with an index
(or metadata) cache of 2^64 - 1 bytes, that is with no limit, without any
error. With the data cache enabled, a negative
`lance_data_cache_disk_capacity_bytes` that happens to be a multiple of 4096
(for example -4096) is accepted the same way as a disk tier of almost 2^64
bytes, and a negative `lance_data_cache_read_block_size_bytes` fails with
lance-c's complaint about the wrapped value instead of the setting.

Root cause: the four settings are `int64_t` BE configs without a range check,
and `LanceSessionManager::_initialize()` passes each of them to lance-c with
`static_cast<uint64_t>`.

Fix: `_initialize()` rejects a negative index or metadata cache size, and,
while the data cache is enabled, a negative disk capacity or read block size,
before any cast, with an InvalidArgument status that names the settings and
tells the operator to fix be.conf (or disable the data cache) and restart.
`open_dataset()` retains this status through `call_once` like every other
session initialization failure, so only Lance scans fail and the rest of the
BE keeps serving. The data cache sizes are still ignored while the data cache
is disabled, as `SessionAndDataCacheConfigurationsAreIndependent` requires.
The remaining constraints (a non-zero multiple of 4096, room for the memory
tier block and two disk blocks) stay with lance-c, which reports them on the
same path.

Before: a negative cache size silently meant an unlimited cache, or a
lance-c error about a huge value. After: the first Lance scan fails with
"... must not be negative. Fix be.conf and restart this BE ...".

branch-4.1 and branch-4.2 pass the same settings through the same casts.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `LanceSessionManagerTest.RejectsNegativeCacheSizesBeforeCreatingTheSession`
      (a negative index cache size and a negative metadata cache size with the
      data cache disabled; a negative disk capacity and a negative read block
      size with the data cache enabled; each fails with INVALID_ARGUMENT
      "must not be negative" and opens no dataset).
    - `run-be-ut.sh --run --filter='Lance*'` on macOS arm64 (ASAN): 107 tests
      pass. Disabling both checks makes the new test, and only it, fail: the
      negative index and metadata sizes open the dataset, and with the data
      cache enabled lance-c reports the wrapped value instead
      (`disk_capacity_bytes=18446744073709551615 must be a multiple of 4096 ...`).
    - clang-format 16, clang-tidy on the changed lines and
      `check-build-hygiene.sh` pass.
- Behavior changed: No (the Lance reader is added by this PR; a negative
  Lance cache size now fails Lance scans instead of meaning no limit)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>

…output block

### What problem does this PR solve?

Issue Number: None

Related PR: apache#69581

Problem Summary:

Reproduction: load one large Arrow IPC batch through Arrow stream load with
a nested column that has NULL rows. For example, take 2^24 LIST<INT> rows
that alternate between [1] and NULL (2^23 child values), read in blocks of
4096 rows. Every block allocates a 1 MiB validity bitmap, zero-fills it and
copies the child's validity into it, so the batch allocates 4 GiB and writes
8 GiB from the query's Arrow pool. A STRUCT field costs N^2/B bits per batch
the same way. So does a NULL list row whose producer left values in its
span.

Root cause: `mask_arrow_children_hidden_by_null_parents()` was added earlier
in this PR so that element SerDes never decode child values hidden by a NULL
list, fixed-size-list, map or struct row. At the first NULL parent row of the
requested range, it allocated a bitmap covering the whole child and copied
the child's validity, before it looked at that row's child span. A NULL row
that owns no slots, as Arrow's list and map builders write it, or only NULL
slots, hides nothing but paid the same. `ArrowStreamReader` keeps each IPC
batch and hands the SerDes one row range per output block, and the mask is
thrown away after each call, so the bitmap was rebuilt for every block. The
map SerDe builds one for the keys and one for the items, and the struct
SerDe one per field. Since the bitmaps come from ExecEnv's accounted Arrow
pool, these temporary allocations also count against the query's memory
limit.

Fix:

1. `mask_arrow_children_hidden_by_null_parents()` reads a NULL row's child
   span first and skips the row when the span is empty or has no set
   validity bit in [offset + begin, offset + end). A child without a validity
   buffer counts as all valid. The bitmap is allocated at the first row that
   really hides a valid value. The check, `has_valid_slot()`, is now shared
   with `mask_unreferenced_slots()`, the slot mask of the Lance path. A NULL
   row that hides a valid value is still masked, and its bitmap still comes
   from ExecEnv's Arrow pool. Offsets are still read through the SerDes'
   unaligned loads, still after their offset validation, and an Arrow Null
   child is still returned as it is.
2. `ArrowStreamReader::_prepare_arrays()` masks each column of a retained
   batch once, after the existing normalization and cast. It uses
   `mask_arrow_descendants_not_visible()` with the reader's accounted pool.
   The per-block SerDe calls then find nothing left to hide and allocate
   nothing. If this mask fails, the error names the column the way the
   per-block errors do. It rejects only offsets that reach outside the child
   or run backwards at a NULL row. `enable_arrow_input_validation` (on by
   default) rejected those batches as well, in the block that read them.

What remains: a MapArray cannot hold NULL keys, so the per-batch mask leaves
the keys alone. A NULL map row that still owns entries therefore costs each
block a key bitmap. Arrow's map builder writes NULL map rows without
entries. The other Arrow readers (ADBC, remote Doris, Lance, the Arrow block
convertor, Python UDTF) read each batch with one SerDe call, so they pay for
the mask at most once per batch. paimon-rust reads row ranges of batches
about the size of a block.

Before: every block of a large nested batch with a NULL row allocated and
filled a bitmap the size of the whole child. After: the blocks allocate
nothing from the SerDes' pool, and each batch is masked at most once per
nested level. In the example above nothing is allocated at all, because
the NULL rows own no values.

Also covered: the NULL-parent case of the DATETIMEV2 / TIMESTAMPTZ range
checks from apache#69697. That PR merged before this one and its tests use visible
values only.

master, branch-4.1 and branch-4.2 have no parent-null masks (this PR adds
them), so nothing needs porting.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - New: `ArrowParentNullsTest.AllocatesNothingForNullRowsThatHideNothing`
      covers LIST rows that alternate [1] and NULL, read two rows per call; a
      NULL MAP row without entries (keys and items); and LIST and
      FIXED_SIZE_LIST NULL rows over NULL slots. Each returns the child
      itself and allocates nothing from the Doris Arrow pool. A NULL row over
      valid values next to them is still masked, with one bitmap.
    - New: `LoadArrowPipeTest.Null{List,FixedSizeList,Map,Struct}BatchReadsInBlocksWithoutPerBlockMasks`
      read one IPC batch of 64 rows four rows per block through
      `ArrowStreamReader`, with every fourth row NULL. No block allocates
      from the pool the SerDes mask with, and every row reads as one SerDe
      call over the whole batch reads it.
    - New: `DataTypeDateTimeV2SerDeTest.IgnoresArrowTimestampsHiddenByNullParents`
      hides 65537-01-01 under a NULL LIST, FIXED_SIZE_LIST, MAP value and
      STRUCT row, as DATETIMEV2 and as TIMESTAMPTZ. It reads as a NULL
      element, while the same value in a visible row fails with the range
      error.
    - Done: `run-be-ut.sh --run --filter=...` (ASAN, macOS arm64) over 24 suites:
      `ArrowParentNullsTest`, `LoadArrowPipeTest`, the DATETIMEV2 / TIMESTAMPTZ,
      Arrow, Arrow validation, array, map and struct SerDe suites,
      `ArrowColumnToDorisColumnTest`, `ArrowBlockConvertorTest`, `AdbcReaderTest`,
      `RemoteDorisV2ReaderTest`, `PaimonRustTableReaderTest` and `Lance*`: 328 of
      328 pass. clang-format 16 and `check-build-hygiene.sh` pass.
    - Mutations: allocating at the first NULL row of each call again (B1 off)
      fails the new `ArrowParentNullsTest` case and the four new
      `LoadArrowPipeTest` cases; dropping the per-batch mask's result (B2 off)
      fails the LIST and STRUCT `LoadArrowPipeTest` cases (the reader casts a
      FIXED_SIZE_LIST to a compacted LIST, and a NULL map row owns no entries,
      so B1 covers those two); returning the child unmasked fails the new
      timestamp case, four `ArrowParentNullsTest` cases and
      `LanceTableReaderTypeTest.IgnoresDate32PayloadHiddenByNullParents`.
- Behavior changed: No (the values read are unchanged; only the
  allocations change)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 <[email protected]>