…70001 (#70083)

Cherry-picked from #70001

---------

Co-authored-by: lihangyu <[email protected]>

…#70189)

cherry-pick #70001

Test adaptation: branch-4.1 defaults `enable_variant_v2=false`, so the
regression case now temporarily enables it via `setFeConfigTemporary`
(suite marked `nonConcurrent`) and additionally asserts
`variant_type(v)`; expected outputs are otherwise unchanged.

### Background

Variant V2 persists each row's sparse leaves as a dotted-path map. Readers binary-search that map by raw path
bytes and merge it in path order, so apache#70416 added a byte-ordered fallback for rows written before the
writer-side sort (apache#70001). The fallback validates a batch once but currently drops the binary-search fast path
for every row of the batch, and the hierarchical assembler can still index paths with unchecked map offsets.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70416, apache#70001
PR Type: Bugfix

Problem Summary:

Mixed batches of conforming and legacy rows pay a linear path scan for every row, and corrupt sparse offsets can
reach path reads instead of failing with a corruption status. This commit records non-conforming rows per batch
and validates offsets in `start_row()`, with unit tests for mixed batches, later runs, missing paths, and bad
offsets. Users see no behavior change for healthy data.

### Details

- **Frequency:** Mixed batches exist only while legacy rows and newly written rows share a segment; corrupt
  offsets come from a damaged segment.
- **Root cause:** the batch-level sortedness flag was consulted per row, and `start_row()` trusted offsets that
  `bind()` already knew could be invalid.
- **Why existing tests missed it:** the fallback tests covered single-row batches only, and no test built
  non-monotonic offsets for the assembler.
- **Fix:** keep a bitmap of non-conforming rows; return `Status::Corruption` for invalid offsets.
- **Other paths and branches:** none; the V1 reader sorts merged sparse paths itself.

### Testing

- `./run-be-ut.sh --run --filter='VariantAssemblerTest.*:BinaryColumnExtractIteratorV2Test.*'` on dev-x (ASAN):
  21/21 tests passed.
- New cases: mixed sorted/legacy/empty batch, three-run legacy row, subtree split across runs, missing path,
  exact path in a later run, and non-monotonic offsets.
- `build-support/check-build-hygiene.sh` passed.

### Release note

None.

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: DeepSeek Harness
- Generated: code, tests, commit message
- Human review: pending

…tion

### Background

Variant V2 readers binary-search sparse keys by raw path bytes. apache#70416 added a byte-ordered fallback for
segments written before the writer-side sort (apache#70001). Vertical variant compaction does not re-shred the
variant column: it copies each raw sparse bucket map into the new segment through
`SparseColumnMergeIterator`.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70416, apache#70001
PR Type: Bugfix

Problem Summary:

Compacting a table that contains legacy out-of-order sparse rows kept the legacy order in the new segments, so
the read fallback would be required forever and a compacted replica would never converge to canonical order.
The merge now reads a non-conforming source row through a byte-ordered view, which canonicalizes compaction
output and matches the writer invariant. Conforming rows keep the zero-copy merge.

### Details

- **Frequency:** Vertical variant compaction (`enable_vertical_compact_variant_subcolumns`, default on) of
  segments written before apache#70001, or of rows assembled from several sorted runs.
- **Root cause:** `SparseColumnMergeIterator::_merge_to` copied source sparse cells in stored order; it is the
  only compaction path that does not go through `VariantShredder`, whose sparse plan is sorted.
- **Why existing tests missed it:** compaction tests write through the fixed shredder, so their source sparse
  maps are already byte-sorted; no test fed an unsorted raw map into the merge iterator.
- **Fix:** validate each source row and merge a non-conforming row through
  `build_variant_sparse_row_sorted_view`.
- **Other paths and branches:** non-vertical compaction and doc compaction shred through the sorted writer;
  queries keep the apache#70416 fallback for segments that have not been compacted yet.

### Testing

- `./run-be-ut.sh --run --filter='VariantAssemblerTest.*:BinaryColumnExtractIteratorV2Test.*:SparseColumnMergeIteratorTest.*'`
  on dev-x (ASAN): 22/22 passed.
- `./run-be-ut.sh --run --filter='VariantColumnWriterReaderTest.*:VariantShredderTest.*:VariantPathBuilderTest.*:IndexStorageVariantCompactionReadTest.*'`:
  102 tests, 98 passed and 4 pre-existing skips.
- `SparseColumnMergeIteratorTest.SortsLegacySparseRowsWhenCompacting` feeds a sorted row, a three-run legacy
  row, and an empty row through the merge; the legacy output is byte-sorted and keeps every cell.
- `build-support/check-build-hygiene.sh` passed.

### Release note

Fix vertical compaction of legacy VARIANT rows so compacted segments store sparse keys in byte order.

### Check List (For Author)

- Behavior changed: Yes (see Details) — compacted legacy rows get canonical key order.
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: DeepSeek Harness
- Generated: code, tests, commit message
- Human review: pending

…tion

### Background

Variant V2 persists each row's sparse leaves as a dotted-path map and every reader
binary-searches those keys by raw path bytes. The writer sorts the sparse plan (apache#70001), but
vertical compaction does not re-shred the variant column: it copies the raw sparse bucket maps
into the new segment through `SparseColumnMergeIterator`.

### What problem does this PR solve?

Issue Number: None

Problem Summary:

Compacting a table that still contains rows persisted before apache#70001 kept the legacy order, so the
compacted replica never converged to the byte-order invariant that readers assume. This change
makes compaction canonicalize such rows and adds debug-only assertions so a producer that violates
the invariant is caught by ASAN/UT builds without a release cost.

### Details

- **Frequency:** Vertical variant compaction (`enable_vertical_compact_variant_subcolumns`,
  default on) of segments written before apache#70001, or of rows assembled from several sorted runs.
- **Root cause:** `SparseColumnMergeIterator::_merge_to` copied source sparse cells in stored
  order; it is the only compaction path that does not go through `VariantShredder`, whose sparse
  plan is sorted.
- **Fix:** merge a non-conforming source row through `build_variant_sparse_row_sorted_view`, so
  compaction output is byte-ordered. Conforming rows keep the zero-copy merge.
- **Debug checks:** `DCHECK(is_variant_sparse_row_sorted(...))` in the V2 row cursor and in the
  exact sparse-path extractor. A producer regression fails the ASAN/UT build; release builds pay
  nothing for the checks.
- **Other paths:** non-vertical and doc compaction shred through the sorted writer.
- **Upgrade behavior:** query-time tolerance is intentionally not added. A legacy row keeps
  failing with `Duplicate Variant object key` until vertical compaction rewrites its segment.

### Testing

- `./run-be-ut.sh --run --filter='SparseColumnMergeIteratorTest.*:VariantAssemblerTest.*:BinaryColumnExtractIteratorV2Test.*'`
  on dev-x (ASAN): passed.
- The new case feeds a sorted row, a three-run legacy row, an empty row, a tail inversion, and a
  byte-first-stored-last row through the merge and asserts the output paths are byte-ordered.
- `build-support/check-build-hygiene.sh` passed.

### Release note

Fix vertical compaction of legacy VARIANT rows so compacted segments store sparse keys in byte
order. Un-compacted legacy segments keep reporting `Duplicate Variant object key` until compaction
rewrites them.

### Check List (For Author)

- Behavior changed: Yes (see Details) — compacted legacy rows get canonical key order.
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: DeepSeek Harness
- Generated: code, tests, commit message
- Human review: pending

### Background

Variant V2 persists each row's sparse leaves as a dotted-path map, and every reader merges or
binary-searches those keys by raw path bytes. The writer sorts the sparse plan (apache#70001) and vertical
compaction copies the cells verbatim, so byte order is a producer obligation. Nothing checked it, so
a producer regression would surface much later as a query failure (`Duplicate Variant object key`).

### What problem does this PR solve?

Issue Number: None

Problem Summary:

Without a check, a writer that stops sorting produces segments whose reads fail, far from the cause.
This change asserts the invariant where it is relied on and keeps the assertion out of release
builds.

### Details

- **Debug-only checks:** `DCHECK(is_variant_sparse_row_sorted(...))` in the V2 row cursor
  (`StorageMapRowCursor::bind`), in the exact sparse-path extractor, and in the compaction merge
  loop. They compile out in release, so query and compaction performance are unchanged: no extra
  scan and no allocation.
- **Why not repair:** query-time and compaction-time tolerance is deliberately not added on this
  branch. Segments must be byte-ordered; a violation is a producer bug.
- **Test:** `VariantAssemblerTest.DebugCheckRejectsUnsortedSparseRows` (death test) feeds a
  two-cell unsorted row and expects the debug check to fire.

### Testing

- `./run-be-ut.sh --run --filter='VariantAssemblerTest.*:BinaryColumnExtractIteratorV2Test.*:VariantShredderTest.*'`
  on dev-x (ASAN): 17 tests passed.
- Negative check: temporarily removed the writer-side `std::ranges::sort` from `VariantShredder`
  (apache#70001). `VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection` then
  failed its byte-order expectation and the new check aborted in `StorageMapRowCursor::bind`
  (`Check failed: is_variant_sparse_row_sorted`), so the assertion does catch a producer that stops
  sorting. The sort was restored afterwards.
- `build-support/check-build-hygiene.sh` passed.

### Release note

None.

### Check List (For Author)

- Behavior changed: No (assertions only, active in debug builds)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: DeepSeek Harness
- Generated: code, tests, commit message
- Human review: pending

### Background

Variant V2 persists each row's sparse leaves as a dotted-path map, and every reader merges or
binary-searches those keys by raw path bytes. The writer sorts the sparse plan (apache#70001), and
vertical compaction merges the source sparse row with the materialized subcolumns, so byte order is
a producer obligation all the way to the compacted segment. Nothing checked it, so a regression
would only surface much later as a query failure (`Duplicate Variant object key`).

### What problem does this PR solve?

Issue Number: None

Problem Summary:

Without a check, a writer or compaction merge that breaks the order produces segments whose reads
fail, far from the cause. This change asserts the invariant where it is relied on and keeps the
assertion out of release builds.

### Details

- **Debug-only checks:** `DCHECK(is_variant_sparse_row_sorted(...))` in
  - `StorageMapRowCursor::bind()` (V2 row cursor used by every hierarchical read),
  - the exact sparse-path extractor,
  - `SparseColumnMergeIterator::_merge_to` for both the source row and the merged destination row,
    because the merge interleaves sparse cells with materialized subcolumn paths.
- **Why not repair:** query-time and compaction-time tolerance is deliberately not added on this
  branch. Segments must be byte-ordered; a violation is a producer bug.
- **Performance:** every check is a `DCHECK`, compiled out in release; queries and compaction keep
  the same scan, sort, and allocation behavior as before.
- **Test:** `VariantAssemblerTest.DebugCheckRejectsUnsortedSparseRows` (death test) feeds a
  two-cell unsorted row and expects the debug check to fire.

### Testing

- `./run-be-ut.sh --run --filter='VariantAssemblerTest.*:BinaryColumnExtractIteratorV2Test.*:VariantShredderTest.*'`
  on dev-x (ASAN): 17 tests passed.
- `./run-be-ut.sh --run --filter='VariantColumnWriterReaderTest.*:IndexStorageVariantCompactionReadTest.*:SparseColumnMergeIteratorTest.*'`
  on dev-x (ASAN): 75 passed, 4 pre-existing skips.
- Negative check: temporarily removed the writer-side `std::ranges::sort` from `VariantShredder`
  (apache#70001). `VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection` failed
  its byte-order expectation and the new check aborted in `StorageMapRowCursor::bind`
  (`Check failed: is_variant_sparse_row_sorted`), so the assertion does catch a producer that stops
  sorting. The sort was restored afterwards.
- `build-support/check-build-hygiene.sh` passed.

### Release note

None.

### Check List (For Author)

- Behavior changed: No (assertions only, active in debug builds)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: DeepSeek Harness
- Generated: code, tests, commit message
- Human review: pending

### Background

Variant V2 persists each row's sparse leaves as a dotted-path map, and every reader merges or
binary-searches those keys by raw path bytes. The writer sorts the sparse plan (apache#70001), and
vertical compaction merges the source sparse row with the materialized subcolumns, so byte order is
a producer obligation all the way to the compacted segment. Nothing checked it, so a regression
would only surface much later as a query failure (`Duplicate Variant object key`).

### What problem does this PR solve?

Issue Number: None

Problem Summary:

Without a check, a writer or compaction merge that breaks the order produces segments whose reads
fail, far from the cause. This change asserts the invariant where it is relied on and keeps the
assertion out of release builds.

### Details

- **Debug-only checks:** `DCHECK(is_variant_sparse_row_sorted(...))` in
  - `StorageMapRowCursor::bind()` (V2 row cursor used by every hierarchical read),
  - the exact sparse-path extractor,
  - `SparseColumnMergeIterator::_merge_to` for both the source row and the merged destination row,
    because the merge interleaves sparse cells with materialized subcolumn paths.
- **Why not repair:** query-time and compaction-time tolerance is deliberately not added on this
  branch. Segments must be byte-ordered; a violation is a producer bug.
- **Performance:** every check is a `DCHECK`, compiled out in release; queries and compaction keep
  the same scan, sort, and allocation behavior as before.
- **Tests:** `VariantAssemblerTest.DebugCheckRejectsUnsortedSparseRows` (death test) feeds a
  two-cell unsorted row and expects the debug check to fire;
  `SparseColumnMergeIteratorTest.KeepsByteOrderWhenMergingMaterializedSubcolumn` merges a
  byte-sorted source row with a materialized `a.d` column and asserts the merged row is
  `a-b, a.c, a.d`. The import side is already covered by
  `VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection`.

### Testing

- `./run-be-ut.sh --run --filter='VariantAssemblerTest.*:BinaryColumnExtractIteratorV2Test.*:VariantShredderTest.*'`
  on dev-x (ASAN): 17 tests passed.
- `./run-be-ut.sh --run --filter='VariantColumnWriterReaderTest.*:IndexStorageVariantCompactionReadTest.*:SparseColumnMergeIteratorTest.*'`
  on dev-x (ASAN): 75 passed, 4 pre-existing skips.
- `./run-be-ut.sh --run --filter='SparseColumnMergeIteratorTest.*:VariantAssemblerTest.*'`
  on dev-x (ASAN): 2 passed (merge interleaving + death test).
- Negative check: temporarily removed the writer-side `std::ranges::sort` from `VariantShredder`
  (apache#70001). `VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection` failed
  its byte-order expectation and the new check aborted in `StorageMapRowCursor::bind`
  (`Check failed: is_variant_sparse_row_sorted`), so the assertion does catch a producer that stops
  sorting. The sort was restored afterwards.
- `build-support/check-build-hygiene.sh` passed.

### Release note

None.

### Check List (For Author)

- Behavior changed: No (assertions only, active in debug builds)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: DeepSeek Harness
- Generated: code, tests, commit message
- Human review: pending

### Background

Variant V2 persists each row's sparse leaves as a dotted-path map, and every reader merges or
binary-searches those keys by raw path bytes. The writer sorts the sparse plan (apache#70001), and
vertical compaction merges the source sparse row with the materialized subcolumns, so byte order is
a producer obligation all the way to the compacted segment. Nothing checked it, so a regression
would only surface much later as a query failure (`Duplicate Variant object key`).

### What problem does this PR solve?

Issue Number: None

Problem Summary:

Without a check, a writer or compaction merge that breaks the order produces segments whose reads
fail, far from the cause. This change asserts the invariant where it is relied on and keeps the
assertion out of release builds.

### Details

- **Debug-only checks:** `DCHECK(is_variant_sparse_row_sorted(...))` in
  - `StorageMapRowCursor::bind()` (V2 row cursor used by every hierarchical read),
  - the exact sparse-path extractor,
  - `SparseColumnMergeIterator::_merge_to` for both the source row and the merged destination row,
    because the merge interleaves sparse cells with materialized subcolumn paths.
- **Why not repair:** query-time and compaction-time tolerance is deliberately not added on this
  branch. Segments must be byte-ordered; a violation is a producer bug.
- **Performance:** every check is a `DCHECK`, compiled out in release; queries and compaction keep
  the same scan, sort, and allocation behavior as before.
- **Tests:** `VariantAssemblerTest.DebugCheckRejectsUnsortedSparseRows` (death test) feeds a
  two-cell unsorted row and expects the debug check to fire;
  `SparseColumnMergeIteratorTest.KeepsByteOrderWhenMergingMaterializedSubcolumn` merges a
  byte-sorted source row with a materialized `a.d` column and asserts the merged row is
  `a-b, a.c, a.d`. The import side is already covered by
  `VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection`.

### Testing

- `./run-be-ut.sh --run --filter='VariantAssemblerTest.*:BinaryColumnExtractIteratorV2Test.*:VariantShredderTest.*'`
  on dev-x (ASAN): 17 tests passed.
- `./run-be-ut.sh --run --filter='VariantColumnWriterReaderTest.*:IndexStorageVariantCompactionReadTest.*:SparseColumnMergeIteratorTest.*'`
  on dev-x (ASAN): 75 passed, 4 pre-existing skips.
- `./run-be-ut.sh --run --filter='SparseColumnMergeIteratorTest.*:VariantAssemblerTest.*'`
  on dev-x (ASAN): 2 passed (merge interleaving + death test).
- Negative check: temporarily removed the writer-side `std::ranges::sort` from `VariantShredder`
  (apache#70001). `VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection` failed
  its byte-order expectation and the new check aborted in `StorageMapRowCursor::bind`
  (`Check failed: is_variant_sparse_row_sorted`), so the assertion does catch a producer that stops
  sorting. The sort was restored afterwards.
- `build-support/check-build-hygiene.sh` passed.

### Release note

None.

### Check List (For Author)

- Behavior changed: No (assertions only, active in debug builds)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: DeepSeek Harness
- Generated: code, tests, commit message
- Human review: pending

…ds (#70416)

### Background

Variant V2 persists each row's sparse leaves as a dotted-path map, and
every reader merges or
binary-searches those keys by raw path bytes. The writer sorts the
sparse plan (#70001), and
vertical compaction merges the source sparse row with the materialized
subcolumns, so byte order is
a producer obligation all the way to the compacted segment. Nothing
checked it, so a regression
would only surface much later as a query failure (`Duplicate Variant
object key`).

### What problem does this PR solve?

Issue Number: None

Problem Summary:

Without a check, a writer or compaction merge that breaks the order
produces segments whose reads
fail, far from the cause. This change asserts the invariant where it is
relied on and keeps the
assertion out of release builds.

### Details

- **Debug-only checks:** `DCHECK(is_variant_sparse_row_sorted(...))` in
- `StorageMapRowCursor::bind()` (V2 row cursor used by every
hierarchical read),
  - the exact sparse-path extractor,
- `SparseColumnMergeIterator::_merge_to` for both the source row and the
merged destination row,
because the merge interleaves sparse cells with materialized subcolumn
paths.
- **Why not repair:** query-time and compaction-time tolerance is
deliberately not added on this
  branch. Segments must be byte-ordered; a violation is a producer bug.
- **Performance:** every check is a `DCHECK`, compiled out in release;
queries and compaction keep
  the same scan, sort, and allocation behavior as before.
- **Tests:** `VariantAssemblerTest.DebugCheckRejectsUnsortedSparseRows`
(death test) feeds a
  two-cell unsorted row and expects the debug check to fire;

`SparseColumnMergeIteratorTest.KeepsByteOrderWhenMergingMaterializedSubcolumn`
merges a
byte-sorted source row with a materialized `a.d` column and asserts the
merged row is
  `a-b, a.c, a.d`. The import side is already covered by

`VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection`.

### Testing

- `./run-be-ut.sh --run
--filter='VariantAssemblerTest.*:BinaryColumnExtractIteratorV2Test.*:VariantShredderTest.*'`
  on dev-x (ASAN): 17 tests passed.
- `./run-be-ut.sh --run
--filter='VariantColumnWriterReaderTest.*:IndexStorageVariantCompactionReadTest.*:SparseColumnMergeIteratorTest.*'`
  on dev-x (ASAN): 75 passed, 4 pre-existing skips.
- `./run-be-ut.sh --run
--filter='SparseColumnMergeIteratorTest.*:VariantAssemblerTest.*'`
  on dev-x (ASAN): 2 passed (merge interleaving + death test).
- Negative check: temporarily removed the writer-side
`std::ranges::sort` from `VariantShredder`
(#70001).
`VariantShredderTest.SparseRowsUseByteOrderWithoutChangingStatisticsSelection`
failed
its byte-order expectation and the new check aborted in
`StorageMapRowCursor::bind`
(`Check failed: is_variant_sparse_row_sorted`), so the assertion does
catch a producer that stops
  sorting. The sort was restored afterwards.
- `build-support/check-build-hygiene.sh` passed.

### Release note

None.

### Check List (For Author)

- Behavior changed: No (assertions only, active in debug builds)
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: DeepSeek Harness
- Generated: code, tests, commit message
- Human review: pending