Skip to content

branch-4.2: [Opt](variant) Stream the VARIANT root column and bound segment-buffering writers - #69362

Merged
yiguolei merged 9 commits into
apache:branch-4.2from
eldenmoon:branch-pick-69221-to-4.2
Oct 21, 2026
Merged

yiguolei merged 9 commits into
apache:branch-4.2from
eldenmoon:branch-pick-69221-to-4.2

Conversation

@eldenmoon

Copy link
Copy Markdown
Member

cherry-pick #69221

…segment (CIR-22037)

ColumnStr<uint32_t> rejects more than 4 GiB of uncompressed chars (E-3113).
The VARIANT V2 writer kept the whole output segment's root JSONB in one
ColumnString before finalize, so a vertical compaction whose segment carried
more than 4 GiB of uncompressed variant data failed deterministically. The
compressed-byte segment budget cannot prevent this for high-compression-ratio
data.

Stream the root column: create the root column writer at init and append every
input batch to it, so the writer never holds a segment-sized ColumnString.

Keep a data-driven segment budget as a guard for the remaining variant buffers:
sum the raw (pre-encoding) data bytes of the input segments' VARIANT column
metas and clamp the row budget with
variant_compaction_max_segment_uncompressed_bytes (2 GiB by default). Segments
written before variant writers reported data bytes keep the compressed-size
budget.

(cherry picked from commit a400b5c)
…s (CIR-22037)

- Assert the root-only writer keeps its buffer close to one append batch instead of
  retaining the whole segment.
- Assert variant writers report the data page bytes they wrote (writer counters equal
  the bytes appended to the file right after write_data).

(cherry picked from commit 7b0958d)
VariantSubcolumnWriter accumulated every value of an output segment in one
VariantPathBuilder column, and materialized a second segment-sized dense column
in finalize(). A VARIANT-typed extracted subcolumn whose non-null payload
exceeds 4 GiB therefore always failed compaction with E-3113.

Accumulate values into bounded chunks (256 MiB / 4M values) instead. Each chunk
keeps the inferred type; finalize() merges the chunk types with
variant_path_least_common_type(), converts every chunk to the final storage type
and flushes it to the underlying writer independently, so neither the builder
nor the dense materialization ever holds a segment-sized ColumnString.

(cherry picked from commit 92db91d)
…sistent (CIR-22037)

- A new chunk starts from the type inferred by the sealed chunks, so the single
  builder's JSONB fallback still applies across chunk boundaries (a bool after
  integers must not be coerced to 1/0 by the least-common-type merge).
- All-NULL ARRAY chunks are flushed through the array writer, which updates the
  offsets, nulls and inverted index together; append_nulls only wrote the data
  column and left the index inconsistent.
- One convertor per column in finalize: the ARRAY convertor carries the
  accumulated item offset, so chunked flushes keep absolute offsets.
- Tests for int-then-bool across a boundary and an ARRAY all-NULL tail chunk.

(cherry picked from commit 720f7f7)
### What problem does this PR solve?

Related PR: apache#69221

Problem Summary: Chunked extracted VARIANT writes can discard valid Decimal values or reduce decimal-array scale. Add native Decimal writer/readback cases comparing chunked and uninterrupted inference, including a later Double and values retained in the active chunk, plus a direct nullable-array type-merge check.

### Release note

None

### Check List (For Author)

- Test: ASAN BE unit tests: all three added tests fail on the unmodified implementation (red); clang-format 16 and build hygiene pass
- Behavior changed: No (tests only)
- Does this need documentation: No

(cherry picked from commit cf1db49)
### What problem does this PR solve?

Related PR: apache#69221

Problem Summary: Extracted VARIANT chunks inherit a type without validating it against earlier values. Decimal widening can therefore filter valid rows as cast NULLs, and nullable-array type merging can reduce element scale. Resolve inferred promotions against retained chunks before committing any cast, preserving whole-path JSONB fallback and the active chunk's original Decimal representation. Unwrap nullable types before array/Decimal dispatch. Keep explicit typed-storage cast-null filtering unchanged.

### Release note

Preserve VARIANT Decimal values and decimal-array precision during chunked writes.

### Check List (For Author)

- Test: ASAN BE unit tests: three new tests fail before the fix and pass afterward; related writer, path-builder, shredder and compatibility tests: 103 passed, 2 skipped because NestedGroup write support is unavailable. Local UT uses GLIBC_COMPATIBILITY=OFF to avoid duplicate toolchain compatibility symbols. clang-format 16, build hygiene and clang-tidy on changed lines passed.
- Behavior changed: Yes, inferred cross-chunk conversions preserve valid values and Decimal array scale
- Does this need documentation: No

(cherry picked from commit fab1e0c)
### What problem does this PR solve?

Related PR: apache#69221

Problem Summary: Keep the branch-4.2 V1 writer dispatch while backporting V2 root streaming and chunked extracted subcolumns, and format the resolved implementation and test includes with clang-format 16.

### Release note

None

### Check List (For Author)

- Test: clang-format 16 and source whitespace checks passed; ASAN build and corresponding cases are in progress.
- Behavior changed: No, formatting only.
- Does this need documentation: No
@eldenmoon
eldenmoon requested a review from yiguolei as a code owner October 21, 2026 03:06
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

### What problem does this PR solve?

Related PR: apache#69221

Problem Summary: branch-4.2 ExtraInfo requires an explicit path_info initializer under the ASAN unit-test warning policy. Carry the extracted column's schema path into the footer-derived readback column used by the new chunked writer tests.

### Release note

None

### Check List (For Author)

- Test: clang-format 16 and whitespace checks passed; rerunning the affected ASAN writer tests after resolving the compile failure.
- Behavior changed: No, test adaptation only.
- Does this need documentation: No
@eldenmoon

Copy link
Copy Markdown
Member Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 76.96% (157/204) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 76.39% (34424/45062)
Line Coverage 61.19% (388651/635140)
Region Coverage 57.94% (327000/564397)
Branch Coverage 58.98% (150999/256010)

@yiguolei
yiguolei merged commit 807c3a0 into apache:branch-4.2 Oct 21, 2026
27 of 29 checks passed
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Oct 21, 2026
@github-actions

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants