Repository navigation
branch-4.2: [Opt](variant) Stream the VARIANT root column and bound segment-buffering writers - #69362
Merged
Merged
Conversation
…segment (CIR-22037) ColumnStr<uint32_t> rejects more than 4 GiB of uncompressed chars (E-3113). The VARIANT V2 writer kept the whole output segment's root JSONB in one ColumnString before finalize, so a vertical compaction whose segment carried more than 4 GiB of uncompressed variant data failed deterministically. The compressed-byte segment budget cannot prevent this for high-compression-ratio data. Stream the root column: create the root column writer at init and append every input batch to it, so the writer never holds a segment-sized ColumnString. Keep a data-driven segment budget as a guard for the remaining variant buffers: sum the raw (pre-encoding) data bytes of the input segments' VARIANT column metas and clamp the row budget with variant_compaction_max_segment_uncompressed_bytes (2 GiB by default). Segments written before variant writers reported data bytes keep the compressed-size budget. (cherry picked from commit a400b5c)
…s (CIR-22037) - Assert the root-only writer keeps its buffer close to one append batch instead of retaining the whole segment. - Assert variant writers report the data page bytes they wrote (writer counters equal the bytes appended to the file right after write_data). (cherry picked from commit 7b0958d)
VariantSubcolumnWriter accumulated every value of an output segment in one VariantPathBuilder column, and materialized a second segment-sized dense column in finalize(). A VARIANT-typed extracted subcolumn whose non-null payload exceeds 4 GiB therefore always failed compaction with E-3113. Accumulate values into bounded chunks (256 MiB / 4M values) instead. Each chunk keeps the inferred type; finalize() merges the chunk types with variant_path_least_common_type(), converts every chunk to the final storage type and flushes it to the underlying writer independently, so neither the builder nor the dense materialization ever holds a segment-sized ColumnString. (cherry picked from commit 92db91d)
…22037) (cherry picked from commit 5314b85)
…sistent (CIR-22037) - A new chunk starts from the type inferred by the sealed chunks, so the single builder's JSONB fallback still applies across chunk boundaries (a bool after integers must not be coerced to 1/0 by the least-common-type merge). - All-NULL ARRAY chunks are flushed through the array writer, which updates the offsets, nulls and inverted index together; append_nulls only wrote the data column and left the index inconsistent. - One convertor per column in finalize: the ARRAY convertor carries the accumulated item offset, so chunked flushes keep absolute offsets. - Tests for int-then-bool across a boundary and an ARRAY all-NULL tail chunk. (cherry picked from commit 720f7f7)
### What problem does this PR solve? Related PR: apache#69221 Problem Summary: Chunked extracted VARIANT writes can discard valid Decimal values or reduce decimal-array scale. Add native Decimal writer/readback cases comparing chunked and uninterrupted inference, including a later Double and values retained in the active chunk, plus a direct nullable-array type-merge check. ### Release note None ### Check List (For Author) - Test: ASAN BE unit tests: all three added tests fail on the unmodified implementation (red); clang-format 16 and build hygiene pass - Behavior changed: No (tests only) - Does this need documentation: No (cherry picked from commit cf1db49)
### What problem does this PR solve? Related PR: apache#69221 Problem Summary: Extracted VARIANT chunks inherit a type without validating it against earlier values. Decimal widening can therefore filter valid rows as cast NULLs, and nullable-array type merging can reduce element scale. Resolve inferred promotions against retained chunks before committing any cast, preserving whole-path JSONB fallback and the active chunk's original Decimal representation. Unwrap nullable types before array/Decimal dispatch. Keep explicit typed-storage cast-null filtering unchanged. ### Release note Preserve VARIANT Decimal values and decimal-array precision during chunked writes. ### Check List (For Author) - Test: ASAN BE unit tests: three new tests fail before the fix and pass afterward; related writer, path-builder, shredder and compatibility tests: 103 passed, 2 skipped because NestedGroup write support is unavailable. Local UT uses GLIBC_COMPATIBILITY=OFF to avoid duplicate toolchain compatibility symbols. clang-format 16, build hygiene and clang-tidy on changed lines passed. - Behavior changed: Yes, inferred cross-chunk conversions preserve valid values and Decimal array scale - Does this need documentation: No (cherry picked from commit fab1e0c)
### What problem does this PR solve? Related PR: apache#69221 Problem Summary: Keep the branch-4.2 V1 writer dispatch while backporting V2 root streaming and chunked extracted subcolumns, and format the resolved implementation and test includes with clang-format 16. ### Release note None ### Check List (For Author) - Test: clang-format 16 and source whitespace checks passed; ASAN build and corresponding cases are in progress. - Behavior changed: No, formatting only. - Does this need documentation: No
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
### What problem does this PR solve? Related PR: apache#69221 Problem Summary: branch-4.2 ExtraInfo requires an explicit path_info initializer under the ASAN unit-test warning policy. Carry the extracted column's schema path into the footer-derived readback column used by the new chunked writer tests. ### Release note None ### Check List (For Author) - Test: clang-format 16 and whitespace checks passed; rerunning the affected ASAN writer tests after resolving the compile failure. - Behavior changed: No, test adaptation only. - Does this need documentation: No
Member
Author
|
run buildall |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Oct 21, 2026
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
PR approved by at least one committer and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cherry-pick #69221