Repository navigation
[fix](be) Count variant data in information_schema.column_data_sizes - #68444
Merged
eldenmoon merged 10 commits intoOct 19, 2026
Merged
Conversation
### What problem does this PR solve? Issue Number: None Related PR: apache#56815 Problem Summary: The segment writer stores each column's data page bytes (compressed, uncompressed and raw) in its footer meta, which `information_schema.column_data_sizes` reads. `VariantColumnWriter`, `VariantSubcolumnWriter` and `VariantDocCompactWriter` returned 0 (TODO) from `get_raw_data_bytes()`, `get_total_uncompressed_data_pages_bytes()` and `get_total_compressed_data_pages_bytes()`, so every VARIANT column reported 0 bytes although its data is written by inner writers: the root, the materialized subcolumns and the sparse or doc value columns. The variant writers now sum the three counters over their inner writers, the same way the struct, array and map writers already do. `ColumnWriter::DataBytesGetter` names the getter that the variant writer layers pass down. The footer now stores real data bytes on variant column metas instead of 0, so the 29 checked-in golden Segment files of the 14 variant cases in `segment_flusher_format_test` are regenerated (each grows by 2 to 4 bytes; only those three counters of the variant metas differ). The other 59 golden cases are byte-identical. The footer raw_data_bytes of a variant root is also the first-batch hint of the segment iterator's adaptive batch size predictor when a query reads the whole variant column; it was 0 for variants. Vertical compaction's footer-based batch size estimate still skips variant columns; its stale comment is updated. ### Release note None ### Check List (For Author) - Test: Unit Test (segment_flusher_format_test with the regenerated golden files, variant writer, compaction and segment writer tests) - Behavior changed: Yes. Variant column metas store their data page bytes instead of 0. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
### What problem does this PR solve? Issue Number: None Related PR: apache#56815 Problem Summary: Right after `write_data()` a test segment file holds only the data pages of the column under test, so a variant writer's compressed data bytes must equal the file's size. Check it in the shared variant segment write helpers (every ordinary and doc mode layout they write, and the extracted subcolumn writer), in `test_write_data_normal` (materialized subcolumns plus bucketized sparse columns, with uncompressed and raw bytes above 0) and in `doc_compact_writer_round_trip` (root-only writer plus doc compact writer). ### Release note None ### Check List (For Author) - Test: Unit Test - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
### What problem does this PR solve? Issue Number: None Related PR: apache#56815 Problem Summary: Compaction writes variant subcolumns, sparse columns and doc value columns as columns of their own. Their footer metas carry data bytes but no unique id, so `information_schema.column_data_sizes` dropped them, and with external column meta (`storage_format = "V3"`) the sparse and doc value metas are embedded in the root variant meta, which the scanner never looked into. After a full compaction a variant reported only its root. The scanner now counts a meta that has data bytes but no unique id to its parent variant column (from its column path info), adds the metas embedded in a variant root meta, and takes COLUMN_TYPE from the schema column, since a subcolumn meta carries the subcolumn's type. Nothing is counted twice: the inner metas of a variant writer carry no data bytes. ### Release note Fix `information_schema.column_data_sizes` reporting 0 bytes for VARIANT columns. Data written before the fix reports the new sizes once compaction rewrites it. ### Check List (For Author) - Test: Regression test (test_query_sys_column_data_sizes_variant, next commit) - Behavior changed: Yes. `information_schema.column_data_sizes` counts compacted variant subcolumns to their variant column. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
…ata_sizes ### What problem does this PR solve? Issue Number: None Related PR: apache#56815 Problem Summary: Cover ordinary and doc mode variants with inline (V2) and external (V3) column meta, before and after a full compaction. The sizes of `id` and `v` must be above 0, and the data pages of all columns must lie within the rowsets' DATA_DISK_SIZE and cover more than half of it; they cover 91%-96%. Without counting the metas embedded in the variant root the V3 tables drop to 35% and 11% after compaction; without resolving the parent unique id the V2 tables drop to 3% and 2%. The table layouts are set explicitly because CI fuzzes the variant defaults and the storage format. ### Release note None ### Check List (For Author) - Test: Regression test - Behavior changed: No - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
eldenmoon
requested review from
airborne12,
csun5285,
gavinchou,
liaoxin01 and
yiguolei
as code owners
September 23, 2026 10:40
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 27854 ms |
Contributor
TPC-DS: Total hot run time: 152881 ms |
Contributor
ClickBench: Total hot run time: 23.77 s |
### What problem does this PR solve? Issue Number: None Related PR: apache#68444 Problem Summary: A wide projected VARIANT can keep fully filtered scans on tiny batches because empty output supplies no predictor history. Add a focused reproducer checking repeated empty batches, unchanged diagnostic footer bytes, scalar footer hints, and feedback from materialized blocks. The reproducer returns 7 rows instead of the expected 4096 on the original PR head. ### Release note None ### Check List (For Author) - Test: ASAN BE UT: original head has 32 passing tests and the new reproducer fails; all 36 selected tests pass with the follow-up fix. UT uses GLIBC_COMPATIBILITY=OFF to avoid a host toolchain duplicate-symbol conflict. Clang-format 16, header hygiene and clang-tidy passed. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#68444 Problem Summary: Complete VARIANT raw-byte statistics shrink the initial scan batch even when a projected VARIANT is deferred and all rows are filtered by a narrow predicate. Empty output never updates the predictor, so the small batch persists until EOF. Keep the correct footer statistics and exclude VARIANT columns from the initial metadata hint, using the existing bounded probe and subsequent measured Block feedback. The focused reproducer changes from 7 to 4096 predicted rows across repeated empty batches. ### Release note Prevent fully filtered scans with wide projected VARIANT columns from remaining on tiny adaptive batches. ### Check List (For Author) - Test: ASAN BE UT: 36 selected tests passed, including the new red/green reproducer and VARIANT writer tests; GLIBC_COMPATIBILITY=OFF avoids a host duplicate-symbol conflict. Clang-format 16 and clang-tidy passed. - Behavior changed: Yes, VARIANT footer statistics no longer constrain initial scan batches; measured feedback and scalar hints remain active. - Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#68444 Problem Summary: The column data size collector looks up its map twice for existing entries and repeats three counter assignments between insertion and update branches. Use try_emplace once, initialize identity fields only on insertion, and add each metadata contribution through one accumulation block. Preserve missing-field/schema checks, parent VARIANT UID attribution, and V3 child metadata traversal. ### Release note None ### Check List (For Author) - Test: ASAN BE UT: all 36 selected tests passed after the refactor; GLIBC_COMPATIBILITY=OFF avoids a host duplicate-symbol conflict. Clang-format 16 and clang-tidy changed-line checks passed with the existing unity-build Rowset forward declaration supplied to the analyzer. - Behavior changed: No - Does this need documentation: No
### What problem does this PR solve? Problem Summary: Restore the original type-independent footer estimate for initial scan batches. Withdraw the VARIANT-specific exclusion and tests added for that policy, while retaining the column data size aggregation simplification. ### Release note None ### Check List (For Author) - Test: Format and header hygiene checks; restored segment iterator code matches the original PR head - Behavior changed: Yes, remove the newly added VARIANT-specific initial batch policy - Does this need documentation: No
### What problem does this PR solve? Related PR: apache#68444 Problem Summary: Merge master ac92748. Regenerate the 16 conflicting segment fixtures using the merged implementation to preserve both the map_agg_v2 metadata from master and the VARIANT data size metadata from this PR. All 132 other generated fixtures match the existing merged tree. ### Release note None ### Check List (For Author) - Test: ASAN BE unit tests 44/44 pass both when generating fixtures and when verifying the checked-in fixtures; clang-format 16 and header hygiene pass - Behavior changed: No, resolve test fixture conflicts while retaining both branches' changes - Does this need documentation: No
Member
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29237 ms |
Contributor
TPC-DS: Total hot run time: 151644 ms |
Contributor
ClickBench: Total hot run time: 24.54 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Member
Author
|
/review |
csun5285
approved these changes
Oct 19, 2026
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
No actionable findings at head 07d1a5c83382cd41c9e9650fdbd4a1c1479ccfc0. Reviewed all 40 changed paths, including 29 binary fixtures, with two complete-scope reviewers and one independent risk reviewer. All returned NO_NEW_VALUABLE_FINDINGS; the review converged in 1 round. There were no existing inline threads or confirmed existing blockers.
Critical checkpoint conclusions:
- Goal and evidence: Writer totals and scanner parent attribution cover VARIANT sizes through load and compaction. UT assertions compare compressed counters with actual appended bytes; regression covers ordinary/doc mode, V2/V3 metadata, and before/after compaction.
- Scope and reuse: The change stays focused on accounting and reuses existing scalar/complex getters through
DataBytesGetter. - Concurrency and locks: No new threads, locks or lock ordering. Scanner aggregation is local to existing rowset snapshots; segment I/O remains outside tablet snapshot locks.
- Lifecycle and initialization: Both production counter-read sites follow successful finalization and data writes. Empty, all-null, root-only, extracted and doc-compaction paths create their inner owners before reads. No new global initialization or ownership cycle.
- Configuration: No production configuration added or changed; tests select existing modes.
- Compatibility: Existing optional footer fields receive real counters; no wire fields, encoding or protocol changes. Historical zero/missing writer counters cannot be reconstructed until segments are rewritten. No new libc/API dependency identified in the source changes.
- Parallel paths: Traced ordinary/doc, sparse single/bucket, root-only/extracted/doc compaction, complex inner writers, V2/V3 metadata, and the common local/cloud scanner path.
- Conditions and errors: Counter-bearing extracted metas carry parent/path information. Inner metas have no separate counters, while V3 embeds separately counted sparse/doc metas once, so the new traversal does not duplicate their aggregates. Existing Status propagation remains intact.
- Test coverage: The exact-byte writer assertions and end-to-end matrix target the accounting boundaries. Empty/all-null ownership and supported typed/complex paths were also checked statically; no new negative-result behavior was introduced.
- Test results and fixtures: All eight ordered two-column regression output groups match the four pre/post-compaction checks. Static wire inspection of all 29 changed fixtures verified both old/new footer CRC32C values, unchanged data/index payloads and other metadata, and only 93 existing VARIANT byte-counter fields changing from zero to positive; files grow by 2 or 4 bytes.
- Observability: This repairs the existing column-size observability surface; no new execution stage requires extra logs or metrics.
- Transactions and persistence: Only existing segment statistics values change. No FE EditLog, master failover, transaction visibility, version or delete-bitmap logic changes.
- Writes and atomicity: Existing segment write/finalize/publication and failure propagation remain intact; statistics are read after successful writes.
- FE/BE variables: None introduced; no additional propagation paths apply.
- Performance and memory: Three linear passes over existing inner writers at finalization add no buffers or retained allocations. Scanner aggregation uses one map insertion/lookup. Query raw-byte hints remain bounded by the initial probe and original row maximum, then measured block sizes replace them; variant compaction retains its fallback.
- Other BE checkpoints: No new includes, hub dependency edges, PCH/gate changes, explicit template instantiations, header renames, or source-cpp test inclusions. No additional substantiated issue found after the final sweep.
User focus: No additional focus was supplied.
Limits: Static review only, as required. No builds, tests, ASAN, performance measurements, compiled ELF audit or live mixed-version execution were performed. Inspected output contents and fixture bytes do not establish their generation provenance. No unresolved correctness candidates or uncovered changed paths remain within this review scope.
Review converged after 1 round(s).
yiguolei
approved these changes
Oct 19, 2026
Contributor
|
PR approved by anyone and no changes requested. |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
yiguolei
pushed a commit
that referenced
this pull request
Oct 20, 2026
…_data_sizes (#69259) cherry-pick #68444 Adapted to branch-4.2: - `be/test/storage/test_data/segment_flusher_format/*`: the fixtures the original commit updates do not exist on branch-4.2 (that test was never backported), so they stay deleted. - branch-4.2 still has the V1 variant writer. `VariantColumnWriterImpl` may own only a V1 writer, so `get_total_data_pages_bytes` dispatches to `VariantV1ColumnWriter`, which sums its root, subcolumn and binary writers, instead of dereferencing a null V2 writer. `VariantSubcolumnWriter` reports 0 before its writer exists. Tested locally with `./run-be-ut.sh --run --filter='VariantColumnWriterReaderTest.*:VariantUtilTest.*'` (branch-4.2 code + this pick, RELEASE build): - 77 tests passed, including `VariantColumnWriterReaderTest.test_write_data_normal`, `doc_compact_writer_round_trip` and the two writer helpers that now assert the variant data page bytes, plus all 21 `VariantUtilTest` tests. - `VariantColumnWriterReaderTest.test_nested_iter` fails, but it fails on branch-4.2 without this change as well (`_process_nested_columns` no longer returns an error for a different array offset), so it is not caused by this pick. The new regression suite `query_p0/system/test_query_sys_column_data_sizes_variant` was not run locally (needs a cluster).
sdk2
pushed a commit
to sdk2/doris
that referenced
this pull request
Oct 21, 2026
…ring writers (apache#69221) ### What problem does this PR solve? Issue Number: CIR-22037 (related: apache#66298) VARIANT compaction can fail forever with `E-3113 string column length is too large`. `ColumnStr<uint32_t>` rejects more han 4 GiB of uncompressed chars, while vertical compaction plans output segments from **compressed** bytes. High-compression-ratio VARIANT data reaches the uint32 offset limit long before the planned segment reaches `vertical_compaction_max_segment_size`. The writer side made this deterministic: the VARIANT V2 writer buffered a whole output segment in memory before flushing it. Reproduced on master with 1.2M rows x 4 KiB of VARIANT payload (4.9 GiB uncompressed, ~8 MiB on disk) and `vertical_compaction_max_segment_size=256MiB`: ``` [E-3113] string column length is too large: total_length=4295296512, limit=4294967295, element_number=1044576, rows=1043584 VariantV2ColumnWriter::append -> ColumnStr<unsigned int>::insert_range_from ``` ### What changed? 1. **Stream the root-only VARIANT writer** (`variant/v2/variant_column_writer.*`). The root column writer is created at `init()` and every input batch is appended to it, so the writer never retains a segment-sized `ColumnString` (`_root_jsonb` is gone). `finalize()` only records the row count; `estimate_buffer_size()` reports the current page instead of the whole segment. 2. **Bound the writers that still buffer** (`compaction/compaction.*`, `common/config.*`). When the compaction schema still selects a segment-buffering writer - the shredder (no extracted subcolumns, e.g. `enable_vertical_compact_variant_subcolumns=false`) or `VariantSubcolumnWriter` (extracted VARIANT subcolumns) - clamp the per-segment row budget using the input segments' VARIANT `raw_data_bytes` and `variant_compaction_max_segment_uncompressed_bytes` (2 GiB default, 0 disables). `raw_data_bytes` is the pre-encoding size the writer buffers; `uncompressed_data_bytes` is the encoded page size and can be orders of magnitude smaller for dictionary-encoded strings (measured 142,560 vs 4,944,000,000 for the same segment). The streaming path is not clamped, so its segment layout does not change. Depends on apache#68444, which makes variant writers report `raw/compressed/uncompressed` data page bytes in the segment footer (first commit is that change, kept here so the branch is testable). ### Tests - `VariantColumnWriterReaderTest.test_variant_root_writer_streams_per_batch` (new): writes 4,000 x 4 KiB root scalars in 1,000-row batches and asserts the buffered bytes stay below two batches. Red/green verified locally with `doris_be_test`: - red (buffering writer): `actual: 16428000 vs 8192000` -> FAILED - green (streamed writer): `[ OK ] ... (156 ms)` -> PASSED - Variant writer data page byte counters equal the bytes appended to the file right after `write_data()` (ordinary, extracted subcolumn, doc compact writer). - `information_schema.column_data_sizes` reports VARIANT compressed/uncompressed/raw bytes (regression suite `test_query_sys_column_data_sizes_variant`). - Manual end-to-end on a single-BE cluster (1.2M x 4 KiB): - default path: **1 segment, no E-3113**, checksum unchanged (count=1200000, sum(len)=4927200000, sum(crc32)=2041942614000000), BE RSS peak +114 MiB (was +5 GiB). - `enable_vertical_compact_variant_subcolumns=false` (shredder): clamp 1,200,001 -> 521,233 rows per segment, **3 segments, no E-3113**, same checksum. - **Sparse columns**: `variant_max_subcolumns_count=1` with a 4 KiB `z_big` path that is not materialized (sparse), extract off. Without the budget: E-3113 at 4,294,970,386 bytes (`serialize_column_to_jsonb_vector`). With the budget: clamp 1,200,001 -> 517,465 rows per segment, 3 segments, no E-3113, checksum identical to a fresh load (count=1200000, sum(len)=4981200000, sum(crc32)=252636363600000). ### Remaining - Non-vertical compaction does not use the segment row budget. - Segments written before the footer raw-byte metadata existed (raw = 0) cannot be estimated; the streamed root path still covers them, the shredder path does not. - Doc mode relies on the raw-byte budget. ### Release note VARIANT compaction no longer buffers a whole output segment in one `ColumnString`.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: None
Related PR: #56815
Problem Summary:
information_schema.column_data_sizesreports 0 for COMPRESSED_DATA_BYTES,UNCOMPRESSED_DATA_BYTES and RAW_DATA_BYTES of every VARIANT column, so a
per-column size breakdown leaves the variant out and overstates the share of
the other columns. Reproduce on master with 200,000 rows whose
jsSTRINGcolumn and
vVARIANT column hold the same JSON:The table reads the data page sizes that the segment writer stores in each
column meta of the segment footer. Two gaps drop the variant:
VariantColumnWriter,VariantSubcolumnWriterandVariantDocCompactWriterreturn 0 (TODO) fromget_raw_data_bytes(),get_total_uncompressed_data_pages_bytes()andget_total_compressed_data_pages_bytes(), although the variant data iswritten by their inner writers: the root, the materialized subcolumns and
the sparse or doc value columns.
as columns of their own. Their metas have no unique id, so the scanner drops
them; with external column meta (
storage_format = "V3") the sparse anddoc value metas are also embedded in the root variant meta, which the
scanner never looks into.
Fix:
same way the struct, array and map writers already do.
ColumnWriter::DataBytesGetternames the getter that the variant writerlayers pass down.
parent variant column (from its column path info), adds the sparse and doc
value metas embedded in a variant root meta, and takes COLUMN_TYPE from the
schema column (a subcolumn meta carries the subcolumn's type). Nothing is
counted twice: the inner metas of a variant writer carry no data bytes.
After the fix, the variant reports the size of all its data pages before and
after compaction, for inline and external column meta, in ordinary and doc
mode. Segments written before the fix keep their old numbers until compaction
rewrites them.
The segment footer now stores real data bytes on variant column metas instead
of 0, so the 29 checked-in golden Segment files of the 14 variant cases in
segment_flusher_format_testare regenerated (each grows by 2 to 4 bytes;only those three counters of the variant metas differ). The other 59 golden
cases are byte-identical.
The footer raw_data_bytes of a variant root is also the first-batch hint of the
segment iterator's adaptive batch size predictor when a query reads the whole
variant column. It was 0 for variants; it now holds the variant's data for
segments written by a load and the root's data for compaction output. It only
affects the first batch, which is capped at the probe rows either way. Vertical
compaction's footer-based batch size estimate still skips variant columns; its
stale comment is updated.
Release note
Fix
information_schema.column_data_sizesreporting 0 bytes for VARIANT columns. Data written before the fix reports the new sizes once compaction rewrites it.Check List (For Author)
Test
BE UT: the variant column writers report exactly the data page bytes they append to the segment file (ordinary and doc mode variant writers, extracted subcolumn writer, doc compact writer); the golden segment tests, variant, compaction and segment writer tests pass.
Regression test
test_query_sys_column_data_sizes_variant: ordinary and doc mode variants with V2 and V3 column meta, before and after full compaction. The data pages of all columns must lie within the rowsets' DATA_DISK_SIZE and cover more than half of it; they cover 91%-96%. Without counting the metas embedded in the variant root, the V3 tables drop to 35% and 11% after compaction; without resolving the parent unique id, the V2 tables drop to 3% and 2%.Behavior changed:
information_schema.column_data_sizesreports the data size of VARIANT columns instead of 0, and variant column metas in the segment footer store their data page bytes instead of 0.Does this need documentation?
Check List (For Reviewer who merge this PR)
🤖 Generated with Claude Code