Repository navigation
[feat](seq mapping) uniq table supports multi-stream merging through sequence mapping - #54936
Merged
Merged
Conversation
HonestManXin
requested review from
CalvinKirs,
dataroaring,
gavinchou,
morningman and
yiguolei
as code owners
August 18, 2025 08:10
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
HonestManXin
force-pushed
the
seq_mapping
branch
2 times, most recently
from
August 18, 2025 09:58
049b532 to
cfe00a4
Compare
Contributor
Author
|
run buildall |
HonestManXin
force-pushed
the
seq_mapping
branch
from
August 18, 2025 10:05
cfe00a4 to
a264265
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 34750 ms |
TPC-DS: Total hot run time: 185399 ms |
ClickBench: Total hot run time: 32.47 s |
HonestManXin
force-pushed
the
seq_mapping
branch
from
September 9, 2025 03:23
a264265 to
637b19e
Compare
HonestManXin
force-pushed
the
seq_mapping
branch
from
September 9, 2025 03:23
637b19e to
9b45113
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
TPC-H: Total hot run time: 34821 ms |
TPC-DS: Total hot run time: 189967 ms |
ClickBench: Total hot run time: 29.65 s |
HonestManXin
force-pushed
the
seq_mapping
branch
2 times, most recently
from
September 9, 2025 07:30
5df34d4 to
1882828
Compare
Contributor
Author
|
run buildall |
1 similar comment
Contributor
Author
|
run buildall |
Contributor
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run feut |
Contributor
Author
|
run beut |
Contributor
FE Regression Coverage ReportIncrement line coverage |
Contributor
Author
|
run external |
Contributor
Author
|
run vault_p0 |
Contributor
Author
|
run external |
HonestManXin
force-pushed
the
seq_mapping
branch
from
December 24, 2025 09:37
22a3449 to
76dda6b
Compare
Contributor
Author
|
run buildall |
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage |
TPC-H: Total hot run time: 36621 ms |
TPC-DS: Total hot run time: 180351 ms |
ClickBench: Total hot run time: 27.34 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
5 tasks
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run vault_p0 |
github-actions
Bot
added
the
approved
Contributor
|
PR approved by at least one committer and no changes requested. |
Contributor
|
PR approved by anyone and no changes requested. |
bobhan1
approved these changes
Dec 30, 2025
dataroaring
pushed a commit
to apache/doris-website
that referenced
this pull request
Jan 5, 2026
## Versions - [ x ] dev - [ ] 3.0 - [ ] 2.1 - [ ] 2.0 ## Languages - [ x ] Chinese - [ x ] English ## Docs Checklist - [ ] Checked by AI - [ ] Test Cases Built apache/doris#54936 Co-authored-by: zhangshixin.1024 <[email protected]>
Contributor
|
@dataroaring @bobhan1 @gavinchou Do we have a plan to apply this feature to mow table? |
This was referenced Aug 18, 2026
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Oct 27, 2026
### What problem does this PR solve? Issue Number: N/A Related PR: apache#35432, apache#54936 Problem Summary: One load that flushes several memtables writes a rowset whose segments overlap on keys, so the same key can sit in two segments of one rowset. Reading such a rowset merges its segments in BetaRowsetReader (VMergeIterator) before VCollectIterator merges the rowsets, and three bugs on that path return wrong rows for merge-on-read tables, in queries and in horizontal compaction. 1. The older row of a key wins (merge-on-read UNIQUE_KEYS, REPLACE columns of AGG_KEYS). The segment merge orders the rows of one key and one sequence value by the data_id of their iterators, so that the segment flushed later supplies the newer row. Since apache#35432 the merge reads LazyInitSegmentIterators, which did not override data_id() and all answered 0, so the tie fell to the heap order and a read could keep the row of the older segment. Vertical compaction orders by iterator position and kept the newer row, so the result of a query also changed after compaction. LazyInitSegmentIterator now answers the id of its segment, the same value SegmentIterator::data_id() reports. 2. A key group cut by a Block boundary is returned twice (AGG_KEYS, and sequence-map tables). The VCollectIterator::Level0Iterator of the lowest-version rowset reads it Block by Block together with the same-key bits of its segment merge, and took the bit of the next row before refresh_current_row() had moved to the next Block, so the first row of every Block lost its bit and BlockReader started a new key group there. The bit is now taken after the refresh. 3. A sequence-map table loses column groups (apache#54936). Such a table needs every row of a key, because BlockReader takes each column group from the row with that group's largest sequence value; VCollectIterator therefore does not skip same-key rows of it, but TabletReader still asked the segment merge to keep only the newest row of a key (is_unique). A column group whose largest sequence value was in another segment lost its values. The segment merge now keeps every row of a sequence-map table. Reproduction (regression test test_overlapping_segments_merge): with the MemTable.need_flush debug point and batch_size = 20, load numbers 0..39 as k = number % 20 into an AGG_KEYS table, a merge-on-read UNIQUE_KEYS table and a sequence-map table, so that keys 0..19 sit in both of two overlapping segments and the second segment holds the newer rows; then read with batch_size = 3. - Before: the AGG_KEYS table returned 7 of the 20 keys twice and its REPLACE column took the older value, the UNIQUE_KEYS table returned the older value of every key, and the sequence-map table returned the older row's v2 group for every key. After a full compaction the AGG_KEYS and UNIQUE_KEYS tables (vertical compaction) were right, while the sequence-map table (horizontal compaction) kept the wrong values. - After: every key comes back once with the newer values, and every column group of the sequence-map table with its largest-sequence values, before and after the full compaction. ### Release note Fix wrong query results and compaction output of merge-on-read UNIQUE KEY tables (including tables with sequence mapping) and AGGREGATE KEY tables when one load wrote the same key into several segments of one rowset. ### Check List (For Author) - Test: Regression test / Unit Test - New BE UT block_reader_overlapping_segments_test.cpp: all 4 tests fail before the fix and pass after it. Also ran BlockReader*, VCollect*, *Changelog*, VerticalBlockReaderTest*, VerticalCompactionTest*, VerticalMergeIteratorTest*, OrderedDataCompactionTest*, SegCompactionTest*, TestRowIdConversion*, BetaRowsetTest*, *SchemaChange*, CumulativeCompaction*, EstimateCollectedEnough* and CollectedEnough*: 254 tests pass. be/test/exprs/function/function_string_test.cpp was left out of the local UT build because it does not compile on master (FunctionContext::FRAGMENT_LOCAL, from apache#69223). - New regression test fault_injection_p0/test_overlapping_segments_merge: run on a build without the fix it returned the wrong rows above; with the fix it passes. - Behavior changed: Yes. Reads of such rowsets return the newer row of a key, fold every key group once, and keep every column group of a sequence-map table. Rowsets that an older version already compacted keep what they were written with. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
csun5285
added a commit
to csun5285/doris
that referenced
this pull request
Nov 4, 2026
### What problem does this PR solve? Issue Number: N/A Related PR: apache#35432, apache#54936 Problem Summary: One load that flushes several memtables writes a rowset whose segments overlap on keys, so the same key can sit in two segments of one rowset. Reading such a rowset merges its segments in BetaRowsetReader (VMergeIterator) before VCollectIterator merges the rowsets, and three bugs on that path return wrong rows for merge-on-read tables, in queries and in horizontal compaction. 1. The older row of a key wins (merge-on-read UNIQUE_KEYS, REPLACE columns of AGG_KEYS). The segment merge orders the rows of one key and one sequence value by the data_id of their iterators, so that the segment flushed later supplies the newer row. Since apache#35432 the merge reads LazyInitSegmentIterators, which did not override data_id() and all answered 0, so the tie fell to the heap order and a read could keep the row of the older segment. Vertical compaction orders by iterator position and kept the newer row, so the result of a query also changed after compaction. LazyInitSegmentIterator now answers the id of its segment, the same value SegmentIterator::data_id() reports. 2. A key group cut by a Block boundary is returned twice (AGG_KEYS, and sequence-map tables). The VCollectIterator::Level0Iterator of the lowest-version rowset reads it Block by Block together with the same-key bits of its segment merge, and took the bit of the next row before refresh_current_row() had moved to the next Block, so the first row of every Block lost its bit and BlockReader started a new key group there. The bit is now taken after the refresh. 3. A sequence-map table loses column groups (apache#54936). Such a table needs every row of a key, because BlockReader takes each column group from the row with that group's largest sequence value; VCollectIterator therefore does not skip same-key rows of it, but TabletReader still asked the segment merge to keep only the newest row of a key (is_unique). A column group whose largest sequence value was in another segment lost its values. The segment merge now keeps every row of a sequence-map table. Reproduction (regression test test_overlapping_segments_merge): with the MemTable.need_flush debug point and batch_size = 20, load numbers 0..39 as k = number % 20 into an AGG_KEYS table, a merge-on-read UNIQUE_KEYS table and a sequence-map table, so that keys 0..19 sit in both of two overlapping segments and the second segment holds the newer rows; then read with batch_size = 3. - Before: the AGG_KEYS table returned 7 of the 20 keys twice and its REPLACE column took the older value, the UNIQUE_KEYS table returned the older value of every key, and the sequence-map table returned the older row's v2 group for every key. After a full compaction the AGG_KEYS and UNIQUE_KEYS tables (vertical compaction) were right, while the sequence-map table (horizontal compaction) kept the wrong values. - After: every key comes back once with the newer values, and every column group of the sequence-map table with its largest-sequence values, before and after the full compaction. ### Release note Fix wrong query results and compaction output of merge-on-read UNIQUE KEY tables (including tables with sequence mapping) and AGGREGATE KEY tables when one load wrote the same key into several segments of one rowset. ### Check List (For Author) - Test: Regression test / Unit Test - New BE UT block_reader_overlapping_segments_test.cpp: all 4 tests fail before the fix and pass after it. Also ran BlockReader*, VCollect*, *Changelog*, VerticalBlockReaderTest*, VerticalCompactionTest*, VerticalMergeIteratorTest*, OrderedDataCompactionTest*, SegCompactionTest*, TestRowIdConversion*, BetaRowsetTest*, *SchemaChange*, CumulativeCompaction*, EstimateCollectedEnough* and CollectedEnough*: 254 tests pass. be/test/exprs/function/function_string_test.cpp was left out of the local UT build because it does not compile on master (FunctionContext::FRAGMENT_LOCAL, from apache#69223). - New regression test fault_injection_p0/test_overlapping_segments_merge: run on a build without the fix it returned the wrong rows above; with the fix it passes. - Behavior changed: Yes. Reads of such rowsets return the newer row of a key, fold every key group once, and keep every column group of a sequence-map table. Rowsets that an older version already compacted keep what they were written with. - Does this need documentation: No Co-Authored-By: Claude Opus 5.5 <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In some business scenarios, business operations need to update different columns in the same wide table through two or more data streams. For example, one data stream will write in real time to update some fields of this table; another data stream will perform imports on demand to update other columns of this table. During the update process, both data stream jobs need to ensure the order of replacement; and during queries, data from all columns should be queryable.
documentation: apache/doris-website#2954