[fix](move-memtable) fix segment number mismatch for erroneously skipped segments #55092
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
sollhui
force-pushed
the
fix_segment_num_mismatch
branch
5 times, most recently
from
b71c278 to
1ce3d0e
Compare
sollhui
changed the title
Contributor
Author
|
run buildall |
sollhui
changed the title
sollhui
force-pushed
the
fix_segment_num_mismatch
branch
from
1ce3d0e to
20257a4
Compare
TPC-H: Total hot run time: 33603 ms |
TPC-DS: Total hot run time: 185614 ms |
ClickBench: Total hot run time: 32.14 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
kaijchen
approved these changes
Aug 27, 2025
Contributor
|
PR approved by anyone and no changes requested. |
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
bot
added
the
approved
Contributor
|
run buildall |
TPC-H: Total hot run time: 35287 ms |
TPC-DS: Total hot run time: 187563 ms |
ClickBench: Total hot run time: 33.24 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
1 similar comment
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
3 similar comments
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
Author
|
run buildall |
TPC-H: Total hot run time: 33990 ms |
TPC-DS: Total hot run time: 186694 ms |
ClickBench: Total hot run time: 33.04 s |
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
BE Regression && UT Coverage ReportIncrement line coverage Increment coverage report
|
github-actions bot
pushed a commit
that referenced
this pull request
Aug 29, 2025
…ped segments (#55092) ### What problem does this PR solve? Fix segment number mismatch caused by erroneously skipped segments during concurrent incremental open on auto-partitioned table： #### Problem During concurrent incremental open on an auto-partitioned table, one sink may incorrectly assume that stream opened by another sink have already been opened and begin writing data while those segments are still being opened. This leads to some segments being silently skipped and results in a segment number mismatch. For example(two instances, 4 BEs: a, b, c, d): | Time | Event | | ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | t0 | `sink1` and `sink2` start incremental open for BEs **a, b, c, d**. | | t1 | `sink1` adds **a, b, c** to `_load_stream_map` and initiates open. | | t2 | `sink2` adds **d** to `_load_stream_map` and initiates open. | | t3 | `sink1` completes open for **a** and **b**; **c** is still in progress. | | t4 | `sink2` successfully opens **d**, assumes **a, b, c** are **all** ready, and starts writing. Because **c** is not yet fully open, its segments are skipped, causing the mismatch. | #### Expected behavior A sink must wait until all stream it depends on are fully opened before starting any write. #### Proposed fix All sinks open the full set of streams (a, b, c, d) instead of a partial subset. Lock on each stream guarantees that: - Duplicate open attempts are prevented:only the first sink performs the actual open; subsequent sinks wait until the open is complete. - Expected behavior is preserved:every sink waits until all streams are fully opened before starting any write, eliminating skipped segments and the resulting segment-number mismatch. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
github-actions bot
pushed a commit
that referenced
this pull request
Aug 29, 2025
…ped segments (#55092) ### What problem does this PR solve? Fix segment number mismatch caused by erroneously skipped segments during concurrent incremental open on auto-partitioned table： #### Problem During concurrent incremental open on an auto-partitioned table, one sink may incorrectly assume that stream opened by another sink have already been opened and begin writing data while those segments are still being opened. This leads to some segments being silently skipped and results in a segment number mismatch. For example(two instances, 4 BEs: a, b, c, d): | Time | Event | | ---- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | t0 | `sink1` and `sink2` start incremental open for BEs **a, b, c, d**. | | t1 | `sink1` adds **a, b, c** to `_load_stream_map` and initiates open. | | t2 | `sink2` adds **d** to `_load_stream_map` and initiates open. | | t3 | `sink1` completes open for **a** and **b**; **c** is still in progress. | | t4 | `sink2` successfully opens **d**, assumes **a, b, c** are **all** ready, and starts writing. Because **c** is not yet fully open, its segments are skipped, causing the mismatch. | #### Expected behavior A sink must wait until all stream it depends on are fully opened before starting any write. #### Proposed fix All sinks open the full set of streams (a, b, c, d) instead of a partial subset. Lock on each stream guarantees that: - Duplicate open attempts are prevented:only the first sink performs the actual open; subsequent sinks wait until the open is complete. - Expected behavior is preserved:every sink waits until all streams are fully opened before starting any write, eliminating skipped segments and the resulting segment-number mismatch. ### Release note None ### Check List (For Author) - Test <!-- At least one of them must be included. --> - [ ] Regression test - [ ] Unit Test - [ ] Manual test (add detailed scripts or steps below) - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason <!-- Add your reason? --> - Behavior changed: - [ ] No. - [ ] Yes. <!-- Explain the behavior change --> - Does this need documentation? - [ ] No. - [ ] Yes. <!-- Add document PR link here. eg: apache/doris-website#1214 --> ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label <!-- Add branch pick label that this PR should merge into -->
morrySnow
pushed a commit
that referenced
this pull request
Sep 4, 2025
…neously skipped segments #55092 (#55471) Cherry-picked from #55092 Co-authored-by: hui lai <[email protected]>
dataroaring
pushed a commit
that referenced
this pull request
Sep 5, 2025
…neously skipped segments #55092 (#55470) Cherry-picked from #55092 Co-authored-by: hui lai <[email protected]>
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Fix segment number mismatch caused by erroneously skipped segments during concurrent incremental open on auto-partitioned table：
Problem
During concurrent incremental open on an auto-partitioned table, one sink may incorrectly assume that stream opened by another sink have already been opened and begin writing data while those segments are still being opened. This leads to some segments being silently skipped and results in a segment number mismatch. For example(two instances, 4 BEs: a, b, c, d):
sink1andsink2start incremental open for BEs a, b, c, d.sink1adds a, b, c to_load_stream_mapand initiates open.sink2adds d to_load_stream_mapand initiates open.sink1completes open for a and b; c is still in progress.sink2successfully opens d, assumes a, b, c are all ready, and starts writing. Because c is not yet fully open, its segments are skipped, causing the mismatch.Expected behavior
A sink must wait until all stream it depends on are fully opened before starting any write.
Proposed fix
All sinks open the full set of streams (a, b, c, d) instead of a partial subset. Lock on each stream guarantees that:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)