Repository navigation
[fix](cloud-mow)Full compaction no need to sync delete bitmap - #43337
Conversation
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
| } | ||
|
|
||
| Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data) { | ||
| Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data, |
There was a problem hiding this comment.
warning: function 'sync_tablet_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]
Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data,
^Additional context
be/src/cloud/cloud_meta_mgr.cpp:386: 181 lines including whitespace and comments (threshold 80)
Status CloudMetaMgr::sync_tablet_rowsets(CloudTablet* tablet, bool warmup_delta_data,
^|
run buildall |
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
|
TeamCity be ut coverage result: |
TPC-H: Total hot run time: 41542 ms |
TPC-DS: Total hot run time: 196012 ms |
ClickBench: Total hot run time: 32.55 s |
sync delete bitmap may cost too much time,which may lead to full compaction fail, but full compaction no need to sync it. pick pr:#43337 Co-authored-by: huanghaibin <[email protected]>
…#43337) sync delete bitmap may cost too much time,which may lead to full compaction fail, but full compaction no need to sync it.
### Background Cloud changelog tablets keep historical events in companion DUP tablets. Delete bitmaps hide physical event rows rejected by the source table's merge-on-write conflict resolution. Full compaction must preserve this logical event set while replacing the physical rowsets. ### What problem does this PR solve? Issue Number: None Related PR: apache#43337 (introduced the full-compaction bitmap-sync optimization), apache#65810 (introduced independent row-binlog tablets) PR Type: Bugfix Problem Summary: A manual full-compaction request can cold-load a changelog tablet's rowsets without its delete bitmaps. The subsequent incremental prepare sync sees no new rowsets and never restores filters for invalid events, allowing a rejected update to appear as a BEFORE/AFTER pair in DETAIL history. Fully synchronize changelog inputs before full compaction; retain incremental synchronization for ordinary tablets. ### Details - Frequency: Requires a changelog tablet with cached rowsets but missing historical delete bitmaps, such as a cold manual full-compaction load after restart. - Root cause: Rowset version progress is used for incremental bitmap retrieval even when the initial load explicitly skipped bitmap synchronization. - Why existing tests missed it: Full-compaction input synchronization lacked coverage for already-cached changelog rowsets with an absent persisted bitmap. - Fix: Use the existing full-sync option for changelog full-compaction preparation. Propagate synchronization failures before selecting input rowsets. - Other paths and branches: Ordinary tablet behavior is unchanged. No protocol, storage format or configuration changes. Existing invalid compaction output is not repaired by this patch. ### Testing - Passed in a Linux x86_64 development environment: repository clang-format/check-format with clang-format 16.0.6, build hygiene, and git diff --check. - Added four BE tests using a loopback Meta Service peer and the real synchronization path: restore a bitmap on physical segment 7; accept an empty bitmap; stop on a bitmap-read error; retain ordinary-tablet incremental sync. - Not run: native compilation, BE tests, before/after runtime comparison, clang-tidy or live regression. Exact thirdparty cache is unavailable and source dependency preparation fails the pinned Arrow ADBC 24 archive checksum. Integrity checks were retained. ### Release note Prevent rejected updates from becoming visible in changelog history when a cloud changelog tablet is cold-loaded for full compaction. ### Check List (For Author) - Test: Static checks passed; four unit tests added but not executed because dependency preparation is blocked. - Behavior changed: Yes (changelog full compaction reloads complete input delete bitmaps). - Does this need documentation: No - Pre-submission status: Partially verified: native compilation, unit tests and runtime comparison remain blocked by exact thirdparty dependencies. ### AI Disclosure - Tool: Codex - Generated: Code, tests, and this description - Human review: Pending
### Background Cloud changelog tablets keep historical events in companion DUP tablets. Delete bitmaps hide physical event rows rejected by the source table's merge-on-write conflict resolution. Full compaction must preserve this logical event set while replacing the physical rowsets. ### What problem does this PR solve? Issue Number: None Related PR: apache#43337 (introduced the full-compaction bitmap-sync optimization), apache#65810 (introduced independent row-binlog tablets) PR Type: Bugfix Problem Summary: A manual full-compaction request can cold-load a changelog tablet's rowsets without its delete bitmaps. The subsequent incremental prepare sync sees no new rowsets and never restores filters for invalid events, allowing a rejected update to appear as a BEFORE/AFTER pair in DETAIL history. Fully synchronize changelog inputs before full compaction; retain incremental synchronization for ordinary tablets. ### Details - Frequency: Requires a changelog tablet with cached rowsets but missing historical delete bitmaps, such as a cold manual full-compaction load after restart. - Root cause: Rowset version progress is used for incremental bitmap retrieval even when the initial load explicitly skipped bitmap synchronization. - Why existing tests missed it: Full-compaction input synchronization lacked coverage for already-cached changelog rowsets with an absent persisted bitmap. - Fix: Use the existing full-sync option for changelog full-compaction preparation. Propagate synchronization failures before selecting input rowsets. - Other paths and branches: Ordinary tablet behavior is unchanged. No protocol, storage format or configuration changes. Existing invalid compaction output is not repaired by this patch. ### Testing - Passed: repository clang-format/check-format with clang-format 16.0.6, build hygiene, and git diff --check. - Added four BE tests using a loopback Meta Service peer and the real synchronization path: restore a bitmap on physical segment 7; accept an empty bitmap; stop on a bitmap-read error; retain ordinary-tablet incremental sync. - Not run: native compilation, BE tests, before/after runtime comparison, clang-tidy or live regression. Exact thirdparty cache is unavailable and source dependency preparation fails the pinned Arrow ADBC 24 archive checksum. Integrity checks were retained. ### Release note Prevent rejected updates from becoming visible in changelog history when a cloud changelog tablet is cold-loaded for full compaction. ### Check List (For Author) - Test: Static checks passed; four unit tests added but not executed because dependency preparation is blocked. - Behavior changed: Yes (changelog full compaction reloads complete input delete bitmaps). - Does this need documentation: No - Pre-submission status: Partially verified: native compilation, unit tests and runtime comparison remain blocked by exact thirdparty dependencies. ### AI Disclosure - Tool: Codex - Generated: Code, tests, and this description - Human review: Pending
### Background Changelog tablets retain historical events. Delete bitmaps exclude physical rows rejected by the source table's conflict resolution. ### What problem does this PR solve? Issue Number: None Related PR: apache#43337, apache#65810 PR Type: Bugfix Problem Summary: After a restart, manual full compaction can expose a rejected update as an extra BEFORE/AFTER pair in DETAIL history. ### Details - Frequency: Cold loading for manual full compaction. - Root cause: The initial load skips bitmaps, and subsequent incremental synchronization sees no new rowsets to fetch their missing filters. - Why existing tests missed it: No coverage for cached changelog rowsets with missing bitmaps. - Fix: Fully synchronize changelog inputs before merging; abort preparation on synchronization errors. - Other paths and branches: Ordinary tablets retain incremental synchronization. Existing incorrect history requires separate repair. ### Testing - Passed: clang-format 16, build hygiene, and git diff --check. - Added four RPC-based BE tests for bitmap restoration, empty bitmaps, read failures, and ordinary-tablet behavior. - Compilation, test execution, and clang-tidy remain blocked: the exact thirdparty cache is unavailable, and the Arrow ADBC 24 source archive fails checksum verification. ### Release note Prevent rejected updates from appearing in changelog history after cold-loaded full compaction. ### Check List (For Author) - Test: See Testing above. - Behavior changed: Yes (complete bitmap synchronization for changelog full compaction). - Does this need documentation: No - Pre-submission status: Partially verified; runtime validation remains blocked. ### AI Disclosure - Tool: Codex - Generated: Code, tests, and description - Human review: Pending
### Background Changelog tablets store historical events. Delete bitmaps hide physical event rows rejected by the source table's Sequence rules. ### What problem does this PR solve? Issue Number: None Related PR: apache#43337 (full-compaction bitmap optimization), apache#65810 (independent changelog tablets) PR Type: Bugfix Problem Summary: After BE restart, manually running full compaction on a cold changelog tablet can make a previously rejected low-Sequence update visible in DETAIL history. One invalid UPDATE row becomes an extra UPDATE_BEFORE/UPDATE_AFTER pair, although the source table remains correct. ### Details - Frequency: When manual full compaction loads a changelog tablet before its delete bitmaps have been restored. - Root cause: The HTTP entry point loads rowsets while skipping bitmap synchronization. This advances the local rowset version: after loading through version 7, preparation requests new rowsets from version 8. An empty response makes bitmap synchronization return without restoring missing filters for older rowsets. - Why existing tests missed it: They did not cover full-compaction preparation with cached changelog rowsets but missing bitmaps. - Fix: Set SyncOptions.full_sync for changelog tablets in prepare_compact(), fetching all visible rowset metadata and associated bitmaps before selecting compaction inputs. Synchronization errors stop preparation. - Other paths and branches: Ordinary tablets keep incremental synchronization. The patch prevents new incorrect output; it does not repair existing history. ### Testing - Passed: clang-format 16, build hygiene, and git diff --check. - Added four RPC-based BE tests covering bitmap restoration, valid empty bitmaps, bitmap-read failures, and ordinary-tablet behavior; not executed. - Compilation, runtime comparison, and clang-tidy are blocked: the exact thirdparty cache is unavailable, and the Arrow ADBC 24 source archive fails checksum verification. ### Release note Prevent rejected updates from appearing in changelog DETAIL history after cold-loaded full compaction. ### Check List (For Author) - Test: Static checks passed; unit tests added but not run. - Behavior changed: Yes (changelog full compaction reloads complete input bitmaps). - Does this need documentation: No - Pre-submission status: Partially verified; compilation and runtime validation remain blocked. ### AI Disclosure - Tool: Codex - Generated: Code, tests, and description - Human review: Pending
sync delete bitmap may cost too much time,which may lead to full compaction fail, but full compaction no need to sync it.