Repository navigation
[branch-4.0] [Opt](cloud) Support packed file for delete bitmap storage - #61007
Merged
Merged
Conversation
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
) Problem Summary: This PR adds packed file support for delete bitmap storage in cloud mode. - Add packed file support for delete bitmap writer/reader - Add write_file_cache flag to PackedAppendContext to control file cache behavior - Delete bitmap files do not use file cache to match original behavior - Add regression tests for packed delete bitmap scenarios
liaoxin01
force-pushed
the
cherry-pick-60411-4.0
branch
from
March 3, 2026 15:43
cb27eb7 to
575a29d
Compare
Contributor
Author
|
run buildall |
Contributor
There was a problem hiding this comment.
Pull request overview
Backports packed-file support for cloud-mode delete bitmap storage to branch-4.0, including persisting packed slice locations, reading/writing delete bitmaps from packed files, recycler cleanup logic, and regression coverage.
Changes:
- Persist packed slice location for delete bitmap storage in metaservice protobuf and use it to read from packed files.
- Add packed-file writer/reader support for delete bitmap files, including disabling file-cache writes for delete bitmaps.
- Extend recycler cleanup to decrement packed-file ref counts for delete bitmap slices; add regression tests for packed delete bitmap scenarios.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| regression-test/suites/unique_with_mow_p0/cloud_delete_bitmap/test_packed_delete_bitmap.groovy | Adds a dedicated regression suite covering packed delete bitmap behavior and BE restart scenarios. |
| regression-test/suites/unique_with_mow_p0/cloud_delete_bitmap/test_load.groovy | Toggles enable_packed_file in an existing delete-bitmap load suite. |
| regression-test/suites/unique_with_mow_p0/cloud_delete_bitmap/test_cu_compaction.groovy | Toggles enable_packed_file in an existing CU compaction suite. |
| regression-test/data/unique_with_mow_p0/cloud_delete_bitmap/test_packed_delete_bitmap.out | Expected output for the new packed delete bitmap regression suite. |
| gensrc/proto/cloud.proto | Extends DeleteBitmapStoragePB to persist packed_slice_location. |
| cloud/src/recycler/recycler.h | Declares recycler helper for decrementing packed-file ref counts for delete bitmap slices. |
| cloud/src/recycler/recycler.cpp | Implements delete bitmap packed-file refcount decrement and skips standalone delete-bitmap file deletion when packed. |
| be/src/io/fs/packed_file_manager.h | Adds write_file_cache control to PackedAppendContext. |
| be/src/io/fs/packed_file_manager.cpp | Honors write_file_cache when deciding whether to populate file cache for packed small files. |
| be/src/cloud/delete_bitmap_file_writer.h | Adds packed-file-aware delete bitmap writer APIs and state. |
| be/src/cloud/delete_bitmap_file_writer.cpp | Wraps delete bitmap writes in PackedFileWriter when enabled and surfaces packed slice location after close. |
| be/src/cloud/delete_bitmap_file_reader.h | Adds a packed-slice-location-based constructor for reading delete bitmap from packed files. |
| be/src/cloud/delete_bitmap_file_reader.cpp | Reads delete bitmap from packed files via PackedFileReader when packed location is present. |
| be/src/cloud/cloud_meta_mgr.cpp | Writes packed slice location into DeleteBitmapStoragePB and selects packed vs standalone reader when loading delete bitmaps. |
Comments suppressed due to low confidence (1)
cloud/src/recycler/recycler.cpp:3268
- Logging
delete bitmap not found, skip packed file ref count decrementat INFO in the recycler can become very noisy because many rowsets may not have any delete-bitmap entry. Recommend downgrading this to a verbose/debug log (or remove it) to avoid flooding logs during routine recycling.
// No delete bitmap for this rowset, nothing to do
LOG_INFO("delete bitmap not found, skip packed file ref count decrement")
.tag("instance_id", instance_id_)
.tag("tablet_id", tablet_id)
.tag("rowset_id", rowset_id);
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Cloud UT Coverage ReportIncrement line coverage Increment coverage report
|
Contributor
FE UT Coverage ReportIncrement line coverage `` 🎉 |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
yiguolei
approved these changes
Mar 12, 2026
Contributor
|
PR approved by at least one committer and no changes requested. |
github-actions
Bot
added
approved
Contributor
|
PR approved by anyone and no changes requested. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
pick：#60411