Skip to content

[Opt](cloud) Support packed file for delete bitmap storage - #60411

Merged
liaoxin01 merged 2 commits into
apache:masterfrom
liaoxin01:feature/delete-bitmap-packed-file
Mar 3, 2026
Merged

[Opt](cloud) Support packed file for delete bitmap storage#60411
liaoxin01 merged 2 commits into
apache:masterfrom
liaoxin01:feature/delete-bitmap-packed-file

Conversation

@liaoxin01

@liaoxin01 liaoxin01 commented Feb 1, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
This PR adds packed file support for delete bitmap storage in cloud mode.

  • Add packed file support for delete bitmap writer/reader
  • Add write_file_cache flag to PackedAppendContext to control file cache behavior
  • Delete bitmap files do not use file cache to match original behavior
  • Add regression tests for packed delete bitmap scenarios

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Copilot AI review requested due to automatic review settings February 1, 2026 15:41
@hello-stephen

hello-stephen commented Feb 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@liaoxin01
liaoxin01 force-pushed the feature/delete-bitmap-packed-file branch from 3652a29 to 66af45e Compare February 1, 2026 15:43
@liaoxin01

liaoxin01 commented Feb 1, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

Copilot AI reviewed Feb 1, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds packed file support for delete bitmap storage in cloud mode to improve storage efficiency by packing small delete bitmap files into larger files when stored in object storage.

Changes:

  • Added packed file read/write support for delete bitmap storage with conditional enablement based on transaction ID
  • Extended PackedAppendContext with write_file_cache flag to control file cache behavior for delete bitmaps
  • Added comprehensive regression tests covering BE restart, multi-rowset packing, and large bitmap fallback scenarios

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
gensrc/proto/cloud.proto Added packed_slice_location field to DeleteBitmapStoragePB message to store packed file metadata
be/src/io/fs/packed_file_manager.h Added write_file_cache flag to PackedAppendContext to control file cache writing behavior
be/src/io/fs/packed_file_manager.cpp Conditionally write to file cache based on write_file_cache flag
be/src/cloud/delete_bitmap_file_writer.h Added constructor and methods to support packed file writing with location tracking
be/src/cloud/delete_bitmap_file_writer.cpp Implemented packed file writing logic with PackedFileWriter wrapper and location storage
be/src/cloud/delete_bitmap_file_reader.h Added constructor to support reading from packed files using PackedSliceLocationPB
be/src/cloud/delete_bitmap_file_reader.cpp Implemented packed file reading with PackedFileReader wrapper
be/src/cloud/cloud_meta_mgr.cpp Integrated packed file support into delete bitmap storage/retrieval flow and added txn_id parameter for conditional packing
regression-test/suites/unique_with_mow_p0/cloud_delete_bitmap/test_packed_delete_bitmap.groovy Added comprehensive test scenarios for packed delete bitmap functionality
regression-test/suites/unique_with_mow_p0/cloud_delete_bitmap/test_load.groovy Added randomized enable_packed_file config to existing test for coverage
regression-test/suites/unique_with_mow_p0/cloud_delete_bitmap/test_cu_compaction.groovy Added randomized enable_packed_file config to compaction test
regression-test/data/unique_with_mow_p0/cloud_delete_bitmap/test_packed_delete_bitmap.out Expected output file for packed delete bitmap regression tests

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread be/src/cloud/cloud_meta_mgr.cpp Resolved
@doris-robot

doris-robot commented Feb 1, 2026

Copy link
Copy Markdown

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.29% (1792/2260)
Line Coverage 64.60% (31865/49323)
Region Coverage 65.36% (15888/24309)
Branch Coverage 55.85% (8434/15100)

@hello-stephen

hello-stephen commented Feb 1, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@doris-robot

doris-robot commented Feb 1, 2026

Copy link
Copy Markdown
TPC-H: Total hot run time: 33098 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66af45e36dc073868e729f7e628c04210278b2db, data reload: false

------ Round 1 ----------------------------------
q1	17669	5314	5231	5231
q2	2049	310	202	202
q3	10487	1346	760	760
q4	10316	803	312	312
q5	9138	2116	1976	1976
q6	226	181	149	149
q7	882	738	604	604
q8	9264	1465	1128	1128
q9	5478	4890	4815	4815
q10	6852	1985	1577	1577
q11	507	314	276	276
q12	398	381	219	219
q13	17815	4070	3235	3235
q14	241	232	216	216
q15	905	841	818	818
q16	680	687	607	607
q17	792	953	505	505
q18	6833	7170	7244	7170
q19	1396	1090	633	633
q20	410	363	233	233
q21	2944	2285	2128	2128
q22	385	329	304	304
Total cold run time: 105667 ms
Total hot run time: 33098 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5630	5631	5475	5475
q2	265	349	260	260
q3	2429	2937	2495	2495
q4	1633	1871	1569	1569
q5	4746	4576	4520	4520
q6	224	193	149	149
q7	2103	1982	1766	1766
q8	2567	2439	2428	2428
q9	7588	7520	7442	7442
q10	2811	2820	2399	2399
q11	539	449	439	439
q12	618	706	584	584
q13	3560	4010	3236	3236
q14	272	286	268	268
q15	831	802	783	783
q16	632	681	639	639
q17	1094	1292	1313	1292
q18	7493	7278	7433	7278
q19	838	799	799	799
q20	1954	2053	1878	1878
q21	4485	4274	4155	4155
q22	587	557	522	522
Total cold run time: 52899 ms
Total hot run time: 50376 ms

@doris-robot

doris-robot commented Feb 1, 2026

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66af45e36dc073868e729f7e628c04210278b2db, data reload: false

query1	0.04	0.05	0.05
query2	0.14	0.07	0.07
query3	0.33	0.08	0.08
query4	1.60	0.11	0.11
query5	0.27	0.29	0.26
query6	1.14	0.68	0.67
query7	0.03	0.02	0.03
query8	0.08	0.06	0.06
query9	0.61	0.50	0.49
query10	0.55	0.56	0.55
query11	0.27	0.14	0.13
query12	0.27	0.14	0.14
query13	0.64	0.62	0.60
query14	0.99	0.98	0.98
query15	0.92	0.82	0.82
query16	0.40	0.40	0.40
query17	1.07	1.07	1.07
query18	0.25	0.22	0.23
query19	1.87	1.86	1.89
query20	0.02	0.02	0.01
query21	15.40	0.29	0.24
query22	4.93	0.11	0.10
query23	15.38	0.41	0.22
query24	2.44	0.51	0.31
query25	0.10	0.10	0.09
query26	0.19	0.18	0.19
query27	0.11	0.09	0.09
query28	3.56	1.20	0.99
query29	12.52	4.11	3.26
query30	0.35	0.13	0.12
query31	2.81	0.72	0.45
query32	3.24	0.67	0.53
query33	3.04	3.11	3.06
query34	15.99	5.24	4.50
query35	4.51	4.47	4.51
query36	0.62	0.50	0.50
query37	0.26	0.09	0.09
query38	0.21	0.06	0.06
query39	0.08	0.04	0.04
query40	0.20	0.17	0.17
query41	0.14	0.07	0.07
query42	0.08	0.04	0.04
query43	0.08	0.06	0.07
Total cold run time: 97.73 s
Total hot run time: 28.23 s

@doris-robot

doris-robot commented Feb 1, 2026

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 5.31% (6/113) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.51% (19287/36727)
Line Coverage 35.98% (179204/498064)
Region Coverage 32.41% (139079/429138)
Branch Coverage 33.34% (60149/180390)

@hello-stephen

hello-stephen commented Feb 1, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 5.31% (6/113) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.99% (20518/36002)
Line Coverage 39.92% (198376/496887)
Region Coverage 36.58% (158590/433548)
Branch Coverage 37.25% (67460/181122)

1 similar comment
@hello-stephen

hello-stephen commented Feb 2, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 5.31% (6/113) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 56.99% (20518/36002)
Line Coverage 39.92% (198376/496887)
Region Coverage 36.58% (158590/433548)
Branch Coverage 37.25% (67460/181122)

@liaoxin01
[feature](cloud) Support packed file for delete bitmap storage
3134ad0 
- Add packed file support for delete bitmap writer/reader
- Add write_file_cache flag to PackedAppendContext to control file cache behavior
- Delete bitmap files do not use file cache to match original behavior
- Add regression tests for packed delete bitmap scenarios
@liaoxin01
liaoxin01 force-pushed the feature/delete-bitmap-packed-file branch from 66af45e to 3134ad0 Compare February 3, 2026 02:37
@liaoxin01

liaoxin01 commented Feb 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@liaoxin01
liaoxin01 marked this pull request as draft February 3, 2026 03:06
@hello-stephen

hello-stephen commented Feb 3, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.29% (1792/2260)
Line Coverage 64.81% (31860/49158)
Region Coverage 65.50% (15903/24280)
Branch Coverage 56.06% (8453/15078)

@doris-robot

doris-robot commented Feb 3, 2026

Copy link
Copy Markdown
TPC-H: Total hot run time: 32107 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3134ad0282136b10babb3050e2039ee4622dc26f, data reload: false

------ Round 1 ----------------------------------
q1	17670	5332	5124	5124
q2	2031	326	189	189
q3	10616	1346	756	756
q4	10324	835	329	329
q5	9560	2183	1951	1951
q6	223	180	146	146
q7	888	733	601	601
q8	9276	1434	1137	1137
q9	5374	4866	4864	4864
q10	6869	1951	1588	1588
q11	547	297	272	272
q12	392	381	226	226
q13	17830	4106	3222	3222
q14	247	239	215	215
q15	909	836	817	817
q16	699	678	631	631
q17	643	820	530	530
q18	6795	6525	6511	6511
q19	1317	994	625	625
q20	399	338	239	239
q21	2801	2144	1859	1859
q22	360	317	275	275
Total cold run time: 105770 ms
Total hot run time: 32107 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5343	5308	5304	5304
q2	255	331	251	251
q3	2161	2703	2244	2244
q4	1384	1754	1307	1307
q5	4262	4170	4635	4170
q6	238	195	154	154
q7	2165	1979	1912	1912
q8	2618	2535	2389	2389
q9	7491	7641	7632	7632
q10	2857	2965	2554	2554
q11	571	468	459	459
q12	736	745	671	671
q13	4049	4454	3492	3492
q14	288	323	296	296
q15	875	879	863	863
q16	720	741	698	698
q17	1172	1443	1396	1396
q18	8120	7878	7881	7878
q19	849	828	866	828
q20	1975	2047	1893	1893
q21	4566	4230	4054	4054
q22	564	523	496	496
Total cold run time: 53259 ms
Total hot run time: 50941 ms

@liaoxin01
liaoxin01 force-pushed the feature/delete-bitmap-packed-file branch 2 times, most recently from 3740e9c to c29a8bc Compare February 3, 2026 05:17
@liaoxin01
liaoxin01 marked this pull request as ready for review February 3, 2026 05:17
@liaoxin01

liaoxin01 commented Feb 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

hello-stephen commented Feb 3, 2026

Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 24.64% (51/207) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.30% (1793/2261)
Line Coverage 64.59% (31882/49362)
Region Coverage 65.26% (15908/24376)
Branch Coverage 55.83% (8453/15140)

@liaoxin01
liaoxin01 force-pushed the feature/delete-bitmap-packed-file branch from c29a8bc to df6b98a Compare February 3, 2026 06:19
@liaoxin01

liaoxin01 commented Feb 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Feb 3, 2026

Copy link
Copy Markdown
TPC-H: Total hot run time: 32231 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit df6b98ac6d3d5bf9ec282ca6f7469e0e338dff94, data reload: false

------ Round 1 ----------------------------------
q1	17673	5235	5072	5072
q2	2037	305	202	202
q3	10233	1313	747	747
q4	10227	895	315	315
q5	7548	2151	1948	1948
q6	201	183	148	148
q7	920	732	598	598
q8	9264	1434	1199	1199
q9	5327	4827	4892	4827
q10	6837	1970	1573	1573
q11	523	292	290	290
q12	337	375	218	218
q13	17793	4063	3232	3232
q14	246	244	219	219
q15	900	841	817	817
q16	668	677	618	618
q17	680	852	429	429
q18	6731	6567	7673	6567
q19	1334	1039	651	651
q20	430	384	267	267
q21	2919	2277	2005	2005
q22	382	315	289	289
Total cold run time: 103210 ms
Total hot run time: 32231 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5592	5754	5526	5526
q2	285	381	268	268
q3	2445	2909	2426	2426
q4	1462	1839	1516	1516
q5	4682	4773	4547	4547
q6	255	185	137	137
q7	2006	1948	1824	1824
q8	2541	2589	2357	2357
q9	7552	7702	7566	7566
q10	2919	2978	2488	2488
q11	535	461	443	443
q12	614	711	551	551
q13	3536	4059	3212	3212
q14	276	287	269	269
q15	843	799	794	794
q16	653	696	651	651
q17	1090	1263	1298	1263
q18	7673	7334	7150	7150
q19	856	808	814	808
q20	2010	2086	1891	1891
q21	4576	4258	4126	4126
q22	593	550	533	533
Total cold run time: 52994 ms
Total hot run time: 50346 ms

@doris-robot

doris-robot commented Feb 3, 2026

Copy link
Copy Markdown
ClickBench: Total hot run time: 27.73 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit df6b98ac6d3d5bf9ec282ca6f7469e0e338dff94, data reload: false

query1	0.05	0.04	0.04
query2	0.12	0.07	0.07
query3	0.31	0.08	0.08
query4	1.61	0.11	0.09
query5	0.26	0.25	0.25
query6	1.15	0.64	0.63
query7	0.03	0.03	0.02
query8	0.07	0.06	0.06
query9	0.60	0.50	0.50
query10	0.55	0.55	0.54
query11	0.26	0.14	0.13
query12	0.26	0.14	0.14
query13	0.63	0.62	0.62
query14	0.99	0.98	0.98
query15	0.91	0.81	0.83
query16	0.39	0.38	0.37
query17	1.05	1.06	1.05
query18	0.24	0.22	0.22
query19	1.97	1.74	1.88
query20	0.02	0.01	0.02
query21	15.38	0.28	0.24
query22	4.93	0.09	0.10
query23	15.39	0.39	0.23
query24	2.37	0.46	0.30
query25	0.10	0.10	0.09
query26	0.18	0.18	0.16
query27	0.10	0.09	0.10
query28	3.40	1.14	0.97
query29	12.55	4.01	3.26
query30	0.33	0.12	0.11
query31	2.79	0.69	0.44
query32	3.23	0.63	0.50
query33	3.06	3.03	3.11
query34	15.91	5.07	4.44
query35	4.40	4.41	4.47
query36	0.63	0.49	0.49
query37	0.25	0.08	0.08
query38	0.21	0.06	0.06
query39	0.08	0.05	0.05
query40	0.20	0.20	0.17
query41	0.12	0.06	0.07
query42	0.08	0.05	0.05
query43	0.08	0.07	0.06
Total cold run time: 97.24 s
Total hot run time: 27.73 s

@doris-robot

doris-robot commented Feb 3, 2026

Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 5.31% (6/113) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.54% (19356/36842)
Line Coverage 36.01% (179735/499177)
Region Coverage 32.41% (139520/430461)
Branch Coverage 33.36% (60332/180826)

@hello-stephen

hello-stephen commented Feb 3, 2026

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 7.08% (8/113) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.32% (26472/36106)
Line Coverage 56.37% (280726/497979)
Region Coverage 53.98% (234756/434875)
Branch Coverage 55.66% (101049/181554)

@hello-stephen

hello-stephen commented Feb 4, 2026

Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage `` 🎉
Increment coverage report
Complete coverage report

@liaoxin01
liaoxin01 requested review from luwei16 and mymeiyi February 4, 2026 08:56
@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Feb 5, 2026
@github-actions

github-actions Bot commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actions Bot commented Feb 5, 2026

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@liaoxin01 liaoxin01 changed the title [feature](cloud) Support packed file for delete bitmap storage [Opt](cloud) Support packed file for delete bitmap storage Mar 3, 2026
sollhui
sollhui approved these changes Mar 3, 2026
View reviewed changes

@sollhui sollhui left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01
liaoxin01 merged commit 59e5f6b into apache:master Mar 3, 2026
30 of 33 checks passed
@liaoxin01
liaoxin01 deleted the feature/delete-bitmap-packed-file branch March 3, 2026 13:14
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Mar 3, 2026
@liaoxin01
[Opt](cloud) Support packed file for delete bitmap storage (apache#60411
cb27eb7 
)

Problem Summary:
This PR adds packed file support for delete bitmap storage in cloud
mode.
- Add packed file support for delete bitmap writer/reader
- Add write_file_cache flag to PackedAppendContext to control file cache
behavior
- Delete bitmap files do not use file cache to match original behavior
- Add regression tests for packed delete bitmap scenarios
@liaoxin01 liaoxin01 mentioned this pull request Mar 3, 2026
Merged
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Mar 3, 2026
@liaoxin01
[Opt](cloud) Support packed file for delete bitmap storage (apache#60411
575a29d 
)

Problem Summary:
This PR adds packed file support for delete bitmap storage in cloud
mode.
- Add packed file support for delete bitmap writer/reader
- Add write_file_cache flag to PackedAppendContext to control file cache
behavior
- Delete bitmap files do not use file cache to match original behavior
- Add regression tests for packed delete bitmap scenarios
yiguolei pushed a commit that referenced this pull request Mar 12, 2026
@liaoxin01
[branch-4.0] [Opt](cloud) Support packed file for delete bitmap stora…
06cb84c 
…ge (#61007)

pick：#60411
@liaoxin01 liaoxin01 mentioned this pull request Oct 27, 2026
Merged
3 of 16 tasks
liaoxin01 added a commit that referenced this pull request Oct 28, 2026
@liaoxin01
[fix](packed-file) Honor write_file_cache option for packed small fil…
bf58c2c 
…es (#69693)

### What problem does this PR solve?

Issue Number: None

Related PR: #59892, #60411, #64995

Problem Summary:

When packed file is enabled, the first segment file (`.dat`) and the
index file (`.idx`) of a rowset are written through `PackedFileSystem`.
The packed path decides whether to cache each small file from
`PackedAppendContext::write_file_cache`, which defaults to `true`.
`RowsetWriterContext` never sets this field.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot
Copilot review effort
Applies to this pull request for everyone.Learn more about Copilot code review.
Copilot left review comments
@gavinchou gavinchou gavinchou approved these changes
@sollhui sollhui sollhui approved these changes
@dataroaring dataroaring Awaiting requested review from dataroaring
@w41ter w41ter Awaiting requested review from w41ter
@mymeiyi mymeiyi Awaiting requested review from mymeiyi
@luwei16 luwei16 Awaiting requested review from luwei16 luwei16 is a code owner

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.5-merged dev/4.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@liaoxin01 @hello-stephen @doris-robot @gavinchou @sollhui @yiguolei