Skip to content

[fix](compaction) skip tablets with compaction score 0 during compaction #55550

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Sep 2, 2025
Merged

[fix](compaction) skip tablets with compaction score 0 during compaction #55550

dataroaring merged 1 commit into from
Sep 2, 2025

Conversation

@luwei16
Copy link
Contributor

@luwei16 luwei16 commented Sep 1, 2025

No description provided.

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 1, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@luwei16
Copy link
Contributor Author

luwei16 commented Sep 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Sep 1, 2025

TPC-H: Total hot run time: 33736 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6430ff38c6238f916348f19e58b0aa6474a8e666, data reload: false

------ Round 1 ----------------------------------
q1	17644	5175	5029	5029
q2	1990	332	226	226
q3	10245	1266	702	702
q4	10239	1029	527	527
q5	7543	2422	2297	2297
q6	193	175	140	140
q7	945	762	638	638
q8	9326	1333	1069	1069
q9	6886	5128	5234	5128
q10	6935	2389	1979	1979
q11	492	323	284	284
q12	347	360	229	229
q13	17780	3640	3000	3000
q14	240	247	219	219
q15	590	534	478	478
q16	431	438	381	381
q17	605	861	353	353
q18	7412	6970	7087	6970
q19	1425	952	551	551
q20	338	331	237	237
q21	3881	2526	2322	2322
q22	1070	1025	977	977
Total cold run time: 106557 ms
Total hot run time: 33736 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5186	5102	5085	5085
q2	254	343	233	233
q3	2161	2683	2257	2257
q4	1318	1749	1330	1330
q5	4229	4178	4552	4178
q6	220	178	134	134
q7	2061	1961	1869	1869
q8	2715	2567	2515	2515
q9	7276	7433	7299	7299
q10	3125	3312	2856	2856
q11	606	506	487	487
q12	670	794	667	667
q13	3528	3906	3351	3351
q14	326	318	288	288
q15	530	486	496	486
q16	468	501	446	446
q17	1195	1594	1391	1391
q18	7834	7520	7736	7520
q19	801	777	844	777
q20	1923	2037	1912	1912
q21	5072	4610	4304	4304
q22	1105	1052	994	994
Total cold run time: 52603 ms
Total hot run time: 50379 ms

@doris-robot
Copy link

doris-robot commented Sep 1, 2025

TPC-DS: Total hot run time: 187198 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6430ff38c6238f916348f19e58b0aa6474a8e666, data reload: false

query1	1032	461	407	407
query2	6580	1774	1745	1745
query3	6759	227	227	227
query4	26369	23488	23170	23170
query5	4387	652	513	513
query6	359	255	240	240
query7	4652	532	304	304
query8	325	280	258	258
query9	8754	2993	3027	2993
query10	520	359	293	293
query11	15974	15079	14892	14892
query12	174	126	121	121
query13	1689	540	436	436
query14	9452	5862	5828	5828
query15	225	197	175	175
query16	7558	651	498	498
query17	1237	761	696	696
query18	2040	445	343	343
query19	202	192	169	169
query20	135	133	129	129
query21	218	131	113	113
query22	4179	4108	3936	3936
query23	33742	33061	33175	33061
query24	8170	2417	2398	2398
query25	600	519	438	438
query26	1247	279	164	164
query27	2734	515	356	356
query28	4358	2339	2290	2290
query29	777	625	488	488
query30	296	217	200	200
query31	932	792	759	759
query32	99	82	80	80
query33	589	410	365	365
query34	824	856	530	530
query35	831	854	762	762
query36	984	1008	906	906
query37	130	108	88	88
query38	4045	4048	4064	4048
query39	1488	1424	1413	1413
query40	230	134	135	134
query41	64	62	62	62
query42	134	132	131	131
query43	538	511	477	477
query44	1349	874	859	859
query45	186	177	175	175
query46	862	1026	651	651
query47	1784	1783	1779	1779
query48	400	430	321	321
query49	748	510	423	423
query50	646	698	414	414
query51	4167	4197	4087	4087
query52	118	115	109	109
query53	246	268	207	207
query54	612	606	557	557
query55	94	96	96	96
query56	341	329	331	329
query57	1231	1208	1133	1133
query58	299	278	282	278
query59	2674	2791	2688	2688
query60	367	358	349	349
query61	170	196	158	158
query62	795	716	708	708
query63	245	196	200	196
query64	4484	1172	827	827
query65	4331	4255	4242	4242
query66	1143	457	354	354
query67	15459	15323	14901	14901
query68	7433	926	582	582
query69	514	328	305	305
query70	1194	1143	1141	1141
query71	514	355	321	321
query72	5830	5194	5241	5194
query73	785	640	363	363
query74	9026	9210	8854	8854
query75	3148	3140	2587	2587
query76	3243	1279	761	761
query77	499	412	350	350
query78	9663	9734	8846	8846
query79	2483	858	606	606
query80	646	590	520	520
query81	507	254	224	224
query82	486	146	111	111
query83	272	267	252	252
query84	263	110	94	94
query85	915	470	438	438
query86	393	343	304	304
query87	4279	4301	4160	4160
query88	3773	2227	2206	2206
query89	416	341	299	299
query90	1915	246	226	226
query91	168	183	133	133
query92	89	81	74	74
query93	1992	996	653	653
query94	728	433	336	336
query95	429	347	334	334
query96	491	587	290	290
query97	2629	2687	2595	2595
query98	252	217	220	217
query99	1371	1417	1345	1345
Total cold run time: 274729 ms
Total hot run time: 187198 ms

@doris-robot
Copy link

doris-robot commented Sep 1, 2025

ClickBench: Total hot run time: 33.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6430ff38c6238f916348f19e58b0aa6474a8e666, data reload: false

query1	0.05	0.05	0.05
query2	0.10	0.06	0.07
query3	0.26	0.08	0.08
query4	1.61	0.12	0.12
query5	0.45	0.43	0.42
query6	1.19	0.64	0.67
query7	0.04	0.03	0.03
query8	0.06	0.05	0.05
query9	0.62	0.53	0.54
query10	0.58	0.59	0.58
query11	0.17	0.11	0.11
query12	0.15	0.13	0.13
query13	0.63	0.63	0.62
query14	0.79	0.83	0.86
query15	0.87	0.89	0.86
query16	0.39	0.40	0.39
query17	1.06	1.06	1.08
query18	0.22	0.20	0.20
query19	1.99	1.89	1.85
query20	0.02	0.01	0.01
query21	15.42	0.95	0.58
query22	0.78	1.30	0.99
query23	14.74	1.33	0.64
query24	6.58	2.57	0.71
query25	0.46	0.22	0.09
query26	0.57	0.16	0.14
query27	0.06	0.05	0.06
query28	10.05	0.92	0.43
query29	12.59	3.91	3.26
query30	3.04	3.04	2.98
query31	2.84	0.58	0.40
query32	3.26	0.55	0.48
query33	3.01	3.13	3.09
query34	16.23	5.43	4.92
query35	4.95	4.94	4.94
query36	0.75	0.51	0.51
query37	0.11	0.07	0.07
query38	0.07	0.05	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 107.15 s
Total hot run time: 33.22 s

dataroaring
dataroaring approved these changes Sep 1, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 1, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Sep 1, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Sep 1, 2025

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Sep 1, 2025

BE UT Coverage Report

Increment line coverage 33.33% (1/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 51.66% (17129/33158)
Line Coverage 37.20% (156552/420847)
Region Coverage 31.86% (119493/375113)
Branch Coverage 33.21% (52485/158034)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 1, 2025

BE Regression && UT Coverage Report

Increment line coverage 33.33% (1/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.51% (22969/32574)
Line Coverage 56.83% (239083/420719)
Region Coverage 52.27% (198966/380630)
Branch Coverage 53.92% (85740/159015)

@hello-stephen
Copy link
Contributor

hello-stephen commented Sep 2, 2025

BE Regression && UT Coverage Report

Increment line coverage 33.33% (1/3) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 70.51% (22969/32574)
Line Coverage 56.82% (239069/420719)
Region Coverage 52.24% (198825/380630)
Branch Coverage 53.91% (85725/159015)

@dataroaring dataroaring merged commit e8cd011 into apache:master Sep 2, 2025
29 of 32 checks passed
github-actions bot pushed a commit that referenced this pull request Sep 2, 2025
@luwei16
[fix](compaction) skip tablets with compaction score 0 during compact…
71da8c0 
…ion (#55550)
@github-actions github-actions bot mentioned this pull request Sep 2, 2025
Merged
github-actions bot pushed a commit that referenced this pull request Sep 2, 2025
@luwei16
[fix](compaction) skip tablets with compaction score 0 during compact…
cbb2378 
…ion (#55550)
@github-actions github-actions bot mentioned this pull request Sep 2, 2025
Merged
morrySnow pushed a commit that referenced this pull request Sep 4, 2025
@github-actions @luwei16
branch-3.1: [fix](compaction) skip tablets with compaction score 0 du…
3a0bef9 
…ring compaction #55550 (#55570)

Cherry-picked from #55550

Co-authored-by: Luwei <[email protected]>
dataroaring pushed a commit that referenced this pull request Sep 4, 2025
@github-actions @luwei16
branch-3.0: [fix](compaction) skip tablets with compaction score 0 du…
6c27926 
…ring compaction #55550 (#55569)

Cherry-picked from #55550

Co-authored-by: Luwei <[email protected]>
@morrySnow morrySnow mentioned this pull request Sep 22, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.x dev/2.1.x-conflict dev/3.0.9-merged dev/3.1.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@luwei16 @hello-stephen @doris-robot @Yukang-Lian @dataroaring @morrySnow