[fix](backup) Support backup meta/job info exceeds 2GB #55608

w41ter · 2025-09-03T02:49:46Z

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

The original implementation does not support deserializing the backupMeta and backupJobInfo because the size exceeds 2GB, since Java byte[] max length is Integer.MAX_VALUE.

This PR compresses the bytes before loading them into the heap.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2025-09-03T02:49:52Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

w41ter · 2025-09-03T02:50:25Z

run buildall

dataroaring

LGTM

github-actions · 2025-09-03T02:59:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2025-09-03T02:59:03Z

PR approved by anyone and no changes requested.

w41ter · 2025-09-03T03:01:40Z

run buildall

dataroaring · 2025-09-03T03:02:14Z

fe/fe-core/src/main/java/org/apache/doris/service/FrontendServiceImpl.java

+            long snapshotSize = snapshot.getMetaSize() + snapshot.getJobInfoSize();
+            if (metaSize + jobInfoSize >= Integer.MAX_VALUE) {
+                String msg = String.format(
+                        "Snapshot %s is too large (%d bytes > 2GB). Please enable compression to continue.",


Please give specific howtos about enable compression.

hello-stephen · 2025-09-03T03:59:34Z

FE UT Coverage Report

Increment line coverage 3.51% (2/57) 🎉
Increment coverage report
Complete coverage report

doris-robot · 2025-09-03T04:21:50Z

TPC-H: Total hot run time: 34324 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2e9cc921de9d1d0d563e3b97abbc9b4edbf23b2, data reload: false

------ Round 1 ----------------------------------
q1	17598	5269	5102	5102
q2	2003	345	218	218
q3	10207	1281	737	737
q4	10250	1039	550	550
q5	7522	2456	2355	2355
q6	194	172	136	136
q7	956	757	633	633
q8	9337	1324	1139	1139
q9	7069	5208	5181	5181
q10	6983	2386	1967	1967
q11	496	309	287	287
q12	385	374	229	229
q13	17785	3636	3056	3056
q14	237	243	251	243
q15	599	501	492	492
q16	442	442	380	380
q17	608	887	366	366
q18	7830	7292	7006	7006
q19	1103	972	579	579
q20	366	355	239	239
q21	3901	3252	2441	2441
q22	1041	1047	988	988
Total cold run time: 106912 ms
Total hot run time: 34324 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5207	5137	5146	5137
q2	256	339	235	235
q3	2154	2685	2314	2314
q4	1368	1813	1374	1374
q5	4262	4512	4532	4512
q6	223	189	133	133
q7	2043	1997	1912	1912
q8	2764	2560	2545	2545
q9	7379	7434	7440	7434
q10	3137	3355	2851	2851
q11	627	521	504	504
q12	721	868	640	640
q13	3587	4009	3421	3421
q14	280	296	297	296
q15	537	488	472	472
q16	458	491	459	459
q17	1209	1535	1450	1450
q18	7826	7697	7578	7578
q19	829	796	855	796
q20	1888	1982	1806	1806
q21	4893	4288	4433	4288
q22	1112	1082	1004	1004
Total cold run time: 52760 ms
Total hot run time: 51161 ms

doris-robot · 2025-09-03T04:33:40Z

TPC-DS: Total hot run time: 187207 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2e9cc921de9d1d0d563e3b97abbc9b4edbf23b2, data reload: false

query1	1064	423	406	406
query2	6554	1747	1771	1747
query3	6750	226	227	226
query4	26145	23116	22887	22887
query5	4406	665	528	528
query6	346	254	251	251
query7	4660	509	306	306
query8	314	260	263	260
query9	8670	2944	2980	2944
query10	507	349	316	316
query11	15799	14933	14716	14716
query12	191	133	125	125
query13	1705	589	457	457
query14	9601	5803	5889	5803
query15	224	189	175	175
query16	7685	695	507	507
query17	1259	770	654	654
query18	2098	447	416	416
query19	220	195	181	181
query20	136	128	123	123
query21	213	133	119	119
query22	4182	4185	3994	3994
query23	33950	33134	33089	33089
query24	8126	2378	2381	2378
query25	576	519	445	445
query26	1239	281	167	167
query27	2687	530	363	363
query28	4339	2306	2277	2277
query29	797	603	491	491
query30	319	226	200	200
query31	938	830	755	755
query32	92	85	80	80
query33	573	407	343	343
query34	806	850	529	529
query35	808	835	764	764
query36	984	1038	909	909
query37	119	111	90	90
query38	4088	4066	4018	4018
query39	1496	1414	1402	1402
query40	230	131	127	127
query41	67	62	66	62
query42	128	123	123	123
query43	522	501	476	476
query44	1369	864	858	858
query45	185	174	170	170
query46	860	1028	657	657
query47	1784	1871	1733	1733
query48	399	444	333	333
query49	756	507	424	424
query50	655	671	405	405
query51	4164	4254	4156	4156
query52	120	128	108	108
query53	243	274	209	209
query54	621	608	547	547
query55	92	95	94	94
query56	359	330	326	326
query57	1197	1224	1125	1125
query58	303	287	286	286
query59	2623	2723	2601	2601
query60	365	369	355	355
query61	167	159	161	159
query62	799	741	667	667
query63	229	203	197	197
query64	4428	1132	916	916
query65	4315	4214	4271	4214
query66	1111	435	400	400
query67	15671	15318	15525	15318
query68	9353	916	590	590
query69	488	336	291	291
query70	1300	1152	1138	1138
query71	565	360	338	338
query72	5790	4897	4918	4897
query73	728	588	370	370
query74	9272	8951	8987	8951
query75	4409	3094	2642	2642
query76	3794	1138	804	804
query77	997	415	337	337
query78	9536	9856	8919	8919
query79	1736	824	609	609
query80	687	594	526	526
query81	539	257	226	226
query82	250	145	114	114
query83	303	275	249	249
query84	302	113	96	96
query85	860	467	422	422
query86	343	326	308	308
query87	4328	4351	4180	4180
query88	2845	2207	2258	2207
query89	400	343	297	297
query90	2070	233	228	228
query91	168	166	139	139
query92	94	77	72	72
query93	1122	995	667	667
query94	697	408	325	325
query95	423	339	336	336
query96	501	590	288	288
query97	2678	2718	2606	2606
query98	243	220	218	218
query99	1447	1420	1330	1330
Total cold run time: 276696 ms
Total hot run time: 187207 ms

doris-robot · 2025-09-03T04:39:20Z

ClickBench: Total hot run time: 32.36 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d2e9cc921de9d1d0d563e3b97abbc9b4edbf23b2, data reload: false

query1	0.06	0.05	0.04
query2	0.10	0.05	0.06
query3	0.25	0.08	0.09
query4	1.61	0.12	0.11
query5	0.47	0.42	0.41
query6	1.17	0.65	0.66
query7	0.03	0.03	0.03
query8	0.05	0.05	0.05
query9	0.60	0.55	0.54
query10	0.59	0.58	0.57
query11	0.18	0.11	0.11
query12	0.16	0.12	0.13
query13	0.63	0.63	0.62
query14	0.81	0.84	0.85
query15	0.89	0.85	0.88
query16	0.40	0.41	0.42
query17	1.07	1.07	1.07
query18	0.23	0.20	0.21
query19	1.94	1.82	1.89
query20	0.02	0.02	0.01
query21	15.40	0.96	0.58
query22	0.78	1.07	0.61
query23	15.15	1.36	0.59
query24	6.89	2.07	0.45
query25	0.32	0.26	0.06
query26	0.56	0.17	0.13
query27	0.06	0.06	0.06
query28	9.15	0.95	0.44
query29	12.60	3.91	3.25
query30	3.10	3.06	3.01
query31	2.82	0.60	0.39
query32	3.23	0.56	0.47
query33	3.21	3.10	3.18
query34	16.12	5.48	4.87
query35	4.93	4.89	4.95
query36	0.74	0.51	0.51
query37	0.10	0.08	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.04	0.03
query42	0.04	0.03	0.03
query43	0.04	0.03	0.04
Total cold run time: 106.86 s
Total hot run time: 32.36 s

hello-stephen · 2025-09-03T07:27:16Z

FE Regression Coverage Report

Increment line coverage 3.51% (2/57) 🎉
Increment coverage report
Complete coverage report

w41ter · 2025-09-03T11:20:40Z

run buildall

doris-robot · 2025-09-03T11:51:55Z

TPC-H: Total hot run time: 33996 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 12a642b30cfabb9d0dcaec5152cd4911054e37dd, data reload: false

------ Round 1 ----------------------------------
q1	17621	5228	5083	5083
q2	2007	308	209	209
q3	10279	1244	714	714
q4	10217	1021	548	548
q5	7529	2484	2336	2336
q6	188	170	139	139
q7	949	751	638	638
q8	9350	1346	1077	1077
q9	6845	5110	5188	5110
q10	6924	2391	1984	1984
q11	502	302	285	285
q12	362	366	234	234
q13	17795	3661	3025	3025
q14	255	240	224	224
q15	561	502	499	499
q16	427	432	406	406
q17	605	853	352	352
q18	7379	7195	7015	7015
q19	1339	980	549	549
q20	336	345	232	232
q21	3792	2504	2326	2326
q22	1085	1012	1011	1011
Total cold run time: 106347 ms
Total hot run time: 33996 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5231	5180	5130	5130
q2	255	354	227	227
q3	2156	2657	2301	2301
q4	1320	1772	1332	1332
q5	4236	4208	4549	4208
q6	231	178	134	134
q7	2055	1905	1919	1905
q8	2744	2678	2562	2562
q9	7450	7502	7247	7247
q10	3094	3385	2846	2846
q11	574	534	489	489
q12	688	752	649	649
q13	3678	3850	3269	3269
q14	314	299	344	299
q15	528	495	473	473
q16	451	569	465	465
q17	1164	1566	1559	1559
q18	7690	7645	7411	7411
q19	819	784	809	784
q20	2015	2047	1916	1916
q21	5238	4437	4412	4412
q22	1105	1020	989	989
Total cold run time: 53036 ms
Total hot run time: 50607 ms

doris-robot · 2025-09-03T12:03:36Z

TPC-DS: Total hot run time: 187081 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 12a642b30cfabb9d0dcaec5152cd4911054e37dd, data reload: false

query1	1079	470	400	400
query2	6558	1752	1797	1752
query3	6749	233	227	227
query4	26194	23653	23036	23036
query5	4410	659	525	525
query6	332	248	223	223
query7	4643	540	295	295
query8	289	253	238	238
query9	8662	2878	2914	2878
query10	512	373	319	319
query11	15849	15331	15012	15012
query12	173	121	123	121
query13	1672	572	436	436
query14	8667	5908	5726	5726
query15	220	197	182	182
query16	7168	648	466	466
query17	1238	753	626	626
query18	2012	449	379	379
query19	210	200	175	175
query20	134	127	125	125
query21	214	134	115	115
query22	4197	4354	3918	3918
query23	33660	32966	32945	32945
query24	8277	2379	2450	2379
query25	573	528	452	452
query26	1245	286	169	169
query27	2739	519	361	361
query28	4349	2265	2240	2240
query29	819	612	491	491
query30	291	239	203	203
query31	916	831	750	750
query32	89	88	78	78
query33	600	415	373	373
query34	808	879	538	538
query35	830	907	761	761
query36	979	1023	899	899
query37	134	113	94	94
query38	4048	4040	4004	4004
query39	1507	1447	1420	1420
query40	225	136	127	127
query41	64	64	60	60
query42	131	118	117	117
query43	561	491	465	465
query44	1348	878	862	862
query45	185	180	173	173
query46	863	1025	656	656
query47	1785	1832	1734	1734
query48	392	431	323	323
query49	765	510	425	425
query50	658	701	411	411
query51	4289	4223	4192	4192
query52	122	116	108	108
query53	251	267	208	208
query54	619	602	553	553
query55	97	93	97	93
query56	340	327	332	327
query57	1193	1204	1137	1137
query58	293	289	308	289
query59	2679	2758	2564	2564
query60	368	365	351	351
query61	185	163	166	163
query62	802	749	685	685
query63	243	198	209	198
query64	4652	1259	957	957
query65	4243	4239	4212	4212
query66	1216	465	378	378
query67	15334	15211	15099	15099
query68	8089	944	591	591
query69	508	349	310	310
query70	1335	1167	1123	1123
query71	581	347	325	325
query72	5923	5061	5402	5061
query73	775	692	363	363
query74	9189	9153	8999	8999
query75	3772	3121	2630	2630
query76	3620	1160	769	769
query77	808	425	335	335
query78	9774	9714	8953	8953
query79	2417	814	590	590
query80	666	579	515	515
query81	503	267	229	229
query82	494	148	115	115
query83	275	263	238	238
query84	282	112	88	88
query85	932	484	439	439
query86	401	322	302	302
query87	4306	4285	4171	4171
query88	3678	2241	2230	2230
query89	416	368	309	309
query90	1857	228	233	228
query91	177	169	132	132
query92	94	81	74	74
query93	1836	1013	652	652
query94	705	425	376	376
query95	417	340	333	333
query96	487	588	284	284
query97	2658	2703	2609	2609
query98	258	219	211	211
query99	1441	1415	1299	1299
Total cold run time: 275623 ms
Total hot run time: 187081 ms

doris-robot · 2025-09-03T12:09:15Z

ClickBench: Total hot run time: 33.02 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 12a642b30cfabb9d0dcaec5152cd4911054e37dd, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.26	0.08	0.08
query4	1.62	0.12	0.12
query5	0.45	0.42	0.40
query6	1.16	0.64	0.65
query7	0.03	0.03	0.02
query8	0.05	0.04	0.05
query9	0.60	0.52	0.54
query10	0.58	0.58	0.57
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.63	0.61
query14	0.80	0.84	0.84
query15	0.88	0.87	0.86
query16	0.39	0.40	0.39
query17	1.05	1.11	1.09
query18	0.22	0.20	0.20
query19	1.91	1.83	1.86
query20	0.02	0.01	0.01
query21	15.40	0.98	0.59
query22	0.79	1.19	0.82
query23	14.80	1.34	0.64
query24	6.70	0.83	1.34
query25	0.61	0.20	0.10
query26	0.69	0.16	0.14
query27	0.07	0.06	0.06
query28	10.12	0.95	0.43
query29	12.61	3.86	3.25
query30	3.09	3.05	2.99
query31	2.83	0.61	0.40
query32	3.23	0.56	0.46
query33	3.03	3.25	3.11
query34	16.14	5.48	4.85
query35	4.96	4.94	4.93
query36	0.71	0.50	0.50
query37	0.10	0.08	0.07
query38	0.06	0.05	0.04
query39	0.04	0.03	0.02
query40	0.18	0.15	0.15
query41	0.09	0.03	0.03
query42	0.03	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 107.43 s
Total hot run time: 33.02 s

hello-stephen · 2025-09-03T14:09:29Z

FE Regression Coverage Report

Increment line coverage 3.51% (2/57) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2025-09-04T05:33:32Z

FE Regression Coverage Report

Increment line coverage 3.51% (2/57) 🎉
Increment coverage report
Complete coverage report

dataroaring

LGTM

github-actions · 2025-09-06T08:45:21Z

PR approved by at least one committer and no changes requested.

…5608 (#55867) cherry-pick #55608

w41ter added the dev/3.1.x label Sep 3, 2025

dataroaring previously approved these changes Sep 3, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 3, 2025

github-actions bot added the reviewed label Sep 3, 2025

w41ter dismissed dataroaring’s stale review via f392160 September 3, 2025 03:00

w41ter force-pushed the backup/support_large_meta_file branch from cea66f3 to f392160 Compare September 3, 2025 03:00

[fix](backup) Support backup meta/job info exceeds 2GB

d2e9cc9

w41ter force-pushed the backup/support_large_meta_file branch from f392160 to d2e9cc9 Compare September 3, 2025 03:01

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 3, 2025

dataroaring reviewed Sep 3, 2025

View reviewed changes

fixup

12a642b

w41ter requested a review from dataroaring September 4, 2025 10:59

dataroaring approved these changes Sep 6, 2025

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 6, 2025

koarz approved these changes Sep 6, 2025

View reviewed changes

dataroaring merged commit 80147f8 into apache:master Sep 6, 2025
26 of 28 checks passed

github-actions bot added the dev/3.1.x-conflict label Sep 6, 2025

wenzhenghu pushed a commit to wenzhenghu/doris that referenced this pull request Sep 8, 2025

[fix](backup) Support backup meta/job info exceeds 2GB (apache#55608)

a8be8d6

w41ter deleted the backup/support_large_meta_file branch September 8, 2025 02:50

w41ter added a commit to w41ter/incubator-doris that referenced this pull request Sep 10, 2025

[fix](backup) Support backup meta/job info exceeds 2GB (apache#55608)

cb002cb

w41ter mentioned this pull request Sep 10, 2025

branch-3.1: [fix](backup) Support backup meta/job info exceeds 2GB #55608 #55867

Merged

w41ter removed the dev/3.1.x-conflict label Sep 10, 2025

morrySnow pushed a commit that referenced this pull request Sep 11, 2025

branch-3.1: [fix](backup) Support backup meta/job info exceeds 2GB #5…

33de374

…5608 (#55867) cherry-pick #55608

morrySnow added dev/3.1.1-merged and removed dev/3.1.x labels Sep 11, 2025

morrySnow mentioned this pull request Sep 22, 2025

Release Note 3.1.1 #56289

Closed

[fix](backup) Support backup meta/job info exceeds 2GB #55608

[fix](backup) Support backup meta/job info exceeds 2GB #55608

Uh oh!

Conversation

w41ter commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Sep 3, 2025

Uh oh!

w41ter commented Sep 3, 2025

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

github-actions bot commented Sep 3, 2025

Uh oh!

w41ter commented Sep 3, 2025

Uh oh!

dataroaring Sep 3, 2025

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Sep 3, 2025

FE UT Coverage Report

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

hello-stephen commented Sep 3, 2025

FE Regression Coverage Report

Uh oh!

w41ter commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

doris-robot commented Sep 3, 2025

Uh oh!

hello-stephen commented Sep 3, 2025

FE Regression Coverage Report

Uh oh!

hello-stephen commented Sep 4, 2025

FE Regression Coverage Report

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

w41ter commented Sep 3, 2025 •

edited

Loading