Skip to content

[opt](iceberg)support read iceberg with tag/branch - #51272

Merged
morningman merged 18 commits into
apache:masterfrom
wuwenchi:opt-iceberg-branch-tag
Jun 6, 2025
Merged

[opt](iceberg)support read iceberg with tag/branch#51272
morningman merged 18 commits into
apache:masterfrom
wuwenchi:opt-iceberg-branch-tag

Conversation

@wuwenchi

@wuwenchi wuwenchi commented May 27, 2025 •
edited by morningman
Loading

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Related Issue: #48285

Problem Summary:

if an iceberg has a branch named branch1 and a tag named tag1, you can query branch1 or tag1 by these ways:

-- query 'branch1'
select * from tb FOR VERSION AS OF 'branch1';
select * from tb@branch(branch1);
select * from tb@branch('name'='branch1')

-- query 'tag1'
select * from tb FOR VERSION AS OF 'tag1';
select * from tb@tag(tag1);
select * from tb@tag('name'='tag1')

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

hello-stephen commented May 27, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi
wuwenchi force-pushed the opt-iceberg-branch-tag branch from 2124445 to b855d75 Compare May 27, 2025 02:08
@wuwenchi

wuwenchi commented May 28, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented May 28, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33951 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ff2f4ca2310d39dd63ba2c4b7450adab8ac8a66f, data reload: false

------ Round 1 ----------------------------------
q1	26291	5159	5019	5019
q2	2098	285	182	182
q3	10562	1259	738	738
q4	10259	1016	552	552
q5	7880	2369	2356	2356
q6	184	164	134	134
q7	904	745	638	638
q8	9315	1333	1126	1126
q9	6827	5079	5050	5050
q10	6882	2317	1887	1887
q11	515	286	275	275
q12	348	349	211	211
q13	17781	3677	3057	3057
q14	222	224	204	204
q15	522	496	487	487
q16	434	429	378	378
q17	624	866	377	377
q18	7764	7241	7210	7210
q19	1578	963	555	555
q20	346	335	223	223
q21	3862	3207	2334	2334
q22	1031	1009	958	958
Total cold run time: 116229 ms
Total hot run time: 33951 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5204	5109	5158	5109
q2	237	326	228	228
q3	2198	2642	2314	2314
q4	1339	1784	1425	1425
q5	4459	4429	4375	4375
q6	220	174	123	123
q7	2026	1910	1763	1763
q8	2566	2596	2543	2543
q9	7323	7133	6998	6998
q10	3080	3180	2753	2753
q11	588	542	504	504
q12	679	774	599	599
q13	3565	3873	3365	3365
q14	317	317	286	286
q15	526	473	473	473
q16	457	476	452	452
q17	1172	1538	1384	1384
q18	7721	7613	7416	7416
q19	848	877	927	877
q20	1996	1983	1805	1805
q21	4879	4523	4451	4451
q22	1111	1084	1029	1029
Total cold run time: 52511 ms
Total hot run time: 50272 ms

@doris-robot

doris-robot commented May 28, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 191897 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ff2f4ca2310d39dd63ba2c4b7450adab8ac8a66f, data reload: false

query1	1410	1073	1049	1049
query2	6064	1809	1807	1807
query3	10996	4627	4426	4426
query4	54607	26336	23479	23479
query5	5197	549	463	463
query6	336	225	204	204
query7	4926	529	287	287
query8	331	246	241	241
query9	5352	2616	2604	2604
query10	438	340	276	276
query11	14999	15112	14761	14761
query12	167	112	107	107
query13	1034	507	411	411
query14	10233	6339	6319	6319
query15	200	212	181	181
query16	7144	641	493	493
query17	1110	742	558	558
query18	1553	428	299	299
query19	191	186	172	172
query20	128	125	123	123
query21	207	131	105	105
query22	4392	4460	4390	4390
query23	34473	33164	33611	33164
query24	6605	2408	2418	2408
query25	466	454	401	401
query26	688	277	153	153
query27	2328	538	363	363
query28	3030	2169	2163	2163
query29	574	579	439	439
query30	274	227	201	201
query31	865	929	750	750
query32	72	63	64	63
query33	451	368	322	322
query34	779	882	521	521
query35	786	837	735	735
query36	945	1001	893	893
query37	110	94	77	77
query38	4198	4236	4240	4236
query39	1499	1411	1461	1411
query40	216	130	110	110
query41	59	58	61	58
query42	133	115	110	110
query43	501	499	474	474
query44	1383	867	853	853
query45	180	171	168	168
query46	866	1036	657	657
query47	1834	1898	1814	1814
query48	403	424	333	333
query49	699	537	422	422
query50	665	700	414	414
query51	4278	4312	4217	4217
query52	120	116	101	101
query53	227	260	183	183
query54	595	575	534	534
query55	89	83	86	83
query56	339	302	310	302
query57	1182	1183	1128	1128
query58	267	277	256	256
query59	2629	2791	2732	2732
query60	328	316	292	292
query61	131	124	121	121
query62	723	767	699	699
query63	225	187	189	187
query64	1708	1064	696	696
query65	4346	4231	4203	4203
query66	719	396	304	304
query67	15851	15556	15268	15268
query68	6180	889	518	518
query69	545	307	260	260
query70	1168	1144	1102	1102
query71	441	319	322	319
query72	5908	4802	4797	4797
query73	1355	633	350	350
query74	9041	9251	8678	8678
query75	3260	3208	2676	2676
query76	3808	1211	760	760
query77	549	381	282	282
query78	10109	10294	9375	9375
query79	2483	811	581	581
query80	653	497	433	433
query81	478	258	221	221
query82	183	124	95	95
query83	267	252	233	233
query84	291	96	90	90
query85	733	356	348	348
query86	352	300	266	266
query87	4385	4485	4265	4265
query88	3395	2290	2280	2280
query89	400	315	281	281
query90	1722	206	202	202
query91	140	145	111	111
query92	68	64	58	58
query93	2130	949	580	580
query94	685	416	299	299
query95	365	299	274	274
query96	487	583	280	280
query97	2726	2760	2650	2650
query98	239	206	200	200
query99	1302	1414	1259	1259
Total cold run time: 295702 ms
Total hot run time: 191897 ms

@doris-robot

doris-robot commented May 28, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ff2f4ca2310d39dd63ba2c4b7450adab8ac8a66f, data reload: false

query1	0.04	0.03	0.03
query2	0.12	0.10	0.11
query3	0.26	0.19	0.19
query4	1.60	0.19	0.19
query5	0.46	0.44	0.44
query6	1.17	0.67	0.64
query7	0.03	0.02	0.02
query8	0.05	0.03	0.04
query9	0.59	0.52	0.53
query10	0.57	0.58	0.56
query11	0.16	0.11	0.11
query12	0.15	0.11	0.11
query13	0.61	0.60	0.59
query14	0.78	0.82	0.79
query15	0.86	0.87	0.87
query16	0.37	0.38	0.41
query17	1.06	1.03	1.00
query18	0.22	0.21	0.21
query19	1.90	1.84	1.76
query20	0.02	0.01	0.01
query21	15.41	0.92	0.56
query22	0.77	1.00	0.75
query23	15.01	1.37	0.63
query24	7.14	1.95	0.34
query25	0.29	0.26	0.09
query26	0.56	0.16	0.14
query27	0.05	0.05	0.06
query28	9.64	0.89	0.41
query29	12.62	3.96	3.28
query30	0.26	0.09	0.06
query31	2.83	0.58	0.40
query32	3.23	0.54	0.47
query33	3.02	3.10	3.05
query34	15.83	5.07	4.48
query35	4.58	4.52	4.49
query36	0.66	0.51	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 103.43 s
Total hot run time: 28.54 s

@wuwenchi

wuwenchi commented May 29, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@wuwenchi
wuwenchi marked this pull request as ready for review May 29, 2025 03:10
@doris-robot

doris-robot commented May 29, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33594 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7ac5d8e8a71d04a2854bf2917f536dd12cdad36, data reload: false

------ Round 1 ----------------------------------
q1	26176	5122	5066	5066
q2	2083	304	185	185
q3	10360	1253	686	686
q4	10220	983	535	535
q5	7526	2372	2320	2320
q6	190	171	131	131
q7	891	798	616	616
q8	9311	1291	1033	1033
q9	6830	5103	5046	5046
q10	6857	2312	1899	1899
q11	495	285	276	276
q12	339	356	212	212
q13	17761	3677	3104	3104
q14	235	235	215	215
q15	523	478	483	478
q16	420	444	373	373
q17	625	855	364	364
q18	7702	7299	6993	6993
q19	1725	962	566	566
q20	348	342	226	226
q21	3698	3163	2323	2323
q22	1020	1007	947	947
Total cold run time: 115335 ms
Total hot run time: 33594 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5222	5089	5117	5089
q2	242	325	230	230
q3	2160	2668	2326	2326
q4	1400	1752	1400	1400
q5	4428	4395	4428	4395
q6	224	170	129	129
q7	2016	1940	1785	1785
q8	2584	2556	2524	2524
q9	7198	7227	6845	6845
q10	3029	3223	2780	2780
q11	561	531	498	498
q12	669	765	624	624
q13	3519	3902	3298	3298
q14	282	304	288	288
q15	524	499	484	484
q16	467	495	432	432
q17	1146	1500	1392	1392
q18	7739	7528	7408	7408
q19	796	777	790	777
q20	2000	2095	1864	1864
q21	4764	4490	4410	4410
q22	1090	1051	1001	1001
Total cold run time: 52060 ms
Total hot run time: 49979 ms

@doris-robot

doris-robot commented May 29, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192202 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7ac5d8e8a71d04a2854bf2917f536dd12cdad36, data reload: false

query1	1389	1084	1054	1054
query2	6299	1831	1861	1831
query3	11031	4643	4442	4442
query4	53588	25133	23060	23060
query5	5089	513	452	452
query6	346	212	204	204
query7	4899	529	289	289
query8	304	239	204	204
query9	5625	2653	2670	2653
query10	438	330	268	268
query11	15012	15105	14781	14781
query12	163	107	104	104
query13	1042	531	437	437
query14	10157	6340	6365	6340
query15	228	204	179	179
query16	7067	625	506	506
query17	1065	756	583	583
query18	1611	407	313	313
query19	187	188	169	169
query20	137	135	119	119
query21	210	126	115	115
query22	4391	4400	4229	4229
query23	34329	33582	33549	33549
query24	6582	2402	2421	2402
query25	468	466	412	412
query26	689	284	157	157
query27	2165	518	356	356
query28	2989	2191	2176	2176
query29	598	580	450	450
query30	272	225	190	190
query31	846	883	760	760
query32	75	70	66	66
query33	471	387	337	337
query34	778	901	548	548
query35	785	868	776	776
query36	963	993	891	891
query37	114	107	86	86
query38	4135	4215	4278	4215
query39	1601	1492	1472	1472
query40	218	129	117	117
query41	69	65	68	65
query42	127	119	122	119
query43	505	522	491	491
query44	1352	863	876	863
query45	189	176	177	176
query46	860	1048	703	703
query47	1855	1844	1798	1798
query48	419	427	330	330
query49	668	490	405	405
query50	662	700	406	406
query51	4237	4327	4169	4169
query52	109	115	105	105
query53	232	271	192	192
query54	581	584	528	528
query55	89	87	83	83
query56	313	299	301	299
query57	1160	1178	1120	1120
query58	275	267	279	267
query59	2743	2835	2775	2775
query60	380	328	304	304
query61	126	124	131	124
query62	743	729	679	679
query63	226	211	194	194
query64	1481	1073	678	678
query65	4288	4219	4238	4219
query66	729	400	302	302
query67	16097	15817	15312	15312
query68	6791	892	527	527
query69	544	304	269	269
query70	1175	1107	1112	1107
query71	504	325	309	309
query72	5871	4884	4797	4797
query73	1460	673	362	362
query74	9078	9269	8698	8698
query75	3821	3187	2692	2692
query76	4230	1185	770	770
query77	633	356	303	303
query78	10094	10159	9279	9279
query79	2459	801	604	604
query80	585	508	449	449
query81	488	256	217	217
query82	433	127	97	97
query83	356	253	229	229
query84	297	112	95	95
query85	788	354	316	316
query86	392	305	294	294
query87	4367	4476	4318	4318
query88	3643	2310	2279	2279
query89	402	316	283	283
query90	1824	202	207	202
query91	144	140	117	117
query92	75	62	64	62
query93	2019	924	581	581
query94	668	375	315	315
query95	361	296	293	293
query96	495	573	281	281
query97	2692	2774	2622	2622
query98	243	206	201	201
query99	1415	1474	1236	1236
Total cold run time: 297218 ms
Total hot run time: 192202 ms

@doris-robot

doris-robot commented May 29, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7ac5d8e8a71d04a2854bf2917f536dd12cdad36, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.12
query3	0.24	0.20	0.20
query4	1.60	0.20	0.19
query5	0.46	0.44	0.45
query6	1.17	0.68	0.65
query7	0.02	0.02	0.02
query8	0.05	0.03	0.03
query9	0.58	0.52	0.53
query10	0.56	0.58	0.57
query11	0.15	0.11	0.11
query12	0.16	0.12	0.11
query13	0.62	0.60	0.60
query14	0.79	0.82	0.80
query15	0.86	0.87	0.85
query16	0.36	0.37	0.36
query17	1.05	1.06	1.04
query18	0.23	0.21	0.21
query19	1.90	1.81	1.76
query20	0.01	0.01	0.01
query21	15.39	0.90	0.52
query22	0.74	1.11	0.67
query23	15.02	1.39	0.64
query24	6.95	2.10	0.34
query25	0.29	0.11	0.10
query26	0.55	0.16	0.13
query27	0.05	0.05	0.05
query28	9.53	0.94	0.45
query29	12.52	3.96	3.30
query30	0.26	0.08	0.06
query31	2.83	0.58	0.38
query32	3.44	0.57	0.48
query33	3.02	3.20	3.11
query34	15.69	5.10	4.46
query35	4.51	4.52	4.52
query36	0.66	0.49	0.48
query37	0.08	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 102.9 s
Total hot run time: 28.63 s

@wuwenchi
wuwenchi force-pushed the opt-iceberg-branch-tag branch from f647790 to 34fba03 Compare May 29, 2025 07:17
@wuwenchi

wuwenchi commented May 29, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented May 29, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33733 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 34fba03ddc1f5c6ba075f8cb157a08af6eea1706, data reload: false

------ Round 1 ----------------------------------
q1	26280	5050	4999	4999
q2	1946	297	184	184
q3	10272	1211	706	706
q4	10222	1015	521	521
q5	7511	2378	2346	2346
q6	181	165	131	131
q7	908	735	603	603
q8	9300	1239	1127	1127
q9	6911	5133	5055	5055
q10	6885	2324	1901	1901
q11	489	300	267	267
q12	350	347	218	218
q13	17811	3646	3055	3055
q14	232	234	212	212
q15	549	483	484	483
q16	425	436	371	371
q17	581	841	342	342
q18	7483	7271	7137	7137
q19	1647	955	551	551
q20	333	331	227	227
q21	3744	3131	2300	2300
q22	1062	1025	997	997
Total cold run time: 115122 ms
Total hot run time: 33733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5135	5157	4992	4992
q2	246	316	218	218
q3	2186	2645	2303	2303
q4	1344	1731	1311	1311
q5	4408	4344	4387	4344
q6	215	167	128	128
q7	1974	1948	1737	1737
q8	2561	2428	2521	2428
q9	7274	7245	6936	6936
q10	3064	3228	2756	2756
q11	569	511	491	491
q12	662	766	616	616
q13	3500	3905	3269	3269
q14	274	308	282	282
q15	532	481	496	481
q16	424	484	455	455
q17	1145	1516	1369	1369
q18	7753	7651	7437	7437
q19	822	758	847	758
q20	2009	2042	1943	1943
q21	4796	4482	4410	4410
q22	1107	1102	1044	1044
Total cold run time: 52000 ms
Total hot run time: 49708 ms

@doris-robot

doris-robot commented May 29, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192168 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 34fba03ddc1f5c6ba075f8cb157a08af6eea1706, data reload: false

query1	1423	1095	1087	1087
query2	6334	1777	1768	1768
query3	11180	4636	4720	4636
query4	25503	23814	23222	23222
query5	4312	622	452	452
query6	319	228	207	207
query7	3988	495	288	288
query8	275	221	234	221
query9	8471	2586	2615	2586
query10	473	325	267	267
query11	15228	15039	14746	14746
query12	155	109	105	105
query13	1549	523	394	394
query14	8593	6185	6156	6156
query15	209	209	174	174
query16	7217	633	485	485
query17	1226	727	572	572
query18	1992	446	327	327
query19	213	197	174	174
query20	138	134	123	123
query21	215	129	116	116
query22	4631	4485	4351	4351
query23	34733	33673	33464	33464
query24	8718	2406	2420	2406
query25	527	477	409	409
query26	1253	268	153	153
query27	2896	509	347	347
query28	4429	2223	2160	2160
query29	760	557	449	449
query30	281	292	197	197
query31	924	842	763	763
query32	75	64	64	64
query33	531	370	322	322
query34	808	910	570	570
query35	840	859	734	734
query36	948	983	881	881
query37	115	102	83	83
query38	4269	4299	4248	4248
query39	1518	1503	1472	1472
query40	237	119	111	111
query41	70	64	62	62
query42	127	111	110	110
query43	484	497	490	490
query44	1396	847	861	847
query45	183	176	175	175
query46	877	1029	657	657
query47	1825	1917	1809	1809
query48	392	432	319	319
query49	725	483	411	411
query50	667	683	398	398
query51	4289	4288	4230	4230
query52	107	109	110	109
query53	232	258	190	190
query54	582	608	512	512
query55	83	82	87	82
query56	334	301	279	279
query57	1199	1195	1093	1093
query58	266	283	262	262
query59	2681	2803	2673	2673
query60	328	336	314	314
query61	130	132	126	126
query62	785	733	677	677
query63	230	192	194	192
query64	4257	1024	692	692
query65	4349	4242	4258	4242
query66	1120	399	316	316
query67	16022	15559	15254	15254
query68	8787	898	546	546
query69	486	293	264	264
query70	1231	1167	1053	1053
query71	465	319	288	288
query72	5657	4705	4769	4705
query73	723	627	361	361
query74	8901	9048	8955	8955
query75	3944	3224	2642	2642
query76	3732	1191	742	742
query77	775	386	306	306
query78	10154	10363	9418	9418
query79	2615	784	583	583
query80	622	513	452	452
query81	472	256	226	226
query82	463	124	94	94
query83	281	242	230	230
query84	297	106	90	90
query85	777	413	309	309
query86	386	318	296	296
query87	4432	4490	4368	4368
query88	3368	2294	2275	2275
query89	412	319	280	280
query90	1804	209	214	209
query91	142	152	109	109
query92	73	71	57	57
query93	1936	955	582	582
query94	665	431	297	297
query95	377	295	281	281
query96	493	583	282	282
query97	2736	2785	2614	2614
query98	227	216	202	202
query99	1439	1375	1241	1241
Total cold run time: 280118 ms
Total hot run time: 192168 ms

@doris-robot

doris-robot commented May 29, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 34fba03ddc1f5c6ba075f8cb157a08af6eea1706, data reload: false

query1	0.04	0.03	0.03
query2	0.13	0.11	0.12
query3	0.25	0.20	0.19
query4	1.60	0.20	0.20
query5	0.47	0.47	0.44
query6	1.14	0.67	0.64
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.58	0.51	0.52
query10	0.58	0.58	0.57
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.79	0.81	0.82
query15	0.88	0.86	0.85
query16	0.38	0.38	0.39
query17	1.05	1.04	1.03
query18	0.22	0.20	0.21
query19	1.95	1.78	1.77
query20	0.01	0.01	0.01
query21	15.40	0.90	0.54
query22	0.76	1.17	0.80
query23	14.82	1.38	0.62
query24	6.87	1.60	0.63
query25	0.46	0.19	0.08
query26	0.60	0.16	0.15
query27	0.05	0.05	0.05
query28	8.71	0.92	0.43
query29	12.55	3.93	3.33
query30	0.26	0.10	0.06
query31	2.81	0.58	0.39
query32	3.23	0.56	0.47
query33	3.13	3.04	3.08
query34	15.71	5.15	4.50
query35	4.53	4.57	4.59
query36	0.67	0.50	0.49
query37	0.09	0.07	0.07
query38	0.05	0.04	0.04
query39	0.03	0.03	0.03
query40	0.18	0.15	0.15
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 102.12 s
Total hot run time: 29.11 s

@doris-robot

doris-robot commented Jun 3, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb2079a3881f88b012f9fa64790d11af4aa5240e, data reload: false

query1	0.05	0.04	0.03
query2	0.12	0.10	0.12
query3	0.24	0.20	0.20
query4	1.60	0.20	0.20
query5	0.46	0.44	0.43
query6	1.17	0.67	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.59	0.52	0.52
query10	0.56	0.58	0.58
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.62	0.59	0.60
query14	0.80	0.82	0.79
query15	0.88	0.87	0.86
query16	0.38	0.39	0.38
query17	1.01	1.03	1.02
query18	0.23	0.21	0.21
query19	1.88	1.84	1.83
query20	0.01	0.01	0.02
query21	15.39	0.92	0.55
query22	0.78	1.28	0.70
query23	14.74	1.36	0.66
query24	7.05	2.28	0.63
query25	0.53	0.16	0.09
query26	0.65	0.16	0.15
query27	0.05	0.05	0.04
query28	9.78	0.84	0.45
query29	12.56	4.14	3.41
query30	0.25	0.10	0.08
query31	2.82	0.59	0.38
query32	3.24	0.55	0.47
query33	3.08	3.07	3.11
query34	15.81	5.12	4.52
query35	4.48	4.50	4.49
query36	0.66	0.50	0.47
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.03
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 103.34 s
Total hot run time: 29.17 s

@wuwenchi

wuwenchi commented Jun 3, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Jun 3, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 34549 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3c0996e79f0a45ed4edaa04b676e95b21b57d1b1, data reload: false

------ Round 1 ----------------------------------
q1	25958	5276	5207	5207
q2	1957	269	173	173
q3	10316	1328	693	693
q4	10223	1035	517	517
q5	7545	2516	2560	2516
q6	193	174	133	133
q7	982	747	596	596
q8	9308	1496	1152	1152
q9	7002	5341	5332	5332
q10	6942	2356	1898	1898
q11	502	294	282	282
q12	354	379	224	224
q13	17784	3705	3058	3058
q14	228	229	219	219
q15	590	488	492	488
q16	426	442	374	374
q17	605	904	376	376
q18	7586	7145	7188	7145
q19	1469	1107	611	611
q20	361	363	226	226
q21	4100	3263	2386	2386
q22	1030	999	943	943
Total cold run time: 115461 ms
Total hot run time: 34549 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5408	5371	5400	5371
q2	237	333	226	226
q3	2175	2721	2280	2280
q4	1331	1930	1398	1398
q5	4490	4385	4357	4357
q6	254	188	132	132
q7	2052	1941	1749	1749
q8	2842	2889	2707	2707
q9	7275	7035	7208	7035
q10	3047	3180	2776	2776
q11	639	529	486	486
q12	713	763	631	631
q13	3515	3963	3360	3360
q14	304	294	279	279
q15	546	477	471	471
q16	464	481	446	446
q17	1137	1706	1431	1431
q18	7825	7489	7421	7421
q19	860	914	1139	914
q20	1956	2004	1822	1822
q21	5015	4539	4467	4467
q22	1113	1061	1041	1041
Total cold run time: 53198 ms
Total hot run time: 50800 ms

@doris-robot

doris-robot commented Jun 3, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192812 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3c0996e79f0a45ed4edaa04b676e95b21b57d1b1, data reload: false

query1	1407	1076	1049	1049
query2	6361	1841	1860	1841
query3	11020	4613	4613	4613
query4	54379	25115	23390	23390
query5	5092	572	476	476
query6	366	213	205	205
query7	4971	512	293	293
query8	304	244	214	214
query9	6011	2654	2654	2654
query10	444	318	281	281
query11	14946	15027	14739	14739
query12	158	113	104	104
query13	1095	538	416	416
query14	10037	6278	6336	6278
query15	202	218	183	183
query16	7005	668	508	508
query17	1102	749	615	615
query18	1541	421	317	317
query19	205	200	181	181
query20	131	133	167	133
query21	214	127	108	108
query22	4514	4591	4310	4310
query23	34402	33593	33527	33527
query24	6673	2448	2464	2448
query25	475	492	411	411
query26	748	271	166	166
query27	2443	511	338	338
query28	2943	2181	2166	2166
query29	610	567	444	444
query30	299	224	191	191
query31	830	863	785	785
query32	78	62	63	62
query33	499	389	302	302
query34	799	886	551	551
query35	791	822	815	815
query36	956	1000	892	892
query37	120	99	80	80
query38	4301	4355	4283	4283
query39	1492	1450	1433	1433
query40	218	129	113	113
query41	61	64	57	57
query42	122	108	111	108
query43	512	500	491	491
query44	1341	840	835	835
query45	194	176	169	169
query46	882	1041	666	666
query47	1871	1905	1783	1783
query48	421	430	326	326
query49	653	484	405	405
query50	685	693	424	424
query51	4225	4335	4185	4185
query52	117	127	105	105
query53	237	269	194	194
query54	601	614	533	533
query55	87	84	83	83
query56	329	305	315	305
query57	1168	1191	1138	1138
query58	277	278	259	259
query59	2799	2863	2751	2751
query60	340	340	310	310
query61	142	146	127	127
query62	753	743	697	697
query63	234	186	188	186
query64	1859	1038	685	685
query65	4266	4140	4164	4140
query66	754	456	315	315
query67	15896	15861	15451	15451
query68	7307	895	529	529
query69	540	306	269	269
query70	1271	1170	1114	1114
query71	511	326	299	299
query72	5833	4799	4895	4799
query73	1377	690	355	355
query74	9159	9116	8653	8653
query75	4009	3224	2702	2702
query76	4332	1198	758	758
query77	688	378	277	277
query78	10081	10144	9305	9305
query79	2620	814	569	569
query80	656	522	451	451
query81	486	256	214	214
query82	461	127	95	95
query83	332	250	293	250
query84	298	111	92	92
query85	817	358	323	323
query86	391	318	296	296
query87	4452	4563	4357	4357
query88	3445	2298	2262	2262
query89	399	312	283	283
query90	1953	205	203	203
query91	212	145	119	119
query92	78	65	52	52
query93	1750	934	587	587
query94	685	425	310	310
query95	366	303	287	287
query96	506	572	282	282
query97	2815	2786	2610	2610
query98	232	215	206	206
query99	1437	1383	1261	1261
Total cold run time: 300507 ms
Total hot run time: 192812 ms

@doris-robot

doris-robot commented Jun 3, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3c0996e79f0a45ed4edaa04b676e95b21b57d1b1, data reload: false

query1	0.04	0.04	0.03
query2	0.12	0.11	0.11
query3	0.24	0.19	0.20
query4	1.61	0.20	0.19
query5	0.45	0.45	0.45
query6	1.51	0.65	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.04
query9	0.57	0.52	0.50
query10	0.57	0.57	0.58
query11	0.16	0.11	0.11
query12	0.15	0.13	0.13
query13	0.61	0.60	0.60
query14	0.78	0.80	0.81
query15	0.88	0.85	0.83
query16	0.37	0.39	0.38
query17	1.04	1.08	1.04
query18	0.23	0.22	0.21
query19	1.87	1.85	1.80
query20	0.01	0.01	0.01
query21	15.39	0.87	0.55
query22	0.77	1.11	0.75
query23	14.91	1.37	0.61
query24	7.36	1.00	0.29
query25	0.30	0.11	0.09
query26	0.62	0.17	0.15
query27	0.06	0.05	0.05
query28	8.89	0.96	0.44
query29	12.53	4.00	3.37
query30	0.25	0.10	0.07
query31	2.81	0.58	0.38
query32	3.23	0.54	0.46
query33	3.06	3.02	3.06
query34	15.71	5.12	4.45
query35	4.51	4.51	4.50
query36	0.66	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.03
query43	0.04	0.03	0.04
Total cold run time: 102.81 s
Total hot run time: 28.66 s

@wuwenchi

wuwenchi commented Jun 4, 2025

Copy link
Copy Markdown
Contributor Author

run p0

@wuwenchi

wuwenchi commented Jun 4, 2025

Copy link
Copy Markdown
Contributor Author

run external

morningman
morningman previously approved these changes Jun 5, 2025
View reviewed changes
@morningman

morningman commented Jun 5, 2025

Copy link
Copy Markdown
Contributor

run buildall

@doris-robot

doris-robot commented Jun 5, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 33981 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fd1140482dabb073709d631b6b3be8e0bb9e6816, data reload: false

------ Round 1 ----------------------------------
q1	26252	5074	5031	5031
q2	1941	274	189	189
q3	10356	1241	723	723
q4	10224	1017	520	520
q5	7515	2351	2363	2351
q6	179	165	134	134
q7	919	740	609	609
q8	9309	1280	1141	1141
q9	6749	5128	5077	5077
q10	6860	2320	1924	1924
q11	488	287	280	280
q12	347	352	214	214
q13	17786	3668	3067	3067
q14	226	235	222	222
q15	576	497	488	488
q16	422	426	371	371
q17	599	856	370	370
q18	7562	7126	7244	7126
q19	1958	984	577	577
q20	326	331	215	215
q21	4078	2637	2373	2373
q22	1054	1026	979	979
Total cold run time: 115726 ms
Total hot run time: 33981 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5186	5101	5117	5101
q2	236	318	223	223
q3	2135	2657	2342	2342
q4	1361	1806	1433	1433
q5	4547	4411	4434	4411
q6	209	174	131	131
q7	1988	1938	1763	1763
q8	2614	2555	2593	2555
q9	7270	7176	6852	6852
q10	3047	3221	2781	2781
q11	583	514	503	503
q12	684	749	617	617
q13	3521	3954	3265	3265
q14	282	326	294	294
q15	507	486	476	476
q16	475	487	452	452
q17	1200	1535	1390	1390
q18	7866	7646	7413	7413
q19	785	799	906	799
q20	1990	2070	2040	2040
q21	5030	4460	4520	4460
q22	1083	1088	1010	1010
Total cold run time: 52599 ms
Total hot run time: 50311 ms

@doris-robot

doris-robot commented Jun 5, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 192976 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fd1140482dabb073709d631b6b3be8e0bb9e6816, data reload: false

query1	1414	1115	1087	1087
query2	6261	1817	1786	1786
query3	11052	4449	4609	4449
query4	55195	24584	23507	23507
query5	5148	549	471	471
query6	380	222	202	202
query7	5027	517	308	308
query8	313	230	215	215
query9	6436	2657	2645	2645
query10	446	342	298	298
query11	15377	14966	14848	14848
query12	166	115	104	104
query13	1118	540	423	423
query14	10652	6517	6571	6517
query15	212	201	186	186
query16	7118	711	482	482
query17	1097	764	623	623
query18	1580	412	329	329
query19	204	198	176	176
query20	138	141	127	127
query21	210	130	109	109
query22	4351	4424	4365	4365
query23	34312	33547	33555	33547
query24	6215	2464	2454	2454
query25	465	471	411	411
query26	728	301	156	156
query27	2002	513	353	353
query28	2844	2191	2192	2191
query29	585	580	444	444
query30	281	222	187	187
query31	839	854	806	806
query32	83	63	67	63
query33	451	365	310	310
query34	785	862	554	554
query35	809	844	740	740
query36	987	1025	913	913
query37	118	99	76	76
query38	4253	4370	4241	4241
query39	1509	1483	1469	1469
query40	213	125	105	105
query41	60	59	57	57
query42	130	114	114	114
query43	509	521	470	470
query44	1341	845	838	838
query45	183	184	179	179
query46	889	1045	647	647
query47	1805	1865	1761	1761
query48	407	457	333	333
query49	683	517	404	404
query50	716	697	415	415
query51	4442	4319	4238	4238
query52	110	108	97	97
query53	229	269	184	184
query54	579	594	527	527
query55	83	84	82	82
query56	306	297	297	297
query57	1151	1162	1091	1091
query58	287	290	264	264
query59	2603	2777	2633	2633
query60	342	345	324	324
query61	133	132	125	125
query62	755	757	654	654
query63	267	195	191	191
query64	1825	1032	683	683
query65	4240	4149	4157	4149
query66	723	389	299	299
query67	15780	15699	15465	15465
query68	6784	831	543	543
query69	546	312	271	271
query70	1148	1107	1099	1099
query71	528	314	294	294
query72	5943	4780	4941	4780
query73	1320	688	358	358
query74	8944	8851	8722	8722
query75	3853	3218	2679	2679
query76	4195	1275	730	730
query77	626	392	293	293
query78	10166	9994	9265	9265
query79	3013	813	584	584
query80	772	519	447	447
query81	481	256	303	256
query82	464	128	96	96
query83	369	248	238	238
query84	316	108	87	87
query85	790	359	314	314
query86	373	309	280	280
query87	4434	4454	4302	4302
query88	3293	2249	2225	2225
query89	398	316	281	281
query90	1799	211	210	210
query91	140	141	117	117
query92	75	62	58	58
query93	2425	950	586	586
query94	651	408	314	314
query95	369	302	291	291
query96	494	578	279	279
query97	2698	2750	2688	2688
query98	229	209	221	209
query99	1457	1430	1282	1282
Total cold run time: 301045 ms
Total hot run time: 192976 ms

@doris-robot

doris-robot commented Jun 5, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 29.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fd1140482dabb073709d631b6b3be8e0bb9e6816, data reload: false

query1	0.04	0.03	0.04
query2	0.12	0.10	0.11
query3	0.26	0.20	0.20
query4	1.59	0.20	0.20
query5	0.45	0.41	0.43
query6	1.16	0.67	0.65
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.60	0.53	0.51
query10	0.58	0.59	0.57
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.62	0.59	0.59
query14	0.78	0.81	0.81
query15	0.87	0.87	0.86
query16	0.38	0.38	0.39
query17	1.02	1.05	1.07
query18	0.22	0.20	0.20
query19	1.88	1.76	1.86
query20	0.02	0.01	0.01
query21	15.40	0.90	0.55
query22	0.74	1.14	0.96
query23	14.69	1.37	0.64
query24	6.84	1.53	0.62
query25	0.48	0.12	0.15
query26	0.52	0.16	0.13
query27	0.05	0.05	0.05
query28	9.98	0.93	0.43
query29	12.59	4.01	3.29
query30	0.26	0.09	0.06
query31	2.83	0.60	0.38
query32	3.24	0.54	0.47
query33	3.00	3.16	3.15
query34	15.86	5.11	4.43
query35	4.50	4.48	4.49
query36	0.68	0.50	0.51
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.07	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 103.08 s
Total hot run time: 29.18 s

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label Jun 5, 2025
@github-actions

github-actions Bot commented Jun 5, 2025

Copy link
Copy Markdown
Contributor

PR approved by at least one committer and no changes requested.

@github-actions

github-actions Bot commented Jun 5, 2025

Copy link
Copy Markdown
Contributor

PR approved by anyone and no changes requested.

@morningman
morningman merged commit fbef303 into apache:master Jun 6, 2025
morningman pushed a commit to morningman/doris that referenced this pull request Jul 3, 2025
@wuwenchi @morningman
[opt](iceberg)support read iceberg with tag/branch (apache#51272)
199613a 
Related Issue: apache#48285

Problem Summary:

if an iceberg has a branch named `branch1` and a tag named `tag1`, you
can query `branch1` or `tag1` by these ways:
```sql
-- query 'branch1'
select * from tb FOR VERSION AS OF 'branch1';
select * from tb@branch(branch1);
select * from tb@branch('name'='branch1')

-- query 'tag1'
select * from tb FOR VERSION AS OF 'tag1';
select * from tb@tag(tag1);
select * from tb@tag('name'='tag1')

```
@morningman morningman mentioned this pull request Jul 3, 2025
Merged
morningman pushed a commit to morningman/doris that referenced this pull request Jul 3, 2025
@wuwenchi @morningman
[opt](iceberg)support read iceberg with tag/branch (apache#51272)
d375e3e 
Related Issue: apache#48285

Problem Summary:

if an iceberg has a branch named `branch1` and a tag named `tag1`, you
can query `branch1` or `tag1` by these ways:
```sql
-- query 'branch1'
select * from tb FOR VERSION AS OF 'branch1';
select * from tb@branch(branch1);
select * from tb@branch('name'='branch1')

-- query 'tag1'
select * from tb FOR VERSION AS OF 'tag1';
select * from tb@tag(tag1);
select * from tb@tag('name'='tag1')

```
morrySnow pushed a commit that referenced this pull request Jul 4, 2025
@morningman @wuwenchi
branch-3.1: [opt](iceberg)support read iceberg with tag/branch #51272 (…
24f4cf2 
…#52740)

bp #51272

Co-authored-by: wuwenchi <[email protected]>
etah000 pushed a commit to etah000/doris that referenced this pull request Jul 7, 2025
@morningman @wuwenchi @etah000
branch-3.1: [opt](iceberg)support read iceberg with tag/branch apache…
9fa5f1e 
…#51272 (apache#52740)

bp apache#51272

Co-authored-by: wuwenchi <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes
@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@wuwenchi @hello-stephen @doris-robot @morningman @CalvinKirs @morrySnow