Skip to content

branch-3.1: [feat](iceberg) support schema change for iceberg #52591 #53588 - #53623

Merged
morrySnow merged 2 commits into
apache:branch-3.1from
morningman:31_bp52591
Jul 21, 2025
Merged

branch-3.1: [feat](iceberg) support schema change for iceberg #52591 #53588#53623
morrySnow merged 2 commits into
apache:branch-3.1from
morningman:31_bp52591

Conversation

@morningman

@morningman morningman commented Jul 21, 2025

Copy link
Copy Markdown
Contributor

suxiaogang223 and others added 2 commits July 20, 2025 21:36
@suxiaogang223 @morningman
[feat](iceberg) support schema change for iceberg (apache#52591)
0508476 
Support schema change ddl for iceberg table:
- Adding, deleting, and renaming top-level columns (not support nested
columns)
- Reordering top-level columns (not support nested columns)
- Widening the type of int, float, and decimal fields
@morningman
[fix](test) fix test_information_schema_external.groovy (apache#53588)
caab7ca
@morningman
morningman requested a review from morrySnow as a code owner July 21, 2025 04:38
@hello-stephen

hello-stephen commented Jul 21, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman

morningman commented Jul 21, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Jul 21, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 40466 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit caab7cafda318aa43e1cb635e6791a00c94c3a1a, data reload: false

------ Round 1 ----------------------------------
q1	17644	6883	6659	6659
q2	2075	230	197	197
q3	10559	1158	1146	1146
q4	10804	752	786	752
q5	7901	3042	2920	2920
q6	236	140	139	139
q7	1005	650	636	636
q8	9376	1979	2029	1979
q9	6702	6511	6469	6469
q10	7081	2286	2358	2286
q11	482	286	280	280
q12	412	231	231	231
q13	17837	3020	2998	2998
q14	244	234	223	223
q15	563	519	512	512
q16	493	395	390	390
q17	999	602	581	581
q18	7532	6788	6723	6723
q19	1359	1027	988	988
q20	496	225	218	218
q21	3994	3286	3114	3114
q22	1119	1028	1025	1025
Total cold run time: 108913 ms
Total hot run time: 40466 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6645	6594	6623	6594
q2	351	259	260	259
q3	3115	2970	2968	2968
q4	2038	1830	1761	1761
q5	5623	5669	5713	5669
q6	213	129	130	129
q7	2211	1796	1783	1783
q8	3345	3501	3506	3501
q9	8860	8877	8914	8877
q10	3614	3556	3566	3556
q11	598	507	507	507
q12	810	618	598	598
q13	11302	3163	3122	3122
q14	318	287	284	284
q15	537	514	493	493
q16	484	463	449	449
q17	1837	1645	1652	1645
q18	8148	7685	7586	7586
q19	1687	1530	1521	1521
q20	2111	1893	1900	1893
q21	5212	5140	5074	5074
q22	1184	1084	1083	1083
Total cold run time: 70243 ms
Total hot run time: 59352 ms

@doris-robot

doris-robot commented Jul 21, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193185 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit caab7cafda318aa43e1cb635e6791a00c94c3a1a, data reload: false

query1	1021	403	382	382
query2	6562	2017	1923	1923
query3	6724	227	235	227
query4	34027	23605	23937	23605
query5	4430	568	561	561
query6	310	213	217	213
query7	4641	336	336	336
query8	419	349	368	349
query9	9769	2617	2612	2612
query10	539	311	303	303
query11	18468	15133	15208	15133
query12	178	123	123	123
query13	1689	484	454	454
query14	9017	7463	6837	6837
query15	280	196	204	196
query16	8163	545	523	523
query17	1844	633	619	619
query18	2325	366	380	366
query19	306	175	168	168
query20	133	126	130	126
query21	235	128	124	124
query22	4730	4669	4606	4606
query23	35655	33888	33653	33653
query24	11798	2892	2903	2892
query25	728	415	421	415
query26	1873	178	178	178
query27	2932	360	353	353
query28	7923	2146	2142	2142
query29	1058	437	448	437
query30	345	168	168	168
query31	1036	832	827	827
query32	107	69	67	67
query33	825	333	359	333
query34	925	510	526	510
query35	876	752	748	748
query36	1094	930	950	930
query37	283	96	80	80
query38	4043	3797	3867	3797
query39	1529	1454	1466	1454
query40	308	119	119	119
query41	94	92	93	92
query42	122	111	110	110
query43	537	496	503	496
query44	1303	816	813	813
query45	206	188	189	188
query46	1166	730	712	712
query47	1981	1884	1878	1878
query48	456	355	352	352
query49	1397	472	469	469
query50	848	439	435	435
query51	7412	7162	7180	7162
query52	114	122	99	99
query53	275	209	206	206
query54	1303	491	501	491
query55	93	89	85	85
query56	297	281	276	276
query57	1342	1184	1197	1184
query58	291	250	246	246
query59	3120	2932	2839	2839
query60	323	297	298	297
query61	135	131	166	131
query62	904	706	707	706
query63	249	209	208	208
query64	5161	708	683	683
query65	3271	3182	3211	3182
query66	1594	465	499	465
query67	15996	15919	15446	15446
query68	5102	605	605	605
query69	467	299	283	283
query70	1198	1138	1135	1135
query71	375	317	282	282
query72	6368	4099	4004	4004
query73	781	374	378	374
query74	10815	9086	8905	8905
query75	3476	2674	2692	2674
query76	3002	1103	1200	1103
query77	574	337	332	332
query78	10651	9619	9538	9538
query79	2433	623	629	623
query80	1247	493	506	493
query81	555	244	238	238
query82	904	109	100	100
query83	262	183	176	176
query84	253	92	93	92
query85	1619	338	348	338
query86	487	325	311	311
query87	4443	4198	4282	4198
query88	4139	2481	2440	2440
query89	415	324	316	316
query90	2185	202	204	202
query91	162	124	127	124
query92	77	64	66	64
query93	1583	564	558	558
query94	1060	327	271	271
query95	393	278	276	276
query96	619	290	289	289
query97	3348	3183	3175	3175
query98	226	217	205	205
query99	1502	1321	1324	1321
Total cold run time: 309859 ms
Total hot run time: 193185 ms

@doris-robot

doris-robot commented Jul 21, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 30.39 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit caab7cafda318aa43e1cb635e6791a00c94c3a1a, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.08	0.07
query4	1.62	0.10	0.11
query5	0.53	0.53	0.52
query6	1.14	0.73	0.73
query7	0.03	0.02	0.01
query8	0.05	0.03	0.04
query9	0.56	0.50	0.54
query10	0.56	0.56	0.57
query11	0.16	0.12	0.12
query12	0.16	0.12	0.11
query13	0.62	0.61	0.61
query14	0.76	0.77	0.81
query15	0.85	0.85	0.83
query16	0.39	0.39	0.40
query17	1.08	1.07	1.07
query18	0.23	0.21	0.22
query19	1.92	1.82	1.88
query20	0.02	0.01	0.01
query21	15.38	0.57	0.57
query22	2.76	2.38	1.68
query23	16.87	1.00	0.87
query24	3.55	1.34	0.98
query25	0.14	0.09	0.05
query26	0.73	0.13	0.13
query27	0.05	0.04	0.05
query28	9.82	0.52	0.44
query29	12.63	3.18	3.22
query30	0.32	0.13	0.15
query31	2.82	0.40	0.39
query32	3.23	0.46	0.46
query33	2.97	3.01	3.07
query34	16.90	4.58	4.54
query35	4.56	4.53	4.60
query36	0.70	0.48	0.49
query37	0.10	0.07	0.07
query38	0.05	0.05	0.05
query39	0.04	0.03	0.04
query40	0.18	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 105.01 s
Total hot run time: 30.39 s

@morrySnow morrySnow changed the title [feat](iceberg) support schema change for iceberg (#52591)(#53588) branch-3.1: [feat](iceberg) support schema change for iceberg #52591 #53588 Jul 21, 2025
@morrySnow
morrySnow merged commit 3fae04a into apache:branch-3.1 Jul 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @hello-stephen @doris-robot @morrySnow @suxiaogang223