Skip to content

branch-3.1: [feat](paimon) support paimon dlf rest catalog (#54523) - #54887

Merged
morningman merged 1 commit into
apache:branch-3.1from
morningman:31_bp54523
Aug 19, 2025
Merged

branch-3.1: [feat](paimon) support paimon dlf rest catalog (#54523)#54887
morningman merged 1 commit into
apache:branch-3.1from
morningman:31_bp54523

Conversation

@morningman

@morningman morningman commented Aug 15, 2025

Copy link
Copy Markdown
Contributor

bp #54523

@morningman
[feat](paimon) support paimon dlf rest catalog (apache#54523)
62decdf 
### What problem does this PR solve?


This PR adds support for Paimon DLF REST catalog integration, enabling
Doris to connect to Alibaba Cloud's Data Lake Formation (DLF) service
through REST API with proper credential management.

Key changes include:
- Implementation of PaimonRestMetaStoreProperties for REST catalog
configuration with DLF token provider support
- Addition of vended credentials provider for dynamic credential
extraction from REST tokens
- Integration of REST catalog type into the existing Paimon external
catalog framework

```
CREATE CATALOG paimon_dlf_test PROPERTIES (
    'type' = 'paimon',
    'paimon.catalog.type' = 'rest',
    'uri' = 'http://cn-beijing-vpc.dlf.aliyuncs.com',
    'warehouse' = 'new_dfl_paimon_catalog',
    'paimon.rest.token.provider' = 'dlf',
    'paimon.rest.dlf.access-key-id' = 'ak',
    'paimon.rest.dlf.access-key-secret' = 'sk'
);
```
@morningman
morningman requested a review from morrySnow as a code owner August 15, 2025 17:26
@Thearas

Thearas commented Aug 15, 2025

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@morningman morningman changed the title [feat](paimon) support paimon dlf rest catalog (#54523) branch-3.1: [feat](paimon) support paimon dlf rest catalog (#54523) Aug 15, 2025
@morningman

morningman commented Aug 16, 2025

Copy link
Copy Markdown
Contributor Author

run buildall

@doris-robot

doris-robot commented Aug 16, 2025

Copy link
Copy Markdown
TPC-H: Total hot run time: 32766 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 62decdf8a47a45dc3f29895d5cd406be5b48f2c6, data reload: false

------ Round 1 ----------------------------------
q1	17611	5510	5466	5466
q2	2025	403	279	279
q3	12078	1253	737	737
q4	10308	860	452	452
q5	9418	2367	2139	2139
q6	183	165	132	132
q7	889	727	613	613
q8	9329	1434	1164	1164
q9	5145	4931	4988	4931
q10	6733	2257	1813	1813
q11	484	282	266	266
q12	340	358	209	209
q13	17780	3655	3034	3034
q14	222	231	214	214
q15	526	471	467	467
q16	433	417	380	380
q17	612	863	356	356
q18	6881	6392	6418	6392
q19	1225	956	538	538
q20	331	338	202	202
q21	2806	2161	2007	2007
q22	1039	1010	975	975
Total cold run time: 106398 ms
Total hot run time: 32766 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5477	5542	5576	5542
q2	236	320	229	229
q3	2200	2642	2319	2319
q4	1364	1804	1353	1353
q5	4508	4961	4956	4956
q6	179	166	134	134
q7	2061	1953	1823	1823
q8	2650	2832	2699	2699
q9	7295	7283	7195	7195
q10	3020	3343	2716	2716
q11	567	513	509	509
q12	653	777	630	630
q13	3391	3833	3145	3145
q14	294	301	279	279
q15	505	467	465	465
q16	443	503	454	454
q17	1225	1752	1277	1277
q18	7652	7491	7295	7295
q19	778	1137	1078	1078
q20	2024	2029	1886	1886
q21	5375	4987	4535	4535
q22	1057	1054	1001	1001
Total cold run time: 52954 ms
Total hot run time: 51520 ms

@doris-robot

doris-robot commented Aug 16, 2025

Copy link
Copy Markdown
TPC-DS: Total hot run time: 193366 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 62decdf8a47a45dc3f29895d5cd406be5b48f2c6, data reload: false

query1	950	403	387	387
query2	6222	1986	1952	1952
query3	8690	204	195	195
query4	33762	23729	24036	23729
query5	4604	595	448	448
query6	288	202	176	176
query7	4226	499	319	319
query8	318	250	248	248
query9	9275	2613	2601	2601
query10	461	322	255	255
query11	17961	15353	15382	15353
query12	153	107	103	103
query13	1566	553	412	412
query14	9386	7533	7016	7016
query15	238	195	175	175
query16	8085	688	512	512
query17	1525	801	628	628
query18	2193	418	329	329
query19	234	189	161	161
query20	119	125	120	120
query21	211	133	108	108
query22	4628	4510	4336	4336
query23	35367	34660	34153	34153
query24	7702	2721	2735	2721
query25	551	506	426	426
query26	787	289	172	172
query27	2074	482	362	362
query28	5009	2187	2203	2187
query29	716	598	464	464
query30	237	206	170	170
query31	1000	932	851	851
query32	90	60	56	56
query33	530	369	331	331
query34	755	850	501	501
query35	774	841	778	778
query36	1024	1083	977	977
query37	107	100	71	71
query38	4021	4034	3975	3975
query39	1559	1484	1497	1484
query40	214	117	100	100
query41	58	50	50	50
query42	130	111	106	106
query43	504	522	491	491
query44	1297	822	825	822
query45	192	184	169	169
query46	882	1090	688	688
query47	1956	1969	1893	1893
query48	419	444	346	346
query49	733	496	414	414
query50	704	693	432	432
query51	7299	7373	7279	7279
query52	105	102	95	95
query53	229	271	194	194
query54	564	561	479	479
query55	81	78	84	78
query56	275	269	290	269
query57	1286	1307	1206	1206
query58	237	227	216	216
query59	3104	3283	3244	3244
query60	293	276	268	268
query61	121	113	151	113
query62	800	785	686	686
query63	242	196	196	196
query64	3816	971	647	647
query65	3370	3369	3332	3332
query66	911	414	332	332
query67	16457	15671	15722	15671
query68	7722	810	549	549
query69	494	320	266	266
query70	1221	1120	1063	1063
query71	407	297	271	271
query72	5356	3793	3829	3793
query73	628	751	355	355
query74	10630	9170	9259	9170
query75	3599	3162	2654	2654
query76	3288	1124	761	761
query77	786	362	275	275
query78	10273	10432	9517	9517
query79	3539	849	593	593
query80	645	514	423	423
query81	506	257	221	221
query82	580	114	91	91
query83	170	161	148	148
query84	287	96	88	88
query85	784	361	297	297
query86	397	302	281	281
query87	4334	4376	4364	4364
query88	4962	2414	2406	2406
query89	411	329	290	290
query90	1750	191	189	189
query91	135	155	109	109
query92	71	58	53	53
query93	2128	897	545	545
query94	646	402	290	290
query95	345	287	276	276
query96	496	612	283	283
query97	3218	3290	3164	3164
query98	237	206	196	196
query99	1551	1422	1327	1327
Total cold run time: 295273 ms
Total hot run time: 193366 ms

@doris-robot

doris-robot commented Aug 16, 2025

Copy link
Copy Markdown
ClickBench: Total hot run time: 28.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 62decdf8a47a45dc3f29895d5cd406be5b48f2c6, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.02
query3	0.24	0.07	0.07
query4	1.64	0.11	0.10
query5	0.51	0.51	0.51
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.52	0.51
query10	0.56	0.56	0.55
query11	0.15	0.12	0.11
query12	0.14	0.11	0.11
query13	0.61	0.61	0.64
query14	0.78	0.79	0.80
query15	0.84	0.82	0.83
query16	0.39	0.41	0.39
query17	1.02	1.00	1.06
query18	0.23	0.22	0.24
query19	1.89	1.78	1.72
query20	0.01	0.02	0.01
query21	15.40	0.88	0.59
query22	0.74	0.75	0.66
query23	15.16	1.41	0.55
query24	3.32	0.48	0.25
query25	0.17	0.07	0.10
query26	0.19	0.14	0.14
query27	0.05	0.04	0.05
query28	12.40	1.15	0.43
query29	12.62	3.99	3.35
query30	0.26	0.09	0.06
query31	2.82	0.59	0.37
query32	3.22	0.55	0.47
query33	2.99	3.01	3.04
query34	16.73	5.18	4.52
query35	4.57	4.56	4.54
query36	0.63	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 102.63 s
Total hot run time: 28.18 s

@morningman
morningman merged commit ddf9805 into apache:branch-3.1 Aug 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@morningman @Thearas @doris-robot @morrySnow