[improvement](catalog) enumerate remote table names once in bypass-mode getTables by u70b3 · Pull Request #66452 · apache/doris

u70b3 · 2026-08-04T16:42:11Z

What problem does this PR solve?

Issue Number: close #66025

Problem Summary:

In the external session-bypass path (e.g. Iceberg REST session=user, where the shared table-name cache is bypassed so one user's visible table set is never served to another), ExternalDatabase.getTables() performs 1 + N remote table-name enumerations for N tables:

getTables() calls getTableNamesWithLock(), which enumerates remote table names once via listLocalTableNamesWithoutCache(sessionContext).
It then iterates the names and calls getTableNullable(tblName) per table.
In bypass mode that enters getTableNullableWithoutCache(sessionContext, tableName).
Which calls findTableNamePairWithoutCache(...) — a full listTableNames(sessionContext) scan per table.

Listing N tables therefore triggers 1 + N remote LIST TABLES calls, which is very slow against a remote catalog with many tables.

Fix: in the bypass branch, enumerate once via listTableNames(sessionContext) and build every table from that single snapshot. Semantics are preserved:

The single enumeration still runs the case-insensitive conflict check inside listTableNames, so conflicting remote names keep failing the whole listing (pinned by a new test).
Local-name dedup keeps the previous Set + findFirst() first-win semantics when two remote names map to the same local name.
Per-table build failures are still logged and skipped, exactly as before.

Verification: a unit test counting remote listings shows 4 enumerations for 3 tables before the fix and exactly 1 after.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-08-04T16:42:16Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

u70b3 · 2026-09-14T08:18:05Z

run buildall

hello-stephen · 2026-09-14T11:09:46Z

TPC-H: Total hot run time: 16936 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 27128b39cf39f25522ed58b76c39d574e1644044, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17571	3039	3034	3034
q2	2065	274	249	249
q3	10228	869	530	530
q4	4674	246	202	202
q5	7679	574	395	395
q6	134	117	96	96
q7	512	504	385	385
q8	9229	880	971	880
q9	3514	2419	2400	2400
q10	6509	865	743	743
q11	394	198	185	185
q12	638	261	198	198
q13	18114	1540	1158	1158
q14	162	152	139	139
q15	q16	444	399	374	374
q17	1909	941	813	813
q18	3104	2344	2280	2280
q19	1109	895	782	782
q20	371	287	203	203
q21	5293	1661	1875	1661
q22	334	276	229	229
Total cold run time: 93987 ms
Total hot run time: 16936 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	3387	3322	3327	3322
q2	496	398	366	366
q3	2255	2341	2219	2219
q4	1239	1194	891	891
q5	2199	2187	2145	2145
q6	172	117	89	89
q7	1021	947	845	845
q8	1584	1409	1405	1405
q9	3228	3210	3180	3180
q10	1901	1868	1658	1658
q11	364	270	249	249
q12	457	427	341	341
q13	1483	1584	1192	1192
q14	172	173	164	164
q15	q16	408	395	368	368
q17	3735	3429	3215	3215
q18	4951	4533	5008	4533
q19	938	880	872	872
q20	1053	999	844	844
q21	3876	3181	3206	3181
q22	375	340	307	307
Total cold run time: 35294 ms
Total hot run time: 31386 ms

hello-stephen · 2026-09-14T11:16:16Z

TPC-DS: Total hot run time: 82953 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 27128b39cf39f25522ed58b76c39d574e1644044, data reload: false

query5	4245	419	331	331
query6	391	132	130	130
query7	4947	407	216	216
query8	281	123	119	119
query9	8689	2873	2872	2872
query10	395	229	187	187
query11	5364	1049	925	925
query12	120	72	68	68
query13	1202	446	323	323
query14	6119	2262	2150	2150
query14_1	2015	2010	2000	2000
query15	171	121	109	109
query16	922	390	362	362
query17	795	445	358	358
query18	2335	317	228	228
query19	161	130	113	113
query20	70	67	70	67
query21	197	99	83	83
query22	5547	5589	5467	5467
query23	6861	6476	6196	6196
query23_1	6191	6060	6088	6060
query24	7267	1091	762	762
query24_1	767	787	762	762
query25	409	273	231	231
query26	1224	235	133	133
query27	2780	410	258	258
query28	4692	1485	1487	1485
query29	950	426	339	339
query30	247	160	122	122
query31	835	408	332	332
query32	123	79	67	67
query33	464	204	182	182
query34	990	815	479	479
query35	400	412	348	348
query36	557	562	530	530
query37	119	81	68	68
query38	1014	855	815	815
query39	494	475	436	436
query39_1	476	453	475	453
query40	202	88	75	75
query41	52	50	51	50
query42	76	72	72	72
query43	239	242	212	212
query44	994	541	533	533
query45	113	108	98	98
query46	764	837	539	539
query47	756	785	738	738
query48	317	317	254	254
query49	536	235	192	192
query50	744	257	196	196
query51	8037	7991	8032	7991
query52	71	67	63	63
query53	193	199	147	147
query54	211	190	152	152
query55	73	57	55	55
query56	224	200	173	173
query57	701	687	663	663
query58	204	152	165	152
query59	1251	1258	1136	1136
query60	235	178	175	175
query61	114	109	127	109
query62	425	212	187	187
query63	168	142	145	142
query64	2700	699	598	598
query65	1735	1628	1652	1628
query66	1862	256	218	218
query67	9999	9974	9819	9819
query68	2755	1239	746	746
query69	355	239	202	202
query70	681	595	616	595
query71	257	174	164	164
query72	2275	1696	1618	1618
query73	656	557	345	345
query74	1574	1265	1148	1148
query75	1192	1144	974	974
query76	2277	707	532	532
query77	257	266	222	222
query78	3978	3759	3365	3365
query79	1631	823	599	599
query80	1373	357	300	300
query81	492	158	134	134
query82	613	128	98	98
query83	294	210	203	203
query84	303	116	95	95
query85	1078	403	276	276
query86	389	199	162	162
query87	1029	994	918	918
query88	2780	2131	2099	2099
query89	282	198	174	174
query90	1917	127	126	126
query91	129	117	96	96
query92	80	64	69	64
query93	1419	1085	740	740
query94	623	243	218	218
query95	526	263	236	236
query96	815	594	282	282
query97	1094	1099	1025	1025
query98	154	140	132	132
query99	431	349	315	315
Total cold run time: 176908 ms
Total hot run time: 82953 ms

hello-stephen · 2026-09-14T11:20:39Z

ClickBench: Total hot run time: 14.97 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 27128b39cf39f25522ed58b76c39d574e1644044, data reload: false

query1	0.01	0.01	0.01
query2	0.08	0.04	0.04
query3	0.25	0.11	0.11
query4	1.61	0.10	0.09
query5	0.19	0.17	0.16
query6	1.24	0.70	0.69
query7	0.03	0.00	0.01
query8	0.05	0.03	0.02
query9	0.29	0.22	0.21
query10	0.37	0.35	0.37
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.31	0.30	0.32
query14	0.46	0.45	0.47
query15	0.36	0.35	0.36
query16	0.24	0.22	0.23
query17	0.69	0.70	0.69
query18	0.18	0.16	0.17
query19	1.22	1.17	1.16
query20	0.02	0.01	0.01
query21	15.46	0.15	0.12
query22	5.07	0.05	0.04
query23	16.18	0.27	0.10
query24	2.96	0.32	0.26
query25	0.11	0.03	0.04
query26	0.77	0.17	0.13
query27	0.03	0.03	0.03
query28	3.65	0.61	0.27
query29	12.42	3.23	2.57
query30	0.26	0.11	0.11
query31	2.76	0.38	0.18
query32	3.52	0.33	0.24
query33	1.49	1.54	1.58
query34	15.40	2.22	1.81
query35	1.78	1.78	1.77
query36	0.46	0.29	0.29
query37	0.06	0.04	0.04
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.12	0.08	0.08
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 90.63 s
Total hot run time: 14.97 s

hello-stephen · 2026-09-14T13:44:38Z

FE Regression Coverage Report

Increment line coverage 0.62% (1/161) 🎉
Increment coverage report
Complete coverage report

…de getTables (apache#66025) In the external session-bypass path (e.g. Iceberg REST session=user), ExternalDatabase.getTables() performed 1 + N remote table-name enumerations for N tables: one in getTableNamesWithLock(), then one more per table inside getTableNullableWithoutCache() -> findTableNamePairWithoutCache(). List the remote names once and build every table from that single snapshot instead. The single enumeration still runs the case-insensitive conflict check, so conflicting remote names keep failing the whole listing, and local-name dedup keeps the previous Set + findFirst() first-win semantics.

### What problem does this PR solve? Issue Number: close apache#66025 Problem Summary: The single-enumeration getTables path preserves first-win remote-to-local name mapping and skips individual table build failures. Add focused coverage for both behaviors and clarify that session-aware listings build uncached table objects. ### Release note None ### Check List (For Author) - Test: Unit Test added; not run locally because thirdparty/installed/bin/protoc is unavailable - Behavior changed: No - Does this need documentation: No

u70b3 mentioned this pull request Aug 4, 2026

[Tracking] Lance feature matrix for Doris 4.2 #66340

Open

68 of 87 tasks

u70b3 force-pushed the fix-external-db-bypass-n-plus-1 branch 3 times, most recently from fe6d568 to a068067 Compare August 6, 2026 17:54

u70b3 force-pushed the fix-external-db-bypass-n-plus-1 branch from a068067 to 7b3f439 Compare August 21, 2026 12:12

u70b3 requested review from 924060929 and CalvinKirs as code owners August 21, 2026 12:12

u70b3 force-pushed the fix-external-db-bypass-n-plus-1 branch 2 times, most recently from ec05ce2 to 27128b3 Compare September 10, 2026 08:05

u70b3 added 2 commits September 16, 2026 09:43

u70b3 force-pushed the fix-external-db-bypass-n-plus-1 branch from 27128b3 to de51ac5 Compare September 16, 2026 01:43

Navigation Menu

Repository navigation

[improvement](catalog) enumerate remote table names once in bypass-mode getTables - #66452

[improvement](catalog) enumerate remote table names once in bypass-mode getTables#66452
u70b3 wants to merge 2 commits into
apache:masterfrom
u70b3:fix-external-db-bypass-n-plus-1

u70b3 commented Aug 4, 2026

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

u70b3 commented Sep 14, 2026

Uh oh!

hello-stephen commented Sep 14, 2026

Uh oh!

hello-stephen commented Sep 14, 2026

Uh oh!

hello-stephen commented Sep 14, 2026

Uh oh!

hello-stephen commented Sep 14, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

u70b3 commented Aug 4, 2026

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 4, 2026

Uh oh!

u70b3 commented Sep 14, 2026

Uh oh!

hello-stephen commented Sep 14, 2026

Uh oh!

hello-stephen commented Sep 14, 2026

Uh oh!

hello-stephen commented Sep 14, 2026

Uh oh!

hello-stephen commented Sep 14, 2026

FE Regression Coverage Report

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants