[Performance] Avoid N+1 remote table enumeration in external session bypass SHOW TABLES path #66025

Summary

In the external session-bypass path, ExternalDatabase.getTables() performs N+1 remote table-name enumerations.

When extCatalog.shouldBypassTableNameCache(sessionContext) is true:

1. getTables() calls getTableNamesWithLock(), which enumerates remote table names once via listLocalTableNamesWithoutCache(sessionContext).
2. Then getTables() iterates all table names and calls getTableNullable(tblName).
3. In bypass mode, getTableNullable(tblName) enters getTableNullableWithoutCache(sessionContext, tableName).
4. getTableNullableWithoutCache(...) calls findTableNamePairWithoutCache(...), which again does a full listTableNames(sessionContext) scan for each table.

As a result, listing N tables triggers 1 + N remote LIST TABLES enumerations.

Affected code

fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalDatabase.java

Current structure:

public List<T> getTables() {
    Set<String> tblNames = getTableNamesWithLock();   // 1 full remote enumeration in bypass mode
    for (String tblName : tblNames) {
        T tbl = getTableNullable(tblName);            // each iteration re-enters bypass path
    }
}

public Set<String> getTableNamesWithLock() {
    if (extCatalog.shouldBypassTableNameCache(sessionContext)) {
        return Sets.newHashSet(listLocalTableNamesWithoutCache(sessionContext));
    }
}

private T getTableNullableWithoutCache(SessionContext sessionContext, String tableName) {
    Optional<Pair<String, String>> matched = findTableNamePairWithoutCache(sessionContext, tableName);
    // findTableNamePairWithoutCache() -> listTableNames(sessionContext)
}

Impact

This affects session-bypass catalogs such as delegated Iceberg REST session catalogs.

For SHOW TABLES / getTables() on a database with many tables, the current implementation scales poorly because every table object construction repeats a full remote name enumeration.

The issue is performance-only, but the overhead can become significant for large databases or expensive remote catalogs.

Scope / origin

This is a pre-existing master/baseline issue, not introduced by PR #65126.

The bypass structure already existed before the external meta cache refactor. PR #65126 carried it forward and added related session-bypass improvements elsewhere, but did not address this getTables() path.

Suggested fix

In bypass mode, reuse a single remote name enumeration inside getTables():

• enumerate (remoteName, localName) pairs once,
• build a local lookup map from the result,
• construct table objects directly from that snapshot instead of re-calling getTableNullable() for every table.

This keeps bypass semantics unchanged while removing the repeated full scans.

Related

• PR [refactor](meta cache) Refactor external metadata cache with MetaCacheEntry #65126
• Commit 48291325300 [fix](fe) Avoid unnecessary session table enumeration addressed a nearby delegated-session existence-check path, but not this getTables() N+1 path.