### What problem does this PR solve?

Issue Number: close apache#62183

Problem Summary: Fix multiple regression failures in the connector-based
JDBC/ES external catalog migration. This commit addresses 8 root causes
covering 20+ test failures from CI External Regression pipeline build 926236.

Fixes include:
- detectDoris() initialization ordering: moved from constructor to
  postInitialize() hook so data source is live before detection runs.
  Added dorisTypeToConnectorType() for Doris-to-Doris type mapping
  including HLL, BITMAP, QUANTILE_STATE types.
- Oracle TIMESTAMP pattern: startsWith("TIMESTAMP") to match
  "TIMESTAMP(6)", "TIMESTAMP(6) WITH LOCAL TIME ZONE" etc.
- Oracle NUMBER: handle null precision/scale (.orElse(0)), scale<=0
  for integer branch, and match old boundary thresholds.
- ClickHouse DB listing: use SHOW DATABASES for full listing;
  fix databaseTermIsCatalog inversion for old drivers.
- DATETIMEV2 precision: ConnectorColumnConverter reads precision
  (not scale) for datetime fractional seconds, matching connector
  encoding convention.
- EXPLAIN cache: invalidate scanNodeProperties after convertPredicate()
  so pushed conjuncts are reflected in EXPLAIN output.
- ExecutionAuthenticator: call initPreExecutionAuthenticator() in
  PluginDrivenExternalCatalog.initLocalObjectsImpl().
- PassthroughQueryTableHandle: add instanceof guards in
  JdbcConnectorMetadata to prevent ClassCastException for TVF.
- isKey: hardcode true for all columns matching legacy behavior.

### Release note

None

### Check List (For Author)

- Test: Regression test (CI External Regression pipeline)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

…faces

Issue Number: close #xxx

Problem Summary: Add the fe-connector module hierarchy (fe-connector-api and
fe-connector-spi) as the foundation for Catalog SPI modularization. This
follows the same pattern as fe-filesystem, enabling external data source
connectors to be developed and deployed as independent plugins.

fe-connector-api defines 30 zero-dependency interfaces including:
- Opaque handle types (ConnectorTableHandle, ConnectorColumnHandle, etc.)
- ConnectorMetadata composed from sub-interfaces (SchemaOps, TableOps,
  PushdownOps, StatisticsOps, WriteOps)
- apply* pushdown negotiation (FilterApplicationResult, ProjectionApplicationResult)
- ConnectorScanPlanProvider for split generation
- ConnectorCapability enum for capability declaration
- Lightweight type system (ConnectorColumn, ConnectorType) independent of fe-core

fe-connector-spi defines the provider SPI:
- ConnectorProvider extends PluginFactory for ServiceLoader discovery
- ConnectorContext for engine-to-connector runtime services

None

- Test: No need to test - module skeleton with interfaces only, no behavioral changes
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorPluginManager and session bridge for connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: This step adds the connector plugin loading infrastructure
to fe-core, mirroring the FileSystemPluginManager pattern. It enables
connector plugins to be discovered via ServiceLoader (builtins) and loaded
from the plugin directory at runtime via DirectoryPluginRuntimeManager with
ChildFirst ClassLoader isolation.

Key additions:
- ConnectorPluginManager: manages ConnectorProvider lifecycle (builtin +
  directory-based plugin loading)
- ConnectorFactory: static factory singleton initialized by Env at startup
- ConnectorSessionBuilder: bridges ConnectContext → ConnectorSession for
  ClassLoader-safe session passing across SPI boundary
- ConnectorSessionImpl: immutable ConnectorSession implementation
- DefaultConnectorContext: minimal ConnectorContext providing catalogName,
  catalogId, and FS access
- Config.connector_plugin_root: configurable plugin directory
- Env.initConnectorPluginManager(): startup initialization hook

None

- Test: No need to test - infrastructure only, no connector plugins exist yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Remove Iceberg SDK dependency from ExternalMetadataOps interface

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: The ExternalMetadataOps interface (a core fe-core interface
for external catalog metadata operations) had a direct import of
org.apache.iceberg.view.View, coupling the interface to Iceberg SDK. The
ExternalMetadataOperations factory class also imported all connector-specific
types (Iceberg, Paimon, MaxCompute, Hive), creating unnecessary coupling.

Changes:
- ExternalMetadataOps.loadView() return type changed from View to Object.
  The method is only overridden by IcebergMetadataOps, and all callers are
  in iceberg-specific code that already imports View for casting.
- Deleted ExternalMetadataOperations factory class entirely. Each catalog
  now directly constructs its own MetadataOps (the factory was a thin
  indirection adding no value since each caller already knows its type).
- Removed unused View import from IcebergMetadataOps (return type is now
  Object; iceberg view.View is only used in IcebergExternalMetaCache).

None

- Test: No need to test - pure refactoring, no behavioral change
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Abstract Hadoop Configuration to Map-based catalog properties

Issue Number: close #xxx

Problem Summary: ExternalCatalog.getConfiguration() returns org.apache.hadoop.conf.Configuration,
which leaks Hadoop types into the core catalog interface. This blocks future SPI extraction of
connector modules since fe-connector-api must be free of Hadoop dependencies.

This commit introduces getHadoopProperties() returning Map<String, String> and a static
buildHadoopConfiguration() utility, then migrates all callers (HudiExternalMetaCache,
IcebergUtils, HiveMetaStoreClientHelper) to the new pattern. The original getConfiguration()
is deprecated.

None

- Test: No need to test - pure refactor, all callers produce identical Configuration objects
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add PluginDrivenExternalCatalog adapter and SPI-based CatalogFactory

Issue Number: close #xxx

Problem Summary: The CatalogFactory uses hardcoded switch-case to create catalogs. This commit
introduces a SPI-first creation path: ConnectorFactory.createConnector() is tried before falling
back to the built-in switch-case. A new PluginDrivenExternalCatalog bridges SPI Connector instances
with the existing ExternalCatalog hierarchy, enabling third-party catalog plugins.

None

- Test: No need to test — no SPI connector plugins exist yet, all catalogs still use the fallback switch-case path. Behavior is unchanged.
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add fe-connector-es module with SPI metadata implementation

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Creates the first real connector plugin module (Elasticsearch)
implementing the connector SPI interfaces. This validates the entire SPI design
with a working metadata-only implementation. The ES scan path remains in fe-core
temporarily — only metadata operations (list databases, list tables, get schema)
are handled via the SPI.

The module includes:
- EsConnectorProvider: SPI entry point discovered via ServiceLoader
- EsConnector: Main connector, creates metadata instances
- EsConnectorMetadata: Lists ES indices as tables, parses mappings to schema
- EsConnectorRestClient: HTTP client adapted from fe-core (no fe-core deps)
- EsTypeMapping: ES type → ConnectorType mapping
- EsTableHandle: Opaque handle for ES indices
- EsConnectorProperties: Property constants and compatibility processing
- Plugin assembly ZIP descriptor for runtime deployment

None

- Test: No need to test — metadata-only SPI implementation, fe-core ES path unchanged
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add shard routing and metadata pipeline to fe-connector-es

Issue Number: close #xxx

Problem Summary: The initial fe-connector-es module (Phase 1) only supported basic
metadata operations (list databases, list tables, get schema). This commit adds
Phase 2 capabilities: shard routing discovery, node info resolution, mapping field
analysis (keyword sniff, doc_value, date compat), and full metadata orchestration.

These are prerequisites for the future ConnectorScanPlanProvider which will generate
TScanRangeLocations from the plugin side.

New files (8):
- EsMajorVersion: ES version detection (adapted from fe-core, zero fe-core deps)
- EsNodeInfo: ES node with HTTP address (host+port instead of TNetworkAddress)
- EsShardRouting: Shard routing entry (host+port instead of TNetworkAddress)
- EsShardPartitions: Shard partition map with _search_shards JSON parsing
- EsFieldContext: Field context holder (fetchFields, docValue, dateCompat maps)
- EsMappingUtils: Mapping parsing + field resolution (from EsUtil + MappingPhase)
- EsMetadataState: Full metadata state (adapted from SearchContext)
- EsMetadataFetcher: Metadata fetch orchestrator (adapted from EsMetaStateTracker)

Enhanced files (3):
- EsConnectorRestClient: added searchShards(), getHttpNodes(), get() methods
- EsConnectorMetadata: added fetchMetadataState() for full metadata pipeline
- EsConnectorProperties: added MAPPING_TYPE constant

None

- Test: No need to test — plugin module only, not activated in production
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-trino plugin module (Step 7 Phase 1)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extract Trino Connector metadata logic from fe-core into an
independent plugin module fe-connector-trino, continuing the Catalog SPI
modularization. This phase migrates the metadata-only layer (schema discovery,
type mapping, Trino plugin bootstrap) while leaving scan/predicate code in
fe-core for Step 16.

Key files:
- TrinoConnectorProvider: SPI entry point (type="trino-connector")
- TrinoDorisConnector: Lazy init, wraps Trino Connector + Session
- TrinoBootstrap: Singleton Trino plugin infrastructure init
- TrinoConnectorDorisMetadata: ConnectorMetadata impl (list/get/schema)
- TrinoTypeMapping: Trino SPI types -> ConnectorType (15+ mappings)
- TrinoPluginManager/TrinoServicesProvider: Adapted from fe-common

Design decisions:
- Copied fe-common classes (~290 lines each) into plugin rather than depending
  on fe-common (which has heavy transitive deps like hadoop-common)
- TrinoBootstrap uses singleton pattern for plugin loading, per-catalog
  connector creation
- Plugin dir resolution: property -> DORIS_HOME/plugins/connectors -> fallback

None

- Test: No need to test (metadata-only extraction, fe-core code unchanged)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-maxcompute plugin module (Step 8)

Problem Summary: Extract MaxCompute connector metadata code into an independent
plugin module as part of the Catalog SPI modularization effort.

Changes:
- Create fe-connector-maxcompute module with 11 Java files + build config
- MaxComputeConnectorProvider (SPI entry, type="max_compute")
- MaxComputeDorisConnector (Odps client lifecycle, lazy init)
- MaxComputeConnectorMetadata (listDatabases, listTables, getTableHandle, getTableSchema)
- MCTypeMapping (15+ ODPS type -> ConnectorType mappings including nested ARRAY/MAP/STRUCT)
- McStructureHelper (namespace schema abstraction, copied from fe-core)
- MCConnectorClientFactory, MCConnectorEndpoint, MCConnectorProperties (utilities)
- MaxComputeTableHandle (opaque handle wrapping ODPS Table + TableIdentifier)
- Enhance ConnectorType API with factory methods (of, arrayOf, mapOf, structOf)
- Make Connector.getScanPlanProvider() a default method (scan planning is optional for Phase 1)

Phase 1 scope: metadata-only. ScanNode, Transaction, DDL operations stay in fe-core.

None

- Test: No need to test - Phase 1 metadata extraction, SPI plugin not yet activated
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-jdbc plugin module (Phase 1: metadata-only)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extracts JDBC connector metadata logic into an independent
fe-connector-jdbc plugin module as part of the Catalog SPI modularization.
This is Step 10 of the connector SPI plan.

The module implements:
- JdbcConnectorProvider (SPI entry point, type="jdbc")
- JdbcDorisConnector (manages HikariCP connection pool lifecycle)
- JdbcConnectorMetadata (delegates to JdbcConnectorClient for metadata discovery)
- JdbcConnectorClient base class with factory method dispatching to 10 DB-specific subclasses
- All 10 DB-specific clients: MySQL, PostgreSQL, Oracle, ClickHouse, SQLServer,
  SAP HANA, Trino, OceanBase, DB2, GBase
- JdbcFieldInfo (column metadata, adapted from JdbcFieldSchema)
- JdbcDbType enum, JdbcTableHandle, JdbcConnectorProperties

Each subclass maps JDBC types to ConnectorType (the SPI type system) instead
of fe-core Type, achieving zero fe-core dependency.

None

- Test: No need to test (Phase 1 metadata-only extraction, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Extract HMS shared module to fe-connector-hms

Issue Number: close #xxx

Problem Summary: The HMS (Hive MetaStore) client code in fe-core is deeply
coupled to internal types (Column, Config, ExecutionAuthenticator, etc.), making
it impossible to directly extract. This commit creates a new clean HMS client
library module (fe-connector-hms) using SPI types, which will be shared by
future fe-connector-hive and fe-connector-hudi plugins.

Key design decisions:
- New clean interfaces using ConnectorType/ConnectorColumn (not code extraction)
- Connection-pooled Thrift client with taint-and-destroy error handling
- Supports HMS/DLF/Glue metastore types via string-based class dispatch
- Auth via functional interface (AuthAction) replacing fe-core ExecutionAuthenticator
- HmsTypeMapping mirrors HiveMetaStoreClientHelper logic but returns ConnectorType

Module contents (9 Java files, ~1440 lines):
- HmsClient: Clean interface for HMS operations
- ThriftHmsClient: Pooled Thrift implementation with ClassLoader safety
- HmsClientConfig/HmsConfHelper: Configuration DTOs and HiveConf creation
- HmsTableInfo/HmsDatabaseInfo/HmsPartitionInfo: Immutable DTOs
- HmsTypeMapping: Hive type string to ConnectorType mapping
- HmsClientException: Runtime exception class

None

- Test: No need to test - library module with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-hive plugin module (Phase 1: metadata-only)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extract Hive connector metadata operations into an independent
fe-connector-hive plugin module as Phase 1 of the Catalog SPI modularization.

This creates the fe-connector-hive plugin that:
- Implements ConnectorProvider (type="hms") for SPI-based catalog creation
- Provides HiveConnectorMetadata with listDatabases, listTables, getTableHandle,
  getTableSchema operations via the shared fe-connector-hms client
- Detects table formats (HIVE/HUDI/ICEBERG) from HMS table metadata
- Supports configurable type mapping options (binary-as-string, timestamp-tz)
- Manages HmsClient lifecycle with lazy initialization and proper shutdown

Files: 10 new files (816 lines), 2 modified files
Dependencies: fe-connector-spi + fe-connector-hms + log4j-api

None

- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-paimon plugin module (Phase 1: metadata-only)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extract Paimon connector metadata operations into an independent
fe-connector-paimon plugin module as Phase 1 of the Catalog SPI modularization.

This creates the fe-connector-paimon plugin that:
- Implements ConnectorProvider (type="paimon") for SPI-based catalog creation
- Creates Paimon Catalog instances directly via Paimon SDK (Options + CatalogContext
  + CatalogFactory) supporting all backends: filesystem, HMS, DLF, REST, JDBC
- Provides PaimonConnectorMetadata with listDatabases, listTables, getTableHandle,
  getTableSchema operations using the Paimon Catalog API
- Includes PaimonTypeMapping that converts Paimon DataType to ConnectorType,
  mirroring existing PaimonUtil.paimonTypeToDorisType logic
- Supports configurable type mapping options (binary-as-varbinary, timestamp-tz)
- Manages Paimon Catalog lifecycle with lazy init and proper shutdown

Files: 8 new files (805 lines), 1 modified file
Dependencies: fe-connector-spi + paimon-core + log4j-api

None

- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add fe-connector-hudi Phase 1 plugin module

Issue Number: close #xxx

Problem Summary: Extract Hudi connector metadata operations into an
independent SPI plugin module (fe-connector-hudi) as part of the
catalog connector SPI modularization effort.

The Hudi connector plugin provides Phase 1 read-only metadata:
- List databases and tables via HMS (shared fe-connector-hms client)
- Get table schema from HoodieTableMetaClient Avro schema
  (authoritative for schema-evolved Hudi tables)
- Detect Hudi table type (COW vs MOR) from input format
- Avro-to-ConnectorType mapping (HudiTypeMapping)

Dependencies: fe-connector-spi + fe-connector-hms + hudi-common +
hudi-hadoop-mr.

Scan planning, incremental query, statistics, and event processing
remain in fe-core temporarily.

None

- Test: No need to test (Phase 1 SPI module, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-iceberg Phase 1 — metadata-only SPI plugin

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Step 15 of the Catalog Connector SPI modularization. Extracts the Iceberg
connector into an independent plugin module (fe-connector-iceberg) that
provides read-only metadata operations through the Connector SPI.

Phase 1 implements:
- IcebergConnectorProvider: SPI entry point, type="iceberg"
- IcebergConnector: Lifecycle management, creates Iceberg SDK Catalog via
  CatalogUtil.buildIcebergCatalog() — supports all 7 backends (REST, HMS,
  Glue, DLF, JDBC, Hadoop, S3Tables)
- IcebergConnectorMetadata: listDatabaseNames (SupportsNamespaces), listTableNames,
  getTableHandle, getTableSchema via Iceberg SDK Catalog API
- IcebergTypeMapping: Iceberg types → ConnectorType (BOOLEAN, INT, LONG, FLOAT,
  DOUBLE, STRING, UUID/BINARY→VARBINARY or STRING, DECIMAL, DATE, TIMESTAMP,
  LIST, MAP, STRUCT) with enableMappingVarbinary and enableMappingTimestampTz flags
- IcebergTableHandle: Opaque handle carrying db+table coordinates
- IcebergConnectorProperties: Property key constants

Key design decisions:
- No fe-connector-hms dependency: Iceberg SDK HiveCatalog handles HMS internally
- Property-driven catalog creation: CatalogUtil.buildIcebergCatalog() handles
  all backends, subclass dispatch is just setting catalog-impl
- Hadoop Configuration built from user properties (hadoop.*, fs.*, dfs.*, hive.*)

6 Java files, ~770 lines added.

None

- Test: No need to test — Phase 1 metadata-only SPI module, not yet wired into fe-core runtime
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add PluginDrivenExternalTable schema support and GSON registration

Issue Number: close #xxx

Problem Summary:
PluginDrivenExternalTable was a minimal stub with no schema retrieval capability.
Plugin-driven catalogs could list databases and tables but could not fetch table
schemas, making them non-functional for queries. Additionally, the PluginDriven*
types were not registered in GsonUtils, preventing GSON serialization/deserialization
of plugin-driven catalog metadata for FE persistence.

This commit implements Step 18 of the Catalog Connector SPI plan:

1. **ConnectorColumnConverter** (new): Converts between the connector SPI type system
   (ConnectorColumn/ConnectorType) and Doris internal types (Column/Type). Handles
   all scalar types plus complex types (ARRAY, MAP, STRUCT) recursively, with proper
   precision/scale handling for CHAR, VARCHAR, DECIMAL, and DATETIMEV2.

2. **PluginDrivenExternalTable**: Overrides initSchema() to fetch table schema from
   the connector SPI. Uses ConnectorMetadata.getTableHandle() + getTableSchema(),
   then converts via ConnectorColumnConverter.

3. **PluginDrivenExternalCatalog**: Makes buildConnectorSession() package-private so
   PluginDrivenExternalTable can build sessions during schema init.

4. **GsonUtils**: Registers PluginDrivenExternalCatalog, PluginDrivenExternalDatabase,
   and PluginDrivenExternalTable in the respective RuntimeTypeAdapterFactory instances
   for proper GSON serialization/deserialization.

None

- Test: No need to test - infrastructure code with no active connector plugins yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorExpression framework and filter pushdown types

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Step 19 of the Catalog SPI modularization. Replaces the
placeholder ConnectorExpression class with a proper expression tree hierarchy
for cross-SPI filter/projection pushdown negotiation. Adds the
ExprToConnectorExpressionConverter in fe-core to convert Doris Expr trees
into ConnectorExpression trees at the SPI boundary.

Changes:
- ConnectorExpression: converted from final class to interface with
  getExprType() and getChildren() methods
- ConnectorExprType: discriminator enum (COLUMN_REF, LITERAL, COMPARISON,
  AND, OR, NOT, IN, BETWEEN, IS_NULL, LIKE, FUNCTION_CALL)
- 11 concrete expression types: ConnectorColumnRef, ConnectorLiteral,
  ConnectorComparison, ConnectorAnd, ConnectorOr, ConnectorNot,
  ConnectorIn, ConnectorBetween, ConnectorIsNull, ConnectorLike,
  ConnectorFunctionCall
- ConnectorRange: range bound with low/high inclusive/exclusive endpoints
- ConnectorDomain: per-column domain (union of ranges + null handling)
  for fast partition pruning
- ConnectorFilterConstraint: redesigned to carry full expression tree +
  per-column domain map (replaces old flat conjuncts list)
- ExprToConnectorExpressionConverter: converts Doris Expr tree to
  ConnectorExpression tree, plus Type to ConnectorType reverse mapping

None

- Test: No need to test - pure API/SPI type definitions and converter
    with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorScanPlanProvider and PluginDrivenScanNode for connector SPI scan planning

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Plugin-driven catalogs need a generic scan node that delegates
scan planning to the connector SPI, rather than requiring connector-specific
ScanNode subclasses in fe-core.

This commit adds Step 20 of the Catalog SPI modularization:

**fe-connector-api enhancements:**
- ConnectorScanRangeType enum: FILE_SCAN, JDBC_SCAN, ES_SCAN, REMOTE_OLAP_SCAN, CUSTOM
- ConnectorDeleteFile: delete file descriptor for Iceberg MOR tables
- ConnectorScanRange: added getRangeType() (mandatory), getFileFormat(),
  getFileSize(), getModificationTime(), getPartitionValues(), getDeleteFiles()
- ConnectorScanPlanProvider: added estimateScanRangeCount() default method

**fe-core additions:**
- PluginDrivenSplit: wraps ConnectorScanRange in FileSplit for the
  FileQueryScanNode pipeline
- PluginDrivenScanNode: extends FileQueryScanNode, delegates scan planning
  to ConnectorScanPlanProvider, uses FORMAT_JNI for BE execution
- ExprToConnectorExpressionConverter: added convertConjuncts() utility
- PhysicalPlanTranslator: dispatch to PluginDrivenScanNode for
  PluginDrivenExternalTable instances

None

- Test: No need to test — SPI infrastructure, no actual connectors activate this path yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorWriteOps & PluginDrivenTableSink (Step 21)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Implements Step 21 of the Catalog SPI modularization plan.
Adds the write operation SPI interfaces and the generic table sink for
plugin-driven external tables.

**SPI layer (fe-connector-api):**
- Enhanced ConnectorWriteOps with full write lifecycle: getWriteConfig,
  beginInsert/finishInsert/abortInsert (with column list), beginDelete/
  finishDelete/abortDelete, beginMerge/finishMerge/abortMerge
- Added ConnectorWriteType enum (FILE_WRITE, JDBC_WRITE, REMOTE_OLAP_WRITE,
  CUSTOM)
- Added ConnectorWriteConfig value object with builder pattern, carrying
  write type, file format, compression, location, partition columns, and
  generic properties
- Added ConnectorDeleteHandle and ConnectorMergeHandle marker interfaces

**Engine layer (fe-core):**
- Created PluginDrivenTableSink extending BaseExternalTableDataSink
- Constructs TDataSink based on ConnectorWriteConfig.getWriteType():
  FILE_WRITE -> THiveTableSink, JDBC_WRITE -> TJdbcTableSink
- Property-driven Thrift construction using well-known keys from
  ConnectorWriteConfig.properties

None

- Test: No need to test - infrastructure interfaces with no runtime behavior change
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add generic Nereids plan nodes for plugin-driven INSERT pipeline

Issue Number: close #xxx

Problem Summary: PluginDrivenExternalCatalog-backed tables cannot participate
in the full Nereids INSERT pipeline because there are no generic plan node
classes for them. Each existing connector (Hive, Iceberg, JDBC, etc.) has its
own typed plan nodes, but the plugin-driven generic path has none.

This commit adds the complete Nereids INSERT pipeline for plugin-driven
catalogs:

1. **Plan nodes**: UnboundConnectorTableSink, LogicalConnectorTableSink,
   PhysicalConnectorTableSink — generic nodes that work with any
   PluginDrivenExternalCatalog without connector-specific plan classes.

2. **Optimizer rules**: BINDING_INSERT_CONNECTOR_TABLE (BindSink),
   LogicalConnectorTableSinkToPhysicalConnectorTableSink (implementation),
   ExpressionRewrite for LogicalConnectorTableSink.

3. **Insert executor**: PluginDrivenInsertExecutor delegates begin/commit/
   abort to ConnectorWriteOps SPI. PluginDrivenInsertCommandContext provides
   the context wrapper.

4. **Transaction manager**: PluginDrivenTransactionManager provides
   lightweight transaction lifecycle bookkeeping; actual commit/rollback
   is handled by ConnectorWriteOps in the insert executor.

5. **Wiring**: PlanType enum entries, RuleType entries, SinkVisitor methods,
   RuleSet registration, UnboundTableSinkCreator dispatch,
   PhysicalPlanTranslator visitor, InsertIntoTableCommand executor selection.

None

- Test: No need to test — infrastructure code, no connector plugins exercise
  this path yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Activate CatalogFactory SPI path with build integration and connectivity testing

Issue Number: close #xxx

Problem Summary: The connector SPI plugins were built but not deployed to
the output directory, making them undiscoverable at runtime. Additionally,
PluginDrivenExternalCatalog lacked connectivity testing support.

This commit:
1. Adds connector module compilation to build.sh FE module list
2. Deploys connector plugin ZIPs to output/fe/plugins/connector/<name>/
   (mirroring the filesystem plugin deployment pattern)
3. Creates ConnectorTestResult value object in fe-connector-api
4. Adds Connector.testConnection(session) default method
5. Overrides checkWhenCreating() in PluginDrivenExternalCatalog to delegate
   connectivity testing to the connector SPI
6. Adds ConnectorFactory.getRegisteredTypes() for diagnostics
7. Enhances logging in CatalogFactory and Env for SPI path transparency

None

- Test: No need to test (build infrastructure + SPI wiring, no behavioral change)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Step 24: ES full migration to connector SPI

Issue Number: close #xxx

Problem Summary: Migrate the ES (Elasticsearch) connector scan path from
legacy fe-core EsScanNode to the plugin-driven connector SPI. This enables
the ES connector to run as an isolated plugin while producing the same
ES_HTTP_SCAN_NODE Thrift types expected by BE.

Key changes:
- Enhanced ConnectorScanPlanProvider with getScanRangeType(),
  getScanNodeProperties(), and getScanNodeMapProperties() to support
  non-file scan types (ES, JDBC) that need custom Thrift structures
- Created EsQueryDslBuilder: ports QueryBuilders from legacy Expr-based
  filter conversion to ConnectorExpression-based ES Query DSL generation
- Created EsScanPlanProvider: generates per-shard scan ranges with ES
  host routing, query DSL, auth info, and field context maps
- Created EsScanRange: ConnectorScanRange impl for ES shard routing
- Created PluginDrivenEsScanNode: extends ExternalScanNode to produce
  TPlanNodeType.ES_HTTP_SCAN_NODE with TEsScanNode/TEsScanRange Thrift
- Wired EsConnector.getScanPlanProvider() to return real provider
- Added EsConnector.testConnection() for catalog connectivity testing
- Fixed BindRelation missing PLUGIN_EXTERNAL_TABLE case (critical:
  without this fix, no plugin-driven table can be queried)
- PhysicalPlanTranslator dispatches to PluginDrivenEsScanNode when
  getScanRangeType() == ES_SCAN
- Removed "es" case from CatalogFactory switch (SPI handles it)

None

- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Step 25: JDBC full migration (scan only) to connector SPI

Issue Number: close #xxx

Problem Summary: Migrate JDBC connector scan path to the connector SPI
framework, enabling JDBC catalogs to run through the PluginDrivenScanNode
pipeline instead of the legacy JdbcScanNode code path.

Key changes:
- JdbcQueryBuilder: converts ConnectorExpression to SQL WHERE clauses with
  DB-specific date formatting (Oracle to_date, Trino date literal, SQL Server
  CONVERT) and LIMIT syntax (MySQL LIMIT, Oracle ROWNUM, SQL Server TOP N)
- JdbcScanRange: ConnectorScanRange with table_format_type="jdbc" and Builder
  pattern for all BE-expected jdbc_params (url, user, password, driver, query_sql,
  connection pool settings)
- JdbcScanPlanProvider: builds SQL query via JdbcQueryBuilder, returns 1 scan
  range with all JDBC connection parameters
- JdbcColumnHandle: carries localName + remoteName for SELECT clause column
  quoting via JdbcIdentifierQuoter
- JdbcConnectorMetadata.getColumnHandles(): returns column handle map for
  PluginDrivenScanNode column selection
- JdbcDorisConnector: wires getScanPlanProvider() and testConnection()
- ConnectorScanRange.getTableFormatType(): default "plugin_driven", JDBC
  overrides to "jdbc" so BE routes to JdbcJniReader
- ConnectorScanPlanProvider: 5-arg planScan() overload with limit parameter
  for JDBC LIMIT pushdown
- PluginDrivenScanNode: uses scanRange.getTableFormatType() instead of
  hardcoded constant; builds column handles via metadata.getColumnHandles();
  passes limit to 5-arg planScan()
- CatalogFactory: removed "jdbc" case from fallback switch (all JDBC catalogs
  now go through SPI path)

None

- Test: Manual test (compilation + checkstyle verified)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Step 26: MaxCompute full migration (scan only) to connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Migrates MaxCompute scan planning from fe-core to the
fe-connector-maxcompute plugin module. This removes the "max_compute" fallback
case from CatalogFactory, routing MaxCompute catalog creation exclusively
through the SPI connector path (PluginDrivenExternalCatalog).

Key changes:
- MaxComputeScanPlanProvider: creates ODPS TableBatchReadSession, generates
  splits (byte_size and row_offset strategies), supports limit optimization
- MaxComputePredicateConverter: converts ConnectorExpression to ODPS Predicate
  (CompoundPredicate, RawPredicate, UnaryPredicate) with datetime timezone
  conversion
- MaxComputeScanRange: ConnectorScanRange carrying serialized session, split
  params, and timeout config with tableFormatType="max_compute"
- MaxComputeColumnHandle: tracks partition vs data column distinction
- MCConnectorEndpoint: region-to-timezone mapping for datetime pushdown
- PluginDrivenScanNode: format-type dispatch in setScanParams() -
  "max_compute" builds TMaxComputeFileDesc, others use generic jdbc_params map
- CatalogFactory: removed "max_compute" fallback case and
  MaxComputeExternalCatalog import
- Added odps-sdk-table-api dependency for table read session API

None

- Test: No need to test - structural migration only, MaxCompute not available
  in dev environment
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Migrate Trino connector scan planning to connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Trino connector scan planning was still handled by the legacy
TrinoConnectorScanNode in fe-core with direct Trino SPI type dependencies.
This commit migrates the full scan flow to the connector SPI plugin module
(fe-connector-trino), keeping all Trino SPI types inside the plugin boundary.

Key changes:
- TrinoScanPlanProvider: core scan planning (beginQuery → applyFilter/Limit/
  Projection → getSplits → JSON-serialize → TrinoScanRange list → cleanupQuery)
- TrinoJsonSerializer: ObjectMapperProvider + HandleJsonModule + BlockJsonSerde
  for serializing Trino SPI objects to JSON
- TrinoPredicateConverter: ConnectorExpression → Trino TupleDomain<ColumnHandle>
- TrinoScanRange: ConnectorScanRange carrying pre-serialized JSON properties
- TrinoColumnHandle: ConnectorColumnHandle for projection pushdown
- PluginDrivenScanNode: "trino_connector" dispatch filling TTrinoConnectorFileDesc
  from properties map (no Trino types needed in fe-core)
- CatalogFactory: removed "trino-connector" legacy fallback case
- TrinoBootstrap: added getHandleResolver()/getTypeRegistry() accessors
- TrinoConnectorDorisMetadata: added getColumnHandles() method
- TrinoDorisConnector: replaced stub with real getScanPlanProvider() + testConnection()

None

- Test: No need to test - structural migration, scan logic ported from existing
  TrinoConnectorScanNode with no behavioral changes
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add Hive scan planning to connector plugin (Step 31)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Adds scan planning capability to the Hive connector plugin
module (fe-connector-hive), enabling file-based scan range generation
through the ConnectorScanPlanProvider SPI. This is the scan-only phase of
the Hive full migration to the connector plugin architecture.

New files in fe-connector-hive:
- HiveScanPlanProvider: Core scan planning - partition resolution via HMS,
  file listing via Hadoop FS, file splitting by configurable target size
- HiveColumnHandle: ConnectorColumnHandle wrapping column name/type/isPartKey
- HiveFileFormat: Enum mapping InputFormat/SerDe classes to format strings
- HiveScanRange: ConnectorScanRange with Builder for file split descriptors
- HiveTextProperties: Extracts text format props from HMS SerDe parameters

Enhanced existing files:
- HiveTableHandle: Rewritten with Builder pattern, now carries inputFormat,
  serializationLib, location, partitionKeyNames, sdParameters, tableParameters,
  and prunedPartitions for scan planning
- HiveConnectorMetadata: Added getColumnHandles(), applyFilter() with
  partition pruning (equality + IN predicates on partition columns)
- HiveConnector: Added getScanPlanProvider() returning HiveScanPlanProvider
- HmsTableInfo: Added sdParameters field for SerDe parameters
- ThriftHmsClient: Populates sdParameters from SerDeInfo
- PluginDrivenScanNode: Major enhancement with property-driven overrides for
  getFileFormatType, getPathPartitionKeys, getLocationProperties, getFileAttributes,
  plus hive/transactional_hive dispatch in setScanParams

Scope: Non-ACID Hive tables (Parquet, ORC, Text). ACID tables, file listing
cache, and table sampling are deferred to future steps.

None

- Test: No need to test (scan planning code path not yet wired to production
  query flow; requires HMS + Hadoop cluster for integration testing)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add Hudi scan planning to connector plugin (Step 32)

Issue Number: close #xxx

Problem Summary: Adds scan planning capability to the fe-connector-hudi plugin
module, following the same pattern established in Step 31 (Hive scan). This
enables the Hudi connector plugin to generate scan ranges that BE can consume,
supporting both COW (native Parquet/ORC reader) and MOR (JNI reader with delta
log merging) table types.

New files:
- HudiScanPlanProvider: Core scan planning — builds MetaClient, resolves
  partitions via HoodieTableMetadata API, generates COW splits (base files
  only) and MOR splits (base + delta logs with dynamic native downgrade)
- HudiScanRange: ConnectorScanRange implementation with Builder pattern,
  carrying both native reader fields and JNI metadata (instant_time, serde,
  delta_logs, column_names/types)
- HudiColumnHandle: Column handle with name, typeName, isPartitionKey

Modified files:
- HudiTableHandle: Rewritten with Builder pattern, added scan-related fields
  (inputFormat, serdeLib, partitionKeyNames, tableParameters, prunedPaths)
- HudiConnectorMetadata: Enhanced getTableHandle with scan fields from HMS,
  added getColumnHandles() and applyFilter() for partition pruning
- HudiConnector: Added getScanPlanProvider()
- PluginDrivenScanNode: Added "hudi" case in setScanParams dispatch with
  setHudiParams() method creating THudiFileDesc for both native/JNI paths

None

- Test: No need to test — plugin scan planning mirrors existing HudiScanNode
  logic; will be validated when end-to-end regression tests are enabled
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add Paimon scan planning to connector plugin (Step 33)

Problem Summary: Migrate Paimon scan planning logic from fe-core PaimonScanNode
to the fe-connector-paimon plugin module, following the same pattern established
by Hive (Step 31) and Hudi (Step 32) scan migrations.

This step adds:
- PaimonColumnHandle: Column handle with name and field index in RowType
- PaimonPredicateConverter: Converts ConnectorExpression to Paimon Predicate
  (PredicateBuilder-based, supports AND/OR/comparison/IN/IS NULL/LIKE prefix)
- PaimonScanRange: Builder-pattern scan range supporting JNI, native, and
  COUNT pushdown paths with dual serialization
- PaimonScanPlanProvider: Core scan planning using Paimon SDK (ReadBuilder →
  TableScan → Split), converting to ConnectorScanRange with JNI/native dispatch
- Modified PaimonTableHandle to carry transient Table reference
- Modified PaimonConnectorMetadata to store Table in handle and add getColumnHandles()
- Modified PaimonConnector to add getScanPlanProvider()
- Modified PluginDrivenScanNode to add paimon dispatch (setPaimonParams),
  getSerializedTable() override, and scan-level paimon params (predicate + options)

None

- Test: No need to test (compile-only migration, runtime wiring not yet active)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Remove old internal ES table code and clean up references

Issue Number: close #xxx

Problem Summary: The old internal Elasticsearch table implementation (EsScanNode,
EsExternalCatalog, EsExternalTable, etc.) has been superseded by the ES Catalog
connector plugin. This commit removes the legacy ES code from fe-core and cleans
up all references throughout the codebase.

Changes:
- Delete 21 old ES datasource files from datasource/es/
- Delete 3 nereids plan classes (LogicalEsScan, PhysicalEsScan, LogicalEsScanToPhysicalEsScan)
- Delete 6 old ES test files from external/elasticsearch/
- Clean up ES references in 13+ files (Env.java, ExternalCatalog.java, ESCatalogAction.java,
  PhysicalPlanTranslator.java, BindRelation.java, RuleSet.java, RelationVisitor.java,
  StatsCalculator.java, CostModel.java, ChildOutputPropertyDeriver.java,
  TopnFilterPushDownVisitor.java, GsonUtils.java, TableIf.java, etc.)
- Preserve EsTable.java and EsResource.java for persistence compatibility
- Add ES stub in SHOW CREATE TABLE for deprecation notice

None

- Test: Manual test - fe-core compiles successfully, ES connector tests pass
- Behavior changed: Yes - old internal ES table queries will no longer work;
  users should use ES Catalog instead
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Add metadata methods to JDBC connector for migration parity

Issue Number: close #xxx

Problem Summary: During JDBC connector modular migration from fe-core to
fe-connector-jdbc, several metadata methods were missing in the new connector
client and SPI interfaces. This commit adds the missing methods to achieve
parity with the old JdbcClient.

None

- Test: No need to test - pure method additions with no behavior change,
  existing integration paths still use old JdbcClient
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Implement JDBC write path through connector plugin pipeline

Issue Number: close #xxx

Problem Summary: The JDBC INSERT INTO write path was not wired through the
new connector plugin pipeline. This commit implements the complete JDBC write
path so that INSERT INTO on PluginDriven JDBC tables produces the correct
TJdbcTableSink thrift structure, matching the old JdbcTableSink behavior.

Changes:
- JdbcConnectorMetadata now implements ConnectorWriteOps with getWriteConfig(),
  beginInsert(), finishInsert() — builds INSERT SQL via JdbcIdentifierQuoter
  and populates all JDBC connection/pool properties
- JdbcIdentifierQuoter gains buildInsertSql() for parameterized INSERT SQL
- PluginDrivenTableSink.bindJdbcWriteSink() now populates ALL TJdbcTable
  fields: catalog_id, driver_checksum, table_name, resource_name, and all
  connection pool settings; also sets TOdbcTableType on TJdbcTableSink
- PluginDrivenInsertExecutor resolves write type from connector metadata and
  returns TransactionType.JDBC for JDBC writes (was hardcoded to HMS)
- PhysicalPlanTranslator passes actual column list to getWriteConfig()
  (was passing empty list)
- ConnectorSessionBuilder now propagates enable_odbc_transcation session var
- Explain output enhanced for JDBC write sinks (shows table type, SQL, txn)

None

- Test: No need to test - infrastructure wiring, no end-to-end JDBC catalog
  available in unit test environment; will be tested with regression tests
  after full migration
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Port JDBC function pushdown and ClickHouse FINAL to connector module

Issue Number: close #xxx

Problem Summary:
The JDBC connector module (fe-connector-jdbc) lacked function pushdown support
and ClickHouse FINAL query support. These features existed only in the old
fe-core JdbcScanNode code path. This commit ports them to the connector SPI
pipeline so PluginDrivenScanNode-based JDBC scans have feature parity.

Key changes:
- New JdbcFunctionPushdownConfig: per-DB function whitelist/blacklist/replacement
  rules with time arithmetic rewriting and JSON-configurable overrides
- Extended JdbcQueryBuilder: ConnectorFunctionCall/Like/Between to SQL conversion,
  conjunct pushdown guards (Oracle NULL, cast, function-containing expressions),
  ClickHouse SETTINGS final=1 support
- Updated JdbcScanPlanProvider to create pushdown config and pass session props
- Plumbed 4 session variables through ConnectorSessionBuilder:
  enable_ext_func_pred_pushdown, jdbc_clickhouse_query_final,
  enable_jdbc_oracle_null_predicate_push_down, enable_jdbc_cast_predicate_push_down

None

- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Route JDBC TVF and CallExecuteStmt through connector SPI

Issue Number: close #xxx

Problem Summary:
JdbcQueryTableValueFunction and CallExecuteStmtFunc only supported the old
JdbcExternalCatalog code path. Newly created JDBC catalogs use
PluginDrivenExternalCatalog via the SPI connector, so these features were
broken for new catalogs.

Key changes:
- New PassthroughQueryTableHandle in connector API for raw query TVF scans
- JdbcQueryTableValueFunction: dual path — uses ConnectorMetadata for schema
  discovery and PluginDrivenScanNode for scan planning when catalog is
  PluginDrivenExternalCatalog
- CallExecuteStmtFunc: routes to connector metadata.executeStmt() for
  PluginDrivenExternalCatalog, falls back to old JdbcExternalCatalog path
- QueryTableValueFunction factory: detects PluginDrivenExternalCatalog
- JdbcScanPlanProvider: handles PassthroughQueryTableHandle for TVF queries

None

- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Migrate JDBC identifier mapping to connector module

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
The JDBC identifier mapping logic (lower_case_meta_names, meta_names_mapping,
lower_case_table_names) was tightly coupled to fe-core via JdbcIdentifierMapping
which depends on Jackson ObjectMapper. This commit migrates the functionality
to the connector-jdbc module as part of the SPI modularization effort.

Key changes:
- Add ConnectorIdentifierOps SPI interface in connector-api with 3 name mapping
  methods (fromRemoteDatabaseName, fromRemoteTableName, fromRemoteColumnName)
- ConnectorMetadata now extends ConnectorIdentifierOps
- Create JdbcIdentifierMapper in connector-jdbc: pure Java reimplementation
  with regex-based JSON parsing (no Jackson dependency), full validation with
  case-conflict detection
- Wire identifier mapping through getColumnHandles() and getWriteConfig() so
  column names are properly mapped in both read and write paths
- Add lower_case_table_names from GlobalVariable to ConnectorSessionBuilder
- PluginDrivenExternalCatalog overrides fromRemoteDatabaseName/TableName to
  delegate to connector metadata
- PluginDrivenExternalTable.initSchema() applies fromRemoteColumnName mapping

None

- Test: Manual test / Compilation verified
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Remove old JDBC connector code after migration to plugin-driven architecture

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: After migrating JDBC connector to fe-connector-jdbc module with
plugin-driven architecture (PluginDrivenExternalCatalog/Database/Table), the old
JDBC code in fe-core/datasource/jdbc/ is dead code. This commit:

1. Adds Gson compatible subtype adapters so persisted metadata with old class names
   (JdbcExternalCatalog/Database/Table) deserializes correctly as PluginDriven* classes
2. Deletes 21 old JDBC files: JdbcExternalCatalog, JdbcExternalDatabase,
   JdbcExternalTable, JdbcNameUtil, JdbcSchemaCacheValue, JdbcTableSink, JdbcScanNode,
   JdbcSplit, JdbcFunctionPushDownRule, IdentifierMapping, JdbcIdentifierMapping,
   UnboundJdbcTableSink, LogicalJdbcTableSink, PhysicalJdbcTableSink,
   JdbcInsertExecutor, JdbcInsertCommandContext,
   LogicalJdbcScanToPhysicalJdbcScan, LogicalJdbcTableSinkToPhysicalJdbcTableSink,
   LogicalJdbcScan, PhysicalJdbcScan
3. Cleans up all references in 23 files across nereids visitors, derivers, rules,
   plan translator, expression rewrite, and external function rules
4. Inlines function pushdown constants from deleted JdbcFunctionPushDownRule into
   ExternalFunctionRules

JdbcClient hierarchy (16 files) is kept for CDC binlog functionality.

None

- Test: No need to test - pure code deletion of dead code after migration
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix connector plugin packaging and ES scan node bugs

Issue Number: close #xxx

Problem Summary:

Three issues causing CI regression test failures (build 926124):

1. **JDBC/Hive/MaxCompute plugin JARs not discovered at runtime**
   The assembly descriptors for these connectors used `includeBaseDirectory=true`,
   creating a nested subdirectory inside the zip (e.g., `fe-connector-jdbc/lib/*.jar`).
   When build.sh unzips into `plugins/connector/jdbc/`, the actual path becomes
   `plugins/connector/jdbc/fe-connector-jdbc/lib/*.jar`, but DirectoryPluginRuntimeManager
   expects JARs at `plugins/connector/jdbc/*.jar` and `plugins/connector/jdbc/lib/*.jar`.
   Fix: Set `includeBaseDirectory=false` and place plugin JAR at root + deps in lib/,
   matching the working ES/iceberg/paimon assembly layout.

2. **ES scan node URL parsing fails for http:// prefixed hosts**
   PluginDrivenEsScanNode.convertToThrift() split host strings by ":" naively,
   so `http://172.16.0.98:29200` was parsed as host="http", port="//172.16.0.98"
   causing NumberFormatException. Fix: Strip scheme prefix before parsing host:port.

3. **ES JSONB type not recognized by ConnectorColumnConverter**
   ES connector returns JSONB type but ScalarType.createType() only handles "JSON".
   ConnectorColumnConverter now maps JSONB to JSON explicitly.

None

- Test: Manual verification of zip layout; CI rerun pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Resolve JDBC driver_url via ConnectorContext environment

Problem Summary: JDBC connector plugin fails to load driver classes because
plain driver filenames (e.g., "mysql-connector-j-8.4.0.jar") are not resolved
to absolute file:// URLs using Config.jdbc_drivers_dir.

The fix adds a generic getEnvironment() method to ConnectorContext SPI,
which fe-core populates with system configs (jdbc_drivers_dir, doris_home).
The JDBC connector uses this environment to resolve driver URLs, keeping
fe-core free of JDBC-specific logic.

None

- Test: Manual test - verified driver URL resolution logic
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Implement toThrift() for PluginDrivenExternalTable via connector SPI

Problem Summary: PluginDrivenExternalTable.toThrift() inherited the base
ExternalTable implementation which returns null, causing NPE during query
fragment serialization when BE needs TTableDescriptor.

The fix adds:
1. ConnectorTableOps.getTableDescriptorProperties() SPI method that lets
   connectors declare the BE table descriptor type and properties
2. JdbcConnectorMetadata returns JDBC_TABLE type with all connection/pool
   properties needed by TJdbcTable
3. EsConnectorMetadata returns ES_TABLE type
4. PluginDrivenExternalTable.toThrift() reads these properties and
   constructs the appropriate typed TTableDescriptor (TJdbcTable/TEsTable)

None

- Test: Manual test — JDBC catalog query no longer hits null TTableDescriptor
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Connector directly builds TTableDescriptor via fe-thrift

Issue Number: close #xxx

Problem Summary: Previously PluginDrivenExternalTable.toThrift() used a Map-based
getTableDescriptorProperties() intermediary to build TTableDescriptor, which required
fe-core to contain connector-specific logic for constructing TJdbcTable/TEsTable.

This refactoring lets each connector module directly depend on fe-thrift (provided scope)
and build its own TTableDescriptor in buildTableDescriptor(), making PluginDrivenExternalTable
a pure delegator with zero connector-specific code.

Key changes:
- ConnectorTableOps: replaced getTableDescriptorProperties() with buildTableDescriptor()
  returning TTableDescriptor directly
- JdbcConnectorMetadata: builds TJdbcTable + TTableDescriptor directly
- EsConnectorMetadata: builds TEsTable + TTableDescriptor directly
- PluginDrivenExternalTable.toThrift(): simplified to pure delegation
- Added fe-thrift as provided dependency to fe-connector-api, fe-connector-jdbc,
  fe-connector-es (Maven provided scope is NOT transitive)
- All 4 plugin assembly ZIPs exclude fe-thrift and libthrift

None

- Test: Manual test (compilation verified, runtime pending BIT/BOOLEAN display fix)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix SPI column attributes, type mapping, EXPLAIN format, and DB/schema mapping

Issue Number: close #xxx

Problem Summary: Fix 13 CI test failures in the External Regression pipeline
caused by the connector migration. This batch addresses 4 root cause categories:

1. **isKey lost (5 tests)**: ConnectorColumn had no isKey field; convertColumn()
   hardcoded false. Added isKey to ConnectorColumn, propagated through
   ConnectorColumnConverter, populated from JDBC getPrimaryKeys().

2. **Type mapping (3 tests)**: HLL/BITMAP/QUANTILE_STATE types were not handled
   in JdbcMySQLConnectorClient.mapSignedType(), falling to UNSUPPORTED. Added
   the missing cases for Doris-to-Doris JDBC catalog support.

3. **DB/schema mapping (3 tests)**: ClickHouse connector ignored databaseterm
   URL parameter; OceanBase only delegated type mapping (not metadata methods);
   PostgreSQL missing bit/varbit/hstore type mappings.

4. **EXPLAIN format (2 tests)**: PluginDrivenScanNode did not output QUERY: line
   in EXPLAIN. Added getNodeExplainString() override that shows pushed-down SQL
   from JdbcScanPlanProvider.getScanNodeProperties().

None

- Test: Regression test (pending CI run)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Wire write path, statistics, row count, and test expectations through connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Fix 5 regression test failures in the connector SPI migration (Batch 2):

1. InsertUtils write sink: Add UnboundConnectorTableSink handling in
   getTargetTableQualified() for write transaction support.

2. Statistics collection: Override createAnalysisTask() in
   PluginDrivenExternalTable to return ExternalAnalysisTask.

3. Row count estimation: Add fetchRowCount() to PluginDrivenExternalTable
   delegating to ConnectorMetadata.getTableStatistics(). Add
   JdbcConnectorMetadata.getTableStatistics() and MySQL-specific
   getRowCount() querying INFORMATION_SCHEMA.

4. CallExecuteStmt test: Update expected exception message from
   "Only support JDBC catalog" to "executeStmt not supported" since
   HMS catalogs now route through PluginDrivenExternalCatalog.

5. Oracle CHAR padding: Update test_oracle_jdbc_catalog.out to match
   current OracleTypeHandler behavior which trims CHAR trailing spaces.
   This is a pre-existing behavior change from master commit 9770ecf8a59
   (JdbcTypeHandler refactor) where the .out file was never updated.

None

- Test: Regression test expectations updated
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix ES URL parsing, predicate pushdown, date compat, and TVF classloader

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Four bugs introduced during the connector SPI migration:

1. **ES double-port URL** (test_es_query_no_http_url): EsNodeInfo seed constructor
   produced "host:port:80" for inputs like "host:9200" without a scheme prefix.
   Fixed by correctly parsing host and port when split(":") yields 2 parts.

2. **ES LIKE not pushed down** (test_es_query): EsQueryDslBuilder.likeToDsl()
   checked column2typeMap BEFORE resolving keyword sub-fields from fieldsContext.
   Text fields with .keyword sub-fields were rejected. Fixed by checking
   fieldsContext.containsKey() first, matching the old QueryBuilders behavior.

3. **ES date equality returns empty** (test_es_query_nereids): compatDefaultDate()
   used ZoneOffset.UTC, but the old Joda-Time code used the system default timezone.
   When JVM runs in CST (UTC+8), "2022-08-08 08:00:00" should map to midnight UTC,
   not 8am UTC. Fixed by using ZoneId.systemDefault().

4. **TVF cross-catalog ClassCastException** (test_query_tvf_cross_catalog,
   test_query_tvf_auth): PassthroughQueryTableHandle (in org.apache.doris.connector.api)
   was loaded by both app classloader and ChildFirstClassLoader, causing instanceof
   to fail. Fixed by adding "org.apache.doris.connector.api." to the parent-first
   package list in ChildFirstClassLoader.

None

- Test: Regression test (CI pipeline)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](be) Fix CTAS ArrayStoreException for MySQL YEAR column

Issue Number: close #xxx

Problem Summary: MySQL YEAR columns mapped to Doris SMALLINT caused
ArrayStoreException: java.sql.Date in JdbcJniScanner.getNext().

Root cause: MySQLTypeHandler.getColumnValue() used untyped
rs.getObject(columnIndex) for TINYINT/SMALLINT, which returns
java.sql.Date for YEAR columns when MySQL Connector/J default
yearIsDateType=true is in effect. The SMALLINT converter then could
not handle java.sql.Date (falls through to return input as-is),
and storing java.sql.Date into Short[] caused ArrayStoreException.

Fix: Use typed rs.getObject(columnIndex, Integer.class) for TINYINT
and SMALLINT to force the JDBC driver to convert to Integer regardless
of the underlying MySQL column type.

None

- Test: Regression test (test_mysql_all_types_ctas)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](build) Change mvn package to install to fix MDEP-187 reactor dependency issue

Problem Summary: The fe-core module copy-dependencies goal fails with
MDEP-187 error when new reactor modules (fe-connector-api,
fe-connector-spi) have not been installed to the local Maven
repository. This is a known Maven limitation where copy-dependencies
cannot resolve reactor artifacts during the same mvn package run.

Fix: Change build.sh from mvn package to mvn install so all reactor
artifacts are installed to the local repo before copy-dependencies runs.

None

- Test: Manual test (build.sh --fe)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](build) Add Apache license headers to SPI service files

Problem Summary: ConnectorProvider service files in META-INF/services/
were missing Apache license headers, causing the license checker to fail.

None

- Test: No need to test (license headers only)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](es) Fix ES date predicate pushdown timezone conversion

Problem Summary: ES date equality queries returned empty results because
compatDefaultDate() could not parse ISO-formatted datetime strings produced
by extractLiteralValue(). The extractLiteralValue() method formats
LocalDateTime as "2022-08-08T08:00:00" (ISO with T separator), but
compatDefaultDate() only tried to parse "yyyy-MM-dd HH:mm:ss" (space
separator). The parse failure caused the value to be sent without timezone
conversion, so ES interpreted it as UTC instead of the local timezone.

Fix: Add ISO_LOCAL_DATE_TIME parsing as the first attempt in
compatDefaultDate(), before falling back to space-separated format and
date-only format.

None

- Test: Regression test (test_es_query_nereids sql63)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix JDBC connector type mapping, auth, and EXPLAIN regressions

Issue Number: close #62183

Problem Summary: Fix multiple regression failures in the connector-based
JDBC/ES external catalog migration. This commit addresses 8 root causes
covering 20+ test failures from CI External Regression pipeline build 926236.

Fixes include:
- detectDoris() initialization ordering: moved from constructor to
  postInitialize() hook so data source is live before detection runs.
  Added dorisTypeToConnectorType() for Doris-to-Doris type mapping
  including HLL, BITMAP, QUANTILE_STATE types.
- Oracle TIMESTAMP pattern: startsWith("TIMESTAMP") to match
  "TIMESTAMP(6)", "TIMESTAMP(6) WITH LOCAL TIME ZONE" etc.
- Oracle NUMBER: handle null precision/scale (.orElse(0)), scale<=0
  for integer branch, and match old boundary thresholds.
- ClickHouse DB listing: use SHOW DATABASES for full listing;
  fix databaseTermIsCatalog inversion for old drivers.
- DATETIMEV2 precision: ConnectorColumnConverter reads precision
  (not scale) for datetime fractional seconds, matching connector
  encoding convention.
- EXPLAIN cache: invalidate scanNodeProperties after convertPredicate()
  so pushed conjuncts are reflected in EXPLAIN output.
- ExecutionAuthenticator: call initPreExecutionAuthenticator() in
  PluginDrivenExternalCatalog.initLocalObjectsImpl().
- PassthroughQueryTableHandle: add instanceof guards in
  JdbcConnectorMetadata to prevent ClassCastException for TVF.
- isKey: hardcode true for all columns matching legacy behavior.

None

- Test: Regression test (CI External Regression pipeline)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix ES LIKE pushdown for Nereids planner

Problem Summary: Nereids planner translates Like/Regexp expressions to
FunctionCallExpr (via ExpressionTranslator.visitScalarFunction) rather
than LikePredicate. ExprToConnectorExpressionConverter only checked for
instanceof LikePredicate, so the LIKE predicate was converted to a
ConnectorFunctionCall instead of ConnectorLike. This caused ES query DSL
builder to not recognize it as a LIKE and fall back to match_all,
returning all documents instead of only those matching the wildcard.

Fix: Detect FunctionCallExpr with function name "like" or "regexp" in
ExprToConnectorExpressionConverter and convert to ConnectorLike.

None

- Test: Regression test (test_es_query sql_5_29/sql_6_29)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Add JDBC URL normalization for connector migration

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: The connector migration moved JDBC catalog logic from
fe-core to fe-connector-jdbc, but the URL normalization step was lost.
The old path called JdbcResource.handleJdbcUrl() which added critical
JDBC driver parameters. Without these, OceanBase tests fail with
timezone shifts (8h offset), lost fractional seconds, BOOLEAN display
as true/false instead of 1/0, and DOUBLE precision loss.

The fix adds JdbcUrlNormalizer in fe-connector-jdbc that replicates the
same parameter injection logic:
- MySQL/OceanBase: yearIsDateType=false, tinyInt1isBit=false,
  useUnicode=true, characterEncoding=utf-8, rewriteBatchedStatements=true
- OceanBase additionally: useCursorFetch=true
- PostgreSQL: reWriteBatchedInserts=true
- SQL Server: useBulkCopyForBatchInsert=true

The normalization is applied once in JdbcDorisConnector constructor,
so all downstream consumers (client, metadata, scan plan provider)
use the normalized URL.

None

- Test: Regression test (test_oceanbase_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Stop filtering ClickHouse system database from catalog listing

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: The new JdbcClickHouseConnectorClient incorrectly filtered
the ClickHouse "system" database in getFilterInternalDatabases(). The old
JdbcClickHouseClient never overrode this method, so it used the base class
default which only filters information_schema, performance_schema, and mysql.
The system database contains useful tables (query_log, processes, etc.) and
should remain accessible to users.

None

- Test: Regression test (test_clickhouse_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix ClickHouse JDBC catalog regression issues in plugin-driven architecture

Issue Number: close #xxx

Problem Summary: Multiple regression issues found after migr…

…faces

Issue Number: close #xxx

Problem Summary: Add the fe-connector module hierarchy (fe-connector-api and
fe-connector-spi) as the foundation for Catalog SPI modularization. This
follows the same pattern as fe-filesystem, enabling external data source
connectors to be developed and deployed as independent plugins.

fe-connector-api defines 30 zero-dependency interfaces including:
- Opaque handle types (ConnectorTableHandle, ConnectorColumnHandle, etc.)
- ConnectorMetadata composed from sub-interfaces (SchemaOps, TableOps,
  PushdownOps, StatisticsOps, WriteOps)
- apply* pushdown negotiation (FilterApplicationResult, ProjectionApplicationResult)
- ConnectorScanPlanProvider for split generation
- ConnectorCapability enum for capability declaration
- Lightweight type system (ConnectorColumn, ConnectorType) independent of fe-core

fe-connector-spi defines the provider SPI:
- ConnectorProvider extends PluginFactory for ServiceLoader discovery
- ConnectorContext for engine-to-connector runtime services

None

- Test: No need to test - module skeleton with interfaces only, no behavioral changes
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorPluginManager and session bridge for connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: This step adds the connector plugin loading infrastructure
to fe-core, mirroring the FileSystemPluginManager pattern. It enables
connector plugins to be discovered via ServiceLoader (builtins) and loaded
from the plugin directory at runtime via DirectoryPluginRuntimeManager with
ChildFirst ClassLoader isolation.

Key additions:
- ConnectorPluginManager: manages ConnectorProvider lifecycle (builtin +
  directory-based plugin loading)
- ConnectorFactory: static factory singleton initialized by Env at startup
- ConnectorSessionBuilder: bridges ConnectContext → ConnectorSession for
  ClassLoader-safe session passing across SPI boundary
- ConnectorSessionImpl: immutable ConnectorSession implementation
- DefaultConnectorContext: minimal ConnectorContext providing catalogName,
  catalogId, and FS access
- Config.connector_plugin_root: configurable plugin directory
- Env.initConnectorPluginManager(): startup initialization hook

None

- Test: No need to test - infrastructure only, no connector plugins exist yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Remove Iceberg SDK dependency from ExternalMetadataOps interface

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: The ExternalMetadataOps interface (a core fe-core interface
for external catalog metadata operations) had a direct import of
org.apache.iceberg.view.View, coupling the interface to Iceberg SDK. The
ExternalMetadataOperations factory class also imported all connector-specific
types (Iceberg, Paimon, MaxCompute, Hive), creating unnecessary coupling.

Changes:
- ExternalMetadataOps.loadView() return type changed from View to Object.
  The method is only overridden by IcebergMetadataOps, and all callers are
  in iceberg-specific code that already imports View for casting.
- Deleted ExternalMetadataOperations factory class entirely. Each catalog
  now directly constructs its own MetadataOps (the factory was a thin
  indirection adding no value since each caller already knows its type).
- Removed unused View import from IcebergMetadataOps (return type is now
  Object; iceberg view.View is only used in IcebergExternalMetaCache).

None

- Test: No need to test - pure refactoring, no behavioral change
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Abstract Hadoop Configuration to Map-based catalog properties

Issue Number: close #xxx

Problem Summary: ExternalCatalog.getConfiguration() returns org.apache.hadoop.conf.Configuration,
which leaks Hadoop types into the core catalog interface. This blocks future SPI extraction of
connector modules since fe-connector-api must be free of Hadoop dependencies.

This commit introduces getHadoopProperties() returning Map<String, String> and a static
buildHadoopConfiguration() utility, then migrates all callers (HudiExternalMetaCache,
IcebergUtils, HiveMetaStoreClientHelper) to the new pattern. The original getConfiguration()
is deprecated.

None

- Test: No need to test - pure refactor, all callers produce identical Configuration objects
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add PluginDrivenExternalCatalog adapter and SPI-based CatalogFactory

Issue Number: close #xxx

Problem Summary: The CatalogFactory uses hardcoded switch-case to create catalogs. This commit
introduces a SPI-first creation path: ConnectorFactory.createConnector() is tried before falling
back to the built-in switch-case. A new PluginDrivenExternalCatalog bridges SPI Connector instances
with the existing ExternalCatalog hierarchy, enabling third-party catalog plugins.

None

- Test: No need to test — no SPI connector plugins exist yet, all catalogs still use the fallback switch-case path. Behavior is unchanged.
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add fe-connector-es module with SPI metadata implementation

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Creates the first real connector plugin module (Elasticsearch)
implementing the connector SPI interfaces. This validates the entire SPI design
with a working metadata-only implementation. The ES scan path remains in fe-core
temporarily — only metadata operations (list databases, list tables, get schema)
are handled via the SPI.

The module includes:
- EsConnectorProvider: SPI entry point discovered via ServiceLoader
- EsConnector: Main connector, creates metadata instances
- EsConnectorMetadata: Lists ES indices as tables, parses mappings to schema
- EsConnectorRestClient: HTTP client adapted from fe-core (no fe-core deps)
- EsTypeMapping: ES type → ConnectorType mapping
- EsTableHandle: Opaque handle for ES indices
- EsConnectorProperties: Property constants and compatibility processing
- Plugin assembly ZIP descriptor for runtime deployment

None

- Test: No need to test — metadata-only SPI implementation, fe-core ES path unchanged
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add shard routing and metadata pipeline to fe-connector-es

Issue Number: close #xxx

Problem Summary: The initial fe-connector-es module (Phase 1) only supported basic
metadata operations (list databases, list tables, get schema). This commit adds
Phase 2 capabilities: shard routing discovery, node info resolution, mapping field
analysis (keyword sniff, doc_value, date compat), and full metadata orchestration.

These are prerequisites for the future ConnectorScanPlanProvider which will generate
TScanRangeLocations from the plugin side.

New files (8):
- EsMajorVersion: ES version detection (adapted from fe-core, zero fe-core deps)
- EsNodeInfo: ES node with HTTP address (host+port instead of TNetworkAddress)
- EsShardRouting: Shard routing entry (host+port instead of TNetworkAddress)
- EsShardPartitions: Shard partition map with _search_shards JSON parsing
- EsFieldContext: Field context holder (fetchFields, docValue, dateCompat maps)
- EsMappingUtils: Mapping parsing + field resolution (from EsUtil + MappingPhase)
- EsMetadataState: Full metadata state (adapted from SearchContext)
- EsMetadataFetcher: Metadata fetch orchestrator (adapted from EsMetaStateTracker)

Enhanced files (3):
- EsConnectorRestClient: added searchShards(), getHttpNodes(), get() methods
- EsConnectorMetadata: added fetchMetadataState() for full metadata pipeline
- EsConnectorProperties: added MAPPING_TYPE constant

None

- Test: No need to test — plugin module only, not activated in production
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-trino plugin module (Step 7 Phase 1)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extract Trino Connector metadata logic from fe-core into an
independent plugin module fe-connector-trino, continuing the Catalog SPI
modularization. This phase migrates the metadata-only layer (schema discovery,
type mapping, Trino plugin bootstrap) while leaving scan/predicate code in
fe-core for Step 16.

Key files:
- TrinoConnectorProvider: SPI entry point (type="trino-connector")
- TrinoDorisConnector: Lazy init, wraps Trino Connector + Session
- TrinoBootstrap: Singleton Trino plugin infrastructure init
- TrinoConnectorDorisMetadata: ConnectorMetadata impl (list/get/schema)
- TrinoTypeMapping: Trino SPI types -> ConnectorType (15+ mappings)
- TrinoPluginManager/TrinoServicesProvider: Adapted from fe-common

Design decisions:
- Copied fe-common classes (~290 lines each) into plugin rather than depending
  on fe-common (which has heavy transitive deps like hadoop-common)
- TrinoBootstrap uses singleton pattern for plugin loading, per-catalog
  connector creation
- Plugin dir resolution: property -> DORIS_HOME/plugins/connectors -> fallback

None

- Test: No need to test (metadata-only extraction, fe-core code unchanged)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-maxcompute plugin module (Step 8)

Problem Summary: Extract MaxCompute connector metadata code into an independent
plugin module as part of the Catalog SPI modularization effort.

Changes:
- Create fe-connector-maxcompute module with 11 Java files + build config
- MaxComputeConnectorProvider (SPI entry, type="max_compute")
- MaxComputeDorisConnector (Odps client lifecycle, lazy init)
- MaxComputeConnectorMetadata (listDatabases, listTables, getTableHandle, getTableSchema)
- MCTypeMapping (15+ ODPS type -> ConnectorType mappings including nested ARRAY/MAP/STRUCT)
- McStructureHelper (namespace schema abstraction, copied from fe-core)
- MCConnectorClientFactory, MCConnectorEndpoint, MCConnectorProperties (utilities)
- MaxComputeTableHandle (opaque handle wrapping ODPS Table + TableIdentifier)
- Enhance ConnectorType API with factory methods (of, arrayOf, mapOf, structOf)
- Make Connector.getScanPlanProvider() a default method (scan planning is optional for Phase 1)

Phase 1 scope: metadata-only. ScanNode, Transaction, DDL operations stay in fe-core.

None

- Test: No need to test - Phase 1 metadata extraction, SPI plugin not yet activated
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-jdbc plugin module (Phase 1: metadata-only)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extracts JDBC connector metadata logic into an independent
fe-connector-jdbc plugin module as part of the Catalog SPI modularization.
This is Step 10 of the connector SPI plan.

The module implements:
- JdbcConnectorProvider (SPI entry point, type="jdbc")
- JdbcDorisConnector (manages HikariCP connection pool lifecycle)
- JdbcConnectorMetadata (delegates to JdbcConnectorClient for metadata discovery)
- JdbcConnectorClient base class with factory method dispatching to 10 DB-specific subclasses
- All 10 DB-specific clients: MySQL, PostgreSQL, Oracle, ClickHouse, SQLServer,
  SAP HANA, Trino, OceanBase, DB2, GBase
- JdbcFieldInfo (column metadata, adapted from JdbcFieldSchema)
- JdbcDbType enum, JdbcTableHandle, JdbcConnectorProperties

Each subclass maps JDBC types to ConnectorType (the SPI type system) instead
of fe-core Type, achieving zero fe-core dependency.

None

- Test: No need to test (Phase 1 metadata-only extraction, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Extract HMS shared module to fe-connector-hms

Issue Number: close #xxx

Problem Summary: The HMS (Hive MetaStore) client code in fe-core is deeply
coupled to internal types (Column, Config, ExecutionAuthenticator, etc.), making
it impossible to directly extract. This commit creates a new clean HMS client
library module (fe-connector-hms) using SPI types, which will be shared by
future fe-connector-hive and fe-connector-hudi plugins.

Key design decisions:
- New clean interfaces using ConnectorType/ConnectorColumn (not code extraction)
- Connection-pooled Thrift client with taint-and-destroy error handling
- Supports HMS/DLF/Glue metastore types via string-based class dispatch
- Auth via functional interface (AuthAction) replacing fe-core ExecutionAuthenticator
- HmsTypeMapping mirrors HiveMetaStoreClientHelper logic but returns ConnectorType

Module contents (9 Java files, ~1440 lines):
- HmsClient: Clean interface for HMS operations
- ThriftHmsClient: Pooled Thrift implementation with ClassLoader safety
- HmsClientConfig/HmsConfHelper: Configuration DTOs and HiveConf creation
- HmsTableInfo/HmsDatabaseInfo/HmsPartitionInfo: Immutable DTOs
- HmsTypeMapping: Hive type string to ConnectorType mapping
- HmsClientException: Runtime exception class

None

- Test: No need to test - library module with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-hive plugin module (Phase 1: metadata-only)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extract Hive connector metadata operations into an independent
fe-connector-hive plugin module as Phase 1 of the Catalog SPI modularization.

This creates the fe-connector-hive plugin that:
- Implements ConnectorProvider (type="hms") for SPI-based catalog creation
- Provides HiveConnectorMetadata with listDatabases, listTables, getTableHandle,
  getTableSchema operations via the shared fe-connector-hms client
- Detects table formats (HIVE/HUDI/ICEBERG) from HMS table metadata
- Supports configurable type mapping options (binary-as-string, timestamp-tz)
- Manages HmsClient lifecycle with lazy initialization and proper shutdown

Files: 10 new files (816 lines), 2 modified files
Dependencies: fe-connector-spi + fe-connector-hms + log4j-api

None

- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-paimon plugin module (Phase 1: metadata-only)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extract Paimon connector metadata operations into an independent
fe-connector-paimon plugin module as Phase 1 of the Catalog SPI modularization.

This creates the fe-connector-paimon plugin that:
- Implements ConnectorProvider (type="paimon") for SPI-based catalog creation
- Creates Paimon Catalog instances directly via Paimon SDK (Options + CatalogContext
  + CatalogFactory) supporting all backends: filesystem, HMS, DLF, REST, JDBC
- Provides PaimonConnectorMetadata with listDatabases, listTables, getTableHandle,
  getTableSchema operations using the Paimon Catalog API
- Includes PaimonTypeMapping that converts Paimon DataType to ConnectorType,
  mirroring existing PaimonUtil.paimonTypeToDorisType logic
- Supports configurable type mapping options (binary-as-varbinary, timestamp-tz)
- Manages Paimon Catalog lifecycle with lazy init and proper shutdown

Files: 8 new files (805 lines), 1 modified file
Dependencies: fe-connector-spi + paimon-core + log4j-api

None

- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add fe-connector-hudi Phase 1 plugin module

Issue Number: close #xxx

Problem Summary: Extract Hudi connector metadata operations into an
independent SPI plugin module (fe-connector-hudi) as part of the
catalog connector SPI modularization effort.

The Hudi connector plugin provides Phase 1 read-only metadata:
- List databases and tables via HMS (shared fe-connector-hms client)
- Get table schema from HoodieTableMetaClient Avro schema
  (authoritative for schema-evolved Hudi tables)
- Detect Hudi table type (COW vs MOR) from input format
- Avro-to-ConnectorType mapping (HudiTypeMapping)

Dependencies: fe-connector-spi + fe-connector-hms + hudi-common +
hudi-hadoop-mr.

Scan planning, incremental query, statistics, and event processing
remain in fe-core temporarily.

None

- Test: No need to test (Phase 1 SPI module, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-iceberg Phase 1 — metadata-only SPI plugin

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Step 15 of the Catalog Connector SPI modularization. Extracts the Iceberg
connector into an independent plugin module (fe-connector-iceberg) that
provides read-only metadata operations through the Connector SPI.

Phase 1 implements:
- IcebergConnectorProvider: SPI entry point, type="iceberg"
- IcebergConnector: Lifecycle management, creates Iceberg SDK Catalog via
  CatalogUtil.buildIcebergCatalog() — supports all 7 backends (REST, HMS,
  Glue, DLF, JDBC, Hadoop, S3Tables)
- IcebergConnectorMetadata: listDatabaseNames (SupportsNamespaces), listTableNames,
  getTableHandle, getTableSchema via Iceberg SDK Catalog API
- IcebergTypeMapping: Iceberg types → ConnectorType (BOOLEAN, INT, LONG, FLOAT,
  DOUBLE, STRING, UUID/BINARY→VARBINARY or STRING, DECIMAL, DATE, TIMESTAMP,
  LIST, MAP, STRUCT) with enableMappingVarbinary and enableMappingTimestampTz flags
- IcebergTableHandle: Opaque handle carrying db+table coordinates
- IcebergConnectorProperties: Property key constants

Key design decisions:
- No fe-connector-hms dependency: Iceberg SDK HiveCatalog handles HMS internally
- Property-driven catalog creation: CatalogUtil.buildIcebergCatalog() handles
  all backends, subclass dispatch is just setting catalog-impl
- Hadoop Configuration built from user properties (hadoop.*, fs.*, dfs.*, hive.*)

6 Java files, ~770 lines added.

None

- Test: No need to test — Phase 1 metadata-only SPI module, not yet wired into fe-core runtime
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add PluginDrivenExternalTable schema support and GSON registration

Issue Number: close #xxx

Problem Summary:
PluginDrivenExternalTable was a minimal stub with no schema retrieval capability.
Plugin-driven catalogs could list databases and tables but could not fetch table
schemas, making them non-functional for queries. Additionally, the PluginDriven*
types were not registered in GsonUtils, preventing GSON serialization/deserialization
of plugin-driven catalog metadata for FE persistence.

This commit implements Step 18 of the Catalog Connector SPI plan:

1. **ConnectorColumnConverter** (new): Converts between the connector SPI type system
   (ConnectorColumn/ConnectorType) and Doris internal types (Column/Type). Handles
   all scalar types plus complex types (ARRAY, MAP, STRUCT) recursively, with proper
   precision/scale handling for CHAR, VARCHAR, DECIMAL, and DATETIMEV2.

2. **PluginDrivenExternalTable**: Overrides initSchema() to fetch table schema from
   the connector SPI. Uses ConnectorMetadata.getTableHandle() + getTableSchema(),
   then converts via ConnectorColumnConverter.

3. **PluginDrivenExternalCatalog**: Makes buildConnectorSession() package-private so
   PluginDrivenExternalTable can build sessions during schema init.

4. **GsonUtils**: Registers PluginDrivenExternalCatalog, PluginDrivenExternalDatabase,
   and PluginDrivenExternalTable in the respective RuntimeTypeAdapterFactory instances
   for proper GSON serialization/deserialization.

None

- Test: No need to test - infrastructure code with no active connector plugins yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorExpression framework and filter pushdown types

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Step 19 of the Catalog SPI modularization. Replaces the
placeholder ConnectorExpression class with a proper expression tree hierarchy
for cross-SPI filter/projection pushdown negotiation. Adds the
ExprToConnectorExpressionConverter in fe-core to convert Doris Expr trees
into ConnectorExpression trees at the SPI boundary.

Changes:
- ConnectorExpression: converted from final class to interface with
  getExprType() and getChildren() methods
- ConnectorExprType: discriminator enum (COLUMN_REF, LITERAL, COMPARISON,
  AND, OR, NOT, IN, BETWEEN, IS_NULL, LIKE, FUNCTION_CALL)
- 11 concrete expression types: ConnectorColumnRef, ConnectorLiteral,
  ConnectorComparison, ConnectorAnd, ConnectorOr, ConnectorNot,
  ConnectorIn, ConnectorBetween, ConnectorIsNull, ConnectorLike,
  ConnectorFunctionCall
- ConnectorRange: range bound with low/high inclusive/exclusive endpoints
- ConnectorDomain: per-column domain (union of ranges + null handling)
  for fast partition pruning
- ConnectorFilterConstraint: redesigned to carry full expression tree +
  per-column domain map (replaces old flat conjuncts list)
- ExprToConnectorExpressionConverter: converts Doris Expr tree to
  ConnectorExpression tree, plus Type to ConnectorType reverse mapping

None

- Test: No need to test - pure API/SPI type definitions and converter
    with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorScanPlanProvider and PluginDrivenScanNode for connector SPI scan planning

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Plugin-driven catalogs need a generic scan node that delegates
scan planning to the connector SPI, rather than requiring connector-specific
ScanNode subclasses in fe-core.

This commit adds Step 20 of the Catalog SPI modularization:

**fe-connector-api enhancements:**
- ConnectorScanRangeType enum: FILE_SCAN, JDBC_SCAN, ES_SCAN, REMOTE_OLAP_SCAN, CUSTOM
- ConnectorDeleteFile: delete file descriptor for Iceberg MOR tables
- ConnectorScanRange: added getRangeType() (mandatory), getFileFormat(),
  getFileSize(), getModificationTime(), getPartitionValues(), getDeleteFiles()
- ConnectorScanPlanProvider: added estimateScanRangeCount() default method

**fe-core additions:**
- PluginDrivenSplit: wraps ConnectorScanRange in FileSplit for the
  FileQueryScanNode pipeline
- PluginDrivenScanNode: extends FileQueryScanNode, delegates scan planning
  to ConnectorScanPlanProvider, uses FORMAT_JNI for BE execution
- ExprToConnectorExpressionConverter: added convertConjuncts() utility
- PhysicalPlanTranslator: dispatch to PluginDrivenScanNode for
  PluginDrivenExternalTable instances

None

- Test: No need to test — SPI infrastructure, no actual connectors activate this path yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorWriteOps & PluginDrivenTableSink (Step 21)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Implements Step 21 of the Catalog SPI modularization plan.
Adds the write operation SPI interfaces and the generic table sink for
plugin-driven external tables.

**SPI layer (fe-connector-api):**
- Enhanced ConnectorWriteOps with full write lifecycle: getWriteConfig,
  beginInsert/finishInsert/abortInsert (with column list), beginDelete/
  finishDelete/abortDelete, beginMerge/finishMerge/abortMerge
- Added ConnectorWriteType enum (FILE_WRITE, JDBC_WRITE, REMOTE_OLAP_WRITE,
  CUSTOM)
- Added ConnectorWriteConfig value object with builder pattern, carrying
  write type, file format, compression, location, partition columns, and
  generic properties
- Added ConnectorDeleteHandle and ConnectorMergeHandle marker interfaces

**Engine layer (fe-core):**
- Created PluginDrivenTableSink extending BaseExternalTableDataSink
- Constructs TDataSink based on ConnectorWriteConfig.getWriteType():
  FILE_WRITE -> THiveTableSink, JDBC_WRITE -> TJdbcTableSink
- Property-driven Thrift construction using well-known keys from
  ConnectorWriteConfig.properties

None

- Test: No need to test - infrastructure interfaces with no runtime behavior change
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add generic Nereids plan nodes for plugin-driven INSERT pipeline

Issue Number: close #xxx

Problem Summary: PluginDrivenExternalCatalog-backed tables cannot participate
in the full Nereids INSERT pipeline because there are no generic plan node
classes for them. Each existing connector (Hive, Iceberg, JDBC, etc.) has its
own typed plan nodes, but the plugin-driven generic path has none.

This commit adds the complete Nereids INSERT pipeline for plugin-driven
catalogs:

1. **Plan nodes**: UnboundConnectorTableSink, LogicalConnectorTableSink,
   PhysicalConnectorTableSink — generic nodes that work with any
   PluginDrivenExternalCatalog without connector-specific plan classes.

2. **Optimizer rules**: BINDING_INSERT_CONNECTOR_TABLE (BindSink),
   LogicalConnectorTableSinkToPhysicalConnectorTableSink (implementation),
   ExpressionRewrite for LogicalConnectorTableSink.

3. **Insert executor**: PluginDrivenInsertExecutor delegates begin/commit/
   abort to ConnectorWriteOps SPI. PluginDrivenInsertCommandContext provides
   the context wrapper.

4. **Transaction manager**: PluginDrivenTransactionManager provides
   lightweight transaction lifecycle bookkeeping; actual commit/rollback
   is handled by ConnectorWriteOps in the insert executor.

5. **Wiring**: PlanType enum entries, RuleType entries, SinkVisitor methods,
   RuleSet registration, UnboundTableSinkCreator dispatch,
   PhysicalPlanTranslator visitor, InsertIntoTableCommand executor selection.

None

- Test: No need to test — infrastructure code, no connector plugins exercise
  this path yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Activate CatalogFactory SPI path with build integration and connectivity testing

Issue Number: close #xxx

Problem Summary: The connector SPI plugins were built but not deployed to
the output directory, making them undiscoverable at runtime. Additionally,
PluginDrivenExternalCatalog lacked connectivity testing support.

This commit:
1. Adds connector module compilation to build.sh FE module list
2. Deploys connector plugin ZIPs to output/fe/plugins/connector/<name>/
   (mirroring the filesystem plugin deployment pattern)
3. Creates ConnectorTestResult value object in fe-connector-api
4. Adds Connector.testConnection(session) default method
5. Overrides checkWhenCreating() in PluginDrivenExternalCatalog to delegate
   connectivity testing to the connector SPI
6. Adds ConnectorFactory.getRegisteredTypes() for diagnostics
7. Enhances logging in CatalogFactory and Env for SPI path transparency

None

- Test: No need to test (build infrastructure + SPI wiring, no behavioral change)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Step 24: ES full migration to connector SPI

Issue Number: close #xxx

Problem Summary: Migrate the ES (Elasticsearch) connector scan path from
legacy fe-core EsScanNode to the plugin-driven connector SPI. This enables
the ES connector to run as an isolated plugin while producing the same
ES_HTTP_SCAN_NODE Thrift types expected by BE.

Key changes:
- Enhanced ConnectorScanPlanProvider with getScanRangeType(),
  getScanNodeProperties(), and getScanNodeMapProperties() to support
  non-file scan types (ES, JDBC) that need custom Thrift structures
- Created EsQueryDslBuilder: ports QueryBuilders from legacy Expr-based
  filter conversion to ConnectorExpression-based ES Query DSL generation
- Created EsScanPlanProvider: generates per-shard scan ranges with ES
  host routing, query DSL, auth info, and field context maps
- Created EsScanRange: ConnectorScanRange impl for ES shard routing
- Created PluginDrivenEsScanNode: extends ExternalScanNode to produce
  TPlanNodeType.ES_HTTP_SCAN_NODE with TEsScanNode/TEsScanRange Thrift
- Wired EsConnector.getScanPlanProvider() to return real provider
- Added EsConnector.testConnection() for catalog connectivity testing
- Fixed BindRelation missing PLUGIN_EXTERNAL_TABLE case (critical:
  without this fix, no plugin-driven table can be queried)
- PhysicalPlanTranslator dispatches to PluginDrivenEsScanNode when
  getScanRangeType() == ES_SCAN
- Removed "es" case from CatalogFactory switch (SPI handles it)

None

- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Step 25: JDBC full migration (scan only) to connector SPI

Issue Number: close #xxx

Problem Summary: Migrate JDBC connector scan path to the connector SPI
framework, enabling JDBC catalogs to run through the PluginDrivenScanNode
pipeline instead of the legacy JdbcScanNode code path.

Key changes:
- JdbcQueryBuilder: converts ConnectorExpression to SQL WHERE clauses with
  DB-specific date formatting (Oracle to_date, Trino date literal, SQL Server
  CONVERT) and LIMIT syntax (MySQL LIMIT, Oracle ROWNUM, SQL Server TOP N)
- JdbcScanRange: ConnectorScanRange with table_format_type="jdbc" and Builder
  pattern for all BE-expected jdbc_params (url, user, password, driver, query_sql,
  connection pool settings)
- JdbcScanPlanProvider: builds SQL query via JdbcQueryBuilder, returns 1 scan
  range with all JDBC connection parameters
- JdbcColumnHandle: carries localName + remoteName for SELECT clause column
  quoting via JdbcIdentifierQuoter
- JdbcConnectorMetadata.getColumnHandles(): returns column handle map for
  PluginDrivenScanNode column selection
- JdbcDorisConnector: wires getScanPlanProvider() and testConnection()
- ConnectorScanRange.getTableFormatType(): default "plugin_driven", JDBC
  overrides to "jdbc" so BE routes to JdbcJniReader
- ConnectorScanPlanProvider: 5-arg planScan() overload with limit parameter
  for JDBC LIMIT pushdown
- PluginDrivenScanNode: uses scanRange.getTableFormatType() instead of
  hardcoded constant; builds column handles via metadata.getColumnHandles();
  passes limit to 5-arg planScan()
- CatalogFactory: removed "jdbc" case from fallback switch (all JDBC catalogs
  now go through SPI path)

None

- Test: Manual test (compilation + checkstyle verified)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Step 26: MaxCompute full migration (scan only) to connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Migrates MaxCompute scan planning from fe-core to the
fe-connector-maxcompute plugin module. This removes the "max_compute" fallback
case from CatalogFactory, routing MaxCompute catalog creation exclusively
through the SPI connector path (PluginDrivenExternalCatalog).

Key changes:
- MaxComputeScanPlanProvider: creates ODPS TableBatchReadSession, generates
  splits (byte_size and row_offset strategies), supports limit optimization
- MaxComputePredicateConverter: converts ConnectorExpression to ODPS Predicate
  (CompoundPredicate, RawPredicate, UnaryPredicate) with datetime timezone
  conversion
- MaxComputeScanRange: ConnectorScanRange carrying serialized session, split
  params, and timeout config with tableFormatType="max_compute"
- MaxComputeColumnHandle: tracks partition vs data column distinction
- MCConnectorEndpoint: region-to-timezone mapping for datetime pushdown
- PluginDrivenScanNode: format-type dispatch in setScanParams() -
  "max_compute" builds TMaxComputeFileDesc, others use generic jdbc_params map
- CatalogFactory: removed "max_compute" fallback case and
  MaxComputeExternalCatalog import
- Added odps-sdk-table-api dependency for table read session API

None

- Test: No need to test - structural migration only, MaxCompute not available
  in dev environment
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Migrate Trino connector scan planning to connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Trino connector scan planning was still handled by the legacy
TrinoConnectorScanNode in fe-core with direct Trino SPI type dependencies.
This commit migrates the full scan flow to the connector SPI plugin module
(fe-connector-trino), keeping all Trino SPI types inside the plugin boundary.

Key changes:
- TrinoScanPlanProvider: core scan planning (beginQuery → applyFilter/Limit/
  Projection → getSplits → JSON-serialize → TrinoScanRange list → cleanupQuery)
- TrinoJsonSerializer: ObjectMapperProvider + HandleJsonModule + BlockJsonSerde
  for serializing Trino SPI objects to JSON
- TrinoPredicateConverter: ConnectorExpression → Trino TupleDomain<ColumnHandle>
- TrinoScanRange: ConnectorScanRange carrying pre-serialized JSON properties
- TrinoColumnHandle: ConnectorColumnHandle for projection pushdown
- PluginDrivenScanNode: "trino_connector" dispatch filling TTrinoConnectorFileDesc
  from properties map (no Trino types needed in fe-core)
- CatalogFactory: removed "trino-connector" legacy fallback case
- TrinoBootstrap: added getHandleResolver()/getTypeRegistry() accessors
- TrinoConnectorDorisMetadata: added getColumnHandles() method
- TrinoDorisConnector: replaced stub with real getScanPlanProvider() + testConnection()

None

- Test: No need to test - structural migration, scan logic ported from existing
  TrinoConnectorScanNode with no behavioral changes
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add Hive scan planning to connector plugin (Step 31)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Adds scan planning capability to the Hive connector plugin
module (fe-connector-hive), enabling file-based scan range generation
through the ConnectorScanPlanProvider SPI. This is the scan-only phase of
the Hive full migration to the connector plugin architecture.

New files in fe-connector-hive:
- HiveScanPlanProvider: Core scan planning - partition resolution via HMS,
  file listing via Hadoop FS, file splitting by configurable target size
- HiveColumnHandle: ConnectorColumnHandle wrapping column name/type/isPartKey
- HiveFileFormat: Enum mapping InputFormat/SerDe classes to format strings
- HiveScanRange: ConnectorScanRange with Builder for file split descriptors
- HiveTextProperties: Extracts text format props from HMS SerDe parameters

Enhanced existing files:
- HiveTableHandle: Rewritten with Builder pattern, now carries inputFormat,
  serializationLib, location, partitionKeyNames, sdParameters, tableParameters,
  and prunedPartitions for scan planning
- HiveConnectorMetadata: Added getColumnHandles(), applyFilter() with
  partition pruning (equality + IN predicates on partition columns)
- HiveConnector: Added getScanPlanProvider() returning HiveScanPlanProvider
- HmsTableInfo: Added sdParameters field for SerDe parameters
- ThriftHmsClient: Populates sdParameters from SerDeInfo
- PluginDrivenScanNode: Major enhancement with property-driven overrides for
  getFileFormatType, getPathPartitionKeys, getLocationProperties, getFileAttributes,
  plus hive/transactional_hive dispatch in setScanParams

Scope: Non-ACID Hive tables (Parquet, ORC, Text). ACID tables, file listing
cache, and table sampling are deferred to future steps.

None

- Test: No need to test (scan planning code path not yet wired to production
  query flow; requires HMS + Hadoop cluster for integration testing)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add Hudi scan planning to connector plugin (Step 32)

Issue Number: close #xxx

Problem Summary: Adds scan planning capability to the fe-connector-hudi plugin
module, following the same pattern established in Step 31 (Hive scan). This
enables the Hudi connector plugin to generate scan ranges that BE can consume,
supporting both COW (native Parquet/ORC reader) and MOR (JNI reader with delta
log merging) table types.

New files:
- HudiScanPlanProvider: Core scan planning — builds MetaClient, resolves
  partitions via HoodieTableMetadata API, generates COW splits (base files
  only) and MOR splits (base + delta logs with dynamic native downgrade)
- HudiScanRange: ConnectorScanRange implementation with Builder pattern,
  carrying both native reader fields and JNI metadata (instant_time, serde,
  delta_logs, column_names/types)
- HudiColumnHandle: Column handle with name, typeName, isPartitionKey

Modified files:
- HudiTableHandle: Rewritten with Builder pattern, added scan-related fields
  (inputFormat, serdeLib, partitionKeyNames, tableParameters, prunedPaths)
- HudiConnectorMetadata: Enhanced getTableHandle with scan fields from HMS,
  added getColumnHandles() and applyFilter() for partition pruning
- HudiConnector: Added getScanPlanProvider()
- PluginDrivenScanNode: Added "hudi" case in setScanParams dispatch with
  setHudiParams() method creating THudiFileDesc for both native/JNI paths

None

- Test: No need to test — plugin scan planning mirrors existing HudiScanNode
  logic; will be validated when end-to-end regression tests are enabled
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add Paimon scan planning to connector plugin (Step 33)

Problem Summary: Migrate Paimon scan planning logic from fe-core PaimonScanNode
to the fe-connector-paimon plugin module, following the same pattern established
by Hive (Step 31) and Hudi (Step 32) scan migrations.

This step adds:
- PaimonColumnHandle: Column handle with name and field index in RowType
- PaimonPredicateConverter: Converts ConnectorExpression to Paimon Predicate
  (PredicateBuilder-based, supports AND/OR/comparison/IN/IS NULL/LIKE prefix)
- PaimonScanRange: Builder-pattern scan range supporting JNI, native, and
  COUNT pushdown paths with dual serialization
- PaimonScanPlanProvider: Core scan planning using Paimon SDK (ReadBuilder →
  TableScan → Split), converting to ConnectorScanRange with JNI/native dispatch
- Modified PaimonTableHandle to carry transient Table reference
- Modified PaimonConnectorMetadata to store Table in handle and add getColumnHandles()
- Modified PaimonConnector to add getScanPlanProvider()
- Modified PluginDrivenScanNode to add paimon dispatch (setPaimonParams),
  getSerializedTable() override, and scan-level paimon params (predicate + options)

None

- Test: No need to test (compile-only migration, runtime wiring not yet active)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Remove old internal ES table code and clean up references

Issue Number: close #xxx

Problem Summary: The old internal Elasticsearch table implementation (EsScanNode,
EsExternalCatalog, EsExternalTable, etc.) has been superseded by the ES Catalog
connector plugin. This commit removes the legacy ES code from fe-core and cleans
up all references throughout the codebase.

Changes:
- Delete 21 old ES datasource files from datasource/es/
- Delete 3 nereids plan classes (LogicalEsScan, PhysicalEsScan, LogicalEsScanToPhysicalEsScan)
- Delete 6 old ES test files from external/elasticsearch/
- Clean up ES references in 13+ files (Env.java, ExternalCatalog.java, ESCatalogAction.java,
  PhysicalPlanTranslator.java, BindRelation.java, RuleSet.java, RelationVisitor.java,
  StatsCalculator.java, CostModel.java, ChildOutputPropertyDeriver.java,
  TopnFilterPushDownVisitor.java, GsonUtils.java, TableIf.java, etc.)
- Preserve EsTable.java and EsResource.java for persistence compatibility
- Add ES stub in SHOW CREATE TABLE for deprecation notice

None

- Test: Manual test - fe-core compiles successfully, ES connector tests pass
- Behavior changed: Yes - old internal ES table queries will no longer work;
  users should use ES Catalog instead
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Add metadata methods to JDBC connector for migration parity

Issue Number: close #xxx

Problem Summary: During JDBC connector modular migration from fe-core to
fe-connector-jdbc, several metadata methods were missing in the new connector
client and SPI interfaces. This commit adds the missing methods to achieve
parity with the old JdbcClient.

None

- Test: No need to test - pure method additions with no behavior change,
  existing integration paths still use old JdbcClient
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Implement JDBC write path through connector plugin pipeline

Issue Number: close #xxx

Problem Summary: The JDBC INSERT INTO write path was not wired through the
new connector plugin pipeline. This commit implements the complete JDBC write
path so that INSERT INTO on PluginDriven JDBC tables produces the correct
TJdbcTableSink thrift structure, matching the old JdbcTableSink behavior.

Changes:
- JdbcConnectorMetadata now implements ConnectorWriteOps with getWriteConfig(),
  beginInsert(), finishInsert() — builds INSERT SQL via JdbcIdentifierQuoter
  and populates all JDBC connection/pool properties
- JdbcIdentifierQuoter gains buildInsertSql() for parameterized INSERT SQL
- PluginDrivenTableSink.bindJdbcWriteSink() now populates ALL TJdbcTable
  fields: catalog_id, driver_checksum, table_name, resource_name, and all
  connection pool settings; also sets TOdbcTableType on TJdbcTableSink
- PluginDrivenInsertExecutor resolves write type from connector metadata and
  returns TransactionType.JDBC for JDBC writes (was hardcoded to HMS)
- PhysicalPlanTranslator passes actual column list to getWriteConfig()
  (was passing empty list)
- ConnectorSessionBuilder now propagates enable_odbc_transcation session var
- Explain output enhanced for JDBC write sinks (shows table type, SQL, txn)

None

- Test: No need to test - infrastructure wiring, no end-to-end JDBC catalog
  available in unit test environment; will be tested with regression tests
  after full migration
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Port JDBC function pushdown and ClickHouse FINAL to connector module

Issue Number: close #xxx

Problem Summary:
The JDBC connector module (fe-connector-jdbc) lacked function pushdown support
and ClickHouse FINAL query support. These features existed only in the old
fe-core JdbcScanNode code path. This commit ports them to the connector SPI
pipeline so PluginDrivenScanNode-based JDBC scans have feature parity.

Key changes:
- New JdbcFunctionPushdownConfig: per-DB function whitelist/blacklist/replacement
  rules with time arithmetic rewriting and JSON-configurable overrides
- Extended JdbcQueryBuilder: ConnectorFunctionCall/Like/Between to SQL conversion,
  conjunct pushdown guards (Oracle NULL, cast, function-containing expressions),
  ClickHouse SETTINGS final=1 support
- Updated JdbcScanPlanProvider to create pushdown config and pass session props
- Plumbed 4 session variables through ConnectorSessionBuilder:
  enable_ext_func_pred_pushdown, jdbc_clickhouse_query_final,
  enable_jdbc_oracle_null_predicate_push_down, enable_jdbc_cast_predicate_push_down

None

- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Route JDBC TVF and CallExecuteStmt through connector SPI

Issue Number: close #xxx

Problem Summary:
JdbcQueryTableValueFunction and CallExecuteStmtFunc only supported the old
JdbcExternalCatalog code path. Newly created JDBC catalogs use
PluginDrivenExternalCatalog via the SPI connector, so these features were
broken for new catalogs.

Key changes:
- New PassthroughQueryTableHandle in connector API for raw query TVF scans
- JdbcQueryTableValueFunction: dual path — uses ConnectorMetadata for schema
  discovery and PluginDrivenScanNode for scan planning when catalog is
  PluginDrivenExternalCatalog
- CallExecuteStmtFunc: routes to connector metadata.executeStmt() for
  PluginDrivenExternalCatalog, falls back to old JdbcExternalCatalog path
- QueryTableValueFunction factory: detects PluginDrivenExternalCatalog
- JdbcScanPlanProvider: handles PassthroughQueryTableHandle for TVF queries

None

- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Migrate JDBC identifier mapping to connector module

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
The JDBC identifier mapping logic (lower_case_meta_names, meta_names_mapping,
lower_case_table_names) was tightly coupled to fe-core via JdbcIdentifierMapping
which depends on Jackson ObjectMapper. This commit migrates the functionality
to the connector-jdbc module as part of the SPI modularization effort.

Key changes:
- Add ConnectorIdentifierOps SPI interface in connector-api with 3 name mapping
  methods (fromRemoteDatabaseName, fromRemoteTableName, fromRemoteColumnName)
- ConnectorMetadata now extends ConnectorIdentifierOps
- Create JdbcIdentifierMapper in connector-jdbc: pure Java reimplementation
  with regex-based JSON parsing (no Jackson dependency), full validation with
  case-conflict detection
- Wire identifier mapping through getColumnHandles() and getWriteConfig() so
  column names are properly mapped in both read and write paths
- Add lower_case_table_names from GlobalVariable to ConnectorSessionBuilder
- PluginDrivenExternalCatalog overrides fromRemoteDatabaseName/TableName to
  delegate to connector metadata
- PluginDrivenExternalTable.initSchema() applies fromRemoteColumnName mapping

None

- Test: Manual test / Compilation verified
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Remove old JDBC connector code after migration to plugin-driven architecture

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: After migrating JDBC connector to fe-connector-jdbc module with
plugin-driven architecture (PluginDrivenExternalCatalog/Database/Table), the old
JDBC code in fe-core/datasource/jdbc/ is dead code. This commit:

1. Adds Gson compatible subtype adapters so persisted metadata with old class names
   (JdbcExternalCatalog/Database/Table) deserializes correctly as PluginDriven* classes
2. Deletes 21 old JDBC files: JdbcExternalCatalog, JdbcExternalDatabase,
   JdbcExternalTable, JdbcNameUtil, JdbcSchemaCacheValue, JdbcTableSink, JdbcScanNode,
   JdbcSplit, JdbcFunctionPushDownRule, IdentifierMapping, JdbcIdentifierMapping,
   UnboundJdbcTableSink, LogicalJdbcTableSink, PhysicalJdbcTableSink,
   JdbcInsertExecutor, JdbcInsertCommandContext,
   LogicalJdbcScanToPhysicalJdbcScan, LogicalJdbcTableSinkToPhysicalJdbcTableSink,
   LogicalJdbcScan, PhysicalJdbcScan
3. Cleans up all references in 23 files across nereids visitors, derivers, rules,
   plan translator, expression rewrite, and external function rules
4. Inlines function pushdown constants from deleted JdbcFunctionPushDownRule into
   ExternalFunctionRules

JdbcClient hierarchy (16 files) is kept for CDC binlog functionality.

None

- Test: No need to test - pure code deletion of dead code after migration
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix connector plugin packaging and ES scan node bugs

Issue Number: close #xxx

Problem Summary:

Three issues causing CI regression test failures (build 926124):

1. **JDBC/Hive/MaxCompute plugin JARs not discovered at runtime**
   The assembly descriptors for these connectors used `includeBaseDirectory=true`,
   creating a nested subdirectory inside the zip (e.g., `fe-connector-jdbc/lib/*.jar`).
   When build.sh unzips into `plugins/connector/jdbc/`, the actual path becomes
   `plugins/connector/jdbc/fe-connector-jdbc/lib/*.jar`, but DirectoryPluginRuntimeManager
   expects JARs at `plugins/connector/jdbc/*.jar` and `plugins/connector/jdbc/lib/*.jar`.
   Fix: Set `includeBaseDirectory=false` and place plugin JAR at root + deps in lib/,
   matching the working ES/iceberg/paimon assembly layout.

2. **ES scan node URL parsing fails for http:// prefixed hosts**
   PluginDrivenEsScanNode.convertToThrift() split host strings by ":" naively,
   so `http://172.16.0.98:29200` was parsed as host="http", port="//172.16.0.98"
   causing NumberFormatException. Fix: Strip scheme prefix before parsing host:port.

3. **ES JSONB type not recognized by ConnectorColumnConverter**
   ES connector returns JSONB type but ScalarType.createType() only handles "JSON".
   ConnectorColumnConverter now maps JSONB to JSON explicitly.

None

- Test: Manual verification of zip layout; CI rerun pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Resolve JDBC driver_url via ConnectorContext environment

Problem Summary: JDBC connector plugin fails to load driver classes because
plain driver filenames (e.g., "mysql-connector-j-8.4.0.jar") are not resolved
to absolute file:// URLs using Config.jdbc_drivers_dir.

The fix adds a generic getEnvironment() method to ConnectorContext SPI,
which fe-core populates with system configs (jdbc_drivers_dir, doris_home).
The JDBC connector uses this environment to resolve driver URLs, keeping
fe-core free of JDBC-specific logic.

None

- Test: Manual test - verified driver URL resolution logic
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Implement toThrift() for PluginDrivenExternalTable via connector SPI

Problem Summary: PluginDrivenExternalTable.toThrift() inherited the base
ExternalTable implementation which returns null, causing NPE during query
fragment serialization when BE needs TTableDescriptor.

The fix adds:
1. ConnectorTableOps.getTableDescriptorProperties() SPI method that lets
   connectors declare the BE table descriptor type and properties
2. JdbcConnectorMetadata returns JDBC_TABLE type with all connection/pool
   properties needed by TJdbcTable
3. EsConnectorMetadata returns ES_TABLE type
4. PluginDrivenExternalTable.toThrift() reads these properties and
   constructs the appropriate typed TTableDescriptor (TJdbcTable/TEsTable)

None

- Test: Manual test — JDBC catalog query no longer hits null TTableDescriptor
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Connector directly builds TTableDescriptor via fe-thrift

Issue Number: close #xxx

Problem Summary: Previously PluginDrivenExternalTable.toThrift() used a Map-based
getTableDescriptorProperties() intermediary to build TTableDescriptor, which required
fe-core to contain connector-specific logic for constructing TJdbcTable/TEsTable.

This refactoring lets each connector module directly depend on fe-thrift (provided scope)
and build its own TTableDescriptor in buildTableDescriptor(), making PluginDrivenExternalTable
a pure delegator with zero connector-specific code.

Key changes:
- ConnectorTableOps: replaced getTableDescriptorProperties() with buildTableDescriptor()
  returning TTableDescriptor directly
- JdbcConnectorMetadata: builds TJdbcTable + TTableDescriptor directly
- EsConnectorMetadata: builds TEsTable + TTableDescriptor directly
- PluginDrivenExternalTable.toThrift(): simplified to pure delegation
- Added fe-thrift as provided dependency to fe-connector-api, fe-connector-jdbc,
  fe-connector-es (Maven provided scope is NOT transitive)
- All 4 plugin assembly ZIPs exclude fe-thrift and libthrift

None

- Test: Manual test (compilation verified, runtime pending BIT/BOOLEAN display fix)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix SPI column attributes, type mapping, EXPLAIN format, and DB/schema mapping

Issue Number: close #xxx

Problem Summary: Fix 13 CI test failures in the External Regression pipeline
caused by the connector migration. This batch addresses 4 root cause categories:

1. **isKey lost (5 tests)**: ConnectorColumn had no isKey field; convertColumn()
   hardcoded false. Added isKey to ConnectorColumn, propagated through
   ConnectorColumnConverter, populated from JDBC getPrimaryKeys().

2. **Type mapping (3 tests)**: HLL/BITMAP/QUANTILE_STATE types were not handled
   in JdbcMySQLConnectorClient.mapSignedType(), falling to UNSUPPORTED. Added
   the missing cases for Doris-to-Doris JDBC catalog support.

3. **DB/schema mapping (3 tests)**: ClickHouse connector ignored databaseterm
   URL parameter; OceanBase only delegated type mapping (not metadata methods);
   PostgreSQL missing bit/varbit/hstore type mappings.

4. **EXPLAIN format (2 tests)**: PluginDrivenScanNode did not output QUERY: line
   in EXPLAIN. Added getNodeExplainString() override that shows pushed-down SQL
   from JdbcScanPlanProvider.getScanNodeProperties().

None

- Test: Regression test (pending CI run)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Wire write path, statistics, row count, and test expectations through connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Fix 5 regression test failures in the connector SPI migration (Batch 2):

1. InsertUtils write sink: Add UnboundConnectorTableSink handling in
   getTargetTableQualified() for write transaction support.

2. Statistics collection: Override createAnalysisTask() in
   PluginDrivenExternalTable to return ExternalAnalysisTask.

3. Row count estimation: Add fetchRowCount() to PluginDrivenExternalTable
   delegating to ConnectorMetadata.getTableStatistics(). Add
   JdbcConnectorMetadata.getTableStatistics() and MySQL-specific
   getRowCount() querying INFORMATION_SCHEMA.

4. CallExecuteStmt test: Update expected exception message from
   "Only support JDBC catalog" to "executeStmt not supported" since
   HMS catalogs now route through PluginDrivenExternalCatalog.

5. Oracle CHAR padding: Update test_oracle_jdbc_catalog.out to match
   current OracleTypeHandler behavior which trims CHAR trailing spaces.
   This is a pre-existing behavior change from master commit 9770ecf8a59
   (JdbcTypeHandler refactor) where the .out file was never updated.

None

- Test: Regression test expectations updated
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix ES URL parsing, predicate pushdown, date compat, and TVF classloader

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Four bugs introduced during the connector SPI migration:

1. **ES double-port URL** (test_es_query_no_http_url): EsNodeInfo seed constructor
   produced "host:port:80" for inputs like "host:9200" without a scheme prefix.
   Fixed by correctly parsing host and port when split(":") yields 2 parts.

2. **ES LIKE not pushed down** (test_es_query): EsQueryDslBuilder.likeToDsl()
   checked column2typeMap BEFORE resolving keyword sub-fields from fieldsContext.
   Text fields with .keyword sub-fields were rejected. Fixed by checking
   fieldsContext.containsKey() first, matching the old QueryBuilders behavior.

3. **ES date equality returns empty** (test_es_query_nereids): compatDefaultDate()
   used ZoneOffset.UTC, but the old Joda-Time code used the system default timezone.
   When JVM runs in CST (UTC+8), "2022-08-08 08:00:00" should map to midnight UTC,
   not 8am UTC. Fixed by using ZoneId.systemDefault().

4. **TVF cross-catalog ClassCastException** (test_query_tvf_cross_catalog,
   test_query_tvf_auth): PassthroughQueryTableHandle (in org.apache.doris.connector.api)
   was loaded by both app classloader and ChildFirstClassLoader, causing instanceof
   to fail. Fixed by adding "org.apache.doris.connector.api." to the parent-first
   package list in ChildFirstClassLoader.

None

- Test: Regression test (CI pipeline)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](be) Fix CTAS ArrayStoreException for MySQL YEAR column

Issue Number: close #xxx

Problem Summary: MySQL YEAR columns mapped to Doris SMALLINT caused
ArrayStoreException: java.sql.Date in JdbcJniScanner.getNext().

Root cause: MySQLTypeHandler.getColumnValue() used untyped
rs.getObject(columnIndex) for TINYINT/SMALLINT, which returns
java.sql.Date for YEAR columns when MySQL Connector/J default
yearIsDateType=true is in effect. The SMALLINT converter then could
not handle java.sql.Date (falls through to return input as-is),
and storing java.sql.Date into Short[] caused ArrayStoreException.

Fix: Use typed rs.getObject(columnIndex, Integer.class) for TINYINT
and SMALLINT to force the JDBC driver to convert to Integer regardless
of the underlying MySQL column type.

None

- Test: Regression test (test_mysql_all_types_ctas)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](build) Change mvn package to install to fix MDEP-187 reactor dependency issue

Problem Summary: The fe-core module copy-dependencies goal fails with
MDEP-187 error when new reactor modules (fe-connector-api,
fe-connector-spi) have not been installed to the local Maven
repository. This is a known Maven limitation where copy-dependencies
cannot resolve reactor artifacts during the same mvn package run.

Fix: Change build.sh from mvn package to mvn install so all reactor
artifacts are installed to the local repo before copy-dependencies runs.

None

- Test: Manual test (build.sh --fe)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](build) Add Apache license headers to SPI service files

Problem Summary: ConnectorProvider service files in META-INF/services/
were missing Apache license headers, causing the license checker to fail.

None

- Test: No need to test (license headers only)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](es) Fix ES date predicate pushdown timezone conversion

Problem Summary: ES date equality queries returned empty results because
compatDefaultDate() could not parse ISO-formatted datetime strings produced
by extractLiteralValue(). The extractLiteralValue() method formats
LocalDateTime as "2022-08-08T08:00:00" (ISO with T separator), but
compatDefaultDate() only tried to parse "yyyy-MM-dd HH:mm:ss" (space
separator). The parse failure caused the value to be sent without timezone
conversion, so ES interpreted it as UTC instead of the local timezone.

Fix: Add ISO_LOCAL_DATE_TIME parsing as the first attempt in
compatDefaultDate(), before falling back to space-separated format and
date-only format.

None

- Test: Regression test (test_es_query_nereids sql63)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix JDBC connector type mapping, auth, and EXPLAIN regressions

Issue Number: close #62183

Problem Summary: Fix multiple regression failures in the connector-based
JDBC/ES external catalog migration. This commit addresses 8 root causes
covering 20+ test failures from CI External Regression pipeline build 926236.

Fixes include:
- detectDoris() initialization ordering: moved from constructor to
  postInitialize() hook so data source is live before detection runs.
  Added dorisTypeToConnectorType() for Doris-to-Doris type mapping
  including HLL, BITMAP, QUANTILE_STATE types.
- Oracle TIMESTAMP pattern: startsWith("TIMESTAMP") to match
  "TIMESTAMP(6)", "TIMESTAMP(6) WITH LOCAL TIME ZONE" etc.
- Oracle NUMBER: handle null precision/scale (.orElse(0)), scale<=0
  for integer branch, and match old boundary thresholds.
- ClickHouse DB listing: use SHOW DATABASES for full listing;
  fix databaseTermIsCatalog inversion for old drivers.
- DATETIMEV2 precision: ConnectorColumnConverter reads precision
  (not scale) for datetime fractional seconds, matching connector
  encoding convention.
- EXPLAIN cache: invalidate scanNodeProperties after convertPredicate()
  so pushed conjuncts are reflected in EXPLAIN output.
- ExecutionAuthenticator: call initPreExecutionAuthenticator() in
  PluginDrivenExternalCatalog.initLocalObjectsImpl().
- PassthroughQueryTableHandle: add instanceof guards in
  JdbcConnectorMetadata to prevent ClassCastException for TVF.
- isKey: hardcode true for all columns matching legacy behavior.

None

- Test: Regression test (CI External Regression pipeline)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix ES LIKE pushdown for Nereids planner

Problem Summary: Nereids planner translates Like/Regexp expressions to
FunctionCallExpr (via ExpressionTranslator.visitScalarFunction) rather
than LikePredicate. ExprToConnectorExpressionConverter only checked for
instanceof LikePredicate, so the LIKE predicate was converted to a
ConnectorFunctionCall instead of ConnectorLike. This caused ES query DSL
builder to not recognize it as a LIKE and fall back to match_all,
returning all documents instead of only those matching the wildcard.

Fix: Detect FunctionCallExpr with function name "like" or "regexp" in
ExprToConnectorExpressionConverter and convert to ConnectorLike.

None

- Test: Regression test (test_es_query sql_5_29/sql_6_29)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Add JDBC URL normalization for connector migration

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: The connector migration moved JDBC catalog logic from
fe-core to fe-connector-jdbc, but the URL normalization step was lost.
The old path called JdbcResource.handleJdbcUrl() which added critical
JDBC driver parameters. Without these, OceanBase tests fail with
timezone shifts (8h offset), lost fractional seconds, BOOLEAN display
as true/false instead of 1/0, and DOUBLE precision loss.

The fix adds JdbcUrlNormalizer in fe-connector-jdbc that replicates the
same parameter injection logic:
- MySQL/OceanBase: yearIsDateType=false, tinyInt1isBit=false,
  useUnicode=true, characterEncoding=utf-8, rewriteBatchedStatements=true
- OceanBase additionally: useCursorFetch=true
- PostgreSQL: reWriteBatchedInserts=true
- SQL Server: useBulkCopyForBatchInsert=true

The normalization is applied once in JdbcDorisConnector constructor,
so all downstream consumers (client, metadata, scan plan provider)
use the normalized URL.

None

- Test: Regression test (test_oceanbase_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Stop filtering ClickHouse system database from catalog listing

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: The new JdbcClickHouseConnectorClient incorrectly filtered
the ClickHouse "system" database in getFilterInternalDatabases(). The old
JdbcClickHouseClient never overrode this method, so it used the base class
default which only filters information_schema, performance_schema, and mysql.
The system database contains useful tables (query_log, processes, etc.) and
should remain accessible to users.

None

- Test: Regression test (test_clickhouse_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix ClickHouse JDBC catalog regression issues in plugin-driven architecture

Issue Number: close #xxx

Problem Summary: Multiple regression issues found after migr…

…faces

Issue Number: close #xxx

Problem Summary: Add the fe-connector module hierarchy (fe-connector-api and
fe-connector-spi) as the foundation for Catalog SPI modularization. This
follows the same pattern as fe-filesystem, enabling external data source
connectors to be developed and deployed as independent plugins.

fe-connector-api defines 30 zero-dependency interfaces including:
- Opaque handle types (ConnectorTableHandle, ConnectorColumnHandle, etc.)
- ConnectorMetadata composed from sub-interfaces (SchemaOps, TableOps,
  PushdownOps, StatisticsOps, WriteOps)
- apply* pushdown negotiation (FilterApplicationResult, ProjectionApplicationResult)
- ConnectorScanPlanProvider for split generation
- ConnectorCapability enum for capability declaration
- Lightweight type system (ConnectorColumn, ConnectorType) independent of fe-core

fe-connector-spi defines the provider SPI:
- ConnectorProvider extends PluginFactory for ServiceLoader discovery
- ConnectorContext for engine-to-connector runtime services

None

- Test: No need to test - module skeleton with interfaces only, no behavioral changes
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorPluginManager and session bridge for connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: This step adds the connector plugin loading infrastructure
to fe-core, mirroring the FileSystemPluginManager pattern. It enables
connector plugins to be discovered via ServiceLoader (builtins) and loaded
from the plugin directory at runtime via DirectoryPluginRuntimeManager with
ChildFirst ClassLoader isolation.

Key additions:
- ConnectorPluginManager: manages ConnectorProvider lifecycle (builtin +
  directory-based plugin loading)
- ConnectorFactory: static factory singleton initialized by Env at startup
- ConnectorSessionBuilder: bridges ConnectContext → ConnectorSession for
  ClassLoader-safe session passing across SPI boundary
- ConnectorSessionImpl: immutable ConnectorSession implementation
- DefaultConnectorContext: minimal ConnectorContext providing catalogName,
  catalogId, and FS access
- Config.connector_plugin_root: configurable plugin directory
- Env.initConnectorPluginManager(): startup initialization hook

None

- Test: No need to test - infrastructure only, no connector plugins exist yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Remove Iceberg SDK dependency from ExternalMetadataOps interface

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: The ExternalMetadataOps interface (a core fe-core interface
for external catalog metadata operations) had a direct import of
org.apache.iceberg.view.View, coupling the interface to Iceberg SDK. The
ExternalMetadataOperations factory class also imported all connector-specific
types (Iceberg, Paimon, MaxCompute, Hive), creating unnecessary coupling.

Changes:
- ExternalMetadataOps.loadView() return type changed from View to Object.
  The method is only overridden by IcebergMetadataOps, and all callers are
  in iceberg-specific code that already imports View for casting.
- Deleted ExternalMetadataOperations factory class entirely. Each catalog
  now directly constructs its own MetadataOps (the factory was a thin
  indirection adding no value since each caller already knows its type).
- Removed unused View import from IcebergMetadataOps (return type is now
  Object; iceberg view.View is only used in IcebergExternalMetaCache).

None

- Test: No need to test - pure refactoring, no behavioral change
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Abstract Hadoop Configuration to Map-based catalog properties

Issue Number: close #xxx

Problem Summary: ExternalCatalog.getConfiguration() returns org.apache.hadoop.conf.Configuration,
which leaks Hadoop types into the core catalog interface. This blocks future SPI extraction of
connector modules since fe-connector-api must be free of Hadoop dependencies.

This commit introduces getHadoopProperties() returning Map<String, String> and a static
buildHadoopConfiguration() utility, then migrates all callers (HudiExternalMetaCache,
IcebergUtils, HiveMetaStoreClientHelper) to the new pattern. The original getConfiguration()
is deprecated.

None

- Test: No need to test - pure refactor, all callers produce identical Configuration objects
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add PluginDrivenExternalCatalog adapter and SPI-based CatalogFactory

Issue Number: close #xxx

Problem Summary: The CatalogFactory uses hardcoded switch-case to create catalogs. This commit
introduces a SPI-first creation path: ConnectorFactory.createConnector() is tried before falling
back to the built-in switch-case. A new PluginDrivenExternalCatalog bridges SPI Connector instances
with the existing ExternalCatalog hierarchy, enabling third-party catalog plugins.

None

- Test: No need to test — no SPI connector plugins exist yet, all catalogs still use the fallback switch-case path. Behavior is unchanged.
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add fe-connector-es module with SPI metadata implementation

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Creates the first real connector plugin module (Elasticsearch)
implementing the connector SPI interfaces. This validates the entire SPI design
with a working metadata-only implementation. The ES scan path remains in fe-core
temporarily — only metadata operations (list databases, list tables, get schema)
are handled via the SPI.

The module includes:
- EsConnectorProvider: SPI entry point discovered via ServiceLoader
- EsConnector: Main connector, creates metadata instances
- EsConnectorMetadata: Lists ES indices as tables, parses mappings to schema
- EsConnectorRestClient: HTTP client adapted from fe-core (no fe-core deps)
- EsTypeMapping: ES type → ConnectorType mapping
- EsTableHandle: Opaque handle for ES indices
- EsConnectorProperties: Property constants and compatibility processing
- Plugin assembly ZIP descriptor for runtime deployment

None

- Test: No need to test — metadata-only SPI implementation, fe-core ES path unchanged
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add shard routing and metadata pipeline to fe-connector-es

Issue Number: close #xxx

Problem Summary: The initial fe-connector-es module (Phase 1) only supported basic
metadata operations (list databases, list tables, get schema). This commit adds
Phase 2 capabilities: shard routing discovery, node info resolution, mapping field
analysis (keyword sniff, doc_value, date compat), and full metadata orchestration.

These are prerequisites for the future ConnectorScanPlanProvider which will generate
TScanRangeLocations from the plugin side.

New files (8):
- EsMajorVersion: ES version detection (adapted from fe-core, zero fe-core deps)
- EsNodeInfo: ES node with HTTP address (host+port instead of TNetworkAddress)
- EsShardRouting: Shard routing entry (host+port instead of TNetworkAddress)
- EsShardPartitions: Shard partition map with _search_shards JSON parsing
- EsFieldContext: Field context holder (fetchFields, docValue, dateCompat maps)
- EsMappingUtils: Mapping parsing + field resolution (from EsUtil + MappingPhase)
- EsMetadataState: Full metadata state (adapted from SearchContext)
- EsMetadataFetcher: Metadata fetch orchestrator (adapted from EsMetaStateTracker)

Enhanced files (3):
- EsConnectorRestClient: added searchShards(), getHttpNodes(), get() methods
- EsConnectorMetadata: added fetchMetadataState() for full metadata pipeline
- EsConnectorProperties: added MAPPING_TYPE constant

None

- Test: No need to test — plugin module only, not activated in production
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-trino plugin module (Step 7 Phase 1)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extract Trino Connector metadata logic from fe-core into an
independent plugin module fe-connector-trino, continuing the Catalog SPI
modularization. This phase migrates the metadata-only layer (schema discovery,
type mapping, Trino plugin bootstrap) while leaving scan/predicate code in
fe-core for Step 16.

Key files:
- TrinoConnectorProvider: SPI entry point (type="trino-connector")
- TrinoDorisConnector: Lazy init, wraps Trino Connector + Session
- TrinoBootstrap: Singleton Trino plugin infrastructure init
- TrinoConnectorDorisMetadata: ConnectorMetadata impl (list/get/schema)
- TrinoTypeMapping: Trino SPI types -> ConnectorType (15+ mappings)
- TrinoPluginManager/TrinoServicesProvider: Adapted from fe-common

Design decisions:
- Copied fe-common classes (~290 lines each) into plugin rather than depending
  on fe-common (which has heavy transitive deps like hadoop-common)
- TrinoBootstrap uses singleton pattern for plugin loading, per-catalog
  connector creation
- Plugin dir resolution: property -> DORIS_HOME/plugins/connectors -> fallback

None

- Test: No need to test (metadata-only extraction, fe-core code unchanged)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-maxcompute plugin module (Step 8)

Problem Summary: Extract MaxCompute connector metadata code into an independent
plugin module as part of the Catalog SPI modularization effort.

Changes:
- Create fe-connector-maxcompute module with 11 Java files + build config
- MaxComputeConnectorProvider (SPI entry, type="max_compute")
- MaxComputeDorisConnector (Odps client lifecycle, lazy init)
- MaxComputeConnectorMetadata (listDatabases, listTables, getTableHandle, getTableSchema)
- MCTypeMapping (15+ ODPS type -> ConnectorType mappings including nested ARRAY/MAP/STRUCT)
- McStructureHelper (namespace schema abstraction, copied from fe-core)
- MCConnectorClientFactory, MCConnectorEndpoint, MCConnectorProperties (utilities)
- MaxComputeTableHandle (opaque handle wrapping ODPS Table + TableIdentifier)
- Enhance ConnectorType API with factory methods (of, arrayOf, mapOf, structOf)
- Make Connector.getScanPlanProvider() a default method (scan planning is optional for Phase 1)

Phase 1 scope: metadata-only. ScanNode, Transaction, DDL operations stay in fe-core.

None

- Test: No need to test - Phase 1 metadata extraction, SPI plugin not yet activated
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-jdbc plugin module (Phase 1: metadata-only)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extracts JDBC connector metadata logic into an independent
fe-connector-jdbc plugin module as part of the Catalog SPI modularization.
This is Step 10 of the connector SPI plan.

The module implements:
- JdbcConnectorProvider (SPI entry point, type="jdbc")
- JdbcDorisConnector (manages HikariCP connection pool lifecycle)
- JdbcConnectorMetadata (delegates to JdbcConnectorClient for metadata discovery)
- JdbcConnectorClient base class with factory method dispatching to 10 DB-specific subclasses
- All 10 DB-specific clients: MySQL, PostgreSQL, Oracle, ClickHouse, SQLServer,
  SAP HANA, Trino, OceanBase, DB2, GBase
- JdbcFieldInfo (column metadata, adapted from JdbcFieldSchema)
- JdbcDbType enum, JdbcTableHandle, JdbcConnectorProperties

Each subclass maps JDBC types to ConnectorType (the SPI type system) instead
of fe-core Type, achieving zero fe-core dependency.

None

- Test: No need to test (Phase 1 metadata-only extraction, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Extract HMS shared module to fe-connector-hms

Issue Number: close #xxx

Problem Summary: The HMS (Hive MetaStore) client code in fe-core is deeply
coupled to internal types (Column, Config, ExecutionAuthenticator, etc.), making
it impossible to directly extract. This commit creates a new clean HMS client
library module (fe-connector-hms) using SPI types, which will be shared by
future fe-connector-hive and fe-connector-hudi plugins.

Key design decisions:
- New clean interfaces using ConnectorType/ConnectorColumn (not code extraction)
- Connection-pooled Thrift client with taint-and-destroy error handling
- Supports HMS/DLF/Glue metastore types via string-based class dispatch
- Auth via functional interface (AuthAction) replacing fe-core ExecutionAuthenticator
- HmsTypeMapping mirrors HiveMetaStoreClientHelper logic but returns ConnectorType

Module contents (9 Java files, ~1440 lines):
- HmsClient: Clean interface for HMS operations
- ThriftHmsClient: Pooled Thrift implementation with ClassLoader safety
- HmsClientConfig/HmsConfHelper: Configuration DTOs and HiveConf creation
- HmsTableInfo/HmsDatabaseInfo/HmsPartitionInfo: Immutable DTOs
- HmsTypeMapping: Hive type string to ConnectorType mapping
- HmsClientException: Runtime exception class

None

- Test: No need to test - library module with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-hive plugin module (Phase 1: metadata-only)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extract Hive connector metadata operations into an independent
fe-connector-hive plugin module as Phase 1 of the Catalog SPI modularization.

This creates the fe-connector-hive plugin that:
- Implements ConnectorProvider (type="hms") for SPI-based catalog creation
- Provides HiveConnectorMetadata with listDatabases, listTables, getTableHandle,
  getTableSchema operations via the shared fe-connector-hms client
- Detects table formats (HIVE/HUDI/ICEBERG) from HMS table metadata
- Supports configurable type mapping options (binary-as-string, timestamp-tz)
- Manages HmsClient lifecycle with lazy initialization and proper shutdown

Files: 10 new files (816 lines), 2 modified files
Dependencies: fe-connector-spi + fe-connector-hms + log4j-api

None

- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-paimon plugin module (Phase 1: metadata-only)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Extract Paimon connector metadata operations into an independent
fe-connector-paimon plugin module as Phase 1 of the Catalog SPI modularization.

This creates the fe-connector-paimon plugin that:
- Implements ConnectorProvider (type="paimon") for SPI-based catalog creation
- Creates Paimon Catalog instances directly via Paimon SDK (Options + CatalogContext
  + CatalogFactory) supporting all backends: filesystem, HMS, DLF, REST, JDBC
- Provides PaimonConnectorMetadata with listDatabases, listTables, getTableHandle,
  getTableSchema operations using the Paimon Catalog API
- Includes PaimonTypeMapping that converts Paimon DataType to ConnectorType,
  mirroring existing PaimonUtil.paimonTypeToDorisType logic
- Supports configurable type mapping options (binary-as-varbinary, timestamp-tz)
- Manages Paimon Catalog lifecycle with lazy init and proper shutdown

Files: 8 new files (805 lines), 1 modified file
Dependencies: fe-connector-spi + paimon-core + log4j-api

None

- Test: No need to test - Phase 1 metadata-only, no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add fe-connector-hudi Phase 1 plugin module

Issue Number: close #xxx

Problem Summary: Extract Hudi connector metadata operations into an
independent SPI plugin module (fe-connector-hudi) as part of the
catalog connector SPI modularization effort.

The Hudi connector plugin provides Phase 1 read-only metadata:
- List databases and tables via HMS (shared fe-connector-hms client)
- Get table schema from HoodieTableMetaClient Avro schema
  (authoritative for schema-evolved Hudi tables)
- Detect Hudi table type (COW vs MOR) from input format
- Avro-to-ConnectorType mapping (HudiTypeMapping)

Dependencies: fe-connector-spi + fe-connector-hms + hudi-common +
hudi-hadoop-mr.

Scan planning, incremental query, statistics, and event processing
remain in fe-core temporarily.

None

- Test: No need to test (Phase 1 SPI module, not yet wired into runtime)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Extract fe-connector-iceberg Phase 1 — metadata-only SPI plugin

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Step 15 of the Catalog Connector SPI modularization. Extracts the Iceberg
connector into an independent plugin module (fe-connector-iceberg) that
provides read-only metadata operations through the Connector SPI.

Phase 1 implements:
- IcebergConnectorProvider: SPI entry point, type="iceberg"
- IcebergConnector: Lifecycle management, creates Iceberg SDK Catalog via
  CatalogUtil.buildIcebergCatalog() — supports all 7 backends (REST, HMS,
  Glue, DLF, JDBC, Hadoop, S3Tables)
- IcebergConnectorMetadata: listDatabaseNames (SupportsNamespaces), listTableNames,
  getTableHandle, getTableSchema via Iceberg SDK Catalog API
- IcebergTypeMapping: Iceberg types → ConnectorType (BOOLEAN, INT, LONG, FLOAT,
  DOUBLE, STRING, UUID/BINARY→VARBINARY or STRING, DECIMAL, DATE, TIMESTAMP,
  LIST, MAP, STRUCT) with enableMappingVarbinary and enableMappingTimestampTz flags
- IcebergTableHandle: Opaque handle carrying db+table coordinates
- IcebergConnectorProperties: Property key constants

Key design decisions:
- No fe-connector-hms dependency: Iceberg SDK HiveCatalog handles HMS internally
- Property-driven catalog creation: CatalogUtil.buildIcebergCatalog() handles
  all backends, subclass dispatch is just setting catalog-impl
- Hadoop Configuration built from user properties (hadoop.*, fs.*, dfs.*, hive.*)

6 Java files, ~770 lines added.

None

- Test: No need to test — Phase 1 metadata-only SPI module, not yet wired into fe-core runtime
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add PluginDrivenExternalTable schema support and GSON registration

Issue Number: close #xxx

Problem Summary:
PluginDrivenExternalTable was a minimal stub with no schema retrieval capability.
Plugin-driven catalogs could list databases and tables but could not fetch table
schemas, making them non-functional for queries. Additionally, the PluginDriven*
types were not registered in GsonUtils, preventing GSON serialization/deserialization
of plugin-driven catalog metadata for FE persistence.

This commit implements Step 18 of the Catalog Connector SPI plan:

1. **ConnectorColumnConverter** (new): Converts between the connector SPI type system
   (ConnectorColumn/ConnectorType) and Doris internal types (Column/Type). Handles
   all scalar types plus complex types (ARRAY, MAP, STRUCT) recursively, with proper
   precision/scale handling for CHAR, VARCHAR, DECIMAL, and DATETIMEV2.

2. **PluginDrivenExternalTable**: Overrides initSchema() to fetch table schema from
   the connector SPI. Uses ConnectorMetadata.getTableHandle() + getTableSchema(),
   then converts via ConnectorColumnConverter.

3. **PluginDrivenExternalCatalog**: Makes buildConnectorSession() package-private so
   PluginDrivenExternalTable can build sessions during schema init.

4. **GsonUtils**: Registers PluginDrivenExternalCatalog, PluginDrivenExternalDatabase,
   and PluginDrivenExternalTable in the respective RuntimeTypeAdapterFactory instances
   for proper GSON serialization/deserialization.

None

- Test: No need to test - infrastructure code with no active connector plugins yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorExpression framework and filter pushdown types

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Step 19 of the Catalog SPI modularization. Replaces the
placeholder ConnectorExpression class with a proper expression tree hierarchy
for cross-SPI filter/projection pushdown negotiation. Adds the
ExprToConnectorExpressionConverter in fe-core to convert Doris Expr trees
into ConnectorExpression trees at the SPI boundary.

Changes:
- ConnectorExpression: converted from final class to interface with
  getExprType() and getChildren() methods
- ConnectorExprType: discriminator enum (COLUMN_REF, LITERAL, COMPARISON,
  AND, OR, NOT, IN, BETWEEN, IS_NULL, LIKE, FUNCTION_CALL)
- 11 concrete expression types: ConnectorColumnRef, ConnectorLiteral,
  ConnectorComparison, ConnectorAnd, ConnectorOr, ConnectorNot,
  ConnectorIn, ConnectorBetween, ConnectorIsNull, ConnectorLike,
  ConnectorFunctionCall
- ConnectorRange: range bound with low/high inclusive/exclusive endpoints
- ConnectorDomain: per-column domain (union of ranges + null handling)
  for fast partition pruning
- ConnectorFilterConstraint: redesigned to carry full expression tree +
  per-column domain map (replaces old flat conjuncts list)
- ExprToConnectorExpressionConverter: converts Doris Expr tree to
  ConnectorExpression tree, plus Type to ConnectorType reverse mapping

None

- Test: No need to test - pure API/SPI type definitions and converter
    with no runtime integration yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorScanPlanProvider and PluginDrivenScanNode for connector SPI scan planning

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Plugin-driven catalogs need a generic scan node that delegates
scan planning to the connector SPI, rather than requiring connector-specific
ScanNode subclasses in fe-core.

This commit adds Step 20 of the Catalog SPI modularization:

**fe-connector-api enhancements:**
- ConnectorScanRangeType enum: FILE_SCAN, JDBC_SCAN, ES_SCAN, REMOTE_OLAP_SCAN, CUSTOM
- ConnectorDeleteFile: delete file descriptor for Iceberg MOR tables
- ConnectorScanRange: added getRangeType() (mandatory), getFileFormat(),
  getFileSize(), getModificationTime(), getPartitionValues(), getDeleteFiles()
- ConnectorScanPlanProvider: added estimateScanRangeCount() default method

**fe-core additions:**
- PluginDrivenSplit: wraps ConnectorScanRange in FileSplit for the
  FileQueryScanNode pipeline
- PluginDrivenScanNode: extends FileQueryScanNode, delegates scan planning
  to ConnectorScanPlanProvider, uses FORMAT_JNI for BE execution
- ExprToConnectorExpressionConverter: added convertConjuncts() utility
- PhysicalPlanTranslator: dispatch to PluginDrivenScanNode for
  PluginDrivenExternalTable instances

None

- Test: No need to test — SPI infrastructure, no actual connectors activate this path yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add ConnectorWriteOps & PluginDrivenTableSink (Step 21)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Implements Step 21 of the Catalog SPI modularization plan.
Adds the write operation SPI interfaces and the generic table sink for
plugin-driven external tables.

**SPI layer (fe-connector-api):**
- Enhanced ConnectorWriteOps with full write lifecycle: getWriteConfig,
  beginInsert/finishInsert/abortInsert (with column list), beginDelete/
  finishDelete/abortDelete, beginMerge/finishMerge/abortMerge
- Added ConnectorWriteType enum (FILE_WRITE, JDBC_WRITE, REMOTE_OLAP_WRITE,
  CUSTOM)
- Added ConnectorWriteConfig value object with builder pattern, carrying
  write type, file format, compression, location, partition columns, and
  generic properties
- Added ConnectorDeleteHandle and ConnectorMergeHandle marker interfaces

**Engine layer (fe-core):**
- Created PluginDrivenTableSink extending BaseExternalTableDataSink
- Constructs TDataSink based on ConnectorWriteConfig.getWriteType():
  FILE_WRITE -> THiveTableSink, JDBC_WRITE -> TJdbcTableSink
- Property-driven Thrift construction using well-known keys from
  ConnectorWriteConfig.properties

None

- Test: No need to test - infrastructure interfaces with no runtime behavior change
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add generic Nereids plan nodes for plugin-driven INSERT pipeline

Issue Number: close #xxx

Problem Summary: PluginDrivenExternalCatalog-backed tables cannot participate
in the full Nereids INSERT pipeline because there are no generic plan node
classes for them. Each existing connector (Hive, Iceberg, JDBC, etc.) has its
own typed plan nodes, but the plugin-driven generic path has none.

This commit adds the complete Nereids INSERT pipeline for plugin-driven
catalogs:

1. **Plan nodes**: UnboundConnectorTableSink, LogicalConnectorTableSink,
   PhysicalConnectorTableSink — generic nodes that work with any
   PluginDrivenExternalCatalog without connector-specific plan classes.

2. **Optimizer rules**: BINDING_INSERT_CONNECTOR_TABLE (BindSink),
   LogicalConnectorTableSinkToPhysicalConnectorTableSink (implementation),
   ExpressionRewrite for LogicalConnectorTableSink.

3. **Insert executor**: PluginDrivenInsertExecutor delegates begin/commit/
   abort to ConnectorWriteOps SPI. PluginDrivenInsertCommandContext provides
   the context wrapper.

4. **Transaction manager**: PluginDrivenTransactionManager provides
   lightweight transaction lifecycle bookkeeping; actual commit/rollback
   is handled by ConnectorWriteOps in the insert executor.

5. **Wiring**: PlanType enum entries, RuleType entries, SinkVisitor methods,
   RuleSet registration, UnboundTableSinkCreator dispatch,
   PhysicalPlanTranslator visitor, InsertIntoTableCommand executor selection.

None

- Test: No need to test — infrastructure code, no connector plugins exercise
  this path yet
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Activate CatalogFactory SPI path with build integration and connectivity testing

Issue Number: close #xxx

Problem Summary: The connector SPI plugins were built but not deployed to
the output directory, making them undiscoverable at runtime. Additionally,
PluginDrivenExternalCatalog lacked connectivity testing support.

This commit:
1. Adds connector module compilation to build.sh FE module list
2. Deploys connector plugin ZIPs to output/fe/plugins/connector/<name>/
   (mirroring the filesystem plugin deployment pattern)
3. Creates ConnectorTestResult value object in fe-connector-api
4. Adds Connector.testConnection(session) default method
5. Overrides checkWhenCreating() in PluginDrivenExternalCatalog to delegate
   connectivity testing to the connector SPI
6. Adds ConnectorFactory.getRegisteredTypes() for diagnostics
7. Enhances logging in CatalogFactory and Env for SPI path transparency

None

- Test: No need to test (build infrastructure + SPI wiring, no behavioral change)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Step 24: ES full migration to connector SPI

Issue Number: close #xxx

Problem Summary: Migrate the ES (Elasticsearch) connector scan path from
legacy fe-core EsScanNode to the plugin-driven connector SPI. This enables
the ES connector to run as an isolated plugin while producing the same
ES_HTTP_SCAN_NODE Thrift types expected by BE.

Key changes:
- Enhanced ConnectorScanPlanProvider with getScanRangeType(),
  getScanNodeProperties(), and getScanNodeMapProperties() to support
  non-file scan types (ES, JDBC) that need custom Thrift structures
- Created EsQueryDslBuilder: ports QueryBuilders from legacy Expr-based
  filter conversion to ConnectorExpression-based ES Query DSL generation
- Created EsScanPlanProvider: generates per-shard scan ranges with ES
  host routing, query DSL, auth info, and field context maps
- Created EsScanRange: ConnectorScanRange impl for ES shard routing
- Created PluginDrivenEsScanNode: extends ExternalScanNode to produce
  TPlanNodeType.ES_HTTP_SCAN_NODE with TEsScanNode/TEsScanRange Thrift
- Wired EsConnector.getScanPlanProvider() to return real provider
- Added EsConnector.testConnection() for catalog connectivity testing
- Fixed BindRelation missing PLUGIN_EXTERNAL_TABLE case (critical:
  without this fix, no plugin-driven table can be queried)
- PhysicalPlanTranslator dispatches to PluginDrivenEsScanNode when
  getScanRangeType() == ES_SCAN
- Removed "es" case from CatalogFactory switch (SPI handles it)

None

- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Step 25: JDBC full migration (scan only) to connector SPI

Issue Number: close #xxx

Problem Summary: Migrate JDBC connector scan path to the connector SPI
framework, enabling JDBC catalogs to run through the PluginDrivenScanNode
pipeline instead of the legacy JdbcScanNode code path.

Key changes:
- JdbcQueryBuilder: converts ConnectorExpression to SQL WHERE clauses with
  DB-specific date formatting (Oracle to_date, Trino date literal, SQL Server
  CONVERT) and LIMIT syntax (MySQL LIMIT, Oracle ROWNUM, SQL Server TOP N)
- JdbcScanRange: ConnectorScanRange with table_format_type="jdbc" and Builder
  pattern for all BE-expected jdbc_params (url, user, password, driver, query_sql,
  connection pool settings)
- JdbcScanPlanProvider: builds SQL query via JdbcQueryBuilder, returns 1 scan
  range with all JDBC connection parameters
- JdbcColumnHandle: carries localName + remoteName for SELECT clause column
  quoting via JdbcIdentifierQuoter
- JdbcConnectorMetadata.getColumnHandles(): returns column handle map for
  PluginDrivenScanNode column selection
- JdbcDorisConnector: wires getScanPlanProvider() and testConnection()
- ConnectorScanRange.getTableFormatType(): default "plugin_driven", JDBC
  overrides to "jdbc" so BE routes to JdbcJniReader
- ConnectorScanPlanProvider: 5-arg planScan() overload with limit parameter
  for JDBC LIMIT pushdown
- PluginDrivenScanNode: uses scanRange.getTableFormatType() instead of
  hardcoded constant; builds column handles via metadata.getColumnHandles();
  passes limit to 5-arg planScan()
- CatalogFactory: removed "jdbc" case from fallback switch (all JDBC catalogs
  now go through SPI path)

None

- Test: Manual test (compilation + checkstyle verified)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Step 26: MaxCompute full migration (scan only) to connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Migrates MaxCompute scan planning from fe-core to the
fe-connector-maxcompute plugin module. This removes the "max_compute" fallback
case from CatalogFactory, routing MaxCompute catalog creation exclusively
through the SPI connector path (PluginDrivenExternalCatalog).

Key changes:
- MaxComputeScanPlanProvider: creates ODPS TableBatchReadSession, generates
  splits (byte_size and row_offset strategies), supports limit optimization
- MaxComputePredicateConverter: converts ConnectorExpression to ODPS Predicate
  (CompoundPredicate, RawPredicate, UnaryPredicate) with datetime timezone
  conversion
- MaxComputeScanRange: ConnectorScanRange carrying serialized session, split
  params, and timeout config with tableFormatType="max_compute"
- MaxComputeColumnHandle: tracks partition vs data column distinction
- MCConnectorEndpoint: region-to-timezone mapping for datetime pushdown
- PluginDrivenScanNode: format-type dispatch in setScanParams() -
  "max_compute" builds TMaxComputeFileDesc, others use generic jdbc_params map
- CatalogFactory: removed "max_compute" fallback case and
  MaxComputeExternalCatalog import
- Added odps-sdk-table-api dependency for table read session API

None

- Test: No need to test - structural migration only, MaxCompute not available
  in dev environment
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Migrate Trino connector scan planning to connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Trino connector scan planning was still handled by the legacy
TrinoConnectorScanNode in fe-core with direct Trino SPI type dependencies.
This commit migrates the full scan flow to the connector SPI plugin module
(fe-connector-trino), keeping all Trino SPI types inside the plugin boundary.

Key changes:
- TrinoScanPlanProvider: core scan planning (beginQuery → applyFilter/Limit/
  Projection → getSplits → JSON-serialize → TrinoScanRange list → cleanupQuery)
- TrinoJsonSerializer: ObjectMapperProvider + HandleJsonModule + BlockJsonSerde
  for serializing Trino SPI objects to JSON
- TrinoPredicateConverter: ConnectorExpression → Trino TupleDomain<ColumnHandle>
- TrinoScanRange: ConnectorScanRange carrying pre-serialized JSON properties
- TrinoColumnHandle: ConnectorColumnHandle for projection pushdown
- PluginDrivenScanNode: "trino_connector" dispatch filling TTrinoConnectorFileDesc
  from properties map (no Trino types needed in fe-core)
- CatalogFactory: removed "trino-connector" legacy fallback case
- TrinoBootstrap: added getHandleResolver()/getTypeRegistry() accessors
- TrinoConnectorDorisMetadata: added getColumnHandles() method
- TrinoDorisConnector: replaced stub with real getScanPlanProvider() + testConnection()

None

- Test: No need to test - structural migration, scan logic ported from existing
  TrinoConnectorScanNode with no behavioral changes
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add Hive scan planning to connector plugin (Step 31)

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: Adds scan planning capability to the Hive connector plugin
module (fe-connector-hive), enabling file-based scan range generation
through the ConnectorScanPlanProvider SPI. This is the scan-only phase of
the Hive full migration to the connector plugin architecture.

New files in fe-connector-hive:
- HiveScanPlanProvider: Core scan planning - partition resolution via HMS,
  file listing via Hadoop FS, file splitting by configurable target size
- HiveColumnHandle: ConnectorColumnHandle wrapping column name/type/isPartKey
- HiveFileFormat: Enum mapping InputFormat/SerDe classes to format strings
- HiveScanRange: ConnectorScanRange with Builder for file split descriptors
- HiveTextProperties: Extracts text format props from HMS SerDe parameters

Enhanced existing files:
- HiveTableHandle: Rewritten with Builder pattern, now carries inputFormat,
  serializationLib, location, partitionKeyNames, sdParameters, tableParameters,
  and prunedPartitions for scan planning
- HiveConnectorMetadata: Added getColumnHandles(), applyFilter() with
  partition pruning (equality + IN predicates on partition columns)
- HiveConnector: Added getScanPlanProvider() returning HiveScanPlanProvider
- HmsTableInfo: Added sdParameters field for SerDe parameters
- ThriftHmsClient: Populates sdParameters from SerDeInfo
- PluginDrivenScanNode: Major enhancement with property-driven overrides for
  getFileFormatType, getPathPartitionKeys, getLocationProperties, getFileAttributes,
  plus hive/transactional_hive dispatch in setScanParams

Scope: Non-ACID Hive tables (Parquet, ORC, Text). ACID tables, file listing
cache, and table sampling are deferred to future steps.

None

- Test: No need to test (scan planning code path not yet wired to production
  query flow; requires HMS + Hadoop cluster for integration testing)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add Hudi scan planning to connector plugin (Step 32)

Issue Number: close #xxx

Problem Summary: Adds scan planning capability to the fe-connector-hudi plugin
module, following the same pattern established in Step 31 (Hive scan). This
enables the Hudi connector plugin to generate scan ranges that BE can consume,
supporting both COW (native Parquet/ORC reader) and MOR (JNI reader with delta
log merging) table types.

New files:
- HudiScanPlanProvider: Core scan planning — builds MetaClient, resolves
  partitions via HoodieTableMetadata API, generates COW splits (base files
  only) and MOR splits (base + delta logs with dynamic native downgrade)
- HudiScanRange: ConnectorScanRange implementation with Builder pattern,
  carrying both native reader fields and JNI metadata (instant_time, serde,
  delta_logs, column_names/types)
- HudiColumnHandle: Column handle with name, typeName, isPartitionKey

Modified files:
- HudiTableHandle: Rewritten with Builder pattern, added scan-related fields
  (inputFormat, serdeLib, partitionKeyNames, tableParameters, prunedPaths)
- HudiConnectorMetadata: Enhanced getTableHandle with scan fields from HMS,
  added getColumnHandles() and applyFilter() for partition pruning
- HudiConnector: Added getScanPlanProvider()
- PluginDrivenScanNode: Added "hudi" case in setScanParams dispatch with
  setHudiParams() method creating THudiFileDesc for both native/JNI paths

None

- Test: No need to test — plugin scan planning mirrors existing HudiScanNode
  logic; will be validated when end-to-end regression tests are enabled
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[feature](fe) Add Paimon scan planning to connector plugin (Step 33)

Problem Summary: Migrate Paimon scan planning logic from fe-core PaimonScanNode
to the fe-connector-paimon plugin module, following the same pattern established
by Hive (Step 31) and Hudi (Step 32) scan migrations.

This step adds:
- PaimonColumnHandle: Column handle with name and field index in RowType
- PaimonPredicateConverter: Converts ConnectorExpression to Paimon Predicate
  (PredicateBuilder-based, supports AND/OR/comparison/IN/IS NULL/LIKE prefix)
- PaimonScanRange: Builder-pattern scan range supporting JNI, native, and
  COUNT pushdown paths with dual serialization
- PaimonScanPlanProvider: Core scan planning using Paimon SDK (ReadBuilder →
  TableScan → Split), converting to ConnectorScanRange with JNI/native dispatch
- Modified PaimonTableHandle to carry transient Table reference
- Modified PaimonConnectorMetadata to store Table in handle and add getColumnHandles()
- Modified PaimonConnector to add getScanPlanProvider()
- Modified PluginDrivenScanNode to add paimon dispatch (setPaimonParams),
  getSerializedTable() override, and scan-level paimon params (predicate + options)

None

- Test: No need to test (compile-only migration, runtime wiring not yet active)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Remove old internal ES table code and clean up references

Issue Number: close #xxx

Problem Summary: The old internal Elasticsearch table implementation (EsScanNode,
EsExternalCatalog, EsExternalTable, etc.) has been superseded by the ES Catalog
connector plugin. This commit removes the legacy ES code from fe-core and cleans
up all references throughout the codebase.

Changes:
- Delete 21 old ES datasource files from datasource/es/
- Delete 3 nereids plan classes (LogicalEsScan, PhysicalEsScan, LogicalEsScanToPhysicalEsScan)
- Delete 6 old ES test files from external/elasticsearch/
- Clean up ES references in 13+ files (Env.java, ExternalCatalog.java, ESCatalogAction.java,
  PhysicalPlanTranslator.java, BindRelation.java, RuleSet.java, RelationVisitor.java,
  StatsCalculator.java, CostModel.java, ChildOutputPropertyDeriver.java,
  TopnFilterPushDownVisitor.java, GsonUtils.java, TableIf.java, etc.)
- Preserve EsTable.java and EsResource.java for persistence compatibility
- Add ES stub in SHOW CREATE TABLE for deprecation notice

None

- Test: Manual test - fe-core compiles successfully, ES connector tests pass
- Behavior changed: Yes - old internal ES table queries will no longer work;
  users should use ES Catalog instead
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Add metadata methods to JDBC connector for migration parity

Issue Number: close #xxx

Problem Summary: During JDBC connector modular migration from fe-core to
fe-connector-jdbc, several metadata methods were missing in the new connector
client and SPI interfaces. This commit adds the missing methods to achieve
parity with the old JdbcClient.

None

- Test: No need to test - pure method additions with no behavior change,
  existing integration paths still use old JdbcClient
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Implement JDBC write path through connector plugin pipeline

Issue Number: close #xxx

Problem Summary: The JDBC INSERT INTO write path was not wired through the
new connector plugin pipeline. This commit implements the complete JDBC write
path so that INSERT INTO on PluginDriven JDBC tables produces the correct
TJdbcTableSink thrift structure, matching the old JdbcTableSink behavior.

Changes:
- JdbcConnectorMetadata now implements ConnectorWriteOps with getWriteConfig(),
  beginInsert(), finishInsert() — builds INSERT SQL via JdbcIdentifierQuoter
  and populates all JDBC connection/pool properties
- JdbcIdentifierQuoter gains buildInsertSql() for parameterized INSERT SQL
- PluginDrivenTableSink.bindJdbcWriteSink() now populates ALL TJdbcTable
  fields: catalog_id, driver_checksum, table_name, resource_name, and all
  connection pool settings; also sets TOdbcTableType on TJdbcTableSink
- PluginDrivenInsertExecutor resolves write type from connector metadata and
  returns TransactionType.JDBC for JDBC writes (was hardcoded to HMS)
- PhysicalPlanTranslator passes actual column list to getWriteConfig()
  (was passing empty list)
- ConnectorSessionBuilder now propagates enable_odbc_transcation session var
- Explain output enhanced for JDBC write sinks (shows table type, SQL, txn)

None

- Test: No need to test - infrastructure wiring, no end-to-end JDBC catalog
  available in unit test environment; will be tested with regression tests
  after full migration
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Port JDBC function pushdown and ClickHouse FINAL to connector module

Issue Number: close #xxx

Problem Summary:
The JDBC connector module (fe-connector-jdbc) lacked function pushdown support
and ClickHouse FINAL query support. These features existed only in the old
fe-core JdbcScanNode code path. This commit ports them to the connector SPI
pipeline so PluginDrivenScanNode-based JDBC scans have feature parity.

Key changes:
- New JdbcFunctionPushdownConfig: per-DB function whitelist/blacklist/replacement
  rules with time arithmetic rewriting and JSON-configurable overrides
- Extended JdbcQueryBuilder: ConnectorFunctionCall/Like/Between to SQL conversion,
  conjunct pushdown guards (Oracle NULL, cast, function-containing expressions),
  ClickHouse SETTINGS final=1 support
- Updated JdbcScanPlanProvider to create pushdown config and pass session props
- Plumbed 4 session variables through ConnectorSessionBuilder:
  enable_ext_func_pred_pushdown, jdbc_clickhouse_query_final,
  enable_jdbc_oracle_null_predicate_push_down, enable_jdbc_cast_predicate_push_down

None

- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Route JDBC TVF and CallExecuteStmt through connector SPI

Issue Number: close #xxx

Problem Summary:
JdbcQueryTableValueFunction and CallExecuteStmtFunc only supported the old
JdbcExternalCatalog code path. Newly created JDBC catalogs use
PluginDrivenExternalCatalog via the SPI connector, so these features were
broken for new catalogs.

Key changes:
- New PassthroughQueryTableHandle in connector API for raw query TVF scans
- JdbcQueryTableValueFunction: dual path — uses ConnectorMetadata for schema
  discovery and PluginDrivenScanNode for scan planning when catalog is
  PluginDrivenExternalCatalog
- CallExecuteStmtFunc: routes to connector metadata.executeStmt() for
  PluginDrivenExternalCatalog, falls back to old JdbcExternalCatalog path
- QueryTableValueFunction factory: detects PluginDrivenExternalCatalog
- JdbcScanPlanProvider: handles PassthroughQueryTableHandle for TVF queries

None

- Test: Manual test / Regression test pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[improvement](fe) Migrate JDBC identifier mapping to connector module

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
The JDBC identifier mapping logic (lower_case_meta_names, meta_names_mapping,
lower_case_table_names) was tightly coupled to fe-core via JdbcIdentifierMapping
which depends on Jackson ObjectMapper. This commit migrates the functionality
to the connector-jdbc module as part of the SPI modularization effort.

Key changes:
- Add ConnectorIdentifierOps SPI interface in connector-api with 3 name mapping
  methods (fromRemoteDatabaseName, fromRemoteTableName, fromRemoteColumnName)
- ConnectorMetadata now extends ConnectorIdentifierOps
- Create JdbcIdentifierMapper in connector-jdbc: pure Java reimplementation
  with regex-based JSON parsing (no Jackson dependency), full validation with
  case-conflict detection
- Wire identifier mapping through getColumnHandles() and getWriteConfig() so
  column names are properly mapped in both read and write paths
- Add lower_case_table_names from GlobalVariable to ConnectorSessionBuilder
- PluginDrivenExternalCatalog overrides fromRemoteDatabaseName/TableName to
  delegate to connector metadata
- PluginDrivenExternalTable.initSchema() applies fromRemoteColumnName mapping

None

- Test: Manual test / Compilation verified
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Remove old JDBC connector code after migration to plugin-driven architecture

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: After migrating JDBC connector to fe-connector-jdbc module with
plugin-driven architecture (PluginDrivenExternalCatalog/Database/Table), the old
JDBC code in fe-core/datasource/jdbc/ is dead code. This commit:

1. Adds Gson compatible subtype adapters so persisted metadata with old class names
   (JdbcExternalCatalog/Database/Table) deserializes correctly as PluginDriven* classes
2. Deletes 21 old JDBC files: JdbcExternalCatalog, JdbcExternalDatabase,
   JdbcExternalTable, JdbcNameUtil, JdbcSchemaCacheValue, JdbcTableSink, JdbcScanNode,
   JdbcSplit, JdbcFunctionPushDownRule, IdentifierMapping, JdbcIdentifierMapping,
   UnboundJdbcTableSink, LogicalJdbcTableSink, PhysicalJdbcTableSink,
   JdbcInsertExecutor, JdbcInsertCommandContext,
   LogicalJdbcScanToPhysicalJdbcScan, LogicalJdbcTableSinkToPhysicalJdbcTableSink,
   LogicalJdbcScan, PhysicalJdbcScan
3. Cleans up all references in 23 files across nereids visitors, derivers, rules,
   plan translator, expression rewrite, and external function rules
4. Inlines function pushdown constants from deleted JdbcFunctionPushDownRule into
   ExternalFunctionRules

JdbcClient hierarchy (16 files) is kept for CDC binlog functionality.

None

- Test: No need to test - pure code deletion of dead code after migration
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix connector plugin packaging and ES scan node bugs

Issue Number: close #xxx

Problem Summary:

Three issues causing CI regression test failures (build 926124):

1. **JDBC/Hive/MaxCompute plugin JARs not discovered at runtime**
   The assembly descriptors for these connectors used `includeBaseDirectory=true`,
   creating a nested subdirectory inside the zip (e.g., `fe-connector-jdbc/lib/*.jar`).
   When build.sh unzips into `plugins/connector/jdbc/`, the actual path becomes
   `plugins/connector/jdbc/fe-connector-jdbc/lib/*.jar`, but DirectoryPluginRuntimeManager
   expects JARs at `plugins/connector/jdbc/*.jar` and `plugins/connector/jdbc/lib/*.jar`.
   Fix: Set `includeBaseDirectory=false` and place plugin JAR at root + deps in lib/,
   matching the working ES/iceberg/paimon assembly layout.

2. **ES scan node URL parsing fails for http:// prefixed hosts**
   PluginDrivenEsScanNode.convertToThrift() split host strings by ":" naively,
   so `http://172.16.0.98:29200` was parsed as host="http", port="//172.16.0.98"
   causing NumberFormatException. Fix: Strip scheme prefix before parsing host:port.

3. **ES JSONB type not recognized by ConnectorColumnConverter**
   ES connector returns JSONB type but ScalarType.createType() only handles "JSON".
   ConnectorColumnConverter now maps JSONB to JSON explicitly.

None

- Test: Manual verification of zip layout; CI rerun pending
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Resolve JDBC driver_url via ConnectorContext environment

Problem Summary: JDBC connector plugin fails to load driver classes because
plain driver filenames (e.g., "mysql-connector-j-8.4.0.jar") are not resolved
to absolute file:// URLs using Config.jdbc_drivers_dir.

The fix adds a generic getEnvironment() method to ConnectorContext SPI,
which fe-core populates with system configs (jdbc_drivers_dir, doris_home).
The JDBC connector uses this environment to resolve driver URLs, keeping
fe-core free of JDBC-specific logic.

None

- Test: Manual test - verified driver URL resolution logic
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Implement toThrift() for PluginDrivenExternalTable via connector SPI

Problem Summary: PluginDrivenExternalTable.toThrift() inherited the base
ExternalTable implementation which returns null, causing NPE during query
fragment serialization when BE needs TTableDescriptor.

The fix adds:
1. ConnectorTableOps.getTableDescriptorProperties() SPI method that lets
   connectors declare the BE table descriptor type and properties
2. JdbcConnectorMetadata returns JDBC_TABLE type with all connection/pool
   properties needed by TJdbcTable
3. EsConnectorMetadata returns ES_TABLE type
4. PluginDrivenExternalTable.toThrift() reads these properties and
   constructs the appropriate typed TTableDescriptor (TJdbcTable/TEsTable)

None

- Test: Manual test — JDBC catalog query no longer hits null TTableDescriptor
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[refactor](fe) Connector directly builds TTableDescriptor via fe-thrift

Issue Number: close #xxx

Problem Summary: Previously PluginDrivenExternalTable.toThrift() used a Map-based
getTableDescriptorProperties() intermediary to build TTableDescriptor, which required
fe-core to contain connector-specific logic for constructing TJdbcTable/TEsTable.

This refactoring lets each connector module directly depend on fe-thrift (provided scope)
and build its own TTableDescriptor in buildTableDescriptor(), making PluginDrivenExternalTable
a pure delegator with zero connector-specific code.

Key changes:
- ConnectorTableOps: replaced getTableDescriptorProperties() with buildTableDescriptor()
  returning TTableDescriptor directly
- JdbcConnectorMetadata: builds TJdbcTable + TTableDescriptor directly
- EsConnectorMetadata: builds TEsTable + TTableDescriptor directly
- PluginDrivenExternalTable.toThrift(): simplified to pure delegation
- Added fe-thrift as provided dependency to fe-connector-api, fe-connector-jdbc,
  fe-connector-es (Maven provided scope is NOT transitive)
- All 4 plugin assembly ZIPs exclude fe-thrift and libthrift

None

- Test: Manual test (compilation verified, runtime pending BIT/BOOLEAN display fix)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix SPI column attributes, type mapping, EXPLAIN format, and DB/schema mapping

Issue Number: close #xxx

Problem Summary: Fix 13 CI test failures in the External Regression pipeline
caused by the connector migration. This batch addresses 4 root cause categories:

1. **isKey lost (5 tests)**: ConnectorColumn had no isKey field; convertColumn()
   hardcoded false. Added isKey to ConnectorColumn, propagated through
   ConnectorColumnConverter, populated from JDBC getPrimaryKeys().

2. **Type mapping (3 tests)**: HLL/BITMAP/QUANTILE_STATE types were not handled
   in JdbcMySQLConnectorClient.mapSignedType(), falling to UNSUPPORTED. Added
   the missing cases for Doris-to-Doris JDBC catalog support.

3. **DB/schema mapping (3 tests)**: ClickHouse connector ignored databaseterm
   URL parameter; OceanBase only delegated type mapping (not metadata methods);
   PostgreSQL missing bit/varbit/hstore type mappings.

4. **EXPLAIN format (2 tests)**: PluginDrivenScanNode did not output QUERY: line
   in EXPLAIN. Added getNodeExplainString() override that shows pushed-down SQL
   from JdbcScanPlanProvider.getScanNodeProperties().

None

- Test: Regression test (pending CI run)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Wire write path, statistics, row count, and test expectations through connector SPI

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
Fix 5 regression test failures in the connector SPI migration (Batch 2):

1. InsertUtils write sink: Add UnboundConnectorTableSink handling in
   getTargetTableQualified() for write transaction support.

2. Statistics collection: Override createAnalysisTask() in
   PluginDrivenExternalTable to return ExternalAnalysisTask.

3. Row count estimation: Add fetchRowCount() to PluginDrivenExternalTable
   delegating to ConnectorMetadata.getTableStatistics(). Add
   JdbcConnectorMetadata.getTableStatistics() and MySQL-specific
   getRowCount() querying INFORMATION_SCHEMA.

4. CallExecuteStmt test: Update expected exception message from
   "Only support JDBC catalog" to "executeStmt not supported" since
   HMS catalogs now route through PluginDrivenExternalCatalog.

5. Oracle CHAR padding: Update test_oracle_jdbc_catalog.out to match
   current OracleTypeHandler behavior which trims CHAR trailing spaces.
   This is a pre-existing behavior change from master commit 9770ecf8a59
   (JdbcTypeHandler refactor) where the .out file was never updated.

None

- Test: Regression test expectations updated
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix ES URL parsing, predicate pushdown, date compat, and TVF classloader

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Four bugs introduced during the connector SPI migration:

1. **ES double-port URL** (test_es_query_no_http_url): EsNodeInfo seed constructor
   produced "host:port:80" for inputs like "host:9200" without a scheme prefix.
   Fixed by correctly parsing host and port when split(":") yields 2 parts.

2. **ES LIKE not pushed down** (test_es_query): EsQueryDslBuilder.likeToDsl()
   checked column2typeMap BEFORE resolving keyword sub-fields from fieldsContext.
   Text fields with .keyword sub-fields were rejected. Fixed by checking
   fieldsContext.containsKey() first, matching the old QueryBuilders behavior.

3. **ES date equality returns empty** (test_es_query_nereids): compatDefaultDate()
   used ZoneOffset.UTC, but the old Joda-Time code used the system default timezone.
   When JVM runs in CST (UTC+8), "2022-08-08 08:00:00" should map to midnight UTC,
   not 8am UTC. Fixed by using ZoneId.systemDefault().

4. **TVF cross-catalog ClassCastException** (test_query_tvf_cross_catalog,
   test_query_tvf_auth): PassthroughQueryTableHandle (in org.apache.doris.connector.api)
   was loaded by both app classloader and ChildFirstClassLoader, causing instanceof
   to fail. Fixed by adding "org.apache.doris.connector.api." to the parent-first
   package list in ChildFirstClassLoader.

None

- Test: Regression test (CI pipeline)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](be) Fix CTAS ArrayStoreException for MySQL YEAR column

Issue Number: close #xxx

Problem Summary: MySQL YEAR columns mapped to Doris SMALLINT caused
ArrayStoreException: java.sql.Date in JdbcJniScanner.getNext().

Root cause: MySQLTypeHandler.getColumnValue() used untyped
rs.getObject(columnIndex) for TINYINT/SMALLINT, which returns
java.sql.Date for YEAR columns when MySQL Connector/J default
yearIsDateType=true is in effect. The SMALLINT converter then could
not handle java.sql.Date (falls through to return input as-is),
and storing java.sql.Date into Short[] caused ArrayStoreException.

Fix: Use typed rs.getObject(columnIndex, Integer.class) for TINYINT
and SMALLINT to force the JDBC driver to convert to Integer regardless
of the underlying MySQL column type.

None

- Test: Regression test (test_mysql_all_types_ctas)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](build) Change mvn package to install to fix MDEP-187 reactor dependency issue

Problem Summary: The fe-core module copy-dependencies goal fails with
MDEP-187 error when new reactor modules (fe-connector-api,
fe-connector-spi) have not been installed to the local Maven
repository. This is a known Maven limitation where copy-dependencies
cannot resolve reactor artifacts during the same mvn package run.

Fix: Change build.sh from mvn package to mvn install so all reactor
artifacts are installed to the local repo before copy-dependencies runs.

None

- Test: Manual test (build.sh --fe)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](build) Add Apache license headers to SPI service files

Problem Summary: ConnectorProvider service files in META-INF/services/
were missing Apache license headers, causing the license checker to fail.

None

- Test: No need to test (license headers only)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](es) Fix ES date predicate pushdown timezone conversion

Problem Summary: ES date equality queries returned empty results because
compatDefaultDate() could not parse ISO-formatted datetime strings produced
by extractLiteralValue(). The extractLiteralValue() method formats
LocalDateTime as "2022-08-08T08:00:00" (ISO with T separator), but
compatDefaultDate() only tried to parse "yyyy-MM-dd HH:mm:ss" (space
separator). The parse failure caused the value to be sent without timezone
conversion, so ES interpreted it as UTC instead of the local timezone.

Fix: Add ISO_LOCAL_DATE_TIME parsing as the first attempt in
compatDefaultDate(), before falling back to space-separated format and
date-only format.

None

- Test: Regression test (test_es_query_nereids sql63)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix JDBC connector type mapping, auth, and EXPLAIN regressions

Issue Number: close #62183

Problem Summary: Fix multiple regression failures in the connector-based
JDBC/ES external catalog migration. This commit addresses 8 root causes
covering 20+ test failures from CI External Regression pipeline build 926236.

Fixes include:
- detectDoris() initialization ordering: moved from constructor to
  postInitialize() hook so data source is live before detection runs.
  Added dorisTypeToConnectorType() for Doris-to-Doris type mapping
  including HLL, BITMAP, QUANTILE_STATE types.
- Oracle TIMESTAMP pattern: startsWith("TIMESTAMP") to match
  "TIMESTAMP(6)", "TIMESTAMP(6) WITH LOCAL TIME ZONE" etc.
- Oracle NUMBER: handle null precision/scale (.orElse(0)), scale<=0
  for integer branch, and match old boundary thresholds.
- ClickHouse DB listing: use SHOW DATABASES for full listing;
  fix databaseTermIsCatalog inversion for old drivers.
- DATETIMEV2 precision: ConnectorColumnConverter reads precision
  (not scale) for datetime fractional seconds, matching connector
  encoding convention.
- EXPLAIN cache: invalidate scanNodeProperties after convertPredicate()
  so pushed conjuncts are reflected in EXPLAIN output.
- ExecutionAuthenticator: call initPreExecutionAuthenticator() in
  PluginDrivenExternalCatalog.initLocalObjectsImpl().
- PassthroughQueryTableHandle: add instanceof guards in
  JdbcConnectorMetadata to prevent ClassCastException for TVF.
- isKey: hardcode true for all columns matching legacy behavior.

None

- Test: Regression test (CI External Regression pipeline)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix ES LIKE pushdown for Nereids planner

Problem Summary: Nereids planner translates Like/Regexp expressions to
FunctionCallExpr (via ExpressionTranslator.visitScalarFunction) rather
than LikePredicate. ExprToConnectorExpressionConverter only checked for
instanceof LikePredicate, so the LIKE predicate was converted to a
ConnectorFunctionCall instead of ConnectorLike. This caused ES query DSL
builder to not recognize it as a LIKE and fall back to match_all,
returning all documents instead of only those matching the wildcard.

Fix: Detect FunctionCallExpr with function name "like" or "regexp" in
ExprToConnectorExpressionConverter and convert to ConnectorLike.

None

- Test: Regression test (test_es_query sql_5_29/sql_6_29)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Add JDBC URL normalization for connector migration

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: The connector migration moved JDBC catalog logic from
fe-core to fe-connector-jdbc, but the URL normalization step was lost.
The old path called JdbcResource.handleJdbcUrl() which added critical
JDBC driver parameters. Without these, OceanBase tests fail with
timezone shifts (8h offset), lost fractional seconds, BOOLEAN display
as true/false instead of 1/0, and DOUBLE precision loss.

The fix adds JdbcUrlNormalizer in fe-connector-jdbc that replicates the
same parameter injection logic:
- MySQL/OceanBase: yearIsDateType=false, tinyInt1isBit=false,
  useUnicode=true, characterEncoding=utf-8, rewriteBatchedStatements=true
- OceanBase additionally: useCursorFetch=true
- PostgreSQL: reWriteBatchedInserts=true
- SQL Server: useBulkCopyForBatchInsert=true

The normalization is applied once in JdbcDorisConnector constructor,
so all downstream consumers (client, metadata, scan plan provider)
use the normalized URL.

None

- Test: Regression test (test_oceanbase_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Stop filtering ClickHouse system database from catalog listing

Issue Number: close #xxx

Related PR: #xxx

Problem Summary: The new JdbcClickHouseConnectorClient incorrectly filtered
the ClickHouse "system" database in getFilterInternalDatabases(). The old
JdbcClickHouseClient never overrode this method, so it used the base class
default which only filters information_schema, performance_schema, and mysql.
The system database contains useful tables (query_log, processes, etc.) and
should remain accessible to users.

None

- Test: Regression test (test_clickhouse_jdbc_catalog)
- Behavior changed: No
- Does this need documentation: No

Co-authored-by: Copilot <[email protected]>

[fix](fe) Fix ClickHouse JDBC catalog regression issues in plugin-driven architecture

Issue Number: close #xxx

Problem Summary: Multiple regression issues found after migr…

…inating PluginDrivenEsScanNode (#62602)

Related PR: #62183

Problem Summary:

Before this change, ES catalog queries went through a dedicated
`PluginDrivenEsScanNode` → `ES_HTTP_SCAN_NODE` → `EsScanner` code path,
while JDBC catalogs used `PluginDrivenScanNode` → `FILE_SCAN_NODE` →
`FileScanner`. The two paths had completely separate plan-generation
logic,
Thrift structures, and BE execution flows, making the connector SPI
harder
to maintain and extend.

This PR routes ES catalog scans through the same `FILE_SCAN_NODE` path
that
JDBC already uses, achieving a single unified scan-node implementation
for
all plugin-driven connectors. The key changes are:

**Thrift layer**
- Add `FORMAT_ES_HTTP = 19` to `TFileFormatType`
- Add `es_params` (per-shard map) to `TTableFormatFileDesc`
- Add `es_properties`, `es_docvalue_context`, `es_fields_context` to
  `TFileScanRangeParams` (shared across shards)

**BE — EsHttpReader (new, native C++)**
- New `GenericReader` subclass in
`be/src/format/table/es_http_reader.{h,cpp}`
- Wraps existing `ESScanReader` (HTTP scroll) and `ScrollParser`
(JSON→columns)
- No JVM overhead — pure C++ HTTP reader, unlike JniReader used by some
formats
- Registered in `FileScanner::_get_next_reader()` for `FORMAT_ES_HTTP`
- Implements `get_columns()` and `fill_all_columns() = true`

**FE — Connector SPI extension**
- Add `ScanNodePropertiesResult` class: typed wrapper carrying both scan
  properties and a `notPushedConjunctIndices` set with explicit
`hasConjunctTracking` flag (distinguishes "no tracking" from "all
pushed")
- Add `getScanNodePropertiesResult()` to `ConnectorScanPlanProvider`

**FE — ES connector adaptation**
- `EsScanRange`: changed to `FILE_SCAN` range type, added `getPath()`,
  `getTableFormatType()`, `getFileFormat()`, BE-compatible property keys
- `EsScanPlanProvider`: builds scan node properties with query_dsl,
  doc_values_mode, auth, docvalue/fields context serialization
- `EsConnectorMetadata`: added `getColumnHandles()` using
`NamedColumnHandle`

**FE — PluginDrivenScanNode enhancement**
- Added ES format mapping (`es_http` → `FORMAT_ES_HTTP`)
- Added `setEsParams()` / `setEsScanLevelParams()` for ES Thrift fields
- Added conjunct pruning via `ScanNodePropertiesResult` (removes
pushed-down
  filters including `esquery()` fake function)
- Added ES-specific EXPLAIN output

**FE — Cleanup**
- Deleted `PluginDrivenEsScanNode.java` (390 lines)
- Simplified `PhysicalPlanTranslator`: removed ES-specific branch, all
  plugin-driven connectors now go through `PluginDrivenScanNode`

### Release note

ES catalog scans now use the unified FILE_SCAN execution path shared
with
JDBC and other plugin-driven connectors. This is an internal
architectural
change — query behavior and results are unchanged. The `esquery()`
function
and doc-value optimization continue to work as before.

### Check List (For Author)

- Test: Regression test — all 7 ES test suites pass
    - test_es_query
    - test_es_query_nereids
    - test_es_flatten_type
    - test_es_keyword_array_type
    - test_es_query_no_http_url
    - test_es_query_predicate_correctness
    - test_es_catalog_http_open_api
- Behavior changed: No
- Does this need documentation: No

…S catalogs to plugin-driven architecture (apache#62183)

### What problem does this PR solve?

Issue Number: close #xxx

Problem Summary:

The existing external catalog implementation in Doris has type-specific
classes
scattered throughout fe-core (JdbcExternalCatalog, EsExternalCatalog,
JdbcExternalTable, EsExternalTable, JdbcScanNode, PhysicalJdbcTableSink,
etc.),
making it difficult to add new connector types and leading to duplicated
logic
across catalog implementations.

This PR introduces a Service Provider Interface (SPI) framework for
external
connectors and migrates JDBC and Elasticsearch catalogs to the new
plugin-driven
architecture. The key changes are:

1. **New `fe-connector` module hierarchy**:
   - `fe-connector-api`: Core interfaces (Connector, ConnectorMetadata,
     ConnectorCapability, ConnectorPushdownOps, ConnectorWriteOps,
     ConnectorSession, pushdown expression tree, handle types)
   - `fe-connector-spi`: Service provider loading (ConnectorProvider,
     ConnectorContext, CatalogFactory)
- `fe-connector-jdbc`: Full JDBC connector implementation migrated from
fe-core
- `fe-connector-es`: Elasticsearch connector implementation migrated
from fe-core

2. **Unified catalog/table/scan classes**:
- PluginDrivenExternalCatalog replaces JdbcExternalCatalog +
EsExternalCatalog
- PluginDrivenExternalTable replaces JdbcExternalTable + EsExternalTable
   - PluginDrivenScanNode replaces JdbcScanNode (and ES scan path)
   - PhysicalConnectorTableSink replaces PhysicalJdbcTableSink

3. **Capability-based feature detection**: Connectors declare
capabilities
(SUPPORTS_INSERT, SUPPORTS_PARALLEL_WRITE, SUPPORTS_PASSTHROUGH_QUERY)
so the
optimizer can make correct distribution and planning decisions without
   type-checking specific catalog classes.

4. **GSON backward compatibility**: GsonUtils.registerCompatibleSubtype
remaps
   persisted JdbcExternalCatalog/EsExternalCatalog metadata to
PluginDrivenExternalCatalog for seamless upgrade from older Doris
versions.

5. **JDBC-specific behavioral preservation**:
- GATHER distribution for transactional write safety (non-parallel
writes)
- query() TVF restricted to connectors with SUPPORTS_PASSTHROUGH_QUERY
   - Property validation (driver_url, jdbc_url, driver_class, etc.)
   - Identifier case mapping scoped per database/table
   - DATETIMEV2 fractional-second precision in predicate pushdown
   - Database-specific boolean literal rendering (TRUE/FALSE vs 1/0)
- Database-specific test queries (DB2: SELECT 1 FROM SYSIBM.SYSDUMMY1)
   - Legacy engine name compatibility for SHOW TABLES/information_schema
   - Driver URL resolution with backward-compatible directory fallback

### Release note

Introduce a pluggable connector SPI framework (fe-connector) that allows
external data source integrations to be developed as independent
modules.
JDBC and Elasticsearch catalogs are migrated to this new architecture.
Existing catalogs created on older versions are automatically upgraded
--
no user action is required. All existing JDBC and ES catalog
functionality
is preserved, including predicate pushdown, identifier mapping, property
validation, and transactional write semantics.

### Check List (For Author)

- Test: Regression test / Unit Test
    - 21 JdbcConnectorProvider validation unit tests
    - 7 PluginDrivenExternalTable engine name unit tests
    - 7 JdbcQueryBuilder datetime fractional-second tests
    - 6 JdbcQueryBuilder boolean literal tests
    - 4 JdbcIdentifierMapper scoped uniqueness tests
    - Concurrency test for PluginDrivenExternalCatalog
- JDBC regression tests (jdbc_catalog_db2, jdbc_catalog_pg,
jdbc_catalog_oracle, etc.)
- Behavior changed: No (backward compatible -- existing catalogs
auto-migrated via GSON)
- Does this need documentation: Yes (connector SPI developer guide -- to
be added separately)

---------

Co-authored-by: Copilot <[email protected]>

…inating PluginDrivenEsScanNode (apache#62602)

Related PR: apache#62183

Problem Summary:

Before this change, ES catalog queries went through a dedicated
`PluginDrivenEsScanNode` → `ES_HTTP_SCAN_NODE` → `EsScanner` code path,
while JDBC catalogs used `PluginDrivenScanNode` → `FILE_SCAN_NODE` →
`FileScanner`. The two paths had completely separate plan-generation
logic,
Thrift structures, and BE execution flows, making the connector SPI
harder
to maintain and extend.

This PR routes ES catalog scans through the same `FILE_SCAN_NODE` path
that
JDBC already uses, achieving a single unified scan-node implementation
for
all plugin-driven connectors. The key changes are:

**Thrift layer**
- Add `FORMAT_ES_HTTP = 19` to `TFileFormatType`
- Add `es_params` (per-shard map) to `TTableFormatFileDesc`
- Add `es_properties`, `es_docvalue_context`, `es_fields_context` to
  `TFileScanRangeParams` (shared across shards)

**BE — EsHttpReader (new, native C++)**
- New `GenericReader` subclass in
`be/src/format/table/es_http_reader.{h,cpp}`
- Wraps existing `ESScanReader` (HTTP scroll) and `ScrollParser`
(JSON→columns)
- No JVM overhead — pure C++ HTTP reader, unlike JniReader used by some
formats
- Registered in `FileScanner::_get_next_reader()` for `FORMAT_ES_HTTP`
- Implements `get_columns()` and `fill_all_columns() = true`

**FE — Connector SPI extension**
- Add `ScanNodePropertiesResult` class: typed wrapper carrying both scan
  properties and a `notPushedConjunctIndices` set with explicit
`hasConjunctTracking` flag (distinguishes "no tracking" from "all
pushed")
- Add `getScanNodePropertiesResult()` to `ConnectorScanPlanProvider`

**FE — ES connector adaptation**
- `EsScanRange`: changed to `FILE_SCAN` range type, added `getPath()`,
  `getTableFormatType()`, `getFileFormat()`, BE-compatible property keys
- `EsScanPlanProvider`: builds scan node properties with query_dsl,
  doc_values_mode, auth, docvalue/fields context serialization
- `EsConnectorMetadata`: added `getColumnHandles()` using
`NamedColumnHandle`

**FE — PluginDrivenScanNode enhancement**
- Added ES format mapping (`es_http` → `FORMAT_ES_HTTP`)
- Added `setEsParams()` / `setEsScanLevelParams()` for ES Thrift fields
- Added conjunct pruning via `ScanNodePropertiesResult` (removes
pushed-down
  filters including `esquery()` fake function)
- Added ES-specific EXPLAIN output

**FE — Cleanup**
- Deleted `PluginDrivenEsScanNode.java` (390 lines)
- Simplified `PhysicalPlanTranslator`: removed ES-specific branch, all
  plugin-driven connectors now go through `PluginDrivenScanNode`

### Release note

ES catalog scans now use the unified FILE_SCAN execution path shared
with
JDBC and other plugin-driven connectors. This is an internal
architectural
change — query behavior and results are unchanged. The `esquery()`
function
and doc-value optimization continue to work as before.

### Check List (For Author)

- Test: Regression test — all 7 ES test suites pass
    - test_es_query
    - test_es_query_nereids
    - test_es_flatten_type
    - test_es_keyword_array_type
    - test_es_query_no_http_url
    - test_es_query_predicate_correctness
    - test_es_catalog_http_open_api
- Behavior changed: No
- Does this need documentation: No

…r write seams

WHAT BROKE: build 1006199 failed 40 non-muted external regression suites (35 iceberg,
5 hive), every one with the same FE error -- "STRUCT requires at least one child type"
/ "MAP requires exactly 2 child type(s), got 0" / "ARRAY requires exactly 1 child
type(s), got 0". The 41st failure (tvf test_hdfs_parquet_group0, MEM_LIMIT_EXCEEDED)
is muted and unrelated.

ROOT CAUSE: both connector write seams in PhysicalPlanTranslator built their
ConnectorColumn list from ConnectorType.of(col.getType().getPrimitiveType().toString()),
i.e. from the column's primitive TAG alone. For an ARRAY/MAP/STRUCT column that yields
the bare string "ARRAY"/"MAP"/"STRUCT" and drops every child type. The seams have been
wrong since they were written (apache#62183 / apache#64688); it stayed invisible because fe-core
silently degrades a childless complex type to ARRAY<NULL>/MAP<NULL,NULL> and no
connector reads types off the write handle (jdbc's buildInsertSql uses names, hive's
buildColumns reads HmsTableInfo, iceberg never reads them). 0ff75bb added the eager
shape check to ConnectorType's canonical constructor, which turned that latent
information loss into a hard failure at plan-translation time.

BLAST RADIUS: the INSERT seam (visitPhysicalConnectorTableSink) breaks any external
write whose target holds a complex column, CTAS included. The row-level DML seam
(buildPluginRowLevelDmlSink) breaks EVERY iceberg DELETE/UPDATE/MERGE, even against an
all-scalar table, because ExternalRowLevelDeletePlanBuilder takes getBaseSchema(true)
and the hidden __DORIS_ICEBERG_ROWID_COL__ is itself a STRUCT. Both seams are generic,
so maxcompute writes with a complex column were exposed to the same throw with no test
to catch it.

FIX: convert the whole type through ConnectorColumnConverter.toConnectorType, which
recurses and is already what the CREATE TABLE path
(CreateTableInfoToConnectorRequestConverter) uses. The shape check is deliberately NOT
relaxed: it caught a real defect, and relaxing it would restore the silence for the
first connector that does read these types.

TEST: the existing tests in PhysicalPlanTranslatorIcebergRowLevelDmlTest use a
scalar-only fixture, which is exactly why they could not catch this. The new test gives
the delete sink a STRUCT row-id column and asserts the ConnectorType keeps its children
and field names. Verified both directions: 5/5 pass with the fix, and reverting the fix
makes the new test fail with the production exception ("STRUCT requires at least one
child type").

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

…r write seams

WHAT BROKE: build 1006199 failed 40 non-muted external regression suites (35 iceberg,
5 hive), every one with the same FE error -- "STRUCT requires at least one child type"
/ "MAP requires exactly 2 child type(s), got 0" / "ARRAY requires exactly 1 child
type(s), got 0". The 41st failure (tvf test_hdfs_parquet_group0, MEM_LIMIT_EXCEEDED)
is muted and unrelated.

ROOT CAUSE: both connector write seams in PhysicalPlanTranslator built their
ConnectorColumn list from ConnectorType.of(col.getType().getPrimitiveType().toString()),
i.e. from the column's primitive TAG alone. For an ARRAY/MAP/STRUCT column that yields
the bare string "ARRAY"/"MAP"/"STRUCT" and drops every child type. The seams have been
wrong since they were written (apache#62183 / apache#64688); it stayed invisible because fe-core
silently degrades a childless complex type to ARRAY<NULL>/MAP<NULL,NULL> and no
connector reads types off the write handle (jdbc's buildInsertSql uses names, hive's
buildColumns reads HmsTableInfo, iceberg never reads them). 0ff75bb added the eager
shape check to ConnectorType's canonical constructor, which turned that latent
information loss into a hard failure at plan-translation time.

BLAST RADIUS: the INSERT seam (visitPhysicalConnectorTableSink) breaks any external
write whose target holds a complex column, CTAS included. The row-level DML seam
(buildPluginRowLevelDmlSink) breaks EVERY iceberg DELETE/UPDATE/MERGE, even against an
all-scalar table, because ExternalRowLevelDeletePlanBuilder takes getBaseSchema(true)
and the hidden __DORIS_ICEBERG_ROWID_COL__ is itself a STRUCT. Both seams are generic,
so maxcompute writes with a complex column were exposed to the same throw with no test
to catch it.

FIX: convert the whole type through ConnectorColumnConverter.toConnectorType, which
recurses and is already what the CREATE TABLE path
(CreateTableInfoToConnectorRequestConverter) uses. The shape check is deliberately NOT
relaxed: it caught a real defect, and relaxing it would restore the silence for the
first connector that does read these types.

TEST: the existing tests in PhysicalPlanTranslatorIcebergRowLevelDmlTest use a
scalar-only fixture, which is exactly why they could not catch this. The new test gives
the delete sink a STRUCT row-id column and asserts the ConnectorType keeps its children
and field names. Verified both directions: 5/5 pass with the fix, and reverting the fix
makes the new test fail with the production exception ("STRUCT requires at least one
child type").

Co-Authored-By: Claude Opus 5 (1M context) <[email protected]>

…ule edit to add a connector, dead-surface deletion, 11 correctness fixes (#66135)

### What problem does this PR solve?

Issue Number: #65185

Problem Summary:

Part of the catalog-SPI migration (#65185). Review round 20, targeting
`branch-catalog-spi`.

This round is one self-contained work line: normalize the interface
design of the two public modules
`fe-connector-api` and `fe-connector-spi`, so that adding a new
connector means implementing declared
interfaces and **does not require editing `fe-core`, `fe-connector-api`
or `fe-connector-spi`**.

It started from a full audit of those two modules (172 findings), which
was turned into 25 numbered
tasks plus 10 separately registered open items. All 35 are now done, one
commit per task. The audit
report, the task documents and the rolling handoff are included under
`plan-doc/connector-public-interface-cleanup/`.

**1. Correctness defects found by the audit and fixed**

| module | defect | observable today? |
| --- | --- | --- |
| trino | an OR predicate with 3+ arms was folded from only the first
two arms | **yes** — silently drops rows; a regression introduced by
this migration |
| trino | STRUCT/complex-type field names were dropped, so the engine
invented `col0` / `col1` | **yes** — a regression introduced by this
migration |
| paimon | `col <=> value` was pushed down as `IS NULL` | latent: a
rewrite rule normally converts it before the connector sees it; drops
rows once that rule is off |
| paimon | a LIKE pattern containing `_`, an escape, or a mid-string `%`
was narrowed to a prefix match | drops rows |
| hudi | the "last modified" field carried the raw instant string
instead of epoch millis, so the query cache never engaged for
partitioned hudi tables | yes, when the session cache switch is on |
| hive/hudi wrappers | two context wrappers forwarded only the methods
that existed when they were written, so every future engine-context
method would silently lose its classloader pin | latent by construction
— fixed at the mechanism (a forwarding base class), not per method |
| fe-core | both connector write seams in `PhysicalPlanTranslator` built
their column list from the primitive tag alone, dropping every child
type of an ARRAY/MAP/STRUCT column | **yes** — pre-existing since #62183
/ #64688, latent while fe-core silently degraded a childless complex
type; the eager shape check added earlier in this line turned it into a
hard failure, and it broke 40 external regression suites (35 iceberg, 5
hive) on build 1006199. Blast radius: any external write to a table with
a complex column (CTAS included), and **every** iceberg
DELETE/UPDATE/MERGE, since the hidden row-id column is itself a STRUCT |
| fe-core | `ConnectorPluginManager`'s load loop had two reject paths
and released the plugin classloader on only one of them | classloader
leak on a rejected plugin registration; `FileSystemPluginManager`, the
sibling it mirrors, discards on both |
| fe-core | the query profile printed two data-source rows that are
never populated | yes, cosmetic |

Two more user-visible gaps, both on the "one HMS catalog serving two
table formats" delegation path:
nested-column DDL was not forwarded to the iceberg sibling (the same
table accepted the statement in a
standalone catalog but reported "not supported" through the gateway),
and an iceberg table's comment
came back empty through the gateway (the getter has no table handle, so
the gateway's handle-based
routing could not apply).

**2. Adding a connector no longer touches the common modules**

- The catalog-type allow-list is deleted: a registered connector plugin
is now reachable by
`CREATE CATALOG` without any common-module edit. Built-in type names
became reserved words, rejected
at plugin registration time, so shadowing is impossible rather than
tolerated (the Trino model: one
  registry, duplicate name refused).
- The remaining source-name branches in `fe-core` are gone. `CREATE
TABLE` is routed by catalog instead
of by engine name, each catalog answers for its own accepted `CREATE
TABLE` engine names, the dead
`MODIFY ENGINE` subsystem is deleted, and the engine name a table
displays is declared by the
connector (`ConnectorProvider.displayEngineName()`, defaulting to the
catalog type name; only
MaxCompute overrides it). **After this round `fe-core` contains no
branch keyed on a data-source
  name.**
- Per-table capabilities moved from loose strings to a typed set, and
the write-trait mirror methods on
  the entry interface are deleted.
- BE file-cache admission and two engine branches previously matched by
type name are now connector
  capability declarations.

**3. Dead interface surface deleted**

Four batches: six pieces of unused connector surface, the LIMIT pushdown
entry point, the unwired
property-descriptor mechanism, the scan-range-type enum family, the
push-model cache-invalidation SPI,
`estimateScanRangeCount`, the create-database mirror switch, and four
zero-consumer SPI groups. Every
deletion was re-scanned across the whole repository (fe-core, 8
connectors, tests,
`be-java-extensions`, service-discovery config, reflection strings) for
zero hits.
`ConnectorTableOps` is split into domain interfaces with a stated
minimum implementation set, SQL
passthrough moved to its own optional interface, and the four `planScan`
overloads collapsed into one
abstract method plus a `ConnectorScanRequest` object (14 connector
overrides became 8). A recorded
surface baseline (`ConnectorMetadataSurfaceTest`, 72 methods) now guards
the interface against silent
growth.

**4. Neutralization**

Elasticsearch-specific surface and the ES branch inside the generic scan
node are relocated, the
partition null sentinel gets a neutral name with the directory-name rule
sunk into the hudi connector,
the scan-node property-key contract is centralized, the distributed
procedure's result columns are
handed back to the connector, and a catalog's storage services are split
into their own context object.

**5. Contracts written down**

Package-level design rules for both public modules, the
predicate-pushdown contract (per-operator
semantics, plus "if you cannot translate it exactly, do not push it
down" — the root cause of three of
the fixes above), the read-transaction lifecycle, the two property
namespaces and their precedence, the
unit of `getLength()` (connector-defined; the engine must not read it as
bytes), and a batch of
interface docs corrected where they contradicted the implementation.

…ule edit to add a connector, dead-surface deletion, 11 correctness fixes (#66135)

Issue Number: #65185

Problem Summary:

Part of the catalog-SPI migration (#65185). Review round 20, targeting
`branch-catalog-spi`.

This round is one self-contained work line: normalize the interface
design of the two public modules
`fe-connector-api` and `fe-connector-spi`, so that adding a new
connector means implementing declared
interfaces and **does not require editing `fe-core`, `fe-connector-api`
or `fe-connector-spi`**.

It started from a full audit of those two modules (172 findings), which
was turned into 25 numbered
tasks plus 10 separately registered open items. All 35 are now done, one
commit per task. The audit
report, the task documents and the rolling handoff are included under
`plan-doc/connector-public-interface-cleanup/`.

**1. Correctness defects found by the audit and fixed**

| module | defect | observable today? |
| --- | --- | --- |
| trino | an OR predicate with 3+ arms was folded from only the first
two arms | **yes** — silently drops rows; a regression introduced by
this migration |
| trino | STRUCT/complex-type field names were dropped, so the engine
invented `col0` / `col1` | **yes** — a regression introduced by this
migration |
| paimon | `col <=> value` was pushed down as `IS NULL` | latent: a
rewrite rule normally converts it before the connector sees it; drops
rows once that rule is off |
| paimon | a LIKE pattern containing `_`, an escape, or a mid-string `%`
was narrowed to a prefix match | drops rows |
| hudi | the "last modified" field carried the raw instant string
instead of epoch millis, so the query cache never engaged for
partitioned hudi tables | yes, when the session cache switch is on |
| hive/hudi wrappers | two context wrappers forwarded only the methods
that existed when they were written, so every future engine-context
method would silently lose its classloader pin | latent by construction
— fixed at the mechanism (a forwarding base class), not per method |
| fe-core | both connector write seams in `PhysicalPlanTranslator` built
their column list from the primitive tag alone, dropping every child
type of an ARRAY/MAP/STRUCT column | **yes** — pre-existing since #62183
/ #64688, latent while fe-core silently degraded a childless complex
type; the eager shape check added earlier in this line turned it into a
hard failure, and it broke 40 external regression suites (35 iceberg, 5
hive) on build 1006199. Blast radius: any external write to a table with
a complex column (CTAS included), and **every** iceberg
DELETE/UPDATE/MERGE, since the hidden row-id column is itself a STRUCT |
| fe-core | `ConnectorPluginManager`'s load loop had two reject paths
and released the plugin classloader on only one of them | classloader
leak on a rejected plugin registration; `FileSystemPluginManager`, the
sibling it mirrors, discards on both |
| fe-core | the query profile printed two data-source rows that are
never populated | yes, cosmetic |

Two more user-visible gaps, both on the "one HMS catalog serving two
table formats" delegation path:
nested-column DDL was not forwarded to the iceberg sibling (the same
table accepted the statement in a
standalone catalog but reported "not supported" through the gateway),
and an iceberg table's comment
came back empty through the gateway (the getter has no table handle, so
the gateway's handle-based
routing could not apply).

**2. Adding a connector no longer touches the common modules**

- The catalog-type allow-list is deleted: a registered connector plugin
is now reachable by
`CREATE CATALOG` without any common-module edit. Built-in type names
became reserved words, rejected
at plugin registration time, so shadowing is impossible rather than
tolerated (the Trino model: one
  registry, duplicate name refused).
- The remaining source-name branches in `fe-core` are gone. `CREATE
TABLE` is routed by catalog instead
of by engine name, each catalog answers for its own accepted `CREATE
TABLE` engine names, the dead
`MODIFY ENGINE` subsystem is deleted, and the engine name a table
displays is declared by the
connector (`ConnectorProvider.displayEngineName()`, defaulting to the
catalog type name; only
MaxCompute overrides it). **After this round `fe-core` contains no
branch keyed on a data-source
  name.**
- Per-table capabilities moved from loose strings to a typed set, and
the write-trait mirror methods on
  the entry interface are deleted.
- BE file-cache admission and two engine branches previously matched by
type name are now connector
  capability declarations.

**3. Dead interface surface deleted**

Four batches: six pieces of unused connector surface, the LIMIT pushdown
entry point, the unwired
property-descriptor mechanism, the scan-range-type enum family, the
push-model cache-invalidation SPI,
`estimateScanRangeCount`, the create-database mirror switch, and four
zero-consumer SPI groups. Every
deletion was re-scanned across the whole repository (fe-core, 8
connectors, tests,
`be-java-extensions`, service-discovery config, reflection strings) for
zero hits.
`ConnectorTableOps` is split into domain interfaces with a stated
minimum implementation set, SQL
passthrough moved to its own optional interface, and the four `planScan`
overloads collapsed into one
abstract method plus a `ConnectorScanRequest` object (14 connector
overrides became 8). A recorded
surface baseline (`ConnectorMetadataSurfaceTest`, 72 methods) now guards
the interface against silent
growth.

**4. Neutralization**

Elasticsearch-specific surface and the ES branch inside the generic scan
node are relocated, the
partition null sentinel gets a neutral name with the directory-name rule
sunk into the hudi connector,
the scan-node property-key contract is centralized, the distributed
procedure's result columns are
handed back to the connector, and a catalog's storage services are split
into their own context object.

**5. Contracts written down**

Package-level design rules for both public modules, the
predicate-pushdown contract (per-operator
semantics, plus "if you cannot translate it exactly, do not push it
down" — the root cause of three of
the fixes above), the read-transaction lifecycle, the two property
namespaces and their precedence, the
unit of `getLength()` (connector-defined; the engine must not read it as
bytes), and a batch of
interface docs corrected where they contradicted the implementation.

…ule edit to add a connector, dead-surface deletion, 11 correctness fixes (#66135)

Issue Number: #65185

Problem Summary:

Part of the catalog-SPI migration (#65185). Review round 20, targeting
`branch-catalog-spi`.

This round is one self-contained work line: normalize the interface
design of the two public modules
`fe-connector-api` and `fe-connector-spi`, so that adding a new
connector means implementing declared
interfaces and **does not require editing `fe-core`, `fe-connector-api`
or `fe-connector-spi`**.

It started from a full audit of those two modules (172 findings), which
was turned into 25 numbered
tasks plus 10 separately registered open items. All 35 are now done, one
commit per task. The audit
report, the task documents and the rolling handoff are included under
`plan-doc/connector-public-interface-cleanup/`.

**1. Correctness defects found by the audit and fixed**

| module | defect | observable today? |
| --- | --- | --- |
| trino | an OR predicate with 3+ arms was folded from only the first
two arms | **yes** — silently drops rows; a regression introduced by
this migration |
| trino | STRUCT/complex-type field names were dropped, so the engine
invented `col0` / `col1` | **yes** — a regression introduced by this
migration |
| paimon | `col <=> value` was pushed down as `IS NULL` | latent: a
rewrite rule normally converts it before the connector sees it; drops
rows once that rule is off |
| paimon | a LIKE pattern containing `_`, an escape, or a mid-string `%`
was narrowed to a prefix match | drops rows |
| hudi | the "last modified" field carried the raw instant string
instead of epoch millis, so the query cache never engaged for
partitioned hudi tables | yes, when the session cache switch is on |
| hive/hudi wrappers | two context wrappers forwarded only the methods
that existed when they were written, so every future engine-context
method would silently lose its classloader pin | latent by construction
— fixed at the mechanism (a forwarding base class), not per method |
| fe-core | both connector write seams in `PhysicalPlanTranslator` built
their column list from the primitive tag alone, dropping every child
type of an ARRAY/MAP/STRUCT column | **yes** — pre-existing since #62183
/ #64688, latent while fe-core silently degraded a childless complex
type; the eager shape check added earlier in this line turned it into a
hard failure, and it broke 40 external regression suites (35 iceberg, 5
hive) on build 1006199. Blast radius: any external write to a table with
a complex column (CTAS included), and **every** iceberg
DELETE/UPDATE/MERGE, since the hidden row-id column is itself a STRUCT |
| fe-core | `ConnectorPluginManager`'s load loop had two reject paths
and released the plugin classloader on only one of them | classloader
leak on a rejected plugin registration; `FileSystemPluginManager`, the
sibling it mirrors, discards on both |
| fe-core | the query profile printed two data-source rows that are
never populated | yes, cosmetic |

Two more user-visible gaps, both on the "one HMS catalog serving two
table formats" delegation path:
nested-column DDL was not forwarded to the iceberg sibling (the same
table accepted the statement in a
standalone catalog but reported "not supported" through the gateway),
and an iceberg table's comment
came back empty through the gateway (the getter has no table handle, so
the gateway's handle-based
routing could not apply).

**2. Adding a connector no longer touches the common modules**

- The catalog-type allow-list is deleted: a registered connector plugin
is now reachable by
`CREATE CATALOG` without any common-module edit. Built-in type names
became reserved words, rejected
at plugin registration time, so shadowing is impossible rather than
tolerated (the Trino model: one
  registry, duplicate name refused).
- The remaining source-name branches in `fe-core` are gone. `CREATE
TABLE` is routed by catalog instead
of by engine name, each catalog answers for its own accepted `CREATE
TABLE` engine names, the dead
`MODIFY ENGINE` subsystem is deleted, and the engine name a table
displays is declared by the
connector (`ConnectorProvider.displayEngineName()`, defaulting to the
catalog type name; only
MaxCompute overrides it). **After this round `fe-core` contains no
branch keyed on a data-source
  name.**
- Per-table capabilities moved from loose strings to a typed set, and
the write-trait mirror methods on
  the entry interface are deleted.
- BE file-cache admission and two engine branches previously matched by
type name are now connector
  capability declarations.

**3. Dead interface surface deleted**

Four batches: six pieces of unused connector surface, the LIMIT pushdown
entry point, the unwired
property-descriptor mechanism, the scan-range-type enum family, the
push-model cache-invalidation SPI,
`estimateScanRangeCount`, the create-database mirror switch, and four
zero-consumer SPI groups. Every
deletion was re-scanned across the whole repository (fe-core, 8
connectors, tests,
`be-java-extensions`, service-discovery config, reflection strings) for
zero hits.
`ConnectorTableOps` is split into domain interfaces with a stated
minimum implementation set, SQL
passthrough moved to its own optional interface, and the four `planScan`
overloads collapsed into one
abstract method plus a `ConnectorScanRequest` object (14 connector
overrides became 8). A recorded
surface baseline (`ConnectorMetadataSurfaceTest`, 72 methods) now guards
the interface against silent
growth.

**4. Neutralization**

Elasticsearch-specific surface and the ES branch inside the generic scan
node are relocated, the
partition null sentinel gets a neutral name with the directory-name rule
sunk into the hudi connector,
the scan-node property-key contract is centralized, the distributed
procedure's result columns are
handed back to the connector, and a catalog's storage services are split
into their own context object.

**5. Contracts written down**

Package-level design rules for both public modules, the
predicate-pushdown contract (per-operator
semantics, plus "if you cannot translate it exactly, do not push it
down" — the root cause of three of
the fixes above), the read-transaction lifecycle, the two property
namespaces and their precedence, the
unit of `getLength()` (connector-defined; the engine must not read it as
bytes), and a batch of
interface docs corrected where they contradicted the implementation.

…ule edit to add a connector, dead-surface deletion, 11 correctness fixes (#66135)

Issue Number: #65185

Problem Summary:

Part of the catalog-SPI migration (#65185). Review round 20, targeting
`branch-catalog-spi`.

This round is one self-contained work line: normalize the interface
design of the two public modules
`fe-connector-api` and `fe-connector-spi`, so that adding a new
connector means implementing declared
interfaces and **does not require editing `fe-core`, `fe-connector-api`
or `fe-connector-spi`**.

It started from a full audit of those two modules (172 findings), which
was turned into 25 numbered
tasks plus 10 separately registered open items. All 35 are now done, one
commit per task. The audit
report, the task documents and the rolling handoff are included under
`plan-doc/connector-public-interface-cleanup/`.

**1. Correctness defects found by the audit and fixed**

| module | defect | observable today? |
| --- | --- | --- |
| trino | an OR predicate with 3+ arms was folded from only the first
two arms | **yes** — silently drops rows; a regression introduced by
this migration |
| trino | STRUCT/complex-type field names were dropped, so the engine
invented `col0` / `col1` | **yes** — a regression introduced by this
migration |
| paimon | `col <=> value` was pushed down as `IS NULL` | latent: a
rewrite rule normally converts it before the connector sees it; drops
rows once that rule is off |
| paimon | a LIKE pattern containing `_`, an escape, or a mid-string `%`
was narrowed to a prefix match | drops rows |
| hudi | the "last modified" field carried the raw instant string
instead of epoch millis, so the query cache never engaged for
partitioned hudi tables | yes, when the session cache switch is on |
| hive/hudi wrappers | two context wrappers forwarded only the methods
that existed when they were written, so every future engine-context
method would silently lose its classloader pin | latent by construction
— fixed at the mechanism (a forwarding base class), not per method |
| fe-core | both connector write seams in `PhysicalPlanTranslator` built
their column list from the primitive tag alone, dropping every child
type of an ARRAY/MAP/STRUCT column | **yes** — pre-existing since #62183
/ #64688, latent while fe-core silently degraded a childless complex
type; the eager shape check added earlier in this line turned it into a
hard failure, and it broke 40 external regression suites (35 iceberg, 5
hive) on build 1006199. Blast radius: any external write to a table with
a complex column (CTAS included), and **every** iceberg
DELETE/UPDATE/MERGE, since the hidden row-id column is itself a STRUCT |
| fe-core | `ConnectorPluginManager`'s load loop had two reject paths
and released the plugin classloader on only one of them | classloader
leak on a rejected plugin registration; `FileSystemPluginManager`, the
sibling it mirrors, discards on both |
| fe-core | the query profile printed two data-source rows that are
never populated | yes, cosmetic |

Two more user-visible gaps, both on the "one HMS catalog serving two
table formats" delegation path:
nested-column DDL was not forwarded to the iceberg sibling (the same
table accepted the statement in a
standalone catalog but reported "not supported" through the gateway),
and an iceberg table's comment
came back empty through the gateway (the getter has no table handle, so
the gateway's handle-based
routing could not apply).

**2. Adding a connector no longer touches the common modules**

- The catalog-type allow-list is deleted: a registered connector plugin
is now reachable by
`CREATE CATALOG` without any common-module edit. Built-in type names
became reserved words, rejected
at plugin registration time, so shadowing is impossible rather than
tolerated (the Trino model: one
  registry, duplicate name refused).
- The remaining source-name branches in `fe-core` are gone. `CREATE
TABLE` is routed by catalog instead
of by engine name, each catalog answers for its own accepted `CREATE
TABLE` engine names, the dead
`MODIFY ENGINE` subsystem is deleted, and the engine name a table
displays is declared by the
connector (`ConnectorProvider.displayEngineName()`, defaulting to the
catalog type name; only
MaxCompute overrides it). **After this round `fe-core` contains no
branch keyed on a data-source
  name.**
- Per-table capabilities moved from loose strings to a typed set, and
the write-trait mirror methods on
  the entry interface are deleted.
- BE file-cache admission and two engine branches previously matched by
type name are now connector
  capability declarations.

**3. Dead interface surface deleted**

Four batches: six pieces of unused connector surface, the LIMIT pushdown
entry point, the unwired
property-descriptor mechanism, the scan-range-type enum family, the
push-model cache-invalidation SPI,
`estimateScanRangeCount`, the create-database mirror switch, and four
zero-consumer SPI groups. Every
deletion was re-scanned across the whole repository (fe-core, 8
connectors, tests,
`be-java-extensions`, service-discovery config, reflection strings) for
zero hits.
`ConnectorTableOps` is split into domain interfaces with a stated
minimum implementation set, SQL
passthrough moved to its own optional interface, and the four `planScan`
overloads collapsed into one
abstract method plus a `ConnectorScanRequest` object (14 connector
overrides became 8). A recorded
surface baseline (`ConnectorMetadataSurfaceTest`, 72 methods) now guards
the interface against silent
growth.

**4. Neutralization**

Elasticsearch-specific surface and the ES branch inside the generic scan
node are relocated, the
partition null sentinel gets a neutral name with the directory-name rule
sunk into the hudi connector,
the scan-node property-key contract is centralized, the distributed
procedure's result columns are
handed back to the connector, and a catalog's storage services are split
into their own context object.

**5. Contracts written down**

Package-level design rules for both public modules, the
predicate-pushdown contract (per-operator
semantics, plus "if you cannot translate it exactly, do not push it
down" — the root cause of three of
the fixes above), the read-transaction lifecycle, the two property
namespaces and their precedence, the
unit of `getLength()` (connector-defined; the engine must not read it as
bytes), and a batch of
interface docs corrected where they contradicted the implementation.

…ule edit to add a connector, dead-surface deletion, 11 correctness fixes (#66135)

Issue Number: #65185

Problem Summary:

Part of the catalog-SPI migration (#65185). Review round 20, targeting
`branch-catalog-spi`.

This round is one self-contained work line: normalize the interface
design of the two public modules
`fe-connector-api` and `fe-connector-spi`, so that adding a new
connector means implementing declared
interfaces and **does not require editing `fe-core`, `fe-connector-api`
or `fe-connector-spi`**.

It started from a full audit of those two modules (172 findings), which
was turned into 25 numbered
tasks plus 10 separately registered open items. All 35 are now done, one
commit per task. The audit
report, the task documents and the rolling handoff are included under
`plan-doc/connector-public-interface-cleanup/`.

**1. Correctness defects found by the audit and fixed**

| module | defect | observable today? |
| --- | --- | --- |
| trino | an OR predicate with 3+ arms was folded from only the first
two arms | **yes** — silently drops rows; a regression introduced by
this migration |
| trino | STRUCT/complex-type field names were dropped, so the engine
invented `col0` / `col1` | **yes** — a regression introduced by this
migration |
| paimon | `col <=> value` was pushed down as `IS NULL` | latent: a
rewrite rule normally converts it before the connector sees it; drops
rows once that rule is off |
| paimon | a LIKE pattern containing `_`, an escape, or a mid-string `%`
was narrowed to a prefix match | drops rows |
| hudi | the "last modified" field carried the raw instant string
instead of epoch millis, so the query cache never engaged for
partitioned hudi tables | yes, when the session cache switch is on |
| hive/hudi wrappers | two context wrappers forwarded only the methods
that existed when they were written, so every future engine-context
method would silently lose its classloader pin | latent by construction
— fixed at the mechanism (a forwarding base class), not per method |
| fe-core | both connector write seams in `PhysicalPlanTranslator` built
their column list from the primitive tag alone, dropping every child
type of an ARRAY/MAP/STRUCT column | **yes** — pre-existing since #62183
/ #64688, latent while fe-core silently degraded a childless complex
type; the eager shape check added earlier in this line turned it into a
hard failure, and it broke 40 external regression suites (35 iceberg, 5
hive) on build 1006199. Blast radius: any external write to a table with
a complex column (CTAS included), and **every** iceberg
DELETE/UPDATE/MERGE, since the hidden row-id column is itself a STRUCT |
| fe-core | `ConnectorPluginManager`'s load loop had two reject paths
and released the plugin classloader on only one of them | classloader
leak on a rejected plugin registration; `FileSystemPluginManager`, the
sibling it mirrors, discards on both |
| fe-core | the query profile printed two data-source rows that are
never populated | yes, cosmetic |

Two more user-visible gaps, both on the "one HMS catalog serving two
table formats" delegation path:
nested-column DDL was not forwarded to the iceberg sibling (the same
table accepted the statement in a
standalone catalog but reported "not supported" through the gateway),
and an iceberg table's comment
came back empty through the gateway (the getter has no table handle, so
the gateway's handle-based
routing could not apply).

**2. Adding a connector no longer touches the common modules**

- The catalog-type allow-list is deleted: a registered connector plugin
is now reachable by
`CREATE CATALOG` without any common-module edit. Built-in type names
became reserved words, rejected
at plugin registration time, so shadowing is impossible rather than
tolerated (the Trino model: one
  registry, duplicate name refused).
- The remaining source-name branches in `fe-core` are gone. `CREATE
TABLE` is routed by catalog instead
of by engine name, each catalog answers for its own accepted `CREATE
TABLE` engine names, the dead
`MODIFY ENGINE` subsystem is deleted, and the engine name a table
displays is declared by the
connector (`ConnectorProvider.displayEngineName()`, defaulting to the
catalog type name; only
MaxCompute overrides it). **After this round `fe-core` contains no
branch keyed on a data-source
  name.**
- Per-table capabilities moved from loose strings to a typed set, and
the write-trait mirror methods on
  the entry interface are deleted.
- BE file-cache admission and two engine branches previously matched by
type name are now connector
  capability declarations.

**3. Dead interface surface deleted**

Four batches: six pieces of unused connector surface, the LIMIT pushdown
entry point, the unwired
property-descriptor mechanism, the scan-range-type enum family, the
push-model cache-invalidation SPI,
`estimateScanRangeCount`, the create-database mirror switch, and four
zero-consumer SPI groups. Every
deletion was re-scanned across the whole repository (fe-core, 8
connectors, tests,
`be-java-extensions`, service-discovery config, reflection strings) for
zero hits.
`ConnectorTableOps` is split into domain interfaces with a stated
minimum implementation set, SQL
passthrough moved to its own optional interface, and the four `planScan`
overloads collapsed into one
abstract method plus a `ConnectorScanRequest` object (14 connector
overrides became 8). A recorded
surface baseline (`ConnectorMetadataSurfaceTest`, 72 methods) now guards
the interface against silent
growth.

**4. Neutralization**

Elasticsearch-specific surface and the ES branch inside the generic scan
node are relocated, the
partition null sentinel gets a neutral name with the directory-name rule
sunk into the hudi connector,
the scan-node property-key contract is centralized, the distributed
procedure's result columns are
handed back to the connector, and a catalog's storage services are split
into their own context object.

**5. Contracts written down**

Package-level design rules for both public modules, the
predicate-pushdown contract (per-operator
semantics, plus "if you cannot translate it exactly, do not push it
down" — the root cause of three of
the fixes above), the read-transaction lifecycle, the two property
namespaces and their precedence, the
unit of `getLength()` (connector-defined; the engine must not read it as
bytes), and a batch of
interface docs corrected where they contradicted the implementation.

…ule edit to add a connector, dead-surface deletion, 11 correctness fixes (#66135)

Issue Number: #65185

Problem Summary:

Part of the catalog-SPI migration (#65185). Review round 20, targeting
`branch-catalog-spi`.

This round is one self-contained work line: normalize the interface
design of the two public modules
`fe-connector-api` and `fe-connector-spi`, so that adding a new
connector means implementing declared
interfaces and **does not require editing `fe-core`, `fe-connector-api`
or `fe-connector-spi`**.

It started from a full audit of those two modules (172 findings), which
was turned into 25 numbered
tasks plus 10 separately registered open items. All 35 are now done, one
commit per task. The audit
report, the task documents and the rolling handoff are included under
`plan-doc/connector-public-interface-cleanup/`.

**1. Correctness defects found by the audit and fixed**

| module | defect | observable today? |
| --- | --- | --- |
| trino | an OR predicate with 3+ arms was folded from only the first
two arms | **yes** — silently drops rows; a regression introduced by
this migration |
| trino | STRUCT/complex-type field names were dropped, so the engine
invented `col0` / `col1` | **yes** — a regression introduced by this
migration |
| paimon | `col <=> value` was pushed down as `IS NULL` | latent: a
rewrite rule normally converts it before the connector sees it; drops
rows once that rule is off |
| paimon | a LIKE pattern containing `_`, an escape, or a mid-string `%`
was narrowed to a prefix match | drops rows |
| hudi | the "last modified" field carried the raw instant string
instead of epoch millis, so the query cache never engaged for
partitioned hudi tables | yes, when the session cache switch is on |
| hive/hudi wrappers | two context wrappers forwarded only the methods
that existed when they were written, so every future engine-context
method would silently lose its classloader pin | latent by construction
— fixed at the mechanism (a forwarding base class), not per method |
| fe-core | both connector write seams in `PhysicalPlanTranslator` built
their column list from the primitive tag alone, dropping every child
type of an ARRAY/MAP/STRUCT column | **yes** — pre-existing since #62183
/ #64688, latent while fe-core silently degraded a childless complex
type; the eager shape check added earlier in this line turned it into a
hard failure, and it broke 40 external regression suites (35 iceberg, 5
hive) on build 1006199. Blast radius: any external write to a table with
a complex column (CTAS included), and **every** iceberg
DELETE/UPDATE/MERGE, since the hidden row-id column is itself a STRUCT |
| fe-core | `ConnectorPluginManager`'s load loop had two reject paths
and released the plugin classloader on only one of them | classloader
leak on a rejected plugin registration; `FileSystemPluginManager`, the
sibling it mirrors, discards on both |
| fe-core | the query profile printed two data-source rows that are
never populated | yes, cosmetic |

Two more user-visible gaps, both on the "one HMS catalog serving two
table formats" delegation path:
nested-column DDL was not forwarded to the iceberg sibling (the same
table accepted the statement in a
standalone catalog but reported "not supported" through the gateway),
and an iceberg table's comment
came back empty through the gateway (the getter has no table handle, so
the gateway's handle-based
routing could not apply).

**2. Adding a connector no longer touches the common modules**

- The catalog-type allow-list is deleted: a registered connector plugin
is now reachable by
`CREATE CATALOG` without any common-module edit. Built-in type names
became reserved words, rejected
at plugin registration time, so shadowing is impossible rather than
tolerated (the Trino model: one
  registry, duplicate name refused).
- The remaining source-name branches in `fe-core` are gone. `CREATE
TABLE` is routed by catalog instead
of by engine name, each catalog answers for its own accepted `CREATE
TABLE` engine names, the dead
`MODIFY ENGINE` subsystem is deleted, and the engine name a table
displays is declared by the
connector (`ConnectorProvider.displayEngineName()`, defaulting to the
catalog type name; only
MaxCompute overrides it). **After this round `fe-core` contains no
branch keyed on a data-source
  name.**
- Per-table capabilities moved from loose strings to a typed set, and
the write-trait mirror methods on
  the entry interface are deleted.
- BE file-cache admission and two engine branches previously matched by
type name are now connector
  capability declarations.

**3. Dead interface surface deleted**

Four batches: six pieces of unused connector surface, the LIMIT pushdown
entry point, the unwired
property-descriptor mechanism, the scan-range-type enum family, the
push-model cache-invalidation SPI,
`estimateScanRangeCount`, the create-database mirror switch, and four
zero-consumer SPI groups. Every
deletion was re-scanned across the whole repository (fe-core, 8
connectors, tests,
`be-java-extensions`, service-discovery config, reflection strings) for
zero hits.
`ConnectorTableOps` is split into domain interfaces with a stated
minimum implementation set, SQL
passthrough moved to its own optional interface, and the four `planScan`
overloads collapsed into one
abstract method plus a `ConnectorScanRequest` object (14 connector
overrides became 8). A recorded
surface baseline (`ConnectorMetadataSurfaceTest`, 72 methods) now guards
the interface against silent
growth.

**4. Neutralization**

Elasticsearch-specific surface and the ES branch inside the generic scan
node are relocated, the
partition null sentinel gets a neutral name with the directory-name rule
sunk into the hudi connector,
the scan-node property-key contract is centralized, the distributed
procedure's result columns are
handed back to the connector, and a catalog's storage services are split
into their own context object.

**5. Contracts written down**

Package-level design rules for both public modules, the
predicate-pushdown contract (per-operator
semantics, plus "if you cannot translate it exactly, do not push it
down" — the root cause of three of
the fixes above), the read-transaction lifecycle, the two property
namespaces and their precedence, the
unit of `getLength()` (connector-defined; the engine must not read it as
bytes), and a batch of
interface docs corrected where they contradicted the implementation.

…68801)

### What problem does this PR solve?

Issue Number: None

Related PR: #64757, #67704, #67153, #68250

Original authors: @yujun777 (#68250: the regression suite and the MTMV
assertion are ported from it), @Gabriel39 (#67153, #67704). The commit
carries Co-authored-by for both.

Problem Summary:

**In short.** These four branch-4.x fixes reached master through
master's own PRs, so master's production code already behaves the same.
Some of the tests the branch-4.x PRs added did not come along, mostly
because the code they test has moved on master. This PR adds that
coverage in master's structure: four unit-test cases and one regression
suite. It does not change production code.

**Background**

| branch-4.x PR | Master PR | Where the code lives on master |
|---|---|---|
| #64757: SQL Server / Oracle booleans pushed down as 1/0 | #62183,
#64756 | `fe-connector-jdbc` (`JdbcQueryBuilder.formatBooleanLiteral`) |
| #67704: fractional seconds in Iceberg `FOR TIME AS OF` | #67705 |
`fe-connector-iceberg` (`IcebergTimeUtils`,
`IcebergConnectorMetadata.resolveTimeTravel`) |
| #67153: fuzzy mode keeps FileScannerV2 on | #67574 |
`SessionVariable.initFuzzyModeVariables` |
| #68250: a base partition without a snapshot does not force a full MV
rebuild | #62606 | `MTMV.hasRefreshSnapshot`, `MTMVTask` refresh
planning |

**The problem, and what it cost**

| Behavior | branch-4.x tests | Master before this PR |
|---|---|---|
| SQL Server `bit_value IN (TRUE, FALSE)` → `[bit_value] IN (1, 0)` |
`JdbcScanNodeTest` | not tested; the IN-list path has no unit test (only
`= FALSE` is covered) |
| SQL Server `= TRUE` → `= 1` | `JdbcScanNodeTest` | not tested |
| `FOR TIME AS OF '... .SSS'` picks the snapshot committed at that
millisecond | `IcebergUtilsTest` | only the parser is tested; nothing
checks that the fraction reaches snapshot selection |
| `enable_file_scanner_v2` defaults to true | `SessionVariablesTest` |
only the fuzzy-mode override is tested |
| An emptied partition snapshot map reads as "no baseline" | `MTMVTest`
| only the non-empty case is tested |
| A new base partition makes REFRESH AUTO refresh only that MV partition
| `test_base_table_add_partition_mtmv` | no suite; the existing ones
only check that the partition exists, refresh it by name, or go through
IVM |

A regression in any of these would reach master's CI unnoticed.

**How this PR fixes it**

- `JdbcQueryBuilderTest`: `testSqlserverBooleanTrueLiteral` and
`testSqlserverBooleanInList`, written against `JdbcQueryBuilder` with
SPI expressions (`ConnectorComparison`, `ConnectorIn`). These replace
the deleted `JdbcScanNodeTest`.
-
`IcebergConnectorMetadataMvccTest.resolveTimestampDatetimeStringKeepsFractionalSeconds`.
The 4.x test mocked two snapshots in the same second, but this module
has no Mockito and its fixture is a real `InMemoryCatalog`. So the new
test uses the fixture's real S2 commit time instead:
- S2's exact millisecond, written with 3 or 6 fractional digits, must
resolve to S2;
  - one millisecond earlier must not.

A parser that rejects the fraction throws, and one that truncates it
lands before S2. The comment of the existing whole-second test still
described a truncating parser and is corrected.
- `SessionVariablesTest`: assert the constructor default before the
fuzzy-mode case.
- `MTMVTest`: after the incomplete-map case, an emptied map must make
`hasRefreshSnapshot()` false.
-
`regression-test/suites/mtmv_p0/test_base_table_add_partition_mtmv.groovy`:
ported from #68250, with the table names hard-coded and the tables
dropped before use rather than after. The `.out` was generated on
master.

**Results**

| Test | Result |
|---|---|
| `JdbcQueryBuilderTest` | 34 run, 0 failures |
| `IcebergConnectorMetadataMvccTest` | 32 run, 0 failures |
| `MTMVTest` | 49 run, 0 failures |
| `SessionVariablesTest` | 19 run, 0 failures |
| `mtmv_p0/test_base_table_add_partition_mtmv` | passes; the generated
`.out` is byte-identical to branch-4.x's (`["p_3"]`, `PARTIAL`, then
`NOT_REFRESH`) |

Co-authored-by: yujun <[email protected]>
Co-authored-by: Gabriel <[email protected]>

…RING

### What problem does this PR solve?

Issue Number: None

Related PR: apache#62183, apache#55651

Problem Summary: Mapping a remote Doris VARBINARY to VARBINARY also
reached the element of an ARRAY column, because the Doris dialect maps
an array element through the same function. A remote view with
array(cast('abc' as varbinary(16))) then became ARRAY<VARBINARY>, and
reading it failed with "Unsupported array child type: VARBINARY": the
remote returns an array as text that spells each VARBINARY element as
0x hex ([0x616263, null, 0x]), and MySQLTypeHandler decodes array
elements from that text with no case for VARBINARY. Before the mapping
change the element was STRING and the column read as that text.
branch-4.x maps such a column to ARRAY<VARBINARY> as well and fails the
same way when it is read.

A VARBINARY element of a remote Doris array, nested or not, maps to
STRING again; only a top-level VARBINARY maps to VARBINARY.

### Release note

None

### Check List (For Author)

- Test: Unit Test / Regression test
    - JdbcLogicalTypeMappingTest.remoteDorisVarbinaryArrayElementsStayString:
      ARRAY<VARBINARY>, ARRAY<VARBINARY(16)> and ARRAY<ARRAY<VARBINARY(16)>>
      keep STRING elements; all fe-connector-jdbc tests (251) pass.
    - test_doris_jdbc_catalog_varbinary also reads a remote view with a
      VARBINARY array; the .out was generated by the script.
- Behavior changed: No (compared with master)
- Does this need documentation: No

Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]>

### What problem does this PR solve?

Issue Number: None

Related PR: #62183, #55651, #68786, #56181

Problem Summary:

**In short.** A JDBC catalog that points at another Doris shows the
remote VARBINARY columns as STRING (`text`). This PR maps them to
VARBINARY again, as the fe-core JDBC client and branch-4.x do; a
VARBINARY element of an array stays STRING, because the scanner reads
array elements only as text. Reading a VARBINARY column through a Java
scanner, which the new mapping now does for a remote Doris, has leaked a
native buffer on every batch since VARBINARY was added to the JNI
vectors, so this PR frees it too. It is not a port of a branch-4.x
change; it came up while comparing the binary type mappings of the two
branches.

**Background.** When the JDBC connector became a plugin (#62183), the
Doris dialect of `JdbcMySQLConnectorClient` mapped a remote `VARBINARY`
to `STRING`. The fe-core `JdbcMySQLClient` it replaced, and branch-4.x,
map it to `VARBINARY` with the column size the remote reports (#55651).
For every other JDBC source, binary types already map to `VARBINARY`
regardless of the deprecated `enable.mapping.*` flags (#68786, pinned by
`JdbcLogicalTypeMappingTest`). A remote Doris exposes VARBINARY through
views and external tables.

A remote Doris returns an `ARRAY` column as text, which spells a
VARBINARY element as `0x` hex (`[0x616263, null, 0x]`), and
`MySQLTypeHandler` decodes the elements from that text: into strings,
numbers and dates, not into bytes.

On the BE side, a Java scanner hands each batch over in a `VectorTable`.
A VARBINARY column of that table keeps a 16-byte `[length, address]`
record per row in an off-heap buffer of its own, and each record points
at a separate block with the row's bytes.

**The problem, and what it cost.**

| A JDBC catalog connected to the local Doris | master | with this PR |
|---|---|---|
| `DESC` of a view with `cast('abc' as varbinary(16))` | `b text` | `b
varbinary(65535)` (the length the remote reports through the JDBC
metadata) |
| `SELECT b` | the value as a string | `0x616263`, the remote bytes;
`hex(b)` = `616263` |
| a view with `array(cast('abc' as varbinary(16)), cast(null as
varbinary(16)), cast('' as varbinary(16)), cast(unhex('00ff') as
varbinary(16)))` | `array<text>`, `["0x616263", null, "0x", "0x00FF"]` |
the same |

Mapping the array element to VARBINARY as well, as branch-4.x does,
makes the column `array<varbinary(2147483647)>`, and reading it fails
with `Unsupported array child type: VARBINARY`.

`VectorColumn.close()` freed the row blocks of a VARBINARY column but
never its record buffer (16 bytes per row of batch capacity, about 64 KB
for a batch of 4064 rows). BE closes the Java table after every batch
(the V1 reader through `releaseTable`, the V2 reader through
`releaseColumn` per column), and a Java UDF closes its output table
after every call, so every JNI read of a binary column (the binary types
of every JDBC source, Paimon, Hudi, MaxCompute) and every Java UDF that
returns VARBINARY lost one such buffer per VARBINARY column per batch.
The allocator behind it is plain malloc / free, so nothing reclaims the
buffer.

**How this PR fixes it.**

- The Doris dialect maps `VARBINARY` to `VARBINARY`, taking the length
from the type name when it carries one and otherwise from the column
size, like the dialect's `VARCHAR` branch and the fe-core client.
- A VARBINARY element of a remote Doris array, nested or not, keeps
mapping to `STRING`.
- `VectorColumn.close()` frees the record buffer of a VARBINARY column
after the row blocks it points to.

**Results.**

- FE UT:
`JdbcLogicalTypeMappingTest#remoteDorisVarbinaryKeepsBytesAndDeclaredLength`
fails without the mapping (STRING) and passes with it;
`#remoteDorisVarbinaryArrayElementsStayString` (`ARRAY<VARBINARY>`,
`ARRAY<VARBINARY(16)>`, `ARRAY<ARRAY<VARBINARY(16)>>`) fails when the
array element follows the scalar mapping. All fe-connector-jdbc tests
(251) pass.
- jni-spi UT:
`VectorTableVarbinaryCloseTest#closeFreesTheRowRecordsOfAVarbinaryColumn`
closes a table with a VARBINARY column, as a scanner does after each
batch, and checks that the record buffer is freed; it fails without the
fix. All jni-spi tests (24) pass. jni-spi gains `mockito-inline` in test
scope to observe the static `OffHeap` calls.
- Regression: `test_doris_jdbc_catalog_varbinary` connects a JDBC
catalog to the local Doris and reads a view with a VARBINARY column and
a view with a VARBINARY array. With the plugin from master the scalar
column is described as `text` and the suite fails; with the scalar
mapping alone the array column is described as
`array<varbinary(2147483647)>` and the suite fails; with this PR it
passes.

### Background

Connector plugins (Hudi, and the coming Lance connector) describe their columns with the SPI's `ConnectorType`, and fe-core's `ConnectorColumnConverter` turns each into a Doris type. Connectors put a temporal column's fractional-second scale in the type's precision field: `("DATETIMEV2", 3, 0)` is `datetime(3)`, and Hudi's `HudiTypeMapping` sends `("TIMEV2", 3, 0)` for an Avro `time-millis` column and `("TIMEV2", 6, 0)` for `time-micros`.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#62183 (introduced the converter without a TIMEV2 case), apache#64304 (moved Hudi onto it)
PR Type: Bugfix

Problem Summary:

Since Hudi became a connector plugin, its `time-millis` and `time-micros` columns show as `time` instead of `time(3)` and `time(6)`, which is what 4.x shows and what `HudiTypeMapping` asks for: the converter drops the scale and builds a scale-0 TIMEV2. The Lance connector, ported in a follow-up, maps Arrow `time32(ms)` and `time64` the same way and would lose its scale too. The converter now reads the TIMEV2 scale from the precision field, as it already does for DATETIMEV2.

### Details

- **Frequency:** Always, for every connector column whose `ConnectorType` is TIMEV2 with a scale; today that is every Hudi `time-millis` / `time-micros` column on master. Found while porting the Lance connector, by reading the converter, and confirmed by the unit test below.
- **Root cause:** `ConnectorColumnConverter.convertScalarType` reads the precision field for CHAR, VARCHAR, VARBINARY, the decimals, DATETIMEV2 and TIMESTAMPTZ, but has no TIMEV2 case, so TIMEV2 falls through to `ScalarType.createType("TIMEV2")`, which returns the scale-0 `ScalarType.TIMEV2`.
- **Why existing tests missed it:** Unit tests: `ConnectorColumnConverterTest` checks decimals, CHAR/VARCHAR lengths and the TIMESTAMPTZ extra info but has no TIMEV2 case, and `HudiTypeMappingTest` asserts the `("TIMEV2", 3, 0)` the plugin emits but stops before fe-core converts it. Regression: no suite under `regression-test/suites` reads a Hudi time column.
- **Fix:** The TIMEV2 case uses the precision field as the scale when one is given; without one the type stays the scale-0 TIMEV2 it has always been. The fix sits in the converter because every connector goes through it. The reverse direction, `toConnectorType`, is unchanged: only the read-only Hudi and Lance connectors produce TIMEV2, and the JDBC connector maps TIME to STRING.
- **Other paths and branches:** `toConnectorType` is not affected for reads (see Fix). branch-4.1 and branch-4.2 are not affected: they have no `ConnectorColumnConverter`, and their fe-core Hudi mapping (`HudiUtils`) builds `createTimeV2Type(3)` / `createTimeV2Type(6)` directly. The bug is on master since apache#64304.

### Testing

- `ConnectorColumnConverterTest#timeScaleRidesInThePrecisionSlot` (new): fails before the fix with `expected: <timev2(3)> but was: <timev2(0)>` and passes after; it also pins that a TIMEV2 without a precision stays scale 0. Run locally on master `f5c993f2c94` with apache#70338 applied, which touches neither file.
- `ConnectorColumnConverterTest` (35 tests) and `HudiTypeMappingTest` (17 tests): pass locally on the same tree.

### Release note

None

### Check List (For Author)

- Behavior changed: No
- Does this need documentation: No
- Pre-submission status: Ready

### AI Disclosure

- Tool: Claude Code (Claude Opus 5.5)
- Generated: code, tests, this description
- Human review: not stated
- Prompts: The author asked to port the branch-4.2 Lance FE read path to master as a connector plugin; this fix came out of that port, and the author's rule for such finds is to fix them in their own PR.

Co-Authored-By: Claude Opus 5.5 <[email protected]>