Repository navigation
Commit b417c55
[refactor](catalog) P6 iceberg: migrate to catalog SPI + cutover + remove legacy fe-core subsystem (#64688)
Migrates the legacy in-tree Iceberg catalog (fe-core
`datasource/iceberg`, its
`property/metastore/*Iceberg*` clusters, and every reverse `instanceof
IcebergExternal*` / `case ICEBERG` coupling scattered across nereids /
planner /
alter) onto the catalog SPI as a self-contained `fe-connector-iceberg`
plugin,
following the paimon full-adopter + cutover template. `iceberg` is added
to
`SPI_READY_TYPES`, so an Iceberg catalog now deserializes to
`PluginDrivenExternalCatalog` and every functional area — read, write,
row-level
DML, `ALTER TABLE EXECUTE` procedures, system tables, DDL and
MVCC/time-travel —
flows through the generic `PluginDriven*` bridge with no
Iceberg-specific branch
in fe-core. (Iceberg tables surfaced *through* a `hive`/HMS catalog stay
on the
hive path until P7; this PR covers the dedicated `type=iceberg`
catalog.)
Iceberg is the widest lakehouse adopter migrated so far — 7 catalog
flavors,
merge-on-read row-level DML, 9 stored procedures, v3 row-lineage /
deletion
vectors — so the cutover is held to behavior-parity with the legacy
path. Unlike
the paimon migration (a cutover PR + a separate removal PR), this PR
does the
build-out, the live cutover **and** the full legacy-subsystem removal on
one
branch: fe-core `datasource/iceberg` and its property clusters are
deleted, not
just left dead.
Part of the catalog-SPI migration tracked in #65185 (phase **P6**).
Full read + write + row-level DML + procedures + system tables + DDL +
MVCC/time-travel, planning native (Parquet/ORC) vs JNI reads. Key seams:
- `IcebergCatalogFactory` (pure flavor switch, classloader-pinned for
child-first
loading), `IcebergCatalogOps` (metadata / DDL ops),
`IcebergConnectorMetadata`
(schema latest / at-snapshot, sys tables, MVCC/time-travel, DDL,
partitions,
statistics, table descriptor), `IcebergConnectorProvider` (ServiceLoader
entry).
- `IcebergScanPlanProvider` + `IcebergScanRange`: predicate/projection
pushdown
(`IcebergPredicateConverter` → Iceberg `Expression`), FileScanTask
byte-offset
sub-splitting, COUNT(\*) collapse, merge-on-read delete attachment
(position
delete / Puffin DV / equality delete), `IcebergColumnHandle` field-id
column
pruning, schema-evolution-safe reads (`IcebergSchemaUtils` field-id
dictionary),
synthetic rowid + v3 row-lineage metadata columns.
- Write path: `IcebergWritePlanProvider` / `IcebergWriteContext` /
`IcebergWriterHelper` / `IcebergConnectorTransaction` — INSERT / INSERT
OVERWRITE
(dynamic + static partition), branch-targeted writes, WRITE ORDERED BY,
per-spec
distribution, optimistic-concurrency conflict detection, one SDK
transaction per
statement, BE commit-fragment → DataFile/DeleteFile conversion.
- Row-level DML: DELETE / UPDATE / MERGE INTO over position-delete (v2)
and
deletion-vector (v3) merge-on-read, baseSnapshot-anchored read.
- `action/`: the 9 `ALTER TABLE EXECUTE` procedures
(rollback_to_snapshot,
rollback_to_timestamp, set_current_snapshot, cherrypick_snapshot,
fast_forward,
expire_snapshots, publish_changes, rewrite_manifests,
rewrite_data_files)
dispatched by `IcebergExecuteActionFactory`; `rewrite/` hosts the
distributed
bin-pack `rewrite_data_files` engine.
- `IcebergSchemaBuilder` (CREATE TABLE), `IcebergTypeMapping` (both
directions),
`IcebergColumnChange` / `IcebergComplexTypeDiff` (schema + partition
evolution),
`IcebergLatestSnapshotCache` / `IcebergManifestCache` (per-catalog
caches on the
long-lived connector), `dlf/` (Aliyun DLF client).
(`fe-connector-metastore-iceberg`)
An Iceberg catalog resolves its backend by `MetaStoreProvider`
ServiceLoader
dispatch on `iceberg.catalog.type` (registered via `META-INF/services`),
instead
of a fe-core switch — **7 flavors**: REST, Hive Metastore, AWS Glue,
Hadoop /
filesystem, JDBC, Aliyun DLF, AWS S3Tables. Adding a backend = one
provider class
+ one services line. The connector self-builds the HMS-metastore
Kerberos
authenticator, so metastore auth is owned entirely by the plugin.
- `CatalogFactory` adds `iceberg` to `SPI_READY_TYPES` and drops the
built-in case.
- `GsonUtils` replay-compat: persisted `IcebergExternalCatalog` →
`PluginDrivenExternalCatalog`, `IcebergExternalDatabase` →
`PluginDrivenExternalDatabase`, `IcebergExternalTable` →
`PluginDrivenMvccExternalTable` on deserialization, so existing clusters
upgrade
without losing catalogs (metadata backward-compat invariant).
- Nereids/DDL/alter generalized off the concrete Iceberg types; SHOW
CREATE
renders LOCATION / PARTITION BY / ORDER BY / PROPERTIES via the generic
path
(iceberg engine name preserved, no credential leak).
Per the 2026-07 architecture decision (fe-core holds **no** property
parsing for
migrated connectors; storage parsing → `fe-filesystem`, metastore
parsing →
`fe-connector`, both plugin-side; the plugin assembles → BE thrift →
hands back to
fe-core), iceberg's property/credential/auth resolution moves entirely
into the
connector:
- Storage credentials bound to `fe-filesystem` on both scan and write
paths (one
source), retiring the second fe-core `StorageProperties.createAll`
parse.
- Vended temporary credentials, `warehouse` → `fs.defaultFS` derivation
(hadoop flavor only, via the neutral `Connector.deriveStorageProperties`
SPI),
and Kerberos doAs are all connector-owned; the fe-core pre-execution
authenticator is a no-op for plugin catalogs.
- fe-core's shared static property/metastore helpers are kept alive only
for the
connectors still on the legacy path (Hive / Hudi / LakeSoul /
HMS-iceberg).
Net **−21.7k** lines from fe-core. Removes the 4 dead entity classes
(`IcebergExternalTable` / `IcebergExternalDatabase` /
`IcebergSysExternalTable` /
`IcebergExternalCatalog`), the fe-core Iceberg `MetastoreProperties`
cluster + its
credential helpers, the 5 fe-core Iceberg connectivity probes and the
coordinator
branches, and un-registers the Iceberg property factories (fail-loud on
any stray
call). The now-dead paimon `MetastoreProperties` cluster left behind by
P5 is
swept in the same pass. `GsonUtils` retains only the three string compat
labels
for old-image upgrade.
Parity fixes surfaced by the SPI cutover (each restores legacy
semantics):
- Classloader: TCCL pinned to the plugin classloader at all three loci
where
fe-core crosses into bundled Iceberg (scan call thread, write/DDL/commit
engine
thread via `TcclPinningConnectorContext`, and Iceberg's own
manifest-writer
worker pool), preventing a duplicate-copy `ClassCastException`.
- Nested schema dictionary field names lowercased so a mixed-case nested
struct
(e.g. Iceberg `DROP_AND_ADD`) can't crash BE with `std::out_of_range`.
- Session time zone: lazy `ZoneId` resolution tolerant of Doris
CST/PST-style
aliases in `FOR TIME AS OF` and datetime partition predicates.
- Storage / URI normalization (`oss/cos/obs/s3a` → `s3`), SHOW CREATE
secret
redaction, and `ALTER TABLE EXECUTE ... WHERE` rejection kept at legacy
wording.
- ~68 FE unit test classes: iceberg connector (offline recording fakes
for the
remote Catalog / Table), metastore-iceberg per-flavor dispatch +
properties, and
the fe-core `PluginDriven*` MVCC / sys-table / scan-node / GSON-replay
surfaces;
many carry mutation checks.
- checkstyle 0 violations; connector import-gate passes (no
`fe-connector-iceberg → fe-core` import); FE main + test compile clean.
- End-to-end iceberg behavior is docker-gated
(`enableIcebergTest=true`); read,
write, row-level DML, procedures, system tables, DDL and time-travel are
covered
by the `external_table_p0/iceberg` (+ `external_table_p2/iceberg`)
regression
suites.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>1 parent 556712a commit b417c55
690 files changed
Lines changed: 79806 additions & 25567 deletions
File tree
- fe
- check/checkstyle
- fe-connector
- fe-connector-api/src
- main/java/org/apache/doris/connector/api
- ddl
- handle
- mvcc
- procedure
- pushdown
- scan
- write
- test/java/org/apache/doris/connector/api
- handle
- mvcc
- procedure
- pushdown
- scan
- write
- fe-connector-cache
- src
- main/java/org/apache/doris/connector/cache
- test/java/org/apache/doris/connector/cache
- fe-connector-es/src/test/java/org/apache/doris/connector/es
- fe-connector-hms/src/main/java/org/apache
- doris/datasource/hive
- hadoop/hive/metastore
- fe-connector-iceberg
- src
- main
- assembly
- java/org/apache
- doris/connector/iceberg
- action
- dlf
- client
- rewrite
- iceberg
- test/java/org/apache/doris/connector/iceberg
- action
- dlf
- rewrite
- fe-connector-jdbc/src
- main/java/org/apache/doris/connector/jdbc
- client
- test/java/org/apache/doris/connector/jdbc
- fe-connector-maxcompute/src
- main/java/org/apache/doris/connector/maxcompute
- test/java/org/apache/doris/connector/maxcompute
- fe-connector-metastore-iceberg
- src
- main
- java/org/apache/doris/connector/metastore/iceberg
- dlf
- glue
- hms
- jdbc
- noop
- rest
- resources/META-INF/services
- test/java/org/apache/doris/connector/metastore/iceberg
- dlf
- glue
- hms
- jdbc
- rest
- fe-connector-metastore-paimon
- src
- main
- java/org/apache/doris/connector/metastore/paimon
- dlf
- fs
- hms
- jdbc
- rest
- resources/META-INF/services
- test/java/org/apache/doris/connector/metastore/paimon
- dlf
- fs
- hms
- jdbc
- rest
- fe-connector-metastore-spi/src
- main/java/org/apache/doris/connector/metastore/spi
- test/java/org/apache/doris/connector/metastore/spi
- fe-connector-paimon
- src
- main/java/org/apache/doris/connector/paimon
- test/java/org/apache/doris/connector/paimon
- fe-connector-spi/src
- main/java/org/apache/doris/connector/spi
- test/java/org/apache/doris/connector/spi
- fe-core/src
- main/java/org/apache/doris
- alter
- catalog
- common/util
- connector
- ddl
- datasource
- connectivity
- credentials
- iceberg
- action
- broker
- dlf
- fileio
- helper
- rewrite
- source
- metacache
- mvcc
- property
- common
- metastore
- systable
- nereids
- analyzer
- glue/translator
- processor
- post
- materialize
- pre
- properties
- rules
- analysis
- expression
- implementation
- rewrite
- trees/plans
- commands
- execute
- info
- insert
- merge
- logical
- physical
- visitor
- persist/gson
- planner
- qe
- statistics
- util
- transaction
- test/java/org/apache/doris
- catalog
- common
- proc
- util
- connector
- ddl
- fake
- datasource
- credentials
- iceberg
- dlf/client
- helper
- rewrite
- property/metastore
- systable
- nereids
- glue/translator
- processor/post/materialize
- rules/analysis
- trees/plans
- commands
- execute
- info
- insert
- logical
- physical
- planner
- statistics
- util
- transaction
- fe-foundation/src
- main/java/org/apache/doris/foundation/util
- test/java/org/apache/doris/foundation/util
- plan-doc
- connectors
- metastore-storage-refactor
- research
- reviews
- tasks
- designs
- regression-test/suites
- external_table_p0
- iceberg
- action
- branch_tag
- write
- jdbc
- kerberos
- paimon
- external_table_p2/refactor_catalog_param
- tools
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
46 | 50 | | |
47 | 51 | | |
48 | 52 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
17 | 17 | | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
| 21 | + | |
20 | 22 | | |
21 | 23 | | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
| 28 | + | |
26 | 29 | | |
| 30 | + | |
27 | 31 | | |
28 | 32 | | |
29 | 33 | | |
| |||
51 | 55 | | |
52 | 56 | | |
53 | 57 | | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
54 | 109 | | |
55 | 110 | | |
56 | 111 | | |
57 | 112 | | |
58 | 113 | | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
59 | 133 | | |
60 | 134 | | |
61 | 135 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
21 | | - | |
22 | | - | |
23 | | - | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
24 | 29 | | |
25 | 30 | | |
26 | | - | |
27 | | - | |
28 | | - | |
29 | | - | |
30 | | - | |
31 | | - | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | 31 | | |
36 | | - | |
37 | | - | |
38 | | - | |
39 | | - | |
40 | | - | |
41 | 32 | | |
42 | | - | |
| 33 | + | |
43 | 34 | | |
44 | | - | |
45 | | - | |
46 | | - | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
47 | 43 | | |
48 | | - | |
49 | | - | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
50 | 48 | | |
51 | | - | |
| 49 | + | |
52 | 50 | | |
53 | | - | |
54 | | - | |
| 51 | + | |
| 52 | + | |
55 | 53 | | |
56 | | - | |
57 | | - | |
58 | | - | |
59 | | - | |
60 | | - | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
61 | 66 | | |
62 | | - | |
63 | | - | |
64 | | - | |
65 | | - | |
66 | | - | |
67 | | - | |
68 | | - | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
69 | 71 | | |
70 | | - | |
| 72 | + | |
71 | 73 | | |
72 | | - | |
73 | | - | |
| 74 | + | |
| 75 | + | |
74 | 76 | | |
75 | | - | |
76 | | - | |
77 | | - | |
78 | | - | |
79 | | - | |
80 | | - | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
81 | 83 | | |
82 | | - | |
| 84 | + | |
83 | 85 | | |
84 | | - | |
| 86 | + | |
85 | 87 | | |
86 | | - | |
87 | | - | |
88 | | - | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
89 | 94 | | |
90 | | - | |
| 95 | + | |
91 | 96 | | |
92 | | - | |
93 | | - | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
94 | 100 | | |
95 | | - | |
96 | | - | |
97 | | - | |
98 | | - | |
99 | | - | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
100 | 124 | | |
101 | | - | |
| 125 | + | |
102 | 126 | | |
0 commit comments