Skip to content

Commit 6117bcb

Browse files
morningmanclaude
andcommitted
[refactor](catalog) fe-connector: replace fat hive-catalog-shade with a slim HMS metastore-client shade (#65721)
## Proposed changes Part of the catalog SPI migration umbrella: #65185. The `fe/fe-connector` closure depended on `org.apache.doris:hive-catalog-shade`, a ~122 MB "kitchen-sink" shade whose Hive metastore-client content is under 2% of its bytes (the rest is a full Hadoop, the paimon bundle, an ancient fastutil, DLF, etc.). This PR replaces it, for the fe-connector modules, with a new plugin-private slim shade `fe-connector-hms-hive-shade` (~15 MB) that bundles **only** the Hive metastore-**client** closure and relocates `org.apache.thrift` to `shade.doris.hive.org.apache.thrift`, mirroring the existing `fe-connector-paimon-hive-shade`. ### What changed - **New module `fe-connector-hms-hive-shade`**: hive-standalone-metastore (api + client), hive-common (`HiveConf`), hive-serde, hive-storage-api, iceberg-hive-metastore (`HiveCatalog`), libthrift/libfb303 0.9.3, codehaus jackson 1.9.2 (HMS JSON event messaging) and commons-lang 2.x (`HiveConf`). It uses an artifactSet `<includes>` **whitelist** so the hadoop-yarn/curator/jersey/kerby tail dragged in by hive-shims stays out; hadoop-common and guava/slf4j/log4j come from each plugin's own deps / the host at runtime. The thrift relocation **reuses** the existing `shade.doris.hive` prefix, so the vendored patch `HiveMetaStoreClient` in `fe-connector-hms` needs no source change. - **`fe-connector-hms`** now depends on the slim shade (compile, transitive to hive/hudi/iceberg) instead of the fat shade; the patch client stays in place. - **`fe-connector-iceberg`** drops its direct fat-shade dependency in the **same commit** (the hms-flavor `HiveCatalog` now arrives transitively via the slim shade), so iceberg is never left with two `HiveCatalog` copies; adds `iceberg-bundled-guava` at compile scope for the vendored `DeleteFileIndex`. - **`fe/pom.xml`** keeps the `hive-catalog-shade` version pin (still used by `be-java-extensions`) with an explanatory comment; stale comments in the iceberg/hudi poms are corrected. ### Size impact The shade artifact drops **122 MB → 15 MB** (to ~12% of its size). Each of the hive / hudi / iceberg plugins bundles its own copy, so each deployed plugin shrinks by **~107 MB** (~322 MB across the three). ### Verification - `fe-connector-hms` / `-hive` / `-hudi` / `-iceberg` build + unit tests green (197 test classes, 0 failures, checkstyle 0). - `dependency:tree` across all 19 fe-connector modules shows no `hive-catalog-shade`. - The three assembled plugin zips contain the slim shade once, no fat shade, no original-package libthrift, and exactly one copy each of the metastore-api `Table` / `HiveConf` / iceberg `HiveCatalog` / relocated thrift; the metastore and `iceberg.hive` bytecode is md5-identical to the fat shade. - **Remaining gate**: heterogeneous-HMS e2e (hive read/write, iceberg-on-HMS INSERT/DELETE/MERGE, hudi-on-HMS read; TCCL / filter-hook / kerberos paths) — to be exercised by CI / a follow-up. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01AkwFY4c7kZkLKZeazy3UMW --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
1 parent 53766ce commit 6117bcb

11 files changed

Lines changed: 800 additions & 21 deletions

File tree

‎fe/fe-connector/fe-connector-hms-hive-shade/pom.xml‎

Lines changed: 386 additions & 0 deletions
Large diffs are not rendered by default.

‎fe/fe-connector/fe-connector-hms/pom.xml‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -59,10 +59,21 @@ under the License.
5959
<version>${project.version}</version>
6060
</dependency>
6161

62-
<!-- Doris shaded Hive MetaStore client (includes Thrift, DLF, Glue) -->
62+
<!-- Slim Hive metastore-client shade — replaces the fat org.apache.doris:hive-catalog-shade
63+
(122MB). Bundles hive-standalone-metastore (metastore api + client), HiveConf, hive-serde,
64+
hive-storage-api, iceberg-hive-metastore's HiveCatalog and libthrift/libfb303 0.9.3, all with
65+
org.apache.thrift relocated to shade.doris.hive.org.apache.thrift so the old-package HMS
66+
client does not collide with the host doris-gen thrift 0.16. compile scope (no <scope>):
67+
propagates transitively to the plugins that reuse this shared plain-HMS client —
68+
fe-connector-hive, fe-connector-hudi, fe-connector-iceberg. The vendored version-aware patch
69+
HiveMetaStoreClient stays in THIS module (its bytecode already references the relocated thrift
70+
prefix; it overlays the unpatched client inside the shade because fe-connector-hms-*.jar sorts
71+
before fe-connector-hms-hive-shade-*.jar in the plugin lib). See
72+
plan-doc/fe-connector-hive-shade-localization/design.md. -->
6373
<dependency>
64-
<groupId>org.apache.doris</groupId>
65-
<artifactId>hive-catalog-shade</artifactId>
74+
<groupId>${project.groupId}</groupId>
75+
<artifactId>fe-connector-hms-hive-shade</artifactId>
76+
<version>${project.version}</version>
6677
</dependency>
6778

6879
<!-- Hadoop client for Configuration and auth -->

‎fe/fe-connector/fe-connector-hudi/pom.xml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -97,8 +97,9 @@ under the License.
9797
org.apache.parquet.avro.AvroSchemaConverter for hudi's FE-side schema read: TableSchemaResolver
9898
reads a data file's parquet footer to resolve the table Avro schema
9999
(ParquetUtils.readAvroSchema -> readSchema -> ParquetFileReader.readFooter(Configuration, Path),
100-
then converts the parquet MessageType to Avro). hudi-common bundles NO parquet, and the
101-
hive-catalog-shade jar carries only a RELOCATED (org.apache.paimon.shade.*) parquet — so the
100+
then converts the parquet MessageType to Avro). hudi-common bundles NO parquet, and no other
101+
jar in this plugin (the fe-connector-hms-hive-shade slim shade carries none either) provides an
102+
un-relocated org.apache.parquet.hadoop.* — so the
102103
un-relocated ParquetFileReader would otherwise resolve from the parent 'app' loader while
103104
ParquetUtils and Configuration come from the child (org.apache.hadoop is child-first here, from the
104105
bundled hadoop-common) — readFooter's Configuration parameter then splits across two loaders and the

‎fe/fe-connector/fe-connector-iceberg/pom.xml‎

Lines changed: 26 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,8 @@ under the License.
6161
</dependency>
6262

6363
<!-- Shared HMS module: supplies Doris's version-aware org.apache.hadoop.hive.metastore.HiveMetaStoreClient
64-
(bundled in fe-connector-hms-*.jar, which sorts before hive-catalog-shade-*.jar in the plugin lib so
65-
it shadows the shaded, unpatched client). Required for the hms flavor: without it, iceberg's
64+
(bundled in fe-connector-hms-*.jar, which sorts before fe-connector-hms-hive-shade-*.jar in the
65+
plugin lib so it shadows the shaded, unpatched client). Required for the hms flavor: without it, iceberg's
6666
HiveCatalog uses the Hive-3 "@cat#" db-name marker that a Hive-1/2 metastore rejects, so list/get
6767
database return empty (see HiveMetaStoreClient header). Same module fe-connector-hive reuses. -->
6868
<dependency>
@@ -127,6 +127,21 @@ under the License.
127127
<version>2.9.3</version>
128128
</dependency>
129129

130+
<!-- iceberg-bundled-guava: needed on the COMPILE classpath for the vendored split-package
131+
org.apache.iceberg.DeleteFileIndex (T08), which imports iceberg's relocated guava
132+
(org.apache.iceberg.relocated.com.google.common.base/collect). iceberg-core brings this
133+
artifact only at RUNTIME scope (so it is in the plugin-zip), and runtime-scoped transitives
134+
are not on the compile classpath — the same treatment as caffeine above. Until the fe-connector
135+
closure moved off the fat hive-catalog-shade, that shade jar carried these relocated-guava
136+
classes at compile scope; the slim fe-connector-hms-hive-shade deliberately excludes
137+
iceberg-bundled-guava (iceberg-core already ships it, avoiding a duplicate), so declare it here
138+
at compile scope. Same version as iceberg-core, deduped at runtime. -->
139+
<dependency>
140+
<groupId>org.apache.iceberg</groupId>
141+
<artifactId>iceberg-bundled-guava</artifactId>
142+
<version>${iceberg.version}</version>
143+
</dependency>
144+
130145
<!-- Iceberg AWS integration: the glue flavor's org.apache.iceberg.aws.glue.GlueCatalog + S3FileIO.
131146
Needs the AWS SDK v2 modules below at runtime. -->
132147
<dependency>
@@ -135,18 +150,15 @@ under the License.
135150
<version>${iceberg.version}</version>
136151
</dependency>
137152

138-
<!-- Doris shaded Hive MetaStore client (D-059 Q1: DLF port-now). Supplies the hms flavor's
139-
org.apache.iceberg.hive.HiveCatalog AND the dlf flavor's reflectively-loaded
140-
com.aliyun.datalake.metastore.hive2.ProxyMetaStoreClient, both at their original package wired
141-
to a relocated thrift (shade.doris.hive.org.apache.thrift) so the old-package HMS client does
142-
NOT collide with the host fe-thrift 0.16. Same dependency fe-connector-hive/hms use; version
143-
managed (${doris.hive.catalog.shade.version}). plugin-zip.xml excludes fe-thrift/libthrift so
144-
no original-package thrift lands in the plugin (the relocated copy inside the shade is
145-
self-contained). -->
146-
<dependency>
147-
<groupId>org.apache.doris</groupId>
148-
<artifactId>hive-catalog-shade</artifactId>
149-
</dependency>
153+
<!-- The hms-flavor org.apache.iceberg.hive.HiveCatalog is NOT a direct dependency here: it is
154+
bundled (with the shared relocated thrift shade.doris.hive.org.apache.thrift) inside
155+
fe-connector-hms-hive-shade and reaches this plugin transitively via fe-connector-hms above —
156+
the same slim shade that supplies the plain-HMS client, so HiveCatalog and the client link
157+
against one identical relocated thrift + metastore-api. The old direct
158+
org.apache.doris:hive-catalog-shade dependency (and the dead dlf-flavor ProxyMetaStoreClient
159+
it carried) was removed atomically with the hms cutover off the fat shade — see
160+
plan-doc/fe-connector-hive-shade-localization/design.md. plugin-zip.xml still excludes
161+
fe-thrift/libthrift so no original-package thrift lands in the plugin. -->
150162

151163
<!-- Hadoop for HadoopCatalog and Configuration -->
152164
<dependency>

‎fe/fe-connector/fe-connector-iceberg/src/main/assembly/plugin-zip.xml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -46,8 +46,9 @@ under the License.
4646
<exclude>org.apache.doris:fe-connector-spi</exclude>
4747
<exclude>org.apache.doris:fe-extension-spi</exclude>
4848
<exclude>org.apache.doris:fe-filesystem-api</exclude>
49-
<!-- Host thrift wins: hive-catalog-shade carries its own relocated thrift, so keep
50-
original-package fe-thrift / libthrift out of the plugin (same as fe-connector-hive). -->
49+
<!-- Host thrift wins: the fe-connector-hms-hive-shade slim shade carries its own
50+
relocated thrift (shade.doris.hive.org.apache.thrift), so keep original-package
51+
fe-thrift / libthrift out of the plugin (same as fe-connector-hive). -->
5152
<exclude>org.apache.doris:fe-thrift</exclude>
5253
<exclude>org.apache.thrift:libthrift</exclude>
5354
<!-- QUIC natives are unused: netty-bom force-manages the classifier jars to

‎fe/fe-connector/pom.xml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -52,6 +52,10 @@ under the License.
5252
<module>fe-connector-trino</module>
5353
<module>fe-connector-maxcompute</module>
5454
<module>fe-connector-jdbc</module>
55+
<!-- Slim HMS metastore-client shade (relocates org.apache.thrift to
56+
shade.doris.hive.org.apache.thrift); MUST build before fe-connector-hms which depends on
57+
it. Replaces the fat org.apache.doris:hive-catalog-shade for the fe-connector closure. -->
58+
<module>fe-connector-hms-hive-shade</module>
5559
<module>fe-connector-hms</module>
5660
<module>fe-connector-hive</module>
5761
<!-- FIX-C: paimon HMS-thrift shade (relocates org.apache.thrift to a paimon-private

‎fe/pom.xml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -844,6 +844,10 @@ under the License.
844844
</exclusion>
845845
</exclusions>
846846
</dependency>
847+
<!-- The fat hive-catalog-shade is no longer consumed by any fe/fe-connector module (the plain-HMS
848+
client + iceberg HiveCatalog moved to the slim fe-connector-hms-hive-shade). This version pin
849+
is kept because be-java-extensions (avro-scanner, java-udf) still depend on hive-catalog-shade
850+
— do NOT delete it. -->
847851
<dependency>
848852
<groupId>org.apache.doris</groupId>
849853
<artifactId>hive-catalog-shade</artifactId>
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# 🤝 Session Handoff — `fe/fe-connector` 剥离 `hive-catalog-shade`
2+
3+
> **滚动文档**:每次 session 结束**覆盖式更新**,只保留下一个 session 必须的上下文。
4+
> 完成明细**不落这里**(在 `git log` + [`progress.md`](./progress.md))。
5+
> 空间索引 [`README.md`](./README.md) · 设计 [`design.md`](./design.md) · 清单 [`tasklist.md`](./tasklist.md)
6+
7+
---
8+
9+
# ✅ Phase 1+2+3 完成(建精简 shade + 切 hms/iceberg + 静态&打包闸门全绿) → 🆕 下一个 session = **Phase 4:e2e(唯一真闸门)**
10+
11+
> 分支 `catalog-spi-hive-shade-12`。行号信 HEAD 不信文档。代码改动已 commit(见 git log 最新一条)。
12+
13+
## 现状(一句话)
14+
`fe/fe-connector/` 已整体脱离 122MB 胖 `hive-catalog-shade`,改用自建 15MB 精简 shade 模块 `fe-connector-hms-hive-shade`(只装 Hive 元数据客户端闭包,重定位 thrift→`shade.doris.hive.org.apache.thrift`)。build+UT(197 测试类全绿)、静态闸门(19 模块 dependency:tree 全空)、打包闸门(三插件 zip 无胖 shade、精简 shade 各 1 份、关键类无重复)、多 agent 对抗 review(零 confirmed)**均已过**。**唯一没做的是 e2e。**
15+
16+
## 第一件事(按顺序)
17+
1. **读** `progress.md` 末段(2026-07-16 Phase 1+2+3 结论,含闸门证据 + 两处现补的缺类)+ `design.md §4`(决策速查)。
18+
2. **动码/跑测前探并发**(`pgrep -af maven|grep wt-catalog-spi` + 近 90s mtime)。
19+
3. **重新构建部署产物**跑 e2e(精简 shade 需重打包插件重部署;旧部署目录可能还是胖 shade)。
20+
21+
## 🎯 Phase 4 要做的(`tasklist.md` FCL-40/41/42)
22+
- **异构 HMS docker 套件**:① 普通 hive catalog 读+写;② **iceberg-on-HMS** INSERT/DELETE/MERGE/read,断言与独立 iceberg 目录同表同结果;③ hudi-on-HMS 读。
23+
- **🔴 TCCL 不回归**(memory `catalog-spi-plugin-tccl-classloader-gotcha` / D5):`test_string_dict_filter` 类用例 + MetaStoreFilterHook/URIResolverHook 按名反射路径 + kerberos HMS(若环境有);FE 启动 + 缓存(MetaCache/StatisticsCache/FileSystemCache)冒烟。
24+
- **专项**:storage-api **2.7.0** 的 write/ACID 路径(本轮从胖 shade 的 2.8.1 换回 3.1.3 原生 2.7.0,review 判无回退但要 e2e 兜底)。
25+
- 结果(含 CI 编号)追加 `progress.md`。
26+
27+
## ⚠️ 白名单 shade 的运行时缺类(Phase 4 可能再遇,按此法补)
28+
精简 shade 用 **白名单 `<includes>`**,只装列出的 hive 客户端闭包;运行时其余靠各插件自带 hadoop-common 闭包/宿主 parent-first 提供。**跑到才暴露的缺类**本轮已补两处(老 Jackson `ObjectMapper`、iceberg `bundled-guava`);e2e 若再报 `NoClassDefFoundError`(尤其 kerberos/filter-hook 路径),**按同法补**:查是哪个 artifact 的类 → 加进 `fe-connector-hms-hive-shade/pom.xml` 的 `<dependency>`(必要时显式 `<scope>compile</scope>` 覆盖 fe/pom.xml 里的 test-scope 管理) + artifactSet `<include>`。**别退回黑名单 excludes**(那会把 122MB junk 又拉回来)。
29+
30+
## ⚙️ 构建/验证坑(本轮实证,直接复用)
31+
- `-pl` 选精简 shade 模块用 **`:fe-connector-hms-hive-shade`**(冒号 artifactId 选择器),裸名 maven 报 "Could not find the selected project"。
32+
- 后台跑 maven **别** `nohup ... &` 套在 `run_in_background` Bash 里 → 脱离 harness 跟踪、误报秒完成;用 `tail --pid=<mvnpid>` 阻塞等真 `BUILD SUCCESS/FAILURE`。
33+
- 依赖 shade 的模块**跑到 `package`**(`test-compile` 假报缺类);`-am` 必填;`-Dmaven.build.cache.enabled=false` 且数 `Running org.apache.doris` 行;`mvn|tail` 后 `$?` 是 tail 的(重定向到文件读 BUILD 行)。
34+
- maven 用绝对 `-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml`。
35+
- `git add` 用 path-whitelist,**严禁 `-A`**(工作树大量非本线程 scratch)。
36+
37+
## ✅ 每个 Phase 收尾
38+
commit(英文)+ 覆盖本 HANDOFF + `progress.md` 追加一行 + 勾 `tasklist.md`。

0 commit comments

Comments
 (0)