You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 6117bcb
Browse filesBrowse the repository at this point in the historyBrowse files
[refactor](catalog) fe-connector: replace fat hive-catalog-shade with a slim HMS metastore-client shade (#65721)
## Proposed changes
Part of the catalog SPI migration umbrella: #65185.
The `fe/fe-connector` closure depended on
`org.apache.doris:hive-catalog-shade`, a ~122 MB "kitchen-sink" shade
whose Hive metastore-client content is under 2% of its bytes (the rest
is a full Hadoop, the paimon bundle, an ancient fastutil, DLF, etc.).
This PR replaces it, for the fe-connector modules, with a new
plugin-private slim shade `fe-connector-hms-hive-shade` (~15 MB) that
bundles **only** the Hive metastore-**client** closure and relocates
`org.apache.thrift` to `shade.doris.hive.org.apache.thrift`, mirroring
the existing `fe-connector-paimon-hive-shade`.
### What changed
- **New module `fe-connector-hms-hive-shade`**:
hive-standalone-metastore (api + client), hive-common (`HiveConf`),
hive-serde, hive-storage-api, iceberg-hive-metastore (`HiveCatalog`),
libthrift/libfb303 0.9.3, codehaus jackson 1.9.2 (HMS JSON event
messaging) and commons-lang 2.x (`HiveConf`). It uses an artifactSet
`<includes>` **whitelist** so the hadoop-yarn/curator/jersey/kerby tail
dragged in by hive-shims stays out; hadoop-common and guava/slf4j/log4j
come from each plugin's own deps / the host at runtime. The thrift
relocation **reuses** the existing `shade.doris.hive` prefix, so the
vendored patch `HiveMetaStoreClient` in `fe-connector-hms` needs no
source change.
- **`fe-connector-hms`** now depends on the slim shade (compile,
transitive to hive/hudi/iceberg) instead of the fat shade; the patch
client stays in place.
- **`fe-connector-iceberg`** drops its direct fat-shade dependency in
the **same commit** (the hms-flavor `HiveCatalog` now arrives
transitively via the slim shade), so iceberg is never left with two
`HiveCatalog` copies; adds `iceberg-bundled-guava` at compile scope for
the vendored `DeleteFileIndex`.
- **`fe/pom.xml`** keeps the `hive-catalog-shade` version pin (still
used by `be-java-extensions`) with an explanatory comment; stale
comments in the iceberg/hudi poms are corrected.
### Size impact
The shade artifact drops **122 MB → 15 MB** (to ~12% of its size). Each
of the hive / hudi / iceberg plugins bundles its own copy, so each
deployed plugin shrinks by **~107 MB** (~322 MB across the three).
### Verification
- `fe-connector-hms` / `-hive` / `-hudi` / `-iceberg` build + unit tests
green (197 test classes, 0 failures, checkstyle 0).
- `dependency:tree` across all 19 fe-connector modules shows no
`hive-catalog-shade`.
- The three assembled plugin zips contain the slim shade once, no fat
shade, no original-package libthrift, and exactly one copy each of the
metastore-api `Table` / `HiveConf` / iceberg `HiveCatalog` / relocated
thrift; the metastore and `iceberg.hive` bytecode is md5-identical to
the fat shade.
- **Remaining gate**: heterogeneous-HMS e2e (hive read/write,
iceberg-on-HMS INSERT/DELETE/MERGE, hudi-on-HMS read; TCCL / filter-hook
/ kerberos paths) — to be exercised by CI / a follow-up.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01AkwFY4c7kZkLKZeazy3UMW
---------
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
0 commit comments